Use Terraform to Create Kubernetes Secrets with GCP Service Account Keys

Managing service account credentials securely is a critical concern when working with Google Cloud Platform and Kubernetes. This guide demonstrates how to use Terraform to create a Kubernetes secret containing a Google service account key, without ever storing sensitive credentials in version control.

The Challenge

The goal is to:

1. Create a Google Cloud service account and key
2. Store the key in GCP Secret Manager
3. Import the key from Secret Manager into a Kubernetes secret
4. Accomplish all of this without storing sensitive credentials in GitHub or version control

Environment

This guide uses the following versions:

• Terraform: v0.14
• Google Provider: v3.60
• Google Beta Provider: v3.60
• Kubernetes: v1.17

Architecture Overview

The flow is:

1. Terraform creates a GCP service account
2. Terraform generates a service account key
3. The key is stored in GCP Secret Manager
4. Terraform creates a Kubernetes secret using the key from Secret Manager

This approach ensures the sensitive key never touches your local filesystem or version control.

Step 1: Create the Service Account

First, create a Google Cloud service account:

resource "google_service_account" "service-account-1" {
  account_id   = "service-account-1"
  display_name = "service-account-1"
  project      = var.project_id
}

Step 2: Generate Service Account Key

Create a key for the service account:

resource "google_service_account_key" "service-account-1-key" {
  service_account_id = google_service_account.service-account-1.name
}

Important: The private_key attribute is base64 encoded by default.

Step 3: Enable Secret Manager API

Before using Secret Manager, ensure the API is enabled:

resource "google_project_service" "secretmanager" {
  provider = google-beta
  project  = var.project_id
  service  = "secretmanager.googleapis.com"

  disable_on_destroy = false
}

Step 4: Create Secret in Secret Manager

Create a secret resource in GCP Secret Manager:

resource "google_secret_manager_secret" "service-acc-1-key" {
  provider  = google-beta
  project   = var.project_id
  secret_id = "service-acc-1-key"

  replication {
    automatic = true
  }

  depends_on = [google_project_service.secretmanager]
}

Step 5: Store the Key in Secret Manager

Create a secret version containing the actual service account key:

resource "google_secret_manager_secret_version" "service-acc-key-v1" {
  provider = google-beta

  secret      = google_secret_manager_secret.service-acc-1-key.id
  secret_data = base64decode(google_service_account_key.service-account-1-key.private_key)
}

Key Point: Use base64decode() because the service account key is base64 encoded, but Secret Manager expects plain text.

Step 6: Create Kubernetes Secret

Finally, create a Kubernetes secret that references the key from Secret Manager:

resource "kubernetes_secret" "service-acc-1-key" {
  metadata {
    name      = "service-acc-1-key"
    namespace = "default"
  }

  data = {
    "key.json" = base64decode(google_service_account_key.service-account-1-key.private_key)
  }

  type = "Opaque"
}

Alternatively, if you want to read from Secret Manager:

data "google_secret_manager_secret_version" "service-acc-1-key" {
  provider = google-beta
  secret   = google_secret_manager_secret.service-acc-1-key.id
}

resource "kubernetes_secret" "service-acc-1-key" {
  metadata {
    name      = "service-acc-1-key"
    namespace = "default"
  }

  data = {
    "key.json" = data.google_secret_manager_secret_version.service-acc-1-key.secret_data
  }

  type = "Opaque"
}

Complete Terraform Configuration

Here's the full configuration in one place:

# Variables
variable "project_id" {
  description = "GCP Project ID"
  type        = string
}

# Enable Secret Manager API
resource "google_project_service" "secretmanager" {
  provider = google-beta
  project  = var.project_id
  service  = "secretmanager.googleapis.com"

  disable_on_destroy = false
}

# Create Service Account
resource "google_service_account" "service-account-1" {
  account_id   = "service-account-1"
  display_name = "service-account-1"
  project      = var.project_id
}

# Create Service Account Key
resource "google_service_account_key" "service-account-1-key" {
  service_account_id = google_service_account.service-account-1.name
}

# Create Secret in Secret Manager
resource "google_secret_manager_secret" "service-acc-1-key" {
  provider  = google-beta
  project   = var.project_id
  secret_id = "service-acc-1-key"

  replication {
    automatic = true
  }

  depends_on = [google_project_service.secretmanager]
}

# Store Key in Secret Manager
resource "google_secret_manager_secret_version" "service-acc-key-v1" {
  provider = google-beta

  secret      = google_secret_manager_secret.service-acc-1-key.id
  secret_data = base64decode(google_service_account_key.service-account-1-key.private_key)
}

# Create Kubernetes Secret
resource "kubernetes_secret" "service-acc-1-key" {
  metadata {
    name      = "service-acc-1-key"
    namespace = "default"
  }

  data = {
    "key.json" = base64decode(google_service_account_key.service-account-1-key.private_key)
  }

  type = "Opaque"
}

Applying the Configuration

1. Initialize Terraform:

terraform init

2. Plan the changes:

terraform plan -var="project_id=your-project-id"

3. Apply the configuration:

terraform apply -var="project_id=your-project-id"

Verification

Verify the Kubernetes secret was created:

kubectl get secret service-acc-1-key -n default
kubectl describe secret service-acc-1-key -n default

Verify the secret in GCP Secret Manager:

gcloud secrets list
gcloud secrets versions list service-acc-1-key

Using the Secret in a Pod

Reference the secret in your pod specification:

apiVersion: v1
kind: Pod
metadata:
  name: my-app
spec:
  containers:
  - name: app
    image: gcr.io/my-project/my-app:latest
    env:
    - name: GOOGLE_APPLICATION_CREDENTIALS
      value: /var/secrets/google/key.json
    volumeMounts:
    - name: google-cloud-key
      mountPath: /var/secrets/google
      readOnly: true
  volumes:
  - name: google-cloud-key
    secret:
      secretName: service-acc-1-key

Security Best Practices

1. Never commit state files: Add *.tfstate* to your .gitignore
2. Use remote state: Store Terraform state in GCS with encryption
3. Restrict Secret Manager access: Use IAM to limit who can read secrets
4. Rotate keys regularly: Implement a key rotation policy
5. Use Workload Identity: Consider using GKE Workload Identity instead of service account keys when possible

Important Notes

• The google_service_account_key resource stores the private key in base64 encoding
• Always use base64decode() when storing in Secret Manager or Kubernetes
• Secret Manager stores data as plain text (you provide the decoding)
• Terraform state contains sensitive data - secure it appropriately

Troubleshooting

Secret Not Found in Kubernetes

Check that your Kubernetes provider is correctly configured:

provider "kubernetes" {
  config_path = "~/.kube/config"
}

Permission Denied in Secret Manager

Ensure your Terraform service account has the secretmanager.admin role:

gcloud projects add-iam-policy-binding PROJECT_ID \
  --member="serviceAccount:terraform@PROJECT_ID.iam.gserviceaccount.com" \
  --role="roles/secretmanager.admin"

Conclusion

This approach provides a secure way to manage Google Cloud service account keys in Kubernetes using Terraform and GCP Secret Manager. The key benefits are:

• No secrets in version control: Keys never touch your repository
• Centralized secret management: GCP Secret Manager provides audit logs and access control
• Infrastructure as Code: Full automation with Terraform
• Kubernetes integration: Seamless secret mounting in pods

While this solution works well, consider migrating to GKE Workload Identity for production workloads, as it eliminates the need for service account keys entirely.