Kubernetes External Secrets

Kubernetes External Secrets allows you to use external secret management systems, like AWS Secrets Manager or HashiCorp Vault, to securely add secrets in Kubernetes. Read more about the design and motivation for Kubernetes External Secrets on the GoDaddy Engineering Blog.

How it works

The project extends the Kubernetes API by adding a ExternalSecrets object using Custom Resource Definition and a controller to implement the behavior of the object itself.

An ExternalSecret declares how to fetch the secret data, while the controller converts all ExternalSecrets to Secrets. The conversion is completely transparent to Pods that can access Secrets normally.

System architecture

1. ExternalSecrets are added in the cluster (e.g., kubectl apply -f external-secret-example.yml)
2. Controller fetches ExternalSecrets using the Kubernetes API
3. Controller uses ExternalSecrets to fetch secret data from external providers (e.g, AWS Secrets Manager)
4. Controller upsert Secrets
5. Pods can access Secrets normally

How to use it

Install with Helm

The official helm chart can be used to create the kubernetes-external-secrets resources and Deployment on a Kubernetes cluster using the Helm package manager.

$ helm repo add external-secrets https://godaddy.github.io/kubernetes-external-secrets/
$ helm install external-secrets/kubernetes-external-secrets

For more details about configuration see the helm chart docs

Install with kubectl

If you don't want to install helm on your cluster and just want to use kubectl to install kubernetes-external-secrets, you could get the helm client cli first and then use the following sample command to generate kubernetes manifests:

$ helm template -f charts/kubernetes-external-secrets/values.yaml --output-dir ./output_dir ./charts/kubernetes-external-secrets/

The generated kubernetes manifests will be in ./output_dir and can be applied to deploy kubernetes-external-secrets to the cluster.

Secrets Manager access

For kubernetes-external-secrets to be able to retrieve your secrets it will need access to your secret backend.

AWS based backends

Access to AWS secrets backends (SSM & secrets manager) can be granted in various ways:

1. Granting your nodes explicit access to your secrets using the node instance role (easy for experimentation, not recommended)

2. IAM roles for service accounts.

3. Per pod IAM authentication: kiam or kube2iam.

4. Directly provide AWS access credentials to the kubernetes-external-secrets pod by environmental variables.

Using AWS access credentials

Set AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY env vars in the kubernetes-external-secrets session/pod. You can use envVarsFromSecret in the helm chart to create these env vars from existing k8s secrets.

Additionally, you can specify a roleArn which will be assumed before retrieving the secret. You can limit the range of roles which can be assumed by this particular namespace by using annotations on the namespace resource. The annotation key is configurable (see above). The annotation value is evaluated as a regular expression and tries to match the roleArn.

kind: Namespace
metadata:
  name: iam-example
  annotations:
    # annotation key is configurable
    iam.amazonaws.com/permitted: "arn:aws:iam::123456789012:role/.*"

Add a secret

Add your secret data to your backend. For example, AWS Secrets Manager:

aws secretsmanager create-secret --name hello-service/password --secret-string "1234"

AWS Parameter Store:

aws ssm put-parameter --name "/hello-service/password" --type "String" --value "1234"

and then create a hello-service-external-secret.yml file:

apiVersion: 'kubernetes-client.io/v1'
kind: ExternalSecret
metadata:
  name: hello-service
spec:
  backendType: secretsManager
  # optional: specify role to assume when retrieving the data
  roleArn: arn:aws:iam::123456789012:role/test-role
  data:
    - key: hello-service/password
      name: password
  # optional: specify a template with any additional markup you would like added to the downstream Secret resource.
  # This template will be deep merged without mutating any existing fields. For example: you cannot override metadata.name.
  template:
    metadata:
      annotations:
        cat: cheese
      labels:
        dog: farfel

or

apiVersion: 'kubernetes-client.io/v1'
kind: ExternalSecret
metadata:
  name: hello-service
spec:
  backendType: systemManager
  data:
    - key: /hello-service/password
      name: password

The following IAM policy allows a user or role to access parameters matching prod-*.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "ssm:GetParameter",
      "Resource": "arn:aws:ssm:us-west-2:123456789012:parameter/prod-*"
    }
  ]
}

The IAM policy for Secrets Manager is similar (see docs):

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "secretsmanager:GetResourcePolicy",
        "secretsmanager:GetSecretValue",
        "secretsmanager:DescribeSecret",
        "secretsmanager:ListSecretVersionIds"
      ],
      "Resource": [
        "arn:aws:secretsmanager:us-west-2:111122223333:secret:aes128-1a2b3c",
        "arn:aws:secretsmanager:us-west-2:111122223333:secret:aes192-4D5e6F",
        "arn:aws:secretsmanager:us-west-2:111122223333:secret:aes256-7g8H9i"
      ]
    }
  ]
}

Save the file and run:

kubectl apply -f hello-service-external-secret.yml

Wait a few minutes and verify that the associated Secret has been created:

kubectl get secret hello-service -o=yaml

The Secret created by the controller should look like:

apiVersion: v1
kind: Secret
metadata:
  name: hello-service
  annotations:
    cat: cheese
  labels:
    dog: farfel
type: Opaque
data:
  password: MTIzNA==

Enforcing naming conventions for backend keys

by default an ExternalSecret may access arbitrary keys from the backend e.g.

  data:
    - key: /dev/cluster1/core-namespace/hello-service/password
      name: password

An enforced naming convention helps to keep the structure tidy and limits the access according to your naming schema.

Configure the schema as regular expression in the namespace using an annotation. This allows ExternalSecrets in core-namespace just to access secrets that start with /dev/cluster1/core-namespace/:

kind: Namespace
metadata:
  name: core-namespace
  annotations:
    # annotation key is configurable
    externalsecrets.kubernetes-client.io/permitted-key-name: "/dev/cluster1/core-namespace/.*"

Deprecations

A few properties has changed name overtime, we still maintain backwards compatbility with these but they will eventually be removed, and they are not validated using the CRD validation.

Old	New
secretDescriptor	spec
spec.type	spec.template.type
spec.properties	spec.data
backendType: secretManager	backendType: secretsManager

Backends

kubernetes-external-secrets supports AWS Secrets Manager, AWS System Manager, Hashicorp Vault and Azure Key Vault.

AWS Secrets Manager

kubernetes-external-secrets supports both JSON objects ("Secret key/value" in the AWS console) or strings ("Plaintext" in the AWS console). Using JSON objects is useful when you need to atomically update multiple values. For example, when rotating a client certificate and private key.

When writing an ExternalSecret for a JSON object you must specify the properties to use. For example, if we add our hello-service credentials as a single JSON object:

aws secretsmanager create-secret --region us-west-2 --name hello-service/credentials --secret-string '{"username":"admin","password":"1234"}'

We can declare which properties we want from hello-service/credentials:

apiVersion: kubernetes-client.io/v1
kind: ExternalSecret
metadata:
  name: hello-service
spec:
  backendType: secretsManager
  # optional: specify role to assume when retrieving the data
  roleArn: arn:aws:iam::123456789012:role/test-role
  data:
    - key: hello-service/credentials
      name: password
      property: password
    - key: hello-service/credentials
      name: username
      property: username
    - key: hello-service/credentials
      name: password
      # Version Stage in Secrets Manager
      versionStage: AWSPREVIOUS
      property: password_previous

alternatively you can use dataFrom and get all the values from hello-service/credentials:

apiVersion: kubernetes-client.io/v1
kind: ExternalSecret
metadata:
  name: hello-service
spec:
  backendType: secretsManager
  # optional: specify role to assume when retrieving the data
  roleArn: arn:aws:iam::123456789012:role/test-role
  dataFrom:
    - hello-service/credentials

data and dataFrom can of course be combined, any naming conflicts will use the last defined, with data overriding dataFrom

apiVersion: kubernetes-client.io/v1
kind: ExternalSecret
metadata:
  name: hello-service
spec:
  backendType: secretsManager
  # optional: specify role to assume when retrieving the data
  roleArn: arn:aws:iam::123456789012:role/test-role
  dataFrom:
    - hello-service/credentials
  data:
    - key: hello-service/migration-credentials
      name: password
      property: password

Hashicorp Vault

kubernetes-external-secrets supports fetching secrets from Hashicorp Vault, using the Kubernetes authentication method.

You will need to set the VAULT_ADDR environment variables so that kubernetes-external-secrets knows which endpoint to connect to, then create ExternalSecret definitions as follows:

apiVersion: 'kubernetes-client.io/v1'
kind: ExternalSecret
metadata:
  name: hello-vault-service
spec:
  backendType: vault
  # Your authentication mount point, e.g. "kubernetes"
  vaultMountPoint: my-kubernetes-vault-mount-point
  # The vault role that will be used to fetch the secrets
  # This role will need to be bound to kubernetes-external-secret's ServiceAccount; see Vault's documentation:
  # https://www.vaultproject.io/docs/auth/kubernetes.html
  vaultRole: my-vault-role
  data:
  - name: password
    # The full path of the secret to read, as in `vault read secret/data/hello-service/credentials`
    key: secret/data/hello-service/credentials
    property: password
  # Vault values are matched individually. If you have several keys in your Vault secret, you will need to add them all separately
  - name: api-key
    key: secret/data/hello-service/credentials
    property: api-key

If Vault uses a certificate issued by a self-signed CA you will need to provide that certificate:

# Create secret with CA
kubectl create secret generic vault-ca --from-file=./ca.pem

# values.yaml
env:
  VAULT_ADDR: https://vault.domain.tld
  NODE_EXTRA_CA_CERTS: "/usr/local/share/ca-certificates/ca.pem"

filesFromSecret:
  certificate-authority:
    secret: vault-ca
    mountPath: /usr/local/share/ca-certificates

Azure Key Vault

kubernetes-external-secrets supports fetching secrets from Azure Key vault

You will need to set these env vars in the deployment of kubernetes-external-secrets:

• AZURE_TENANT_ID
• AZURE_CLIENT_ID
• AZURE_CLIENT_SECRET

The SP configured will require get and list access policies on the AZURE_KEYVAULT_NAME.

apiVersion: kubernetes-client.io/v1
kind: ExternalSecret
metadata:
  name: hello-keyvault-service
spec:
  backendType: azureKeyVault
  keyVaultName: hello-world
  data:
    - key: hello-service/credentials
      name: password
      property: value

GCP Secret Manager

kubernetes-external-secrets supports fetching secrets from GCP Secret Manager

A service account is required to grant the controller access to pull secrets. Instructions are her: Enable Workload Identity

Alternatively you can create and mount a kubernetes secret containing google service account credentials and set the GOOGLE_APPLICATION_CREDENTIALS env variable.

apiVersion: kubernetes-client.io/v1
kind: ExternalSecret
metadata:
  name: gcp-secrets-manager-example
spec:
  backendType: gcpSecretsManager
  data:
    - key: projects/111122223333/secrets/my-secret/versions/latest
      name: password
      property: value

Due to the way Azure handles binary files, you need to explicitly let the ExternalSecret know that the secret is binary. You can do that with the isBinary field on the key. This is necessary for certificates and other secret binary files.

apiVersion: kubernetes-client.io/v1
kind: ExternalSecret
metadata:
  name: hello-keyvault-service
spec:
  backendType: azureKeyVault
  keyVaultName: hello-world
  data:
    - key: hello-service/credentials
      name: password
      isBinary: true

Metrics

kubernetes-external-secrets exposes the following metrics over a prometheus endpoint:

Metric	Description	Example
sync_calls	This metric counts the number of sync calls by backend, secret name and status	sync_calls{name="foo",namespace="example",backend="foo",status="success"} 1

Development

Minikube is a tool that makes it easy to run a Kubernetes cluster locally.

Start minikube and the daemon. This creates the CustomerResourceDefinition, and starts to process ExternalSecrets:

minikube start

npm run nodemon

Development with localstack

Localstack mocks AWS services locally so you can test without connecting to AWS.

Run localstack in a seperate terminal window

npm run localstack

Start minikube as above

minikube start

Run the daemon with localstack

npm run local

Add secrets using the AWS cli (example)

AWS_ACCESS_KEY_ID=foobar AWS_SECRET_ACCESS_KEY=foobar aws --region=us-west-2 --endpoint-url=http://localhost:4584 secretsmanager create-secret --name hello-service/password --secret-string "1234"