Manage Kubernetes Secrets using AWS Secrets Manager

Let’s go ahead and install ESO in a Kubernetes cluster using the official Helm chart.

TL;DR

helm repo add external-secrets https://charts.external-secrets.io
helm repo update

Install ESO in external-secrets namespace

helm upgrade --namespace external-secrets --create-namespace --install --wait external-secrets external-secrets/external-secrets

Verify ESO installation

kubectl -n external-secrets get all

aws iam attach-user-policy --user-name external-secrets --policy-arn arn:aws:iam::aws:policy/SecretsManagerReadWrite

Now, generate the AWS access and secret key.

aws iam create-access-key --user-name external-secrets

Place the access key and secret key onto a file and create a secret in default namespace called awssm-secret from the file.

echo -n "REPLACE_ME_WITH_YOUR_ACCESS_KEY" > access-key
echo -n "REPLACE_ME_WITH_YOUR_SECRET_KEY" > secret-access-key

Go ahead and create a demo app secret in us-east-1 region of the AWS Secrets Manager via AWS CLI, which will be fetched later by External Secrets and exposed to the application Pod via environment variable:

aws secretsmanager create-secret --name app-secret --secret-string '{"username":"bob","password":"abc123xyz456"}' --region us-east-1

And verify the secret from the AWS Secrets Manager console.

As mentioned before, a cluster-scoped secret store allows referencing the secret store from any namespaces, which is convenient to use as a central gateway to the secret provider, rather than creating a secret store per namespace.

Go ahead, and save the following YAML as cluster-secret-store.yaml 

cat > cluster-secret-store.yaml <<EOF
apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: global-secret-store
spec:
  provider:
    aws:
      service: SecretsManager
      region: us-east-1
      auth:
        secretRef:
          accessKeyIDSecretRef:
            name: awssm-secret
            key: access-key
            namespace: default
          secretAccessKeySecretRef:
            name: awssm-secret
            key: secret-access-key
            namespace: default
EOF

And apply the manifest:

kubectl apply -f cluster-secret-store.yaml

And, verify the cluster secret store is created successfully and shows the message “store validated”.

kubectl describe clustersecretstore global-secret-store

We’re now ready to create the ExternalSecret resource and fetch the app-secret demo secret data from AWS Secrets Manager by referencing the cluster-scoped secret store.

Firstly, create a namespace called app

kubectl create namespace app

And save the following YAML as app-secret.yaml where ExternalSecret object specification is structured as such:

• spec.refreshInterval is set to 1 minute, meaning reconciliation will take place every minute.
• spec.secretStoreRefis set to ClusterSecretStore named global-secret-store which we created before.
• spec.target.name specifies the name of the secret object that will be created in the same namespace, where ExternalSecret is created.
• spec.dataFrom specifies the secret name in the AWS Secrets Manager as key and extracttells it to retrieve all key/value secrets.

cat > app-secret.yaml <<EOF
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: app-secret
spec:
  refreshInterval: 1m
  secretStoreRef:
    name: global-secret-store
    kind: ClusterSecretStore
  target:
    name: app-secret
    creationPolicy: Owner
  dataFrom:
  - extract:
      key: app-secret
EOF

Finally, go ahead and apply the manifest:

kubectl -n app apply -f app-secret.yaml

If everything went as planned, we should get SecretSynced status for ExternalSecret resource and a new Kubernetes Secret resource called app-secret should be created.

kubectl -n app get externalsecret
kubectl -n app get secret app-secret

What just happened?

• ExternalSecret was able to authenticate to AWS Secrets Manager using the cluster-scoped secret store.
• ExternalSecret was able to fetch the secret data and create the Kubernetes Secret object.

Now, that the ExternalSecret was able to retrieve the secrets from AWS Secrets Manager and on-the-fly create the Kubernetes Secret for us. We can reference this secret app-secret as usual from the Pod, without any application modification.

Let’s go ahead and apply the following Pod manifest and see if our container has the secret data in its environment variable.

cat > app-pod.yaml <<EOF
apiVersion: v1
kind: Pod
metadata:
  name: app-pod
spec:
  containers:
    - name: app
      image: k8s.gcr.io/busybox
      command: [ "/bin/sh", "-c", "env" ]
      envFrom:
      - secretRef:
          name: app-secret
EOF

And apply the manifest:

kubectl -n app apply -f app-pod.yaml

Once, applied. The Pod will create a single container and runs env command and exits. We can check the Pod log to verify that our secret data was indeed available as an environment variable.

kubectl -n app logs app-pod | egrep 'username|password'

External Secrets Operator is a mature and popular open source project with great community engagement. It is well-tested and stable for production workloads with high-availability features for managing and synchronizing large-scale secrets and supports many popular external secrets providers like HashiCorp Vault, Google Secrets Manager, Azure Key Vault, and many more.

Since External Secrets provides the same ease of use as Kubernetes native Secret objects, it doesn’t require any application modifications, and existing application leveraging Secret object works out of the box without any changes.