Kubernetes secret management using the External Secrets Operator-EKS

In this blog post, I am going to examine the External Secrets Operator and demonstrate how to store your secrets externally on the AWS Secrets Manager.

Kubernetes includes native capabilities for managing secrets in the form of Kubernetes Secrets to satisfy the requirement of safely delivering secrets to running applications. To improve security, administration, and the ability to track how secrets are used, centralized secret management can be carried out outside of Kubernetes clusters using an external secret store such as Hashicorp Vault, AWS Secrets Manager, etc. One way to store secrets outside the K8s cluster is with the help of the External Secrets Operator open-source project. 

What Is the Function of the External Secrets Operator?

The External Secrets Operator's objective is to synchronize secrets from external APIs with Kubernetes. The ESO manages secrets via Custom Resource Definitions. ExternalSecret, SecretStore, and ClusterSecretStore are user-friendly wrappers around the external API that store and manage secrets on your behalf.

Your secrets are managed by the "ExternalSecret" CRD and the controller uses ExternalSecret’s data to create secrets. When you use External Secrets to read secrets from an external secret store, the data is stored in the Kubernetes control plane as native Kubernetes Secrets. 

SecretStore and ExternalSecret CRDs are important to understand this demo and how ESO works.

SecretStore CRD is used for the authentication and access management of the external secret store.

ExternalSecret CRD is used for the specific secret that is going to be pulled. The controller uses ExternalSecret’s data to create secrets and sync them at a time interval you choose.

Let's Practice a Little Bit

This tutorial will show you how to sync a secret from the AWS Secrets Manager to your EKS cluster using the External Secrets Operator.

To properly follow this tutorial, make sure you have installed the following tools:

• Helm version 3
• kubectl
• eksctl
• AWS CLI

Creating an EKS Cluster

If you already have an EKS Cluster, you can skip this step. I am going to deploy my own EKS Cluster using eksctl, which is a simple CLI tool for creating and managing EKS Clusters.

The following commands will create my EKS Cluster.

eksctl create cluster -f cluster-config.yaml

The code example below provides an overview of the cluster-config.yaml file.

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: eso-test-cluster
  region: eu-central-1

nodeGroups:
  - name: ng-1
    instanceType: t3.medium
    desiredCapacity: 2
    volumeSize: 20
    ssh:
      allow: true

Use the following command to get kube-config.

aws eks update-kubeconfig --name=eso-test-cluster --region=eu-central-1

After these steps are complete, run the following command to verify.

kubectl get nodes

As there are no errors, the EKS Cluster is ready for usage.

Installing the External Secrets Operator

The External Secrets Operator offers Helm Charts for deployment convenience, and I am going to use Helm for deploying the External Secrets Operator.

The following commands will deploy an External Secrets Operator to my EKS Cluster.

helm repo add external-secrets https://charts.external-secrets.io

helm install external-secrets \
   external-secrets/external-secrets \
   -n external-secrets \
   --create-namespace \
   --set installCRDs=true \
   --set webhook.port=9443 

After the `external-secrets has been deployed successfully!` message, run the following command to verify external secret operator resources.

kubectl get pods -n external-secrets

Expected Output:

IAM Roles For Service Accounts (IRSA)

For the external-secrets operator to be able to get secrets from the AWS Secrets Manager, I have to set a few configurations. You can manage the credentials for your applications with EKS's feature IRSA. This is similar to how Amazon EC2 instance profiles give credentials for Amazon EC2 instances. Instead of distributing AWS credentials or using the Amazon EC2 role, you can map an IAM role to a Kubernetes service account and set up your pods to use this service account. To use IRSA, it is mandatory to create an IAM OIDC Provider for the EKS cluster.

The following command will create an OIDC provider and associate it with my cluster.

$ eksctl utils associate-iam-oidc-provider --cluster=eso-test-cluster --approve --region eu-central-1

I am going to use eksctl to create a service account. The following command 'eksctl create iamserviceaccount' takes an IAM policy arn as an argument, creates an IAM role associated with the given policy, and maps a service account to that role.

eksctl create iamserviceaccount \
    --name esoblogsa \
    --namespace default \
    --cluster  \
    --role-name "esoBlogRole" \
    --attach-policy-arn <polıcy_arn> \
    --approve \
    --override-existing-serviceaccounts
</polıcy_arn>

I have already created my secret in the AWS Secrets Manager and an IAM policy that lets it be retrieved and decrypted. The following image shows a secret with the name test/eso/testSecret.

test/eso/testSecret

esoBlogSecretReadPolicy

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "secretsmanager:GetSecretValue",
                "secretsmanager:DescribeSecret"
            ],
            "Resource":≤secret_arn≥
        }
    ]
}

After these steps are completed, run the following command to verify the service account. 

$ kubectl describe sa esoblogsa

Sync External Secrets to the AWS Secrets Manager Secret

I am going to create a SecretStore that references AWS Secrets Manager with a service account that I have already created, 'esoblogsa'. Run the following command to create the SecretStore.

cat <<EOF | kubectl apply -f -
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: eso-blog-secret-store
spec:
  provider:
    aws:
      service: SecretsManager
      region: eu-central-1
      auth:
        jwt:
          serviceAccountRef:
            name: esoblogsa
EOF

Run the following command to create an ExternalSecret. The Controller is going to create a secret with the name of "spec.target.name".

cat <<EOF | kubectl apply -f -
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: blogdemo
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: eso-blog-secret-store
    kind: SecretStore
  target:
    name: esoblogsecret
    creationPolicy: Owner
  data:
  - secretKey: esoblog-test-password
    remoteRef:
      key: test/eso/testSecret #AWS secret name
      property: password #AWS secret key

Run the following command to verify the Kubernetes secret.

kubectl describe secret esoblogsecret

Name:         esoblogsecret
Namespace:    default
Labels:       
Annotations:  reconcile.external-secrets.io/data-hash: 2ffa008bbbf55886723ea27d84ed84c4

Type:  Opaque

Data
====
esoblog-test-password:  12 bytes

My ExternalSecret is synced to the secret stored in AWS Secrets Manager. The controller has already created the Kubernetes Secret and I can use that secret similar to how I use kubernetes secrets regularly. 

Destroy

Don’t forget to destroy your resources after all of these steps. Run the following commands to destroy your resources.

kubectl delete externalsecret.external-secrets.io/blogdemo
kubectl delete secretstore eso-blog-secret-store
eksctl delete iamserviceaccount \
    --name  \
    --namespace default \
    --cluster 
helm uninstall external-secrets -n external-secrets

#If you didn't use eksctl, you can pass this line.
eksctl delete cluster --name  --region 

Alternative solution

AWS Secrets Manager and Config Provider for Secret Store CSI Driver (ASCP) :Secrets from AWS Secrets Manager are exposed to your pods as a mounted storage volume via ASCP using the Secrets Store CSI driver. The CSI Secret Store driver runs as a DaemonSet and DaemonSets are not supported on Fargate.

Conclusion

The External Secrets Operator is a useful tool if you want to utilize your external secrets in your K8s cluster. Rather than other solutions, ESO is compatible with a wide range of cloud secret store services such as AWS Secrets Manager, GCP Secret Manager, Azure Key Vault, etc., and for the EKS perspective, it is compatible with Fargate nodes too, which is not possible with ASCP.