Manage Kubernetes Secrets with External Secrets Operator through AWS System Manager Parameter Store: A Comprehensive Hands-On Guide

Introduction

Kubernetes secrets is an object that by default allows to store confidential data such as a password, a token, or a key in the API server's underlying data store called etcd. Using Kubernetes secrets, you will decouple hard-coded sensitive data from your application code, container image, or pod definition.

However, Kubernetes secrets often considered insufficient since:

• Security: The sensitive data is only base64-encoded, meaning anyone with cluster access can easily decode and read any secret. This makes Kubernetes' secrets not fully secure.
• Lifecycle of secrets: As of this writing, Kubernetes  does not have native built-in capabilities to rotate secrets automatically. So secret rotation can become complex, especially if you want a fully automated and seamless process.
• Secret Management: Managing and synchronizing secrets across multiple environments can be challenging. Secrets may differ between environments, making it difficult to ensure consistency while customizing them per environment.

As the number of secrets grows, you may require additional tools to effectively manage them within a Kubernetes cluster and better manage the centralization of credentials, secrets and security policies.

…

This guide will demonstrate how to leverage the External Secrets Operator to securely manage and synchronize Kubernetes secrets with AWS Systems Manager Parameter Store. I’ll focus on how you can manage your secrets, how many different ways to fetch our secrets, and how we can  transform them. You’ll store your secrets on AWS Parameter Store with Secure String parameter type, which allows us to fetch secrets efficiently and at no extra cost for this demo.

Overview of External Secrets Operator

External Secrets Operator (ESO) is a Kubernetes operator that integrates with external secret management systems such as AWS Parameter Store, AWS Secret Manager, HashiCorp Vault, Google Cloud Secret Manager and many more. The operator manages secrets in a secure and scalable way by extending Kubernetes with Custom Resources that define where secrets live and how to synchronize them. The custom API resources provide an abstraction for the external APIs that stores and manages the lifecycle of the secrets. The controller fetches secrets from an external API and creates Kubernetes secrets. If the secret from the external API changes, the controller will reconcile the state in the cluster and update the secrets.

AWS Systems Manager Parameter Store Overview

Parameter Store, a capability of AWS Systems Manager, provides a secure, hierarchical storage for configuration data management and secrets management. You can store data such as passwords, database strings, Amazon Machine Image (AMI) IDs, and license codes as parameter values. You can store values as plain text or encrypted data.

Parameter type: SecureString

A SecureString parameter is any sensitive data that needs to be stored and referenced in a secure manner. If you have data that you don't want users to alter or reference in plaintext, such as passwords or license keys, you can create those parameters using the SecureString data type.

The AWS Systems Manager Parameter Store API is charged by throughput and is available in different tiers, if you consider using AWS Parameter Store in a real-life project see pricing.

Prerequisites

Before we start, you need to have the following prerequisites:

• AWS Account
• Helm installed
• AWS CLI installed
• KUBECTL installed
• EKSCTL installed

Getting Started

1. Set Secrets in AWS Systems Manager Parameter Store

As I mentioned above, you’ll store our secrets on AWS Parameter Store with the Secure String parameter type via the secure string option. It'll be automatically encrypted using AWS Key Management Service (KMS) when stored.

In real-world scenarios, using separate namespaces, service accounts, and policies can significantly enhance security by isolating sensitive credentials. Imagine you’re managing a cluster that hosts multiple products developed by different teams. Each product is deployed in its own namespace, with secrets that are entirely distinct and restricted to their respective teams. This isolation ensures that neither developers nor applications can access secrets that don’t belong to them.

For instance, let’s say you’re working on an application called the Booking Service in the development cluster that has MongoDB dependency. You’ve created a dedicated database user for the booking app, and its credentials need to remain specific to this application. By leveraging the hierarchical structure of AWS Parameter Store, you can grant granular access to the IAM Role for Service Accounts (IRSA) used by the SecretStore. This approach ensures that only the Booking Service team can access the relevant secrets, while other teams remain isolated from them. This setup not only enhances security but also simplifies secret management across teams.

Now, let’s walk through a step-by-step example to set this up in your own cluster. We’ll cover how to create an EKS cluster, install the External Secrets Operator, configure the service account, define the secrets in AWS Parameter Store, and grant access using IRSA. Finally, we’ll fetch the secrets from AWS Parameter Store using ESO’s Custom Resource Definitions (CRDs) to securely retrieve them in your application.

By the end of this exercise, you should have a better understanding of how to use the External Secrets Operator in your environment.

Create MONGODB_USERNAME and MONGODB_PASSWORD for your database user and put these parameters in a common path called /app/booking_service. Naming convention of the secrets in AWS Parameter Store would be like this:

• /app/booking_service/MONGODB_USERNAME

• /app/booking_service/MONGODB_PASSWORD

2. Create EKS Cluster

2.1 Create a Configuration File

Create a file named eks-cluster-config.yaml for your EKS cluster configuration.

cat < eks-cluster-config.yaml
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
  name: eks-external-secrets-operator
  region: eu-west-1
iam:
  withOIDC: true
nodeGroups:
  - name: ng-1
    instanceType: t3.small
    desiredCapacity: 1
    volumeSize: 8
EOF

Enabling by setting iam: withOIDC: true is essential for integrating Amazon EKS with IAM Roles for Service Accounts (IRSA). By enabling OIDC, EKS automatically creates an OpenID Connect (OIDC) identity provider for the cluster. This identity provider allows you to securely associate Kubernetes service accounts with IAM roles. This association makes it possible for the workloads running on the EKS cluster to assume IAM roles without requiring direct access to AWS credentials. This enhances security and flexibility.

2.2 Create the Cluster with eksctl

Run the following command to create the cluster:

 
eksctl create cluster -f eks-cluster-config.yaml

This command creates an EKS cluster with OIDC enabled based on your configuration file.

2.3 Verify the OIDC Provider

Once the cluster is created, you can verify that OIDC is enabled by running:

 
eksctl utils associate-iam-oidc-provider --cluster eks-external-secrets-operator --approve

The output should be: IAM Open ID Connect provider is already associated with cluster "eks-external-secrets-operator" in "eu-west-1”

3. Install External Secrets Operator from chart repository

Install External Secrets Operator with the following commands:

 
# Add helm repo to local
helm repo add external-secrets https://charts.external-secrets.io
helm repo update

# Install External Secrets Operator 
helm install external-secrets \
   external-secrets/external-secrets \
    -n external-secrets \
    --create-namespace

Wait until pod is ready:

It’s Time to Dive Into External Secrets Operator Resources!

SecretStore and ClusterSecretStore

Secret Store and Cluster Secret Store resources specify how to access an external API, including the external secret management service to use and the authentication configuration required for access.It contains references to secrets which hold credentials to access the external API.

So the question in mind may be what is the difference between these two CRDs? The answer is their scope.

The SecretStore is namespaced. SecretStores are bound to unique namespaces and can not reference resources across namespaces.

The ClusterSecretStore is on the other hand is a cluster scoped SecretStore that can be referenced by all ExternalSecrets from all namespaces.

That means, like in this article, if you prefer to use a service account with IRSA role annotation as authentication to AWS Parameter Store, in each region you use SecretStore, you are going to need to create a service account.

If you have different products managed by different developers spread across namespaces, and you want to separate AWS Parameter Store permissions of your applications, you can use SecretStore which is namespace scoped. Then you need to create different service accounts in each namespace, and you need to create different IRSA roles that have more granular policy to AWS Parameter Store secrets. In this way you can restrict the permission in that namespace by giving more granular policies through your IRSA.

ExternalSecret

The ExternalSecret describes what data should be fetched, how the data should be transformed and stored as a Kubernetes Secret object in your cluster. It has a reference to a SecretStore which knows how to access that data. The controller uses that ExternalSecret as a blueprint to create secrets.

It’s time to see the resources in action.

1. Create the Namespace

Run the following command to create a namespace.

 
kubectl create namespace booking-service

It’s time to see the resources in action.

2. Create booking-service Service Account IRSA configuration

You need to create a service account with the IRSA annotation for the External Secrets Operator to authenticate and fetch the required secrets from AWS Parameter Store.

2.1 Create the IAM Role with Trust Policy

Before running the following command replace <oidc-provider-url> and  <oidc-provider-arn> with your actual values. The namespace will be the booking-service. You can create trust policy file using the command line:

 
cat < trust-policy.json
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Federated": ""
            },
            "Action": "sts:AssumeRoleWithWebIdentity",
            "Condition": {
                "StringEquals": {
                    ":sub": "system:serviceaccount:booking-service:booking-service",
                    ":aud": "sts.amazonaws.com"
                }
            }
        }
    ]
}
EOF

You can retrieve the OIDC provider URL using the following command:

 
aws eks describe-cluster --name eks-external-secrets-operator --query "cluster.identity.oidc.issuer" --output text
EOF

2.2 Create the IAM Role Use the following command to create the IAM role:

 
aws iam create-role \
  --role-name external-secrets-operator-role \
  --assume-role-policy-document file://trust-policy.json

2.3 Create the IAM Policy

When it comes to policies, it's crucial to follow the Principle of Least Privilege. It is important to ensure that each role is granted only the minimal permissions necessary to perform its intended functions. Granting broader permissions can expose your resources to potential risks. The policy below shows the required permissions for fetching parameters from AWS Parameter Store. This policy permits pinning down access to secrets with a path matching under /app/booking_service/. The second statement will be required in the “6.4 Fetch Nested and Rewriting Keys in DataFrom” part. Before running the following command replace with your actual values. You can create this file using the command line:

 
cat < custom-policy.json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "ssm:GetParameter",
        "ssm:GetParameters",
        "ssm:GetParametersByPath",
        "ssm:DescribeParameters"
      ],
      "Resource": "arn:aws:ssm:eu-west-1::parameter/app/booking_service/*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "ssm:DescribeParameters"
      ],
      "Resource": "arn:aws:ssm:eu-west-1::*"
    }
  ]
}
EOF

2.4 Create the IAM Policy

Use the following command to create the IAM policy:

 
aws iam create-policy \
  --policy-name external-secrets-operator-policy \
  --policy-document file://custom-policy.json

2.5 Attach the Custom Policy to the IAM Role

Attach your user-defined policy to the role.

 
aws iam attach-role-policy \
  --role-name external-secrets-operator-role \
  --policy-arn arn:aws:iam:::policy/external-secrets-operator-policy

3. Create the Service Account

Run the following command to create a service account:

 
kubectl create serviceaccount booking-service -n booking-service

4. Annotate the Service Account with the IAM Role

Run the following command to annotate the created service account with the IAM role ARN:

 kubectl annotate serviceaccount booking-service \
eks.amazonaws.com/role-arn=arn:aws:iam:::role/external-secrets-operator-role \
  --namespace booking-service

When you check the service account, you should see that annotation is passed:

5. Create a SecretStore

Now it’s time to create a SecretStore.

 cat < secretstore.yaml
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: aws-parameter-store
  namespace: booking-service
spec:
  provider:
	aws:
  	  service: ParameterStore
  	  region: eu-west-1
  	  auth:
    	    jwt:
           serviceAccountRef:
        	   name: booking-service
EOF

- You specified our external API which is AWS Parameter Store, in spec.provider field

-You specified the service account that you created above in spec.provider.aws.auth field

Run the following command to create a SecretStore:

 kubectl apply -f secretstore.yaml

When you check, we should see that SecretStore status is valid:

6. Create a ExternalSecrets

There are a couple of ways that you can fetch your secrets from external API using ExternalSecrets resource. Let’s take a look at them.

6.1 Fetch Directly

You can fetch our secret by specifying directly:

 cat < external-secrets.yaml
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: booking-service
  namespace: booking-service
spec:
  refreshInterval: 1h
  secretStoreRef:
	name: aws-parameter-store
	kind: SecretStore
  target:
	name: aws-parameter-store-secrets
	creationPolicy: Owner
data:
	- secretKey: MONGODB_USERNAME
  	  remoteRef:
    	    key: "/app/booking_service/MONGODB_USERNAME"
	- secretKey: MONGODB_PASSWORD
  	  remoteRef:
    	    key: "/app/booking_service/MONGODB_PASSWORD"
EOF

- RefreshInterval indicates the frequency of reading the values from the SecretStore provider, external API. Since it is set to 1h, if you change the value of the secret in the AWS Parameter Store when the values just refreshed, it will reflect the secret in an hour.

- You specified your SecretStore that created previously in spec.secretStoreRef

- spec.target.name is the Kubernetes secret object name of the resource that will be created in the cluster

- spec.target.creationPolicy specifies the ExternalSecret ownership details in the created Secret. Since you set it to Owner, that means if the ExternalSecret is deleted, the Secret will also be deleted.

- spec.data defines the connection between the Kubernetes Secret keys and the Provider data

Above example shows how you can fetch secrets from the AWS Parameter Store one by one. In this example it will fetch the value of the secret from "/app/booking_service/MONGODB_USERNAME" of AWS Parameter Store and store the value in the Kubernetes Secret called aws-parameter-store-secrets with MONGODB_USERNAME key. In the same way, it will fetch the secret value from "/app/booking_service/MONGODB_PASSWORD" path and store it with MONGODB_PASSWORD key.

Run the following command to create a ExternalSecret:

 
kubectl apply -f external-secrets.yaml

When you check, you should see the secrets in base64 format:

You need to pass the secrets reference to the pod to achieve that secrets are accessible by pods. Here the snipped part of a pod configuration:

 
...
    env:
      - name: "MONGODB_USERNAME"
        valueFrom:
          secretKeyRef:
            name: aws-parameter-store-secrets
            key: "MONGODB_USERNAME"
...

It is also possible to import all keys from a single Kubernetes Secret into the container’s environment. You can find an example about this on “6.4. Fetch Nested and Rewriting Keys in DataFrom” section.

6.2 Dependent Environment Variables

You can also combine the value of our secrets to create dependent environment variables for our application. Let’s say your application requires an environment variable called MONGODB_CONNECTION_SECRET and the format of the connection secret is: mongodb+srv://<username>:<password>@beyondthebasics.abcde.mongodb.net/test. And you deployed the ExternalSecret resource like above.

You will create a Pod that runs one container. And pass secret references and combine them under the env: section of the pod. Here is the configuration manifest for the Pod:

 
apiVersion: v1
kind: Pod
metadata:
  name: nginx
spec:
  containers:
  - name: nginx
	image: nginx:1.14.2
	ports:
	- containerPort: 80
	env:
  	  - name: "MONGODB_USERNAME"
    	    valueFrom:
           secretKeyRef:
        	    name: aws-parameter-store-secrets
        	    key: "MONGODB_USERNAME"
  	  - name: "MONGODB_PASSWORD"
  valueFrom:
           secretKeyRef:
        	   name: aws-parameter-store-secrets
        	   key: "MONGODB_PASSWORD"
  	  - name: MONGODB_CONNECTION_STRING
    	    value: "mongodb+srv://$(MONGODB_USERNAME):$(MONGODB_PASSWORD)@beyondthebasics.abcde.mongodb.net/test"

As you can see, you gave the references of the secrets which ExternalSecrets fetches from AWS Parameter Store and then we combine them together. When you print the environment variables of the pod, you could see secrets have passed to the pod:

6.3 Fetch Directly and Manipulate the Secret Name

It is possible to transform the data from the external secret provider before being stored as Kubernetes secret. You can specify how the secret should look like by specifying under the spec.target.template.

 
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: booking-service
spec:
  refreshInterval: 1h
  secretStoreRef:
	name: aws-parameter-store
	kind: SecretStore
  target:
	name: aws-parameter-store-secrets
	creationPolicy: Owner
	template:
  	  engineVersion: v2
  	  data:
    	    db_password: ""
  data:
	- secretKey: MONGODB_PASSWORD
  	  remoteRef:
    	    key: "/app/booking_service/MONGODB_PASSWORD"

In this example it will fetch the secret from the AWS Parameter store and before storing it in Kubernetes Secret, it will manipulate the key value pair and add the secret key as db_password in Kubernetes Secret.

Since the Helm Chart templates are written in the Go programming language, if you deploy ExternalSecrets via Helm, the template must be escaped so that Helm will not try to render it. In that case the .spec.target.template.data part should be passed like this: " }}"

You should see that secrets passed to Kubernetes Secret with the db_password key!

This is also helpful if you are working with one of the common Kubernetes secret types like .dockerconfigjson. As an example:

 
...
target:
  name: secret-to-be-created
  creationPolicy: Owner
  template:
	type: kubernetes.io/dockerconfigjson
	engineVersion: v2
	data:
  	  .dockerconfigjson: ""
data:
- secretKey: mysecret
  remoteRef:
	key: docker-config-example  

6.4 Fetch Nested and Rewriting Keys in DataFrom

In this example you will fetch the secret’s nested keys by path and remove a common path from the secret keys with dataFrom.find using regex! You can get all values using spec.dataFrom from the AWS Parameter Store.

To do this, you will define a rewrite operation using dataFrom.rewrite through the use of regular expressions.

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: booking-service
spec:
  refreshInterval: 1h
  secretStoreRef:
	name: aws-parameter-store
	kind: SecretStore
  target:
	name: aws-parameter-store-secrets
	creationPolicy: Owner
  dataFrom:
  - find:
      path: "/app"
  	 name:
       regexp: "booking_service"
    rewrite:
    - regexp:
        source: "/app/booking_service/(.*)"
    	   target: "$1"

In this example, since you have the following secrets available in AWS Parameter Store:

/app/booking_service/MONGODB_USERNAME

/app/booking_service/MONGODB_USERNAME

You will get all the secrets matching /app/booking_service/* and then rewrite them by removing the common path away.

This is how the secret look like:

It is possible to define environment variables in bulk from a ConfigMap or Secret in the pod to make secrets accessible by pods. Here the snipped part of a pod configuration for that:

...
    envFrom:
      - secretRef:
          name: aws-parameter-store-secrets
...

You will get all the secrets matching /app/booking_service/* and then rewrite them by removing the common path away.

Conclusion

Managing Kubernetes secrets securely and efficiently is critical, especially when working with applications that handle sensitive data. By using External Secrets Operator with AWS Systems Manager Parameter Store, you can securely centralize secret management, enforce role-based access, and automate secret synchronization and rotation without exposing sensitive information directly within Kubernetes clusters. This approach improves security, simplifies secret management across multiple environments, and enables scaling while maintaining consistency. Through this hands-on guide, we have covered how to integrate External Secrets Operator with AWS Parameter Store, to configure an EKS cluster with OIDC, to create and manage IAM roles, and to define custom resources for securely fetching and managing secrets in Kubernetes. By following these steps, you can build a more secure and scalable system for managing secrets in your Kubernetes environment.

References

• https://external-secrets.io
• https://docs.aws.amazon.com/systems-manager/latest/userguide/systems-manager-parameter-store.html
• https://kubernetes.io/docs/tasks/inject-data-application/define-interdependent-environment-variables/
• https://kubernetes.io/docs/concepts/workloads/pods/