Kubernetes secrets is an object that by default allows to store confidential data such as a password, a token, or a key in the API server's underlying data store called etcd. Using Kubernetes secrets, you will decouple hard-coded sensitive data from your application code, container image, or pod definition.
However, Kubernetes secrets often considered insufficient since:
As the number of secrets grows, you may require additional tools to effectively manage them within a Kubernetes cluster and better manage the centralization of credentials, secrets and security policies.
…
This guide will demonstrate how to leverage the External Secrets Operator to securely manage and synchronize Kubernetes secrets with AWS Systems Manager Parameter Store. I’ll focus on how you can manage your secrets, how many different ways to fetch our secrets, and how we can transform them. You’ll store your secrets on AWS Parameter Store with Secure String parameter type, which allows us to fetch secrets efficiently and at no extra cost for this demo.
External Secrets Operator (ESO) is a Kubernetes operator that integrates with external secret management systems such as AWS Parameter Store, AWS Secret Manager, HashiCorp Vault, Google Cloud Secret Manager and many more. The operator manages secrets in a secure and scalable way by extending Kubernetes with Custom Resources that define where secrets live and how to synchronize them. The custom API resources provide an abstraction for the external APIs that stores and manages the lifecycle of the secrets. The controller fetches secrets from an external API and creates Kubernetes secrets. If the secret from the external API changes, the controller will reconcile the state in the cluster and update the secrets.
Parameter Store, a capability of AWS Systems Manager, provides a secure, hierarchical storage for configuration data management and secrets management. You can store data such as passwords, database strings, Amazon Machine Image (AMI) IDs, and license codes as parameter values. You can store values as plain text or encrypted data.
A SecureString parameter is any sensitive data that needs to be stored and referenced in a secure manner. If you have data that you don't want users to alter or reference in plaintext, such as passwords or license keys, you can create those parameters using the SecureString data type.
The AWS Systems Manager Parameter Store API is charged by throughput and is available in different tiers, if you consider using AWS Parameter Store in a real-life project see pricing.
Before we start, you need to have the following prerequisites:
As I mentioned above, you’ll store our secrets on AWS Parameter Store with the Secure String parameter type via the secure string option. It'll be automatically encrypted using AWS Key Management Service (KMS) when stored.
In real-world scenarios, using separate namespaces, service accounts, and policies can significantly enhance security by isolating sensitive credentials. Imagine you’re managing a cluster that hosts multiple products developed by different teams. Each product is deployed in its own namespace, with secrets that are entirely distinct and restricted to their respective teams. This isolation ensures that neither developers nor applications can access secrets that don’t belong to them.
For instance, let’s say you’re working on an application called the Booking Service in the development cluster that has MongoDB dependency. You’ve created a dedicated database user for the booking app, and its credentials need to remain specific to this application. By leveraging the hierarchical structure of AWS Parameter Store, you can grant granular access to the IAM Role for Service Accounts (IRSA) used by the SecretStore. This approach ensures that only the Booking Service team can access the relevant secrets, while other teams remain isolated from them. This setup not only enhances security but also simplifies secret management across teams.
Now, let’s walk through a step-by-step example to set this up in your own cluster. We’ll cover how to create an EKS cluster, install the External Secrets Operator, configure the service account, define the secrets in AWS Parameter Store, and grant access using IRSA. Finally, we’ll fetch the secrets from AWS Parameter Store using ESO’s Custom Resource Definitions (CRDs) to securely retrieve them in your application.
By the end of this exercise, you should have a better understanding of how to use the External Secrets Operator in your environment.
Create MONGODB_USERNAME and MONGODB_PASSWORD for your database user and put these parameters in a common path called /app/booking_service. Naming convention of the secrets in AWS Parameter Store would be like this:
Create a file named eks-cluster-config.yaml for your EKS cluster configuration.
cat < eks-cluster-config.yaml
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
name: eks-external-secrets-operator
region: eu-west-1
iam:
withOIDC: true
nodeGroups:
- name: ng-1
instanceType: t3.small
desiredCapacity: 1
volumeSize: 8
EOF
Enabling by setting iam: withOIDC: true is essential for integrating Amazon EKS with IAM Roles for Service Accounts (IRSA). By enabling OIDC, EKS automatically creates an OpenID Connect (OIDC) identity provider for the cluster. This identity provider allows you to securely associate Kubernetes service accounts with IAM roles. This association makes it possible for the workloads running on the EKS cluster to assume IAM roles without requiring direct access to AWS credentials. This enhances security and flexibility.
Run the following command to create the cluster:
eksctl create cluster -f eks-cluster-config.yaml
This command creates an EKS cluster with OIDC enabled based on your configuration file.
Once the cluster is created, you can verify that OIDC is enabled by running:
eksctl utils associate-iam-oidc-provider --cluster eks-external-secrets-operator --approveThe output should be: IAM Open ID Connect provider is already associated with cluster "eks-external-secrets-operator" in "eu-west-1”
Install External Secrets Operator with the following commands:
# Add helm repo to local
helm repo add external-secrets https://charts.external-secrets.io
helm repo update
# Install External Secrets Operator
helm install external-secrets \
external-secrets/external-secrets \
-n external-secrets \
--create-namespace
Wait until pod is ready:
Secret Store and Cluster Secret Store resources specify how to access an external API, including the external secret management service to use and the authentication configuration required for access.It contains references to secrets which hold credentials to access the external API.
So the question in mind may be what is the difference between these two CRDs? The answer is their scope.
The SecretStore is namespaced. SecretStores are bound to unique namespaces and can not reference resources across namespaces.
The ClusterSecretStore is on the other hand is a cluster scoped SecretStore that can be referenced by all ExternalSecrets from all namespaces.
That means, like in this article, if you prefer to use a service account with IRSA role annotation as authentication to AWS Parameter Store, in each region you use SecretStore, you are going to need to create a service account.
If you have different products managed by different developers spread across namespaces, and you want to separate AWS Parameter Store permissions of your applications, you can use SecretStore which is namespace scoped. Then you need to create different service accounts in each namespace, and you need to create different IRSA roles that have more granular policy to AWS Parameter Store secrets. In this way you can restrict the permission in that namespace by giving more granular policies through your IRSA.
The ExternalSecret describes what data should be fetched, how the data should be transformed and stored as a Kubernetes Secret object in your cluster. It has a reference to a SecretStore which knows how to access that data. The controller uses that ExternalSecret as a blueprint to create secrets.
It’s time to see the resources in action.
Run the following command to create a namespace.
kubectl create namespace booking-service
It’s time to see the resources in action.
You need to create a service account with the IRSA annotation for the External Secrets Operator to authenticate and fetch the required secrets from AWS Parameter Store.
Before running the following command replace <oidc-provider-url> and <oidc-provider-arn> with your actual values. The namespace will be the booking-service. You can create trust policy file using the command line:
cat < trust-policy.json
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Federated": ""
},
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {
"StringEquals": {
":sub": "system:serviceaccount:booking-service:booking-service",
":aud": "sts.amazonaws.com"
}
}
}
]
}
EOF
You can retrieve the OIDC provider URL using the following command:
aws eks describe-cluster --name eks-external-secrets-operator --query "cluster.identity.oidc.issuer" --output text
EOF
aws iam create-role \
--role-name external-secrets-operator-role \
--assume-role-policy-document file://trust-policy.json
When it comes to policies, it's crucial to follow the Principle of Least Privilege. It is important to ensure that each role is granted only the minimal permissions necessary to perform its intended functions. Granting broader permissions can expose your resources to potential risks. The policy below shows the required permissions for fetching parameters from AWS Parameter Store. This policy permits pinning down access to secrets with a path matching under /app/booking_service/. The second statement will be required in the “6.4 Fetch Nested and Rewriting Keys in DataFrom” part. Before running the following command replace with your actual values. You can create this file using the command line:
cat < custom-policy.json
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"ssm:GetParameter",
"ssm:GetParameters",
"ssm:GetParametersByPath",
"ssm:DescribeParameters"
],
"Resource": "arn:aws:ssm:eu-west-1::parameter/app/booking_service/*"
},
{
"Effect": "Allow",
"Action": [
"ssm:DescribeParameters"
],
"Resource": "arn:aws:ssm:eu-west-1::*"
}
]
}
EOF
Use the following command to create the IAM policy:
aws iam create-policy \
--policy-name external-secrets-operator-policy \
--policy-document file://custom-policy.json
Attach your user-defined policy to the role.
aws iam attach-role-policy \
--role-name external-secrets-operator-role \
--policy-arn arn:aws:iam:::policy/external-secrets-operator-policy
Run the following command to create a service account:
kubectl create serviceaccount booking-service -n booking-service
Run the following command to annotate the created service account with the IAM role ARN:
kubectl annotate serviceaccount booking-service \
eks.amazonaws.com/role-arn=arn:aws:iam:::role/external-secrets-operator-role \
--namespace booking-service
When you check the service account, you should see that annotation is passed:
Now it’s time to create a SecretStore.
cat < secretstore.yaml
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
name: aws-parameter-store
namespace: booking-service
spec:
provider:
aws:
service: ParameterStore
region: eu-west-1
auth:
jwt:
serviceAccountRef:
name: booking-service
EOF
- You specified our external API which is AWS Parameter Store, in spec.provider field
-You specified the service account that you created above in spec.provider.aws.auth field
Run the following command to create a SecretStore:
kubectl apply -f secretstore.yaml
When you check, we should see that SecretStore status is valid:
There are a couple of ways that you can fetch your secrets from external API using ExternalSecrets resource. Let’s take a look at them.
You can fetch our secret by specifying directly:
cat < external-secrets.yaml
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: booking-service
namespace: booking-service
spec:
refreshInterval: 1h
secretStoreRef:
name: aws-parameter-store
kind: SecretStore
target:
name: aws-parameter-store-secrets
creationPolicy: Owner
data:
- secretKey: MONGODB_USERNAME
remoteRef:
key: "/app/booking_service/MONGODB_USERNAME"
- secretKey: MONGODB_PASSWORD
remoteRef:
key: "/app/booking_service/MONGODB_PASSWORD"
EOF
- RefreshInterval indicates the frequency of reading the values from the SecretStore provider, external API. Since it is set to 1h, if you change the value of the secret in the AWS Parameter Store when the values just refreshed, it will reflect the secret in an hour.
- You specified your SecretStore that created previously in spec.secretStoreRef
- spec.target.name is the Kubernetes secret object name of the resource that will be created in the cluster
- spec.target.creationPolicy specifies the ExternalSecret ownership details in the created Secret. Since you set it to Owner, that means if the ExternalSecret is deleted, the Secret will also be deleted.
- spec.data defines the connection between the Kubernetes Secret keys and the Provider data
Above example shows how you can fetch secrets from the AWS Parameter Store one by one. In this example it will fetch the value of the secret from "/app/booking_service/MONGODB_USERNAME" of AWS Parameter Store and store the value in the Kubernetes Secret called aws-parameter-store-secrets with MONGODB_USERNAME key. In the same way, it will fetch the secret value from "/app/booking_service/MONGODB_PASSWORD" path and store it with MONGODB_PASSWORD key.
Run the following command to create a ExternalSecret:
kubectl apply -f external-secrets.yaml
When you check, you should see the secrets in base64 format:
You need to pass the secrets reference to the pod to achieve that secrets are accessible by pods. Here the snipped part of a pod configuration:
...
env:
- name: "MONGODB_USERNAME"
valueFrom:
secretKeyRef:
name: aws-parameter-store-secrets
key: "MONGODB_USERNAME"
...
It is also possible to import all keys from a single Kubernetes Secret into the container’s environment. You can find an example about this on “6.4. Fetch Nested and Rewriting Keys in DataFrom” section.
You can also combine the value of our secrets to create dependent environment variables for our application. Let’s say your application requires an environment variable called MONGODB_CONNECTION_SECRET and the format of the connection secret is: mongodb+srv://<username>:<password>@beyondthebasics.abcde.mongodb.net/test. And you deployed the ExternalSecret resource like above.
You will create a Pod that runs one container. And pass secret references and combine them under the env: section of the pod. Here is the configuration manifest for the Pod:
apiVersion: v1
kind: Pod
metadata:
name: nginx
spec:
containers:
- name: nginx
image: nginx:1.14.2
ports:
- containerPort: 80
env:
- name: "MONGODB_USERNAME"
valueFrom:
secretKeyRef:
name: aws-parameter-store-secrets
key: "MONGODB_USERNAME"
- name: "MONGODB_PASSWORD"
valueFrom:
secretKeyRef:
name: aws-parameter-store-secrets
key: "MONGODB_PASSWORD"
- name: MONGODB_CONNECTION_STRING
value: "mongodb+srv://$(MONGODB_USERNAME):$(MONGODB_PASSWORD)@beyondthebasics.abcde.mongodb.net/test"
As you can see, you gave the references of the secrets which ExternalSecrets fetches from AWS Parameter Store and then we combine them together. When you print the environment variables of the pod, you could see secrets have passed to the pod:
It is possible to transform the data from the external secret provider before being stored as Kubernetes secret. You can specify how the secret should look like by specifying under the spec.target.template.
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: booking-service
spec:
refreshInterval: 1h
secretStoreRef:
name: aws-parameter-store
kind: SecretStore
target:
name: aws-parameter-store-secrets
creationPolicy: Owner
template:
engineVersion: v2
data:
db_password: ""
data:
- secretKey: MONGODB_PASSWORD
remoteRef:
key: "/app/booking_service/MONGODB_PASSWORD"
In this example it will fetch the secret from the AWS Parameter store and before storing it in Kubernetes Secret, it will manipulate the key value pair and add the secret key as db_password in Kubernetes Secret.
Since the Helm Chart templates are written in the Go programming language, if you deploy ExternalSecrets via Helm, the template must be escaped so that Helm will not try to render it. In that case the .spec.target.template.data part should be passed like this: " }}"
You should see that secrets passed to Kubernetes Secret with the db_password key!
This is also helpful if you are working with one of the common Kubernetes secret types like .dockerconfigjson. As an example:
...
target:
name: secret-to-be-created
creationPolicy: Owner
template:
type: kubernetes.io/dockerconfigjson
engineVersion: v2
data:
.dockerconfigjson: ""
data:
- secretKey: mysecret
remoteRef:
key: docker-config-example In this example you will fetch the secret’s nested keys by path and remove a common path from the secret keys with dataFrom.find using regex! You can get all values using spec.dataFrom from the AWS Parameter Store.
To do this, you will define a rewrite operation using dataFrom.rewrite through the use of regular expressions.
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: booking-service
spec:
refreshInterval: 1h
secretStoreRef:
name: aws-parameter-store
kind: SecretStore
target:
name: aws-parameter-store-secrets
creationPolicy: Owner
dataFrom:
- find:
path: "/app"
name:
regexp: "booking_service"
rewrite:
- regexp:
source: "/app/booking_service/(.*)"
target: "$1"
In this example, since you have the following secrets available in AWS Parameter Store:
/app/booking_service/MONGODB_USERNAME
/app/booking_service/MONGODB_USERNAME
You will get all the secrets matching /app/booking_service/* and then rewrite them by removing the common path away.
This is how the secret look like:
It is possible to define environment variables in bulk from a ConfigMap or Secret in the pod to make secrets accessible by pods. Here the snipped part of a pod configuration for that:
...
envFrom:
- secretRef:
name: aws-parameter-store-secrets
...
You will get all the secrets matching /app/booking_service/* and then rewrite them by removing the common path away.
Managing Kubernetes secrets securely and efficiently is critical, especially when working with applications that handle sensitive data. By using External Secrets Operator with AWS Systems Manager Parameter Store, you can securely centralize secret management, enforce role-based access, and automate secret synchronization and rotation without exposing sensitive information directly within Kubernetes clusters. This approach improves security, simplifies secret management across multiple environments, and enables scaling while maintaining consistency. Through this hands-on guide, we have covered how to integrate External Secrets Operator with AWS Parameter Store, to configure an EKS cluster with OIDC, to create and manage IAM roles, and to define custom resources for securely fetching and managing secrets in Kubernetes. By following these steps, you can build a more secure and scalable system for managing secrets in your Kubernetes environment.