The Helm and GitOps Groundwork Before Self-Service Is Possible-Part 1

The Helm and GitOps groundwork that has to exist before self-service is possible, and the details that decide whether it holds up.

Part 1 of 2. This post is about the building blocks: one shared chart, one place that declares which service runs where, and a way to migrate onto both without a cutover weekend. Part 2, Building a Golden Path, is about the self-service layer that generates them.

Every Kubernetes estate that grew organically looks the same from the inside. Each service carries its own automation/helm directory. Each deployment.yaml is almost identical to the others, and different in one detail nobody remembers deciding. Every team invented its own way of getting secrets out of AWS. None of that is anybody's fault; it is what happens when 40 services are added one at a time by people doing their actual jobs.

It becomes a problem the moment you want self-service. A golden path is the supported, opinionated route from "I need a service" to "it is running": conventions made executable, usually behind a form, always with a maintained road underneath. Automate conventions that were never agreed and the template becomes the place they get invented, which leaves it as the only record of a decision nobody remembers making.

So the wizard is the second problem. This post is the first one: consolidating dozens of backend services, four environments and two AWS regions onto shared building blocks.

What's Inside?

1. One chart instead of forty a shared chart and thin umbrella charts per service and environment
2. Secrets sync-wave ordering, and what a rotated value still needs to reach the process
3. Owning a shared chart versioning, rendering tests, consumer diffs
4. App-of-apps which services exist in which environment, as toggle files
5. The migration switch moving services onto the shared chart one environment at a time

Note: The estate here is Kubernetes, and the artifacts follow from that. If yours mixes runtimes serverless functions, Linux services on VMs, managed databases that is what the design step has to resolve, because each resource type produces a different deployable artifact and usually a different delivery tool. Decide up front which of them the standardization covers.

One Chart Instead of Forty

A single general-purpose chart, common-chart, is replacing every service's private copy: Deployment, Service, Ingress, HPA, PodDisruptionBudget, Jobs and CronJobs, ServiceAccount and RBAC, ConfigMaps, and the External Secrets wiring. It is published to a private OCI registry and versioned with release-please.

Services do not fork it. They depend on it through a thin umbrella chart, one directory per service per environment:

Because common-chart is a dependency, everything service-specific nests one level down under the chart's own name. The root of the values file is where other subcharts sit, so common-chart: is one key among them:

The umbrella holds a version pin and a values file. Everything that renders lives in the shared chart, which is what makes a service's whole deployment description something a form can generate.

Two names that look alike: common-chart is the chart in the OCI registry, common-helm-chart is the Git repository holding both its source and every service's umbrella directory.

A design question usually follows: why a subchart dependency rather than a Helm library chart? A library chart ships helpers, not manifests, so every umbrella would carry template files of its own that call them. Here the umbrella carries no templates at all, which is what a generator needs: producing a values.yaml is templating, while producing template code is code generation.

One detail worth copying: the Deployment only sets replicas when autoscaling is off.

Emit it unconditionally and Git and the HPA fight forever: the GitOps controller reverts the HPA's scale, the HPA scales again, and the Application never settles as synced. The usual patch is an ignoreDifferences block on every Application. Not rendering the field is cheaper, and removes the cause instead of hiding the symptom.

Secrets: Declaring Where a Value Comes From

The chart absorbs secrets on top of the External Secrets Operator. A variable declares where it comes from, inside the same nested values file as everything else:

The chart scans every env entry for parameterStoreRef and secretsManagerRef, builds an ExternalSecret with one data entry per reference, and rewrites the pod spec into a plain secretKeyRef. Env var name and AWS path stay independent, so one parameter can surface in three services under three different names.

There is a second mode. autoEnvs: true fetches everything under the service's path with a single dataFrom block and injects it through envFrom. Ours runs autoEnvs in every long-lived environment, with explicit references mainly in preview environments where a name has to be chosen deliberately. Both cost something: explicit references let you see what is exposed and rename it, while autoEnvs means adding a parameter in AWS needs no chart change at all.

Ordering, and What Carries It

The ClusterSecretStore renders at argocd.argoproj.io/sync-wave: "-2", the ExternalSecret at "-1", the Deployment in wave 0. What makes that work is not the numbers on their own. ArgoCD moves to the next wave once the current one reports healthy, and health for a custom resource depends on a registered health check. ArgoCD ships one for external-secrets.io/ExternalSecret, so the resource is not healthy until the operator has populated it, and wave 0 waits.

What is worth checking is that you have not replaced it. A resource.customizations entry covering that group, including one inherited from a broad wildcard, overrides the built-in check, and a custom function that returns healthy immediately silently removes the ordering. Note too that the Kubernetes Secret is created by the operator, is not part of the Application, and is never tracked by the sync: the sequence works because the ExternalSecret's health stands in for the Secret's existence.

The store and the ExternalSecret are ordered by sync waves. The Secret is written by the operator, outside the GitOps controller's view, and the pod reads it once at process start.

What a Rotation Still Needs

refreshInterval: 1m keeps the Kubernetes Secret within a minute of the source parameter, and that is the easy half. Secrets arrive in the container as environment variables, and an env var is read once at process start. Refreshing the Secret alone leaves every running pod on the old value, and the failure is quiet: pods pick up the new secret whenever they happen to restart, so a rotation spreads at the speed of unrelated deploys.

The obvious Helm answer does not close it. A checksum/secret pod annotation is computed at render time over what the chart renders, and this Secret is materialized by the operator, so the values in Git are byte-identical before and after a rotation and the checksum never moves. The GitOps controller has nothing to act on either, because the Deployment spec did not change.

Closing it takes a controller that watches the Secret object itself and rolls the Deployment when its contents change. That controller is not wired into our clusters yet, so today a rotation is finished by a deliberate restart rather than by the platform. It is the next thing going in, and the chart is already shaped for it: a workload opts in through podAnnotations, keeping it a per-service decision, and the roll is safe to automate because probes and an optional PodDisruptionBudget already ship with the chart. Under autoEnvs a Deployment references the whole parameter path, so blanket auto-reload restarts a service for keys it never reads — which is why the annotation goes on the services whose secrets actually rotate first, and why busy paths are better moved to explicit references. Until then, "rotate the parameter, then restart these Deployments" belongs in a runbook rather than in someone's memory.

Note the polling bill. refreshInterval: 1m is per ExternalSecret, and services generate more than one. Multiply by an estate and you are making a steady stream of Parameter Store calls all day, against an account-level throughput limit that is modest by default and chargeable to raise. Throttled reads look like unrelated startup failures. Set the interval per sensitivity: a minute for things that genuinely rotate, much longer for the rest.

Two Things to Decide Deliberately

Where the read boundary sits. A ClusterSecretStore is cluster-scoped, and one authenticating identity behind it means any values file in the cluster can name any path that identity can read. Rendering a cluster-scoped object per service compounds it: its lifecycle belongs to whichever Application rendered it, and its name must stay unique cluster-wide. If services need isolation from each other, the better default is a namespaced SecretStore per namespace with its own IRSA role, so the boundary is enforced by IAM instead of by convention.

What the chart is not responsible for. No ServiceMonitor, no PrometheusRule: observability is wired at cluster level. A defensible scope boundary, worth stating so nobody reads "shared chart" as "production-ready service".

Clean-up runs in two directions. creationPolicy: Owner puts an owner reference on the Kubernetes Secret, so removing the ExternalSecret removes the Secret. deletionPolicy governs what happens when the data disappears at the provider, and it fires when every entry the ExternalSecret asks for has gone missing, not when one key of several does:

Retain fails safe rather than emptying a live Secret, and it is permanent: a pod restart re-reads the same stale value. Merge is the one to reach for when something else also writes into that Secret. Retain is usually right, and should still be written down rather than inherited.

Owning a Shared Chart

A chart 40 services depend on is a product with users. Four habits carried most of the weight:

Version it, and pin it. Semantic versioning driven by release-please, and every umbrella pins an exact version. A floating dependency means an unrelated merge in the chart repository decides what deploys tonight.

Test the rendering, not the syntax. helm lint catches almost nothing that matters. CI renders the chart against representative values files, validates output with kubeconform against the Kubernetes version actually running, asserts on what must not move with helm unittest snapshots, and runs policy checks over the rendered manifests: no latest tags, resource limits present, runAsNonRoot set.

Diff every change against real consumers. Render the previous version and the candidate against the same values files, and post the diff on the pull request. A change that silently drops a volumeMount is invisible in a review and obvious in a diff.

Write the changelog for the person upgrading, and keep a generated report of which service sits on which version. Otherwise the tail of services pinned to an old minor becomes permanent. Renovate against the umbrella charts turns that report into pull requests instead of a spreadsheet.

Note the OCI registry tax. Registry authorization tokens expire, ECR's after 12 hours. Whatever renders your charts, the GitOps controller's repo-server or CI, needs a credential refreshed on a schedule rather than created once. It is a Day 1 problem you discover on Day 2, as a repo-server that renders fine all morning and cannot pull the dependency after lunch.

App-of-Apps: Which Services Exist Where

A second repository answers a different question: not how a service deploys, but which services exist in which environment.

It is an ArgoCD app-of-apps. One Application is applied by hand, once, per cluster, and renders a Helm chart whose templates are the other Applications, one file per service behind an enable guard. Every environment becomes a toggle file:

Each region has its own values directory because each region is its own cluster running its own ArgoCD, which is also why Application names stay simple.

The chart's defaults are deliberately unusable: not just enable: false, but environment: "fake" and branch: "fake", so an environment file that forgets a service cannot quietly inherit a real branch and deploy something nobody asked for. A safe default is one that cannot half-work. Autosync is per service and per environment: dev self-heals, anything closer to production is a deliberate human sync, and that policy lives in a values file rather than in somebody's memory.

Why Not an ApplicationSet?

The lazy answer is that a git directory generator only enumerates directories while this file carries policy. That answer is wrong, and anyone who knows ArgoCD will say so: a git files generator reads a per-service config file, goTemplate plus templatePatch render conditional fields, and matrix and merge generators combine sources. Almost everything in these toggle files is expressible.

The real reason is that expressing it buys nothing here. A files generator moves the same per-service block from one YAML file into another, and pays for it with a generator template that has to absorb every shape the estate contains, including services that render a different set of Applications from their neighbours. That is a rewrite of a working, readable layer in exchange for the same information in a less obvious place. Where a generator genuinely wins is when the entries stop differing which is exactly what part 2 produces.

What the Sync Policy Does Not Cover

Every generated Application carries syncOptions: [CreateNamespace=true] plus automated.selfHeal: true where the toggle asks for it. Three absences matter:

No prune. Pruning deletes resources removed from the desired state. Without it, a resource you take out of a chart stays in the cluster, reported out of sync and otherwise untouched. selfHeal does not help and is often misread as if it did: it re-applies desired state to correct drift, so it restores a resource somebody deleted by hand, and has nothing to say about one that is no longer in the desired state at all. Remove a CronJob from a chart without pruning and it keeps running.

No finalizer. Deleting an Application removes only the Application object unless it carries resources-finalizer.argocd.argoproj.io, which is what cascades the delete to everything it created.

No ServerSideApply. Client-side apply stores the previous manifest in the kubectl.kubernetes.io/last-applied-configuration annotation, and annotations cap at 256 KB. Large CRDs cross that line and fail to apply. Server-side apply avoids the annotation entirely.

The first two interact in a way that changes what a toggle does. Turning enable: false removes an Application from what the parent renders; if the parent also syncs without pruning, the child Application is not deleted either. It stops being managed and keeps running, so switching a service off is a two-step operation today: flip the toggle, then delete the Application by hand. Enabling pruning on the parent makes the first step delete the Application, and adding the finalizer makes that delete carry the workloads with it. Decide both together, because pruning without the finalizer gives you the confusing middle state where the Application disappears from the UI while the pods keep serving traffic.

The Migration Switch

Forty services cannot move in one weekend. When a service migrates off its own automation/helm directory, we do not rewrite its Application template. We add a branch to it:

Migration and rollback are the same line in the same file. The Application object itself never changes, which is why this can be done one service and one environment at a time.

Migration becomes one line in one environment values file, and rollback is the same line. Migrate dev, sit on it for a week, leave prod alone.

Now look at the last line, because that is the one that bit us. targetRevision sits outside the conditional and reuses a single branch value for both cases, which quietly means two different things: a branch in the service repository before migration, and a branch in the chart repository after. A service tracking release/2024-11 migrated onto a chart-repo branch of that name, which did not exist, and the Application went straight to a comparison error.

The immediate fix brings the revision inside the same conditional that moved the repository:

targetRevision: dev

That is where several of our templates are today, and it is worth being exact. It removes the failure above, because a migrated service no longer looks for its own release branch in a repository that has never heard of it. What it does not remove is the coupling: a hardcoded revision pins every migrated environment to one branch of the chart repository. While dev is the only migrated environment that is precisely what you want, which is why the line reads this way. The moment a second environment follows it becomes the same mistake one level deeper, with production reading a branch chosen for development.

So the sequence matters more than the snippet. Moving an environment onto the shared chart means giving the revision its own per-environment value at the same time, not after, and giving that value a deliberately unusable default the way environment and branch already have one. A missing revision does not fail loudly: Helm renders an empty targetRevision and the Application tracks whatever the chart repository's default branch happens to be that day.

The rule is the part to carry away. If a switch changes where a source lives, it has to change every field that means something different in the new location, and a field that only looks correct because one environment has migrated is a deferred bug rather than a solved one.

Note: Pair the switch with a mechanical habit. Render the old chart and the new chart against the real values, diff them, do the same for the rendered Application manifests, and only then flip the flag. A migration that produces an empty diff is boring, which is the whole idea.

The switch is scaffolding rather than a fixture, and an estate mid-consolidation shows all three stages at once: services still on their own chart with the flag absent, services carrying it in the environments they have moved, and finished services whose conditional has been removed entirely because a branch nobody will flip back is dead code in a file dozens of services read.

Finally, be precise about what "rollback is a revert" buys. Reverting the flag puts the old manifests back, but it is not helm rollback: a GitOps controller applies desired state, it does not replay releases. If the new chart version created resources the old one does not manage, reverting leaves them behind until pruning is enabled or somebody deletes them.

Final Thoughts

None of this is glamorous work. It is a chart, a values file and a boolean, and the payoff is measured in pull requests that never had to be opened.

• One chart to change instead of one per service, with the blast radius visible as a rendered diff before it merges.
• A new environment for an existing service that is a directory and a version pin, rather than a copied templates/ tree nobody will keep in step.
• Secret wiring declared next to the variable that uses it, ordered by sync waves, with the one remaining manual step named rather than assumed.
• A migration that moves one service in one environment and is reversible by the line that moved it.

And then the part that surprised us: none of it made creating a new service self-service. Everything was standardized and a developer still could not do it alone, because standardization and discoverability are different problems. Every step existed somewhere in a repository. Not one of them was discoverable by someone who had not done it before.

That gap, and the golden path we built on these building blocks to close it, is part 2: Building a Golden Path: From Shared Helm Charts to Developer Self-Service.

References

• ArgoCD: Cluster Bootstrapping (App of Apps)
• ArgoCD: Sync Phases and Waves
• ArgoCD: Automated sync and pruning
• ArgoCD: Sync options, including Server-Side Apply
• ArgoCD: App deletion and the resources finalizer
• ArgoCD: ApplicationSet generators
• External Secrets Operator
• External Secrets: ExternalSecret API, including deletionPolicy
• Helm: Subcharts and global values
• Helm: Library charts
• Kloia: Managing Kubernetes Clusters with GitOps
• Kloia: Manage Kubernetes Secrets with External Secrets Operator and AWS SSM