Building a Golden Path: Kubernetes Self-Service with Backstage-Part 2

How a standardized Kubernetes estate becomes a platform where shipping a new service is a form instead of a ticket, and why the interesting part is not the wizard but where the platform stops writing.

Part 2 of 2. Part 1, One Chart Instead of Forty, covers the building blocks this post generates. The first section recaps them, so this one stands on its own.

Ask a platform team how deployment works at their company and you usually get a confident answer. One chart. One pipeline. One GitOps repository. Everything is standardized.

Then ask a developer who joined three weeks ago to deploy a new service on their own.

What actually happens is a Slack thread. Somebody senior pastes a link to an existing service and says "copy this one, but change the IAM role." Two days later there are four pull requests across three repositories the repository that declares which services exist takes two of them on its own, one adding the service and a later one flipping the toggle that enables it. They get merged in the wrong order, and a pod sits in CreateContainerConfigError because nobody mentioned the sync-wave annotation.

None of that is a standardization failure. The chart really was shared, the conventions really did exist, the senior engineer was right. Standardization and self-service are simply different problems, and a lot of platform work quietly stalls in the gap between them.

A golden path closes that gap. It is the supported, opinionated route from "I need a service" to "it is running": conventions made executable and maintained as a product, usually reached through a form, but defined by the road rather than the door.

We built ours on a Backstage-based developer portal. The wizard turned out to be the least interesting part. The rule underneath it mattered far more: the platform writes to Git, and stops.

What's Inside?

1. The ground it stands on what had to be standardized first
2. The gap nobody talks about four steps, all standardized, none discoverable
3. The step the golden path deletes a directory generator instead of a toggle
4. Templates belong to the platform consumer
5. Three writes, all to Git and who writes the second deploy
6. Field notes partial failures and honest failures

The Ground It Stands On

A golden path is a code generator for your conventions, so it can only be as good as the conventions underneath it. Ours came out of the consolidation in part 1, which left three artifacts and those three are exactly what the template has to write.

One chart instead of forty. A single common-chart covering Deployment, Service, Ingress, HPA, Jobs, RBAC and secret wiring, published to a private OCI registry. Services depend on it through a thin umbrella chart: one directory per service per environment, holding a Chart.yaml that pins an exact version and a values.yaml where everything service-specific nests under common-chart:. The umbrella carries no templates of its own, which is what makes a whole deployment description something a form can generate.

Declarative secret wiring. On top of the External Secrets Operator, a variable declares where it comes from and the chart builds the ExternalSecret, then rewrites the pod spec into a plain secretKeyRef. Ordering comes from sync waves: the store at -2, the ExternalSecret at -1, the Deployment in wave 0.

One place that says which services run where. An app-of-apps whose templates are the other Applications, one file per service behind an enable guard, so every environment is a toggle file carrying that service's branch, autosync policy and region.

If those do not exist in your estate yet, build them before the wizard. A template written against conventions that do not exist becomes the place the conventions get invented, and then nobody can find them.

The Gap Nobody Talks About

With all of that in place, creating a new service still meant touching three repositories in a specific order:

1. The service repository create it, add the standard CI workflow, wire up the OIDC role so the pipeline can push to the registry.

2. The chart repository add a deploy/<service>/<env>/ directory with the version pin, the image, the probes, the resources, the IAM role annotation and every environment variable with its AWS path.

3. The bootstrapper add the Application template file, with the right namespace, project and source path.

4. The bootstrapper, again add the toggle block to every environment values file the service should run in.

Then merge them in that order. Step 2 has to land before step 4, because the moment the toggle flips, the delivery tool creates the Application and goes looking for a chart directory that does not exist yet. What you get is not a helpful error. It is an Application sitting in the UI unable to compare against anything, in a repository you were not the one editing.

Every step standardized. Not one of them discoverable.

None of it is written down anywhere except in the services that already did it, which is why the honest advice really is "copy the newest one." Someone doing this for the first time does not know about the ordering constraint, or that the ExternalSecret has to land a sync wave ahead of the Deployment so the Secret exists before the pod mounts it. They find out when the pod crash-loops, usually right around the time they hoped to be done.

So they ask the platform team, and the platform team turns into a ticket queue whose actual job is assembling pull requests from templates by hand. Which is string interpolation performed by a person.

That is the job a golden path takes over.

The Step the Golden Path Deletes

Steps 3 and 4 are the ones a scaffolder cannot do well. Rendering a new directory is easy: a template and a destination. Adding a block to an existing dev-values.yaml is different work read the file, parse it, insert in the right place, write it back, and do all of that on a file two other pull requests may be editing this afternoon. A scaffolder renders files. It is not a YAML patching engine, and building one inside a template is how templates become unmaintainable.

So new services skip the toggle model entirely. The delivery side is an ApplicationSet with a git generator, applied once, which turns an apps/<service>/<env>/ directory in the GitOps repository into an Application:

Two things there are load-bearing. goTemplate: true is what makes the index .path.segments syntax work at all. And there are two generators rather than one apps/*/* glob, because the Application name is a contract with the catalog: the argocd/app-name annotation has to match it exactly. A single glob would derive both apps/checkout/prod and apps/checkout/staging from segment 1 and produce two Applications called checkout in the same controller. Suffix the environment, as here, or run one ApplicationSet per cluster and keep the bare name.

The directory is the enablement. There is no second file to patch, so the ordering constraint above does not get automated it stops existing. One pull request adds the chart directory, and that same directory is what the generator discovers.

By hand, the enablement lives in a file that has to be edited after the chart directory exists. Generated, the chart directory is the enablement, so there is no second write and no order to get wrong.

Deletion is where the two models really diverge, and it is the strongest argument for this one. Part 1 ends up somewhere awkward: a hand-written Application without resources-finalizer.argocd.argoproj.io gets deleted without taking its workloads down, so switching a service off is a two-step job. The ApplicationSet controller handles that itself. Deleting a generated Application cascades to the resources it created unless you ask otherwise with preserveResourcesOnDeletion: true, so removing a directory from Git removes the workloads. The default is already the behaviour you want; write the field out anyway, because a reader of the manifest should not have to know the default to know what happens.

One more annotation earns its place on the generated template: argocd.argoproj.io/manifest-generate-paths: '.'. It tells ArgoCD that an Application only needs re-rendering when files under its own path change, so a commit touching one service does not drag every Application in the repository through manifest generation. On a few hundred services that is the difference between a merge and a stampede. It applies to both webhook-triggered and polled refreshes in current ArgoCD versions, having started life as a webhook-only optimisation, so confirm it against the version you run before leaning on it at that scale.

Templates Belong to the Platform Consumer

We treated the platform as a product rather than an installation: one codebase, one Docker image, one config file per consumer. Onboarding must never require a code change. If it does, that is a bug in the platform, not an onboarding task.

The decision people push back on hardest is that templates do not live in the platform's repository. They live in the consumer's own platform-templates repository, discovered by a catalog provider:

The image is identical everywhere. A per-consumer profile names the organization and the template repository, so the next consumer is a config file rather than a build.

A golden path already encodes consumer-specific knowledge: which registry, which cluster layout, which review rules, which IAM naming convention. Everything in part 1 is that kind of knowledge. Bake it into the platform image and you will have a fork per consumer within two quarters. The platform ships the building blocks a template can call; the template belongs to whoever owns the conventions.

Three Writes, All to Git

Here is the shape of a real template. Eight steps, three of which write anything:

Every write the scaffolder makes lands in a Git repository. Merging the pull request is the handoff. From there the delivery tool takes over, and the only thing coming back is read-only state.

To be exact about the heading: three steps write to Git, and step 8 writes to the platform's own catalog and sends a notification. Neither of those touches the consumer's infrastructure, which is the boundary that matters.

Step 5 is the handoff. Past that point the deployment belongs to the consumer's delivery tool, and the scaffolder holds no cluster credentials at all, not even scoped ones. That is what turns "the platform is read-only" from a policy you have to trust into something you can verify by reading the deployment manifest.

It does hold a GitHub App installation, which is the credential that actually matters here. An App scoped to the organization, with repository-creation and pull-request permissions and nothing else, is auditable per installation and revocable in one place. A personal access token belonging to whoever set the platform up is neither. Removing cluster credentials does not make the scaffolder harmless; it makes the blast radius Git-shaped, and Git-shaped blast radii are the ones your branch protection already knows how to contain.

Auto-merge in step 6 defaults to off, and that default is a design statement rather than caution. The pull request waiting for a human is the review gate for the deployment. Turning it on for staging is reasonable. Turning it on everywhere means you built a deploy button and called it GitOps. And if the GitOps repository has required status checks, which it should, auto-merge either waits for them or needs bypass rights and granting the platform bypass rights quietly undoes the gate you thought you kept.

The Second Deploy

Provisioning is the part demos show. The part teams live with is every deploy after the first, and the platform has nothing to do with it.

The umbrella chart the template renders carries a placeholder image tag, and the pull request says so in as many words, because a tag nobody pushed leaves the pod in ImagePullBackOff. From the first real build onward, the service's own CI writes the new tag into the GitOps repository a commit updating one line in one values.yaml and the delivery tool notices and syncs.

So Git has two writers: the scaffolder, once, at creation, and CI, on every merge to a service repository. That is not a hole in the boundary, it is the boundary working as intended. "The platform writes to Git and stops" is a claim about the platform's blast radius, not a claim that nothing else may commit. What matters is that both writers end at a merge, both are auditable in a pull-request log, and neither holds a kubeconfig.

Field Notes: Real-World Gotchas

1. One Workspace, Two Destinations

The GitOps content in step 4 renders into a ./gitops subdirectory rather than the workspace root, so it does not get swept into the application repository published in step 3:

repoUrl, branchName, title and description are all required by the action; sourcePath and targetPath are the optional pair doing the real work. The description is not decoration either — it is the last screen anyone reads before the deployment becomes real, and the only thing standing between a first-time user and a pod stuck on a tag nobody pushed.

Miss sourcePath and your brand-new service repository ships with somebody's Helm values inside it.

2. There Is No Transaction

Step 3 creates the repository. Step 5 opens the pull request. If step 5 fails a rate limit, a protected branch, a bad token you are left with an empty repository, no deployment, and a run that cannot simply be retried, because validate-name now finds the repository it created a minute ago and refuses.

The scaffolder has no rollback, and pretending otherwise would mean writing compensating deletes into a template, which is how a wizard learns to delete repositories. We keep the failure loud and the cleanup documented instead: the log names the repository, the error says the pull request never opened, and recovery is to delete the empty repository and run again. Where you have the choice, order the writes so the cheapest one fails first, because an orphaned pull request costs nothing to close and an orphaned repository costs a conversation. What you should not do is make validate-name tolerant of existing repositories. The check that stops two teams claiming the same service name is worth more than the retry convenience.

3. Validate the Name Twice

serviceName becomes the repository name, the Helm release name, the workload name and the Application name, and it cannot be changed afterwards. So it is checked twice: a debounced field in the wizard calls a validation endpoint, and the authoritative validate-name step fails the run for real. Front-end validation is a courtesy; back-end validation is the contract.

The endpoint has a third answer besides taken and free. If no GitHub organization is configured it returns 501 Not Implemented naming the missing key, rather than probing an organization it guessed, so the wizard shows no tick instead of a wrong one. That choice has a cost worth naming: 501 is a 5xx, so an unconfigured instance shows up in error dashboards as if something broke. The alternative 200 with {"configured": false} keeps monitoring quiet but requires every caller to remember the branch, and a caller that forgets renders a green tick for an answer nobody gave. We would rather explain a noisy dashboard than debug a confident lie.

4. Be Honest About Optional Tools

This is the pattern we would most like other platform teams to steal, and it came out of a bug that embarrassed us.

Step 7 nudges ArgoCD to re-read Git instead of waiting for its polling interval. It is a GET with ?refresh=hard: read-only, creates nothing, syncs nothing. Remove it and you change latency, not outcome. But it is the one tool-specific step in an otherwise portable template. So what happens when a consumer runs Flux, or plain CI, and a template calls argocd:refresh?

We shipped both obvious answers and disliked both:

What we do now is register the action id unconditionally and resolve it at startup. Tool configured, you get the real implementation. Not configured, you get a stub under the same id that throws with instructions:

The registry behind it is split in two. Generic actions, the Git operations every consumer has, register unconditionally. Delivery actions declare which tool they need:

Git actions skip the check entirely. Everything tool-specific is resolved once at startup, so a template calling a step the consumer cannot run fails with an explanation rather than a green tick.

Two smaller lessons ended up in the detection function. It walks every appLocatorMethod and every instance inside it looking for one with both a url and a token, rather than stopping at the first, because otherwise a consumer whose only working instance sits second in the list silently gets a stub. And every lookup uses the optional config accessors, so a half-filled configuration resolves to "not configured" instead of throwing during startup. The difference between a degraded feature and a dead platform can be one getOptionalString.

5. An Empty Tab Is Usually Half a Configuration

Once the pull request merges and the consumer's tool deploys, the platform switches to reading. Entity pages grow tabs based on annotations in a service's catalog-info.yaml, and it helps to know where each is defined when you go looking for documentation. github.com/project-slug is a well-known annotation in the catalog itself. backstage.io/kubernetes-id belongs to the Kubernetes feature and argocd/app-name to the ArgoCD plugin, so despite the backstage.io prefix on one of them, both are documented with their plugin.

A working tab needs both the annotation and the matching config block on the platform side. The template handles its half: the catalog-info.yaml it renders carries the annotations, and the Application name the generator produces is the same string argocd/app-name points at. Get that pair out of sync and the tab renders an empty state while the metrics collector attributes the deploys to nothing. Which leaves one failure mode in practice the annotation is right and the platform has no config block for that tool. When someone reports an empty tab, that is the half to check first.

Final Thoughts

A golden path is a maintained product, not a delivered project. Every convention it encodes will drift, and when the base chart reaches 2.0 the template still pinning 1.7.1 keeps generating outdated services without saying a word. It will not cover everything either: ours handles a stateless HTTP service, not a StatefulSet and not anything with unusual networking. Stretching one template over every case gives you a wizard with 40 fields, which is worse than the ticket it replaced.

Done with those limits in mind, what you get is:

• A new service from a form: what took 4 pull requests across 3 repositories, merged in a specific order, is now 1 pull request and a repository created for you.
• A write boundary you can verify, because the scaffolder holds a scoped GitHub App and no cluster credentials at all.
• An ordering constraint that stopped existing, because the directory the template writes is the enablement.
• One image for every consumer, with the golden paths owned by the teams whose conventions they encode.

The measure of a golden path is not how sophisticated the wizard is. It is whether someone who has never deployed anything at your company can ship a service alone, and whether the platform tells them honestly what it did and what it did not do. A step that goes green while doing nothing is worse than a red step that explains itself, because it teaches people not to trust the platform. Once they stop trusting it they go back to Slack, and you are a ticket queue again with extra infrastructure to maintain.

References

• Backstage: Software Templates
• Backstage: Well-known annotations
• ArgoCD: ApplicationSet git generator
• ArgoCD: ApplicationSet controller
• ArgoCD: Sync Phases and Waves
• External Secrets Operator
• Kloia: Managing Kubernetes Clusters with GitOps
• Kloia: Manage Kubernetes Secrets with External Secrets Operator and AWS SSM