Provisioning a 3rd-party / in-tree-chart app¶

TL;DR. From a clean state, "I want headlamp at headlamp.apps.local.example.net with a bearer token" is one prompt to the agent. The agent drives the provision-new-3p-app skill, which in turn calls the same blueprint-new-app and blueprint-publish-app CLIs you see in this doc — it just decides when to ask vs. when to run, and surfaces failures for you.
you → "I want to install headlamp at
       headlamp.apps.local.example.net with a bearer token."
agent   ↳ invokes .agents/skills/provision-new-3p-app/
        ↳ proposes steps 1 → 4 in chat
        ↳ runs each on your "go ahead"
        ↳ surfaces any user-action needed
you  →  "go ahead" (per step)
After ~2 minutes, the URL is live. The blueprint-publish-app exit code is 0 on success and 2 if a user-action is required (OpenBao sealed, runner not registered, etc.) — the message body says what to do.

Prefer to run it yourself? The end-to-end sequence the agent walks is the same sequence you can run by hand — see the step-by-step section. The two CLI entry points (blueprint-new-app, blueprint-publish-app) are idempotent and re-runnable; the agent just adds the choreography on top.

This page is the end-user runbook for adding any upstream-helm-chart application to the cluster as a GitOps-managed GitLab project. It complements the provision-new-3p-app agent skill (the AI-driven flow); the two are kept in lockstep — the skill's "Run" section mirrors the steps here.

Audience and assumptions¶

You're either a human operator or an AI coding agent walking through this. Either way:

Phase 2 has run — GitLab, Runner, OpenBao, the apps-tier Gateway listener (*.apps.<domain>), and the ci-deployer ServiceAccount all exist. If they don't, run uv run blueprint-bootstrap --phase 2 first. The blueprint-publish-app CLI checks for each and surfaces a clear "missing prerequisite" message if anything is absent.
Phase 3 has run at least once — the bootstrap-installed admin PAT lives at secret/gitlab/bootstrap/admin_token in OpenBao, and infra/logs/ci-deployer.kubeconfig (server: https://kubernetes.default.svc, SA-token auth) is on disk. These are the two artifacts blueprint-publish-app consumes; without them, the publisher can't talk to GitLab or to the in-cluster apiserver.
You know which app you want. A user-initiated decision. The agent cannot pick a chart for you — it can scaffold, sync, and surface failure modes, but "headlamp vs. kubernetes-dashboard vs. rancher" is a decision you'll be asked to confirm.

End-to-end sequence¶

What follows is the canonical sequence the agent walks when you hand it a sentence like "I want headlamp at headlamp.apps.local.example.net with a bearer token". The same flow runs the same CLIs in the same order whether the agent drives it or you do — the diagram's only job is to make the participants and the "where does the agent yield to me?" boundary obvious.

Hold "Alt" / "Option" to enable pan & zoom

sequenceDiagram
    autonumber
    actor User as You
    participant Agent as "Agent: skill provision-new-3p-app"
    participant NewApp as blueprint-new-app
    participant PubApp as blueprint-publish-app
    participant OpenBao as "OpenBao: bootstrap-installed, ns=openbao"
    participant GitLab as "GitLab: in-cluster, chart-managed"
    participant Runner as "Runner: GitLab Runner build pod"
    participant Cluster as "Cluster: kind apps-tier Gateway"

    User->>Agent: request headlamp at headlamp.apps.local.example.net with bearer token
    Note over Agent: proposal mode — no commands run yet

    Agent-->>User: step 1 — propose helm repo add and helm search repo for chart metadata
    User->>Agent: go ahead
    Agent->>Agent: helm repo add and helm search repo

    Agent-->>User: step 2 — propose JSON entry for VERSIONS.json with chart_version and values_overrides
    User->>Agent: go ahead
    Note right of Agent: pre-flight: jq -e . VERSIONS.json
    Agent->>Agent: jq upsert helm_repositories.headlamp

    Agent-->>User: step 3 — propose uv run blueprint-new-app headlamp
    User->>Agent: go ahead
    Agent->>NewApp: invoke scaffolder
    NewApp->>NewApp: helm pull and untar
    Note right of NewApp: idempotent re-runs are no-op
    NewApp->>OpenBao: seed secret/apps/headlamp/bearer-token
    NewApp-->>Agent: scaffold summary

    Agent-->>User: step 4 — propose edit to apps/headlamp/values.yaml with the diff printed first
    User->>Agent: go ahead
    Agent->>Agent: write values.yaml

    Agent-->>User: step 5 — propose uv run blueprint-publish-app headlamp
    User->>Agent: go ahead
    Agent->>PubApp: invoke publisher

    PubApp->>OpenBao: read secret/gitlab/bootstrap/admin_token
    PubApp->>GitLab: ensure group and project (idempotent)
    PubApp->>GitLab: sync shared-code templates
    Note right of GitLab: working tree: apps-local/group/headlamp
    PubApp->>PubApp: rsync apps/headlamp/ into working tree
    PubApp->>PubApp: render .gitlab-ci.yml from the AppKind.IN_TREE_CHART template
    PubApp->>GitLab: set CI vars including KUBECONFIG_FILE, CI_REPOSITORY_URL, and others
    PubApp->>GitLab: commit and push
    PubApp->>GitLab: trigger pipeline

    loop poll every 5s, 10-min timeout
        PubApp->>GitLab: GET /pipelines/:id
        GitLab-->>PubApp: status
    end

    GitLab->>Runner: assign job
    Runner->>Cluster: preflight context
    Runner->>GitLab: fetch deploy-info.yaml content
    Runner->>Cluster: helm diff upgrade

    alt no-op
        Runner-->>PubApp: helm-diff no-op gate
        PubApp->>GitLab: set FORCE_DEPLOY=1 and re-trigger
    else drift or new chart
        Runner->>Cluster: deploy with helm upgrade --install
        Runner->>Cluster: smoke auth via curl /api/healthz
    end

    alt pipeline success
        PubApp-->>Agent: exit 0 with pipeline URL and summary
        Agent-->>User: live at headlamp.apps.local.example.net
    else pipeline failed and auto-fix exhausted
        PubApp-->>Agent: exit 2 with user-action message
        Agent-->>User: openbao sealed: run uv run blueprint-bootstrap --unseal-openbao
    end

Legend — the participants are the same in either driving mode:

Participant	Role
You	Initiator. Says "go ahead" / "run it" / "commit + push" per step. Source of truth for which chart, which values, when to commit.
Agent	The orchestrator. Reads the provision-new-3p-app skill on GitHub; proposes each command on chat; runs it after you say "go ahead". Surfaces failures verbatim from the CLIs.
`blueprint-new-app`	The scaffolder. Pure host-side work (no GitLab, no cluster). Vendor the chart, write `apps/<name>/{deploy-info.yaml, values.yaml, README.md, helm-chart/}`, upsert `apps_manifest.yaml`, seed the bearer token in OpenBao. Idempotent.
`blueprint-publish-app`	The publisher. The per-app sibling of `blueprint-phase3`. Talks to GitLab + OpenBao; renders the per-`AppKind` `.gitlab-ci.yml`; sets CI variables; commits + pushes; triggers + watches the pipeline; auto-fixes the helm-diff no-op gate or surfaces a user-action.
OpenBao	Two paths: `secret/gitlab/bootstrap/admin_token` (admin PAT for the publisher to authenticate to GitLab) and `secret/apps/<name>/bearer-token` (the per-app token the smoke step uses).
GitLab	In-cluster chart-managed GitLab. The publisher creates the `<group>/<name>` project (default group `blueprint-apps`; override with `--group Services`).
GitLab Runner	A Kubernetes executor build pod. Reads `deploy-info.yaml`, runs `helm diff` → `helm upgrade --install` → bearer-token smoke against the apps-tier Gateway.
kind cluster	The 5-node cluster. The apps-tier Gateway (`*.apps.<domain>`) routes to the `Service` fronting the chart.

The two places the agent explicitly yields to you:

Between "discover" and "decide" — after helm search repo surfaces the chart's metadata, the agent pastes the helm_repositories.<name> JSON dict for you to confirm. Why this is yours, not the agent's: pinning a chart version is a reviewable change with cluster-compat implications (your CI runner's k8s version, RBAC). You own the bump.
Before any destructive step — rm -rf apps/<name>/, helm uninstall, tofu destroy, git reset --hard, git push --force, bearer-token rotation, GitLab project deletion, lane switches (helm → tf, GitOps → direct). The agent prints a 1-paragraph proposal with the exact command and waits for "yes" — even if you just said "go ahead" a turn earlier.

Everything else in the diagram — helm repo add, blueprint-new-app, blueprint-publish-app, the jq upsert, the pipeline polling, the smoke step — is fair game for the agent on a plain "go ahead".

The 30-second mental model¶

A "3rd-party app" in this blueprint is an upstream helm chart that:

lives vendored in this repo under apps/<name>/helm-chart/ (so a rebuild of the kind cluster needs no internet and a version bump is one PR),
gets a dedicated GitLab project under a group of your choice (default blueprint-apps; override with --group Services for a dedicated third-party lane),
has a per-project CI that runs helm diff → helm upgrade --install → bearer-token smoke, on every push to main.

The deploy-info.yaml file is the contract between the chart and the CI. It carries the app's name, ingress hostname, and auth contract (OpenBao path + header format). The CI reads it at job time — the chart itself is never modified by us.

apps_manifest.yaml is the per-app metadata that ties everything together: which app, which app_kind, which group, which helm release name, which namespace, which registry image. It's the single source of truth for the metadata; template content lives as real YAML files in apps/shared-code/templates/.

blueprint-new-app writes the source-of-truth files (apps/<name>/{deploy-info.yaml, values.yaml, README.md, helm-chart/} + a manifest entry + a bearer token in OpenBao). blueprint-publish-app is the publisher that turns that source of truth into a live deployment: GitLab project + .gitlab-ci.yml + CI variables + commit + push + pipeline trigger. The two commands are siblings — neither boots the other, neither depends on the other being run first. Run blueprint-new-app once, then blueprint-publish-app whenever you want to (re)sync.

Step-by-step¶

You're reading the manual path. The default flow is "prompt the agent" (see TL;DR + End-to-end sequence above). The same five steps below are what the agent walks for you when you say "go ahead" — rephrased here as paste-ready shell commands you can run yourself.

Step 1 — declare the chart source (once per chart)¶

Edit infra/scripts/bootstrap/VERSIONS.json and add an entry under helm_repositories. The structure is locked-in so the scaffolder can find the chart without guessing:

"headlamp": {
  "name": "headlamp",
  "url": "https://kubernetes-sigs.github.io/headlamp/",
  "chart": "headlamp",
  "chart_version": "0.43.0"
}

The url is the helm repo URL (the same one helm repo add would consume); chart is the chart name inside that repo; chart_version is pinned (the scaffold won't bump it for you — that's a deliberate decision to keep upgrades reviewable).

Why this isn't automated: pinning a chart version is a reviewable change. The user owns the version bump; the agent only proposes the JSON.

Step 2 — scaffold the source-of-truth files¶

uv run blueprint-new-app headlamp

Idempotent. Re-running is cheap and safe. On a fresh app, the scaffolder does:

Verifies the manifest entry in infra/scripts/bootstrap/phase3/apps_manifest.yaml — creates one if missing, leaves it alone if present.
Pulls the chart from the helm repo declared in step 1, into infra/helm-charts/headlamp-0.43.0.tgz. The VERSIONS.json entry's chart_version decides the version; the .cache-stamp next to the chart caches the sha256, so re-runs skip the network unless the version changed.
Unpacks the tgz into apps/headlamp/helm-chart/. Unmodified — what the upstream chart author shipped is what you vendor.
Writes apps/headlamp/deploy-info.yaml (the per-app contract; CI reads this at job time), apps/headlamp/values.yaml (a starter values.yaml you edit; the scaffolder never overwrites it on re-runs), and apps/headlamp/README.md (the URL table).
Upserts the entry in apps_manifest.yaml if it didn't exist; leaves it alone otherwise.
Idempotently seeds a bearer token at secret/apps/headlamp/bearer-token in OpenBao. The token is a 32-byte hex value; the seeder skips if a non-empty value already exists (so re-runs don't invalidate a deployed token).

What you get after a successful run:

[headlamp] chart cache hit: /…/infra/helm-charts/headlamp-0.43.0.tgz
[headlamp] extracted chart → /…/apps/headlamp/helm-chart
[headlamp] manifest entry already up to date
[headlamp] bearer token already at secret/apps/headlamp/bearer-token; leaving in place
[headlamp] scaffold summary:
  • apps/<name>/deploy-info.yaml written
  …

Step 3 — hand-edit `apps/headlamp/values.yaml`¶

The scaffolder writes a starter values file. The starting contents are deliberately minimal so a git diff shows only what the human chose to add. Common edits:

An ingress.host (most charts use this) — already populated by the scaffolder from the manifest entry's ingress_host.
A service.port if the chart's default doesn't match what the apps-tier Gateway expects (typically 80 / 443 for HTTP, 8080 for dashboards).
An httpRoute: block (newer charts using the Gateway API directly) — Headlamp uses this; see apps/headlamp/values.yaml for the canonical shape.

The scaffolder will not touch this file on re-runs. Your edits are sacred.

Step 4 — publish¶

uv run blueprint-publish-app headlamp

This is the new sibling CLI of blueprint-phase3 (the bulk 6-step pipeline) and blueprint-bootstrap (Phase 1+2). It does the per-app piece of Phase 3 in isolation: ensure GitLab group + project, clone/pull, rsync, render .gitlab-ci.yml, set CI variables, commit + push, trigger a pipeline. Repeatable — re-run after a chart bump, a values edit, or an upstream renovate PR. Optional flags:

Flag	Use
`--group Services`	Land the project in a different GitLab group (e.g. a dedicated `Services` group for third-party apps). The `include:` in the per-project template resolves to `<group>/shared-code`, so the project has to live in a group that has a `shared-code` neighbour.
`--dry-run`	Print every action without touching disk or GitLab. Re-running is a no-op.

The publisher's loop is:

Seed the host-routable GitLab URL by loading the admin PAT from OpenBao (secret/gitlab/bootstrap/admin_token). Without this, git clone / git push would fall back to the in-cluster Service DNS (gitlab-webservice-default.gitlab.svc) which isn't resolvable from the host.
Sync the shared-code templates to GitLab's blueprint-apps/shared-code repo. The per-project pipeline include: resolves against the live GitLab repo; if canonical templates are renamed but the GitLab side is on a stale revision, every downstream project's CI fails with "Could not resolve include".
Ensure the GitLab group + project under the chosen group path. Idempotent — re-runs no-op.
Clone or pull the GitLab project into the host's working tree (apps-local/<group>/<name>/). The local working tree's origin is rewritten to the host-routable URL on every run (a prior blueprint-phase3 run left it pointing at the in-cluster DNS; without the rewrite, push would silently fail).
Sync canonical apps/<name>/ into the working tree via rsync --delete. The user's hand-edits live in apps/<name>/; the working tree is the publish target.
Render .gitlab-ci.yml from the per-app-kind template in apps/shared-code/templates/. The dispatch is typed: AppKind.IN_HOUSE_BUILD → app.gitlab-ci.yml.tpl, AppKind.CUSTOM_CHART → redis.gitlab-ci.yml.tpl, AppKind.IN_TREE_CHART → chart.gitlab-ci.yml.tpl. No if/else on booleans — a missing app_kind is a load-time error.
Set CI/CD variables on the GitLab project: KUBECONFIG_FILE (the build-pod-friendly ci-deployer.kubeconfig with server: https://kubernetes.default.svc), CI_CLUSTER_CA_CERT (the cluster wildcard CA, file-type), CI_REPOSITORY_URL (in-cluster GitLab Service), CI_REGISTRY_IMAGE, CI_HELM_CHART_DIR, CI_INSECURE_REGISTRY, and the redis/redis-tag/dockerfile-path keys where applicable.
Commit + push if there are local changes. Author is blueprint-phase3@local so every commit is visibly from the bootstrap. Empty trees are no-ops.
Trigger a pipeline and watch it. The publisher polls every 5 seconds for a terminal state, with a 10-minute timeout. On failure, it inspects the failed jobs' names
failure reasons and either:
Auto-fixes known patterns (currently: helm-diff no-op gate → set FORCE_DEPLOY=1 and re-trigger), or
Surfaces a user-action (e.g. "no GitLab Runner can pick up jobs for <group>/<name>; run uv run blueprint-bootstrap --phase 2 to re-register the runner").
Prints a summary with the pipeline URL, the auto-fix history (if any), and the user-action string (if any). Exit code is 0 on success, 2 for "user-action required", 3 for "pipeline failed and we couldn't auto-fix".

Step 5 — verify the URL¶

After the publisher reports success, the app should be reachable on the apps-tier ingress within a few seconds (Gateway program latency). The smoke step in the per-project CI also curls the URL from a build pod — that's the end-to-end check, but you can verify from the host first:

curl -k --resolve headlamp.apps.local.example.net:443:127.0.0.1 \
     -H "Authorization: Bearer $(…)" \
     https://headlamp.apps.local.example.net/api/healthz
# expect: 200 OK

The --resolve is a host-side port-forward trick; the host routes 127.0.0.1:443 to the kind control-plane's NodePort-mapping of the chart-managed Envoy Gateway. From inside the cluster, the in-cluster kubernetes.default.svc apiserver-routed *.apps.<domain> listener answers directly.

The URL https://headlamp.apps.local.example.net/ is the documented "this is where the app lives" string. It serves from the chart's Service (fronted by a gateway.networking.k8s.io/v1.HTTPRoute).

Iteration loop¶

You want to…	Run
Re-publish after a `values.yaml` edit	`uv run blueprint-publish-app headlamp`
Bump the chart version	edit `VERSIONS.json` → `uv run blueprint-new-app headlamp` → `uv run blueprint-publish-app headlamp`
Rotate the bearer token (the `secret/apps/headlamp/bearer-token` value)	`kubectl -n openbao exec openbao-0 -- bao kv put -tls-skip-verify secret/apps/headlamp/bearer-token value=$(openssl rand -hex 32)` then `uv run blueprint-publish-app headlamp` to push the new pipeline (no new secret in CI, but the next deploy reads the new value)
Add a 2nd 3rd-party app	repeat from step 1 with a new name
Wipe one app from the cluster	`helm uninstall <name> -n <name>` (CI won't re-deploy it on the next publish unless you also remove it from `apps_manifest.yaml` and `apps/<name>/`)
Wipe one app from GitLab	`glab project delete blueprint-apps/<name>` (the publisher will re-create it on the next run because `ensure_project` is idempotent)
See the publisher's actions without running them	`uv run blueprint-publish-app headlamp --dry-run`

All operations are idempotent. Re-running either blueprint-new-app or blueprint-publish-app after a partial failure resumes from the failed step — no global "wipe and re-provision" is ever required.

File / CLI reference¶

What	Where
The per-app contract	`apps/<name>/deploy-info.yaml`
The chart (vendored, unmodified)	`apps/<name>/helm-chart/`
The user's hand-edits (never overwritten by re-runs)	`apps/<name>/values.yaml`
Chart version pinning	`infra/scripts/bootstrap/VERSIONS.json` → `helm_repositories.<name>`
The per-app metadata (CI template selection + CI variables)	`infra/scripts/bootstrap/phase3/apps_manifest.yaml`
The per-app-kind CI templates (real YAML, no Python string literals)	`apps/shared-code/templates/*.yml.tpl`
The reusable pipeline bodies (each kind has one)	`apps/shared-code/templates/*-pipeline.yml`
The build-pod kubeconfig (server: `https://kubernetes.default.svc`)	`infra/logs/ci-deployer.kubeconfig`
The GitLab admin PAT (consumed by `blueprint-publish-app`)	OpenBao `secret/gitlab/bootstrap/admin_token`
The bearer token the smoke step sends to the app	OpenBao `secret/apps/<name>/bearer-token`
The agent entry point	provision-new-3p-app skill on GitHub

Design rationale (for the next maintainer)¶

These are the why questions that come up the second time someone modifies the templates or the publisher.

Two templates per app kind, not one with branches. The pipeline defines the stages; branches would have to be evaluated at CI time against an attribute that's not in the triggering commit. Picking at scaffold time (and rendering the right .gitlab-ci.yml based on AppKind) keeps the evaluation out of the runner.
deploy-info.yaml instead of values injection. The alternative — template a values.<env>.yaml and overlay the auth via helm hooks — couples the chart to our convention. Reading deploy-info.yaml in the deploy-chart job keeps the chart pristine. It also makes swapping to ArgoCD or Flux tomorrow a one-file change (the contract is what stays).
Static bearer token, not OAuth. OAuth requires a callback URL + client-id + refresh, all of which live outside the chart. The premise of this lane is "I want headlamp, not IdP integration".
Per-app publish CLI, not bulk Phase 3. The 4 in-house cluster apps (guestbook, redis, redis-slave, shared-code) are bootstrapped together once via blueprint-phase3 — that's their lifecycle. Third-party / in-tree-chart apps follow a different lifecycle: added one at a time, on demand, by humans or agents, sometimes in a different group. Booting Phase 3 to "re-publish headlamp" is the wrong tool. blueprint-publish-app is the right tool — reuses Phase 3's helpers (GitlabClient.ensure_group / ensure_project / set_variable, AppsLocalFS.clone / pull_ff / commit_and_push, ci_render.render_for) but composes them for one app at a time.
Build-pod kubeconfig points at https://kubernetes.default.svc, not https://localhost:35423. The latter is the kind port-forward on the host's loopback — only reachable from the operator's workstation. The build pods run inside the kind cluster and use the in-cluster Service DNS to reach the apiserver. infra/logs/ci-deployer.kubeconfig (minted by Phase 3's step 4b — a ci-deployer ServiceAccount + cluster-admin binding + 1-year token) is the build-pod-friendly kubeconfig. blueprint-publish-app reads it and bakes it as the KUBECONFIG_FILE CI variable on every publish.

Common failure modes¶

These are the same as the skill's "Common pitfalls" section, written for the end user rather than the agent.

Re-running the scaffolder is cheap, but pulls-from-network isn't. The .cache-stamp next to the chart is what makes a re-run a no-op when nothing changed. If you delete apps/<name>/helm-chart/ without also deleting the .cache-stamp, the next blueprint-new-app <name> will NOT re-pull — and you'll be deploying a stale chart. Delete both, or update VERSIONS.json to a new version explicitly.
Renamed upstream values key. If upstream renames service.port → service.portNumber, your apps/<name>/values.yaml silently stops overriding. CI doesn't fail. The CI's helm diff step prints a warning (or the deploy is a no-op). Catch this by reading the upstream release notes before bumping chart_version.
OpenBao sealed at CI time. If Phase 2 ever re-sealed (laptop restart, or someone scaled the openbao-0 StatefulSet to zero), the GitLab Runner can't read the bearer token from OpenBao and the smoke step fails with permission denied. Run uv run blueprint-bootstrap --unseal-openbao on the host, then re-trigger the pipeline.
Drift between apps/<name>/ and the cluster. The diff:helm-upgrade job does helm diff upgrade first and refuses on no-op. This is intentional — drift is a sign of a manual edit on the cluster, and silently re-deploying would clobber the operator's intent. Set FORCE_DEPLOY=1 on the GitLab project's CI variables to bypass the gate (or fix the drift by hand).
smoke:auth fails with Failed to connect to <host>:443. The build pod can't reach the apps-tier Gateway listener because the runner config has no hostAliases for the gateway's ClusterIP. The deploy:helm job succeeded regardless; only the post-deploy smoke step is affected. The app is still reachable from the host at the documented URL. Fix at the runner level by adding [[runners.kubernetes.host_aliases]] ip = "<gateway ClusterIP>" hostnames = ["<each-apps-tld>"] to the runner helm values and re-applying.

Out of scope (deliberately deferred)¶

Multi-env overlays. The publisher treats main as the single deployment. A values-qa.yaml / values-prod.yaml split would need branch→env mapping in CI; tracked as a follow-up.
ArgoCD / Flux adoption. Once deploy-chart works, replacing it with a GitOps operator is a one-file swap. The deploy-info.yaml contract is what stays.
Auto-unseal for OpenBao. Handled separately by blueprint-bootstrap --unseal-openbao. The publisher just surfaces the unseal-needed error if it sees one.
PR-based workflow. Today, the publisher pushes directly to main on the GitLab project. A future enhancement is a --branch <name> flag that pushes to a feature branch and opens a merge request; the user owns the merge.

Acceptance — when is this provisioning "done"?¶

A single sentence from a user — "I want Headlamp at headlamp.apps.local.example.net with a bearer token from OpenBao" — results, in under five minutes of human-in-the-loop edits, in:

A new branch with apps/headlamp/{README.md, deploy-info.yaml, values.yaml, helm-chart/} + one entry in apps_manifest.yaml.
A successful uv run blueprint-publish-app headlamp that sets up blueprint-apps/headlamp on GitLab with the per-app-kind template and exits 0.
A green CI pipeline on main that runs helm diff → helm upgrade --install → bearer-token smoke.
curl -H "Authorization: Bearer $(…)" https://headlamp.apps.local.example.net/api/healthz returning 200 OK.

If (1) needs more than five file edits by the human, or if (3) needs more than one auto-fix re-run, the lane is not done. The auto-fix loop is intentionally conservative — it only fires when the failure mode is unambiguous (helm-diff no-op gate). Anything else is surfaced as a user-action with the exact recovery command.