Skip to content

Blueprint for a Local GitLab + Kubernetes CI/CD Stack

2. Architecture Overview

bruj0/k8s-cicd-gitlab

Architecture Overview¶

The blueprint decomposes into three concentric rings: an Infrastructure ring (IaC-owned, run by a human via tofu), an Application ring (cluster-resident services, run by the bootstrap), and a Source-of-truth ring (the GitLab-side repos the runner deploys from). They are connected by hand-offs at narrow boundaries — tofu apply → bootstrap --phase 2 → blueprint-phase3 → git push → pipeline → workload.

The whole system is fully local: there is no cloud account, no managed K8s, no public DNS. Domain trust is local CA + /etc/hosts. The cluster-side data plane is NodePort (not LoadBalancer) because kind has no MetalLB.

System Context — C4 Level 1¶

Hold "Alt" / "Option" to enable pan & zoom

flowchart TD
  DEV["Developer<br/>(on the host)"]

  BP["Blueprint<br/>Local GitLab + k8s + CI/CD stack<br/>(this repo, on the developer's machine)"]

  subgraph External["External actors + systems"]
    HOSTS["/etc/hosts<br/>(host-side DNS mapping)"]
    CA["Host trust store<br/>(local CA anchor)"]
    REG["Docker Hub<br/>(image pulls)"]
  end

  DEV -->|"edits apps/, runs uv / tofu"| BP
  DEV -->|"opens browser, clones, pushes"| BP
  BP -->|"trusts *.local.example.net"| HOSTS
  BP -->|"ships 10-yr self-signed CA"| CA
  BP -->|"pulls kindest/node, helm charts"| REG

Container Architecture — C4 Level 2¶

The C4 Level 2 view is split across two diagrams. The first shows the host-side processes + the GitLab-side repo group (who runs what, where the user interacts); the second zooms into the inside of the kind cluster (cluster infrastructure, application stack, GitLab stack). The two together answer "where does this thing live?".

Host processes, kind cluster, and the blueprint-apps group¶

Hold "Alt" / "Option" to enable pan & zoom

flowchart TD
  DEV["Developer host<br/>Ubuntu 22.04/24.04 · Fedora 41 · Arch · macOS"]

  subgraph HostSide["Host processes (Python venv)"]
    BB["blueprint-bootstrap<br/>CLI: install / destroy / port-forward"]
    BS["blueprint-secrets<br/>CLI: read OpenBao, open UI"]
    BP3["blueprint-phase3<br/>CLI: GitLab project provisioning"]
    Tofu["tofu<br/>(apply is manual, never automated)"]
    Docker["Docker / Podman<br/>(kind container runtime)"]
  end

  subgraph Cluster["kind cluster (5 nodes: 1 cp + 3 gitlab + 1 runner)"]
    direction TB
    CLUSTDOTS["… (see next diagram)"]
  end

  subgraph GlProjects["GitLab repos (blueprint-apps/ group)"]
    SC_repo["shared-code"]
    GB_repo["guestbook"]
    R_repo["redis"]
    RS_repo["redis-slave"]
  end

  DEV --> BB
  DEV --> BS
  DEV --> BP3
  DEV --> Tofu
  DEV --> Docker
  Docker -->|"runs the containers"| Cluster
  Tofu -->|"provisions"| Cluster

  BB -->|"writes (post-install)"| Cluster
  BP3 -->|"python-gitlab + requests"| GlProjects
  BP3 -->|"PATs in OpenBao"| Cluster
  GlProjects -->|"webhook → pipelines"| Cluster

  DEV -->|"https://*.local.example.net"| Cluster

Why this layer stops here: the host is where the developer-facing CLIs (blueprint-bootstrap, blueprint-secrets, blueprint-phase3) and the Docker daemon live; the cluster is one opaque box in this view, and the GitLab-side repos are a separate layer that lives behind the bootstrap's Phase 3 wiring. The next diagram zooms into the cluster.

Inside the kind cluster¶

Hold "Alt" / "Option" to enable pan & zoom

flowchart TD
  KUBEDOTS[… tofu-provisioned kind cluster]

  subgraph K8sInfra["Cluster infrastructure"]
    SC["local-path StorageClass<br/>(default)"]
    PV["PV/PVC pairs<br/>(infra/data/shared/stable/)"]
  end

  subgraph K8sApps["Application stack (Phase 2)"]
    CNPG["CloudNativePG<br/>(standalone, single instance)"]
    Redis["Redis single-node<br/>(bitnami/redis)"]
    Minio["MinIO single-node<br/>(11 GitLab buckets)"]
    OpenBao["Bootstrap OpenBao<br/>(openbao ns)"]
    TLS["Wildcard TLS Secrets<br/>(gitlab/registry/kas/minio)"]
  end

  subgraph K8sGitlab["GitLab stack"]
    EG["Envoy Gateway<br/>(chart subchart)"]
    GL["GitLab CE 19.x<br/>(webservice, sidekiq, gitaly, kas, registry)"]
    GR["GitLab Runner<br/>(Kubernetes executor)"]
    CBO["Chart-bundled OpenBao<br/>(gitlab-openbao Deployment)"]
  end

  SC --> PV
  PV --> CNPG
  PV --> Redis
  PV --> Minio
  PV --> OpenBao
  PV --> GR

  EG --> GL
  GL --> CBO
  GL -->|"global.psql.host"| CNPG
  GL -->|"global.redis.host"| Redis
  GL -->|"global.appConfig.object_store.*"| Minio
  GL -->|"global.openbao.psql.host"| CNPG

  GR -->|"http://gitlab-webservice-default.gitlab.svc:8181"| GL

  EG -->|"HTTPRoute → listener Secret"| TLS
  TLS -->|"routes to"| GL

Where host + cluster meet: the bootstrap writes the chart-managed Secrets into the gitlab namespace (handled by the host BB node in the previous diagram), and OpenBao's secrets_cli.py host-driven client reads them back through the auto-port-forward. The Envoy data-plane is exposed from this side via the chart's NodePort service; the host reaches it via kubectl port-forward (see docs/phase-2.md § 14).

Component Architecture — C4 Level 3¶

The bootstrap is the component-rich module. Everything else is thin glue. The level-3 view is split across two diagrams: first the composition roots + cross-cutting protocols + IaC target (the wiring around the installers), then the installers themselves (Phase 1 prep, Phase 2, Phase 3).

Composition, Protocols, IaC¶

Hold "Alt" / "Option" to enable pan & zoom

graph TB
  subgraph COMPOSITION["Composition roots"]
    APPA["BootstrapApp<br/>(Phase 1+2, infra/scripts/bootstrap/app.py)"]
    APP3["Phase3App<br/>(Phase 3, infra/scripts/bootstrap/app_phase3.py)"]
  end

  subgraph CLI_LAYER["CLI wrappers (click)"]
    CLI1["cli.py → blueprint-bootstrap"]
    CLI2["secrets_cli.py → blueprint-secrets"]
    CLI3["phase3_cli.py → blueprint-phase3"]
  end

  subgraph PROTOCOLS["Cross-cutting protocols"]
    LOG["Logger<br/>(Console / Null)"]
    SHELL["CommandRunner<br/>(Subprocess / DryRun)"]
    INV["Installer Strategy<br/>(Arch / Debian / Rhel / Darwin)"]
    VER["versions.py + VERSIONS.json"]
    PF["PortForward<br/>(generic 127.0.0.1:<svc>:port helper)"]
  end

  subgraph TOFU_CFG["infra/tofu/ (IaC)"]
    CLUST["kind_cluster.cicd<br/>(5 nodes, hostPath mounts)"]
    WD["null_resource.wipe_data<br/>(destroy-time data sweep)"]
  end

  subgraph SHARED["Shared secret + port-forward clients"]
    OBC["OpenBaoClient<br/>(hvac + auto port-forward)"]
  end

  CLI1 --> APPA
  CLI2 --> APPA
  CLI3 --> APP3

  APPA -.uses.-> LOG
  APPA -.uses.-> SHELL
  APPA -.uses.-> OBC
  APP3 -.uses.-> LOG
  APP3 -.uses.-> SHELL
  APP3 -.uses.-> OBC
  OBC -.uses.-> PF

Phase 1 prep¶

BootstrapApp (composition root for Phase 1 + Phase 2) calls into three single-responsibility prep modules and hands off to the Phase 2 pipeline (which is shown in the next diagram below). The PrepRegistry + HelmChartCache + TofuRunner trio is everything Phase 1 needs to verify prereqs, download chart tarballs, and validate IaC — leaving the actual cluster apply to the user (tofu apply per the hard rule in AGENTS.md).

Hold "Alt" / "Option" to enable pan & zoom

graph LR
  subgraph P1["Phase 1 prep"]
    APPA["BootstrapApp<br/>(Phase 1+2)"]
    PREREQ["PrereqRegistry<br/>(Docker, kubectl, kind, helm, tofu, openssl)"]
    HC["HelmChartCache<br/>(downloads to infra/helm-charts/)"]
    TR["TofuRunner<br/>(init + validate + next_steps)"]
  end

  subgraph HANDOFF["Handoff to Phase 2"]
    PIPE["Phase2Pipeline<br/>(see next diagram)"]
  end

  APPA --> PREREQ
  APPA --> HC
  APPA --> TR
  APPA -->|"runs"| PIPE

Phase 3 modules¶

Phase3App (separate composition root) wires a 6-step pipeline over the GitLab + apps-local machinery. The OpenBaoClient (shown as a compact shared node here, fully expanded in the Composition + Protocols diagram above) is what the GitlabClient uses to mint the admin PAT that survives tofu destroy.

Hold "Alt" / "Option" to enable pan & zoom

graph LR
  subgraph P3["Phase 3 modules (phase3/)"]
    APP3["Phase3App<br/>(Phase 3)"]
    P3P["Phase3Pipeline<br/>(6-step orchestrator)"]
    AM["AppsManifest<br/>(4 project specs from apps_manifest.yaml)"]
    AL["AppsLocalFS<br/>(apps-local/ working tree)"]
    GC["GitlabClient<br/>(python-gitlab + requests, no glab)"]
    GLC["glab_client<br/>(auth login seeder only)"]
    CR["ci_render<br/>(substitute YAML templates)"]
  end

  subgraph SHARED["Shared client (see other diagrams)"]
    OBC["OpenBaoClient<br/>(hvac + auto port-forward)"]
  end

  APP3 --> P3P
  APP3 --> AM
  APP3 --> AL
  APP3 --> GC
  APP3 --> GLC

  GC -->|"reads PAT from"| OBC
  P3P --> GC
  P3P --> AL
  P3P --> CR
  P3P --> GLC

Phase 2 installers — infrastructure foundation¶

The seven infrastructure-foundation installers orchestrated by Phase2Pipeline. Each one owns either a Kubernetes primitive (StorageClass, CRDs, PV/PVCs), a backing service that GitLab talks to over global.*.host, or a low-level cluster secret (PG role passwords, MinIO root creds). External dependencies — OpenBaoClient for secret material and the IaC-provisioned kind_cluster.cicd — are shown as compact stubs alongside the group.

Hold "Alt" / "Option" to enable pan & zoom

graph TB
  subgraph P2A["Infrastructure foundation installers"]
    PIPE["Phase2Pipeline"]
    PIPE --> CRDS["GatewayCRDsInstaller"]
    PIPE --> LPP["LocalPathProvisionerInstaller"]
    PIPE --> SS["StableStorageInstaller<br/>(PV/PVC + CNPG annotations)"]
    PIPE --> PG["CloudNativePGInstaller<br/>(operator + Cluster + role/db)"]
    PIPE --> RDS["RedisInstaller"]
    PIPE --> MIN["MinioInstaller<br/>(+ 11 GitLab buckets)"]
    PIPE --> OBI["OpenBaoInstaller<br/>(+ init + unseal)"]
  end

  OBC["OpenBaoClient<br/>(hvac + auto port-forward)"]
  CLUST["kind_cluster.cicd<br/>(5 nodes, hostPath mounts)"]

  PG -->|"uses"| OBC
  OBI -->|"uses"| OBC
  PG -->|"owned by"| CLUST
  LPP -->|"StorageClass for"| CLUST

Phase 2 installers — GitLab-facing¶

The five installers that shape the GitLab deployment itself: wildcard TLS, the chart-managed Secrets snapshot/restore, the GitLab CE chart (which sub-installs Envoy Gateway and bundled OpenBao), the GitLab Runner, and the CloudNativePGInstaller reference for the chart's global.psql.host. See deep-dive/Phase 2 Pipeline.md for the per-step rationale.

Hold "Alt" / "Option" to enable pan & zoom

graph TB
  subgraph P2B["GitLab-facing installers"]
    PIPE["Phase2Pipeline"]
    PIPE --> WC["WildcardCertsInstaller<br/>(+ 4 Gateway listener Secrets)"]
    PIPE --> PS["PersistentSecretsInstaller<br/>(snapshot + restore chart-managed Secrets)"]
    PIPE --> GLI["GitlabInstaller<br/>(chart + bundled Envoy + bundled OpenBao)"]
    PIPE --> RUN["GitLabRunnerInstaller"]
    PG2["CloudNativePGInstaller<br/>(referenced from this diagram)"]
  end

  OBC["OpenBaoClient<br/>(hvac + auto port-forward)"]

  GLI -->|"consumes chart-managed Secrets"| PS
  GLI -->|"tls via listener Secrets"| WC
  GLI -->|"global.psql.host"| PG2
  WC -->|"stores CA in"| OBC
  GLI -->|"stores initial_root_password in"| OBC
  RUN -->|"registration_token from"| OBC

Architectural Patterns¶

1. Composition Root + Single-Responsibility Installer (SOLID)¶

Each installer class takes only the paths and version catalog it needs (Dependency Inversion). The composition root (app.py:BootstrapApp, app_phase3.py:Phase3App) wires dependencies. Adding a new installer means one new class + one wiring line + one CLI option if exposed — not a parallel hierarchy or a new top-level command. The package layout (infra/scripts/bootstrap/phase<N>/<installer>.py) enforces it.

2. Idempotent Orchestrator / Pipeline¶

Phase2Pipeline is a 13-step orchestrator where each step delegates to one installer. Every step is idempotent — re-running resumes from the failed step, success is a no-op. The pipeline owns ordering and error reporting; each installer owns its own install logic. Same pattern at one level higher for Phase3Pipeline (6 steps).

3. Prepare / Apply Boundary (IaC vs. Application)¶

A hard rule encoded in the codebase:

TofuRunner exposes init(), validate(), next_steps() — no apply().
Phase 1 bootstrap never runs OpenTofu; the user runs tofu apply.
Phase 2/3 bootstrap may call helm install, kubectl apply, gitlab-rails runner, register the Runner, etc.
The cluster lifecycle (kind create/delete, host bind mounts) is only tofu. Never kind create cluster ad-hoc, never docker rm -f kind-cicd-*, never kubectl delete ns "to tidy up".

This is enforced at the type level: there is simply no apply() method to call.

4. Bidirectional Mode Toggle (preserve state vs. destructive)¶

var.preserve_stateful_data (default false since 2026-07) flips three coupled knobs:

Kind extra_mounts.propagation — Bidirectional (destructive) vs. default HostToContainer (preserve).
null_resource.wipe_data destroy provisioner — active (destructive) vs. no-op (preserve).
Local-path provisioner's teardown script — upstream rm -rf (destructive) vs. mv to .preserved-* (preserve).

The bootstrap CLI mirrors this with --destroy --preserve-data. The flags must agree between tofu apply and bootstrap --destroy or the next install sees a divergent contract.

5. Templates on Disk, Not in Code (Rule 0)¶

Every rendered YAML (per-project .gitlab-ci.yml, helm values, cluster PostgreSQL, Gateway API CRDs) lives in apps/shared-code/templates/*.yml.tpl or infra/scripts/bootstrap/phase2/references/*.yaml. Substitution happens via string.Template.safe_substitute (or .substitute when missing vars must raise). There is a regression test (tests/test_ci_render.py::test_no_yaml_literals_in_ci_render) that catches any yaml.dump({...}) creeping back into Python.

6. Branch by Abstraction (Protocols)¶

Three Protocol classes decouple the bootstrap from the side effects:

Logger → ConsoleLogger / NullLogger
CommandRunner → SubprocessRunner / DryRunRunner
PortForward → used by OpenBaoClient, GitlabClient, the CLI

This is why blueprint-bootstrap --dry-run and --check work without conditional code sprinkled everywhere — the dry-run wrapper is the runner the production code holds.

7. Dual-Key Secret (cross-tool schema mismatch)¶

The gitlab-rails-storage Secret in the gitlab namespace carries two keys on the same Opaque Secret:

connection — Rails-side object-store parsers (Fog/AWS provider schema)
config — Docker registry native s3: block

Don't try to "consolidate" — the two consumers want different YAML schemas for the same MinIO bucket, and a single schema satisfying both would also confuse one.

Key Design Decisions¶

Decision 1 — NodePort data-plane, no LoadBalancer / MetalLB¶

What: Chart 10.x's bundled envoy-gateway defaults the data-plane Service to LoadBalancer; kind has no provisioner, so the Service never settles and Gateway reports AddressNotUsable.

Rationale: Two-file fix (no new control plane to debug): 1. infra/tofu/cluster.tf — extraPortMappings host 80/443/22 → container 30080/30443/30022. 2. phase2/references/helm-values-gitlab.yaml — gatewayApiResources.envoy.proxySpec.provider.kubernetes.envoyService.type: NodePort plus matching nodePort: <30080|30443|30022>.

Trade-offs: Single-host load only (no multi-node kind scaling). Acceptable because Blueprint is local and the repo's whole model is "single developer laptop".

Decision 2 — Bootstrap is Python, not shell¶

What: The whole installer is a uv-managed Python package with click CLIs, dataclasses, and Protocol-based runners. No bootstrap.sh for anything non-trivial.

Rationale: Spec rules shell out for Python; the package is class-based + SOLID. Single responsibility per installer; append-only history under phase<N>/.

Trade-offs: Heavier than a 50-line shell script. Payoff: idempotency probes, retries, structured logs, --dry-run everywhere.

Decision 3 — Pin everything in `VERSIONS.json`¶

What: Every tool, helm chart, helm repo URL lives in infra/scripts/bootstrap/VERSIONS.json. No class hardcodes a version.

Rationale: Single bump point; versions.py:load_versions() is read at composition-root construction.

Trade-offs: An extra file read at start-up (negligible). Payoff: rg "v0.27.0" returns one line, period.

Decision 4 — Bootstrap mints a fresh PAT via `gitlab-rails runner`¶

What: blueprint-phase3 exec's into the gitlab-toolbox pod, runs gitlab-rails runner to mint a PAT with the exact scopes (api, read_api, read_repository, write_repository, read_registry, write_registry, sudo, …) and writes it to OpenBao at secret/gitlab/bootstrap/admin_token.

Rationale: Replicate the entire stack from tofu destroy with no manual glab step. The whole provisioning is one command.

Trade-offs: Requires the toolbox pod to be schedulable; subject to RBAC in the chart. Resolved by relying on the chart's default toolbox ServiceAccount.

Decision 5 — External services for CNPG / Redis / MinIO (chart 10.x dropped bundled subcharts)¶

What: Instead of installing chart-managed subcharts (which no longer exist), Blueprint installs cloudnative-pg, bitnami/redis, and minio/minio as standalone helm charts in their own namespaces, and the GitLab chart wires to them via global.psql.host, global.redis.host, global.appConfig.object_store.*.connection.secret.

Rationale: Chart 10.x upgrade window. As a side effect, those services now survive the chart-managed Service tier (they have their own StatefulSets and PVCs bound to the stable hostPath).

Trade-offs: 4 namespaces to manage instead of 1. Mitigated by the bootstrap owning the install order (install before, point GitLab at them, then the chart sees them already up).

Decision 6 — Pods use Service DNS, never `*.local.example.net`¶

What: Runner URL is http://gitlab-webservice-default.gitlab.svc:8181 (plain HTTP, port 8181). OpenBao URL from the CLI is openbao.openbao.svc:8200. Never gitlab.local.example.net from inside a pod.

Rationale: No CoreDNS rewrite inside the cluster for *.local.example.net. Pods resolving that hostname hit the host's /etc/hosts (127.0.0.1) and break. Plus: TLS terminates at Envoy Gateway above the cluster-internal path — pod-to-pod traffic on port 8181 is plain HTTP.

Trade-offs: Two parallel naming schemes (gateway FQDN for humans, Service DNS for pods). Mitigated by AGENTS.md rule + a code comment in every installer.

Module Breakdown¶

`infra/scripts/bootstrap/` — the installer¶

Module	Purpose
`app.py`	`BootstrapApp` composition root. Wires Phase 1 prep (`prereq` + `helm_cache` + `tofu`) and Phase 2 `Phase2Pipeline`.
`app_phase3.py`	`Phase3App` composition root. Mirrors `app.py` for `Phase3Pipeline` (separate file because Phase 3 has nothing to do with cluster bootstrap).
`cli.py` / `secrets_cli.py` / `phase3_cli.py`	Three click wrappers — `blueprint-bootstrap`, `blueprint-secrets`, `blueprint-phase3`.
`app_installer.py`	`HelmAppInstaller` generic + `HeadlampInstaller` subclass + `installer_for(...)` factory.
`helm_cache.py`	`HelmChartCache` — downloads `helm repo add` + `helm pull` results into `infra/helm-charts/<name>-<version>.tgz`.
`tofu.py`	`TofuRunner` — exposes `init()`, `validate()`, `next_steps()`. No `apply()`.
`prereq.py`	`PrereqTool` ABC + `Docker`/`Kubectl`/`Kind`/`Helm`/`Tofu` checks + `PrereqRegistry`.
`versions.py`	Loads `VERSIONS.json`, exposes `versions()`, `helm_repo(name)`, `chart_version(name)`.
`paths.py`	Resolved `Paths` dataclass (chart dir, secrets dir, infra/tls/, bootstrap-dir resolved).
`logger.py`	`Logger` Protocol + `ConsoleLogger` / `NullLogger`.
`shell.py`	`CommandRunner` Protocol + `SubprocessRunner` / `DryRunRunner`.
`os_detect.py` + `installer.py`	OSFamily detection (Arch/Debian/RHEL/Darwin/other) + per-family `installer.run()` Strategy.
`port_forward.py`	`PortForward` generic (namespace, Service, port + kubeconfig path). Used by OpenBao + GitLab + the CLI dispatcher.

`infra/scripts/bootstrap/phase2/` — 13 installers + pipeline¶

(see deep-dive/Phase 2 Pipeline.md)

`infra/scripts/bootstrap/phase3/` — 6-step GitLab provisioner¶

(see deep-dive/Phase 3 Pipeline.md)

`infra/tofu/` — IaC¶

File	Purpose
`providers.tf`	`kind ~> 0.11`, `helm ~> 3.0`, `local ~> 2.5`, `null ~> 3.2`.
`variables.tf`	`cluster_name`, `kubernetes_version`, `node_shapes`, `kubeconfig_path`, `data_root`, `domain`, `preserve_stateful_data`.
`locals.tf`	Resolves absolute paths + node shapes into kind-style node specs.
`cluster.tf`	`kind_cluster.cicd` + per-node hostPath bind mounts + containerd config patches (registry TLS trust) + `null_resource.wipe_data` destroy provisioner.
`outputs.tf`	`kubeconfig_path`, `ca_`, `wildcard_`, `phase_ready`.
`tofu.tfvars.example`	Committed template; copy to `tofu.tfvars` (gitignored) for local overrides.

`apps/` — Canonical GitLab-side source¶

Path	Purpose
`apps/shared-code/`	Cross-app CI templates + helper scripts. Lives in its own GitLab project under `blueprint-apps/`; consumed via cross-project `include:` from the others.
`apps/guestbook/`	Classic k8s guestbook: Go app (`guestbook-go/`), Dockerfile, helm chart.
`apps/redis/`	Redis master: helm chart (no Dockerfile; uses upstream `redis:7.4.1`).
`apps/redis-slave/`	Redis slave workload: helm chart (no Dockerfile; uses upstream `redis:7.4.1`).

`apps/shared-code/templates/` — per-project CI templates¶

Template	Used by
`per-project-shared-code.gitlab-ci.yml` (literal)	`shared-code` project (no substitution).
`per-project-app.gitlab-ci.yml.tpl`	`guestbook` (build_image=true). Substitutes `$name`, `$build`, `$release`, `$namespace`.
`per-project-redis.gitlab-ci.yml.tpl`	`redis`, `redis-slave` (build_image=false, uses upstream image). Substitutes `$name`, `$build`, `$release`, `$namespace`, `$redis_host`, `$redis_tag`.

phase3/ci_render.py does the substitution. The rule "templates must have their own files" is asserted by a regression test that fails on any yaml.dump literal in ci_render.py.

`.agents/skills/provision-phase-{1,2,3}/`¶

Each SKILL.md follows the canonical 10-section template:

Pre-flight — what to verify before starting
Install — the one-liner
Smoke tests — checkable invariants
URLs you can reach after install
Iteration loop — when something is off
Canonical (known-good) pinned versions
Common pitfalls (frozen — append, don't rewrite)
Rules of thumb (apply when adding Phase-N pieces)
When the install is green — the exact "you're done" output
How to undo — symmetric teardown

These are the runbooks. AI agents read them as soon as the repo opens (the agentskills.io open standard is supported by Copilot, Cursor, Claude Code, Codex).

Phase 3 invariants (summary — see Phase-3 doc for full contract)¶

A full tofu destroy && tofu apply && uv run blueprint-bootstrap --phase 2 && uv run blueprint-phase3 cycle reproduces the exact end state:

*.local.example.net URLs respond with the same TLS fingerprint (CA regenerated).
GitLab root password identical (chart-bundled initial_root password re-minted by the chart).
4 GitLab projects under blueprint-apps/ with the same IDs and CI/CD variables.
4 pipelines (one per project) green.
The secret/gitlab/bootstrap/admin_token PAT is re-minted in OpenBao — blueprint-phase3 does the gitlab-rails runner exec automatically.
The CI/CD variables (CI_KUBECONFIG_B64, CI_REGISTRY_IMAGE, CI_HELM_CHART_DIR, CI_INSECURE_REGISTRY) match the pre-wipe values.

There is no manual glab step required to rebuild — automation is the invariant.