Skip to content

Workflow Overview

The blueprint has three top-level workflows, one per phase, plus the day-to-day inner loop (git push → running workload). Every workflow has a sequence diagram showing who-does-what-when, and a state diagram for the underlying objects (cluster, GitLab projects, OpenBao secrets) so the boundary between "manual" and "automated" steps stays visible.

Core Workflows

Workflow 1 — Phase 1: Cluster bring-up

End state: a 5-node kind cluster reachable via the bootstrap-managed kubeconfig, with infra/data/shared/ hostPath mounts on every node and a Headlamp chart pre-cached locally.

Hold "Alt" / "Option" to enable pan & zoom 
sequenceDiagram
  participant Dev as Developer
  participant Bootstrap as blueprint-bootstrap (--phase 1)
  participant OS as OS package manager
  participant Docker as Docker daemon
  participant Tofu as tofu
  participant Helm as helm

  Dev->>Bootstrap: uv run blueprint-bootstrap --phase 1
  Bootstrap->>Bootstrap: detect OSFamily (arch/debian/rhel/darwin)
  Bootstrap->>OS: install missing tools (docker, kubectl, kind, helm, tofu)
  OS-->>Bootstrap: tools on PATH
  Bootstrap->>Tofu: tofu -chdir=infra/tofu init
  Bootstrap->>Tofu: tofu -chdir=infra/tofu validate
  Bootstrap->>Helm: helm repo add + helm pull (Headlamp chart)
  Helm-->>Bootstrap: infra/helm-charts/headlamp-0.43.0.tgz
  Bootstrap-->>Dev: print [user] next commands<br/>(tofu apply, helm install headlamp)
  Dev->>Tofu: tofu -chdir=infra/tofu apply -auto-approve
  Tofu->>Docker: create kindest/node containers (1 cp + 3 gitlab + 1 runner)
  Docker-->>Tofu: 5 nodes Ready
  Tofu-->>"Dev: kubernetes API up, kubeconfig at infra/tofu/kubeconfig"
  Dev->>Helm: helm install headlamp infra/helm-charts/headlamp-0.43.0.tgz (optional)

Steps (idempotent):

  1. uv sync — installs the Python package and three entry points (uv.lock ensures reproducibility).
  2. uv run blueprint-bootstrap --phase 1 — runs prereq checks (with auto-install of missing tools), seeds infra/tofu/tofu.tfvars, runs tofu init + tofu validate, downloads Headlamp chart to infra/helm-charts/headlamp-0.43.0.tgz. Prints the next commands.
  3. You run tofu -chdir=infra/tofu apply -auto-approve — the bootstrap never applies infrastructure.
  4. You optionally run helm install for Headlamp (the bootstrap's HeadlampInstaller prints the command; per spec, Phase 1 installers print rather than execute).

Failure paths: - Prereq missing → PrereqRegistry calls installer_for(os_family).install(tool_name). - tofu init fails → check infra/tofu/.terraform.lock.hcl is present and tofu.tfvars exists. - kind cluster never settles → infra/tofu/cluster.tf:extra_mounts.propagation toggled to Bidirectional is the destructive (2026-07+) default; if kind + Docker disagree on bind semantics, set -var=preserve_stateful_data=true and re-apply.

Workflow 2 — Phase 2: Application stack install

End state: Gateway API CRDs + CNPG + Redis + MinIO + OpenBao + GitLab (with chart-bundled Envoy Gateway + chart-bundled OpenBao subchart) + GitLab Runner all Running, all secrets in OpenBao, wildcard TLS for *.local.example.net materialised.

Hold "Alt" / "Option" to enable pan & zoom 
sequenceDiagram
  participant Dev as Developer
  participant Bootstrap as blueprint-bootstrap (--phase 2)
  participant Helm as helm
  participant Kube as kubectl
  participant OpenBao as OpenBao (in-cluster)

  Dev->>Bootstrap: uv run blueprint-bootstrap --phase 2
  Bootstrap->>Kube: cluster-info, helm version
  Bootstrap->>Kube: apply Gateway API CRDs (standard + Envoy CRDs)
  Bootstrap->>Helm: install local-path-provisioner + mark default SC
  Bootstrap->>Kube: pre-create PV/PVC pairs (CNPG/Redis/MinIO/OpenBao/Gitaly)
  Bootstrap->>Helm: install CloudNativePG operator + Cluster/postgresql-cnpg
  Bootstrap->>Kube: create DBs (gitlabhq_production, openbao) + roles
  Bootstrap->>Helm: install bitnami/redis single-node
  Bootstrap->>Kube: snapshot redis-password.txt
  Bootstrap->>Helm: install MinIO single-node
  Bootstrap->>Kube: create 11 GitLab buckets via in-cluster mc
  Bootstrap->>Helm: install OpenBao (openbao ns)
  Bootstrap->>Kube: bao operator init (writes openbao-init.json)
  Bootstrap->>Kube: bao operator unseal
  Bootstrap->>OpenBao: write chart-managed secrets (gitlab/postgres/redis/minio/rails/gitaly/kas)
  Bootstrap->>Kube: mint wildcard TLS via openssl (cfssl)
  Bootstrap->>Kube: materialise 4 Gateway listener Secrets
  Bootstrap->>Helm: install gitlab chart 10.1.1 (bundled Envoy + bundled OpenBao + Gitaly)
  Bootstrap->>Kube: wait for gitlab-webservice + registry + kas
  Bootstrap->>Helm: install gitlab-runner chart 0.71.0
  Bootstrap->>OpenBao: write runner registration token
  Bootstrap-->>Dev: Phase 2 install complete.<br/>/etc/hosts + CA trust + secrets CLI instructions

The 13-step order inside the pipeline is strict — see the Phase 2 Pipeline deep dive for the rationale (must-restore-before-chart-install, must-wildcard-before-Gitlab, etc.).

Steps (idempotent):

  1. Pre-flight — kubectl cluster-info + helm version --short.
  2. Gateway API CRDs — standard v1.5.0 + chart-shipped Envoy CRDs (EnvoyProxy, ClientTrafficPolicy, plus experimental TCPRoute, BackendTLSPolicy).
  3. local-path StorageClass + mark as default + configure pathBase on the host side.
  4. Stable PV/PVC pairs for CNPG + Redis + MinIO + OpenBao + Gitaly. Stamps CNPG-specific PVC annotations + ownerReferences[controller=true].
  5. CloudNativePG operator + Cluster/postgresql-cnpg — single instance, 8Gi; creates databases gitlabhq_production + openbao; mints gitlab + openbao PG roles (passwords → infra/secrets/cnpg-role-passwords.json).
  6. Redis single-node (architecture=standalone); password snapshotted to infra/secrets/redis-password.txt.
  7. MinIO single-node + create 11 GitLab buckets (lfs, artifacts, uploads, packages, backups, terraform-state, ci-secure-files, pages, dependency-proxy, snippets) via in-cluster mc.
  8. OpenBao (bootstrap-installed) — helm install + bao operator init + bao operator unseal; root token + unseal key in infra/secrets/openbao-init.json (0600).
  9. Wildcard TLS — self-signed CA + cert for *.local.example.net minted via openssl; 4 Gateway listener Secrets materialised (gitlab-wildcard-tls, registry-tls, kas-tls, minio-tls). MUST complete before GitLab install.
  10. Persistent secrets restore — pull infra/secrets/gitlab-runtime-secrets.yaml and re-apply any chart-managed Secrets that match the on-disk data.
  11. GitLab chart install — chart 10.1.1 sub-installs gateway-helm (Envoy Gateway) + chart-bundled OpenBao; consumes the external CNPG/Redis/MinIO we just stood up.
  12. Persistent secrets snapshot — pull anything left in the chart's Secrets for the next wipe+reinstall.
  13. GitLab Runner — chart 0.71.0, registers against http://gitlab-webservice-default.gitlab.svc:8181 using the registration token we captured into OpenBao.

Failure paths (each maps to a single installer + line range): - Gateway not Programmed → check kind_node_trust.py (containerd cert symlinks). - CNPG Cluster not Ready → check cloudnative_pg.py PVC ownership annotations. - GitLab not reachable via https://gitlab.local.example.net → check wildcard_certs.py listener Secret + /etc/hosts mapping. - Runner never appears → check runner.py registration token re-read from OpenBao.

Workflow 3 — Phase 3: GitLab-side project provisioning

End state: 4 GitLab projects under the blueprint-apps/ group (shared-code, guestbook, redis, redis-slave), each with a rendered .gitlab-ci.yml, CI/CD variables set, and one smoke pipeline green.

Hold "Alt" / "Option" to enable pan & zoom 
sequenceDiagram
  participant Dev as Developer
  participant Bootstrap as blueprint-phase3
  participant OpenBao as OpenBao
  participant GitLab as GitLab (in-cluster)
  participant Runner as GitLab Runner
  participant Cluster as kind cluster

  Dev->>Bootstrap: uv run blueprint-phase3
  Bootstrap->>OpenBao: read secret/gitlab/bootstrap/admin_token
  alt token missing
    Bootstrap->>GitLab: exec gitlab-rails runner (PAT mint)
    GitLab-->>Bootstrap: PAT (api, sudo, registry, ...)
    Bootstrap->>OpenBao: write secret/gitlab/bootstrap/admin_token
  end
  Bootstrap->>GitLab: glab auth login --token <PAT> (seeder only)
  Bootstrap->>GitLab: ensure blueprint-apps group exists (visibility=internal)
  loop for each entry in apps_manifest.yaml (shared-code first)
    Bootstrap->>GitLab: create/find project
    Bootstrap->>Bootstrap: git clone into apps-local/blueprint-apps/<name>/
    Bootstrap->>Bootstrap: rsync apps/<name>/ → working tree
    Bootstrap->>Bootstrap: render .gitlab-ci.yml (from apps/shared-code/templates/)
    Bootstrap->>GitLab: set CI/CD variables (CI_KUBECONFIG_B64, CI_REGISTRY_IMAGE, ...)
    Bootstrap->>Bootstrap: git commit + push
    Bootstrap->>GitLab: trigger pipeline on main
  end
  loop for each project
    Bootstrap->>GitLab: poll pipeline status (up to 10 min)
    GitLab-->>Runner: schedule job
    Runner->>Runner: helm lint + helm template
    Runner->>Runner: kaniko build → 127.0.0.1:5000/blueprint-apps/<name>
    Runner->>Runner: helm upgrade --install (using CI_KUBECONFIG_B64)
    Runner->>Cluster: deploy workload
    GitLab-->>Bootstrap: pipeline status = success
  end
  Bootstrap-->>Dev: 4/4 smoke pipelines green, URLs reachable

Steps (idempotent, six total):

  1. Pre-flight — glab auth works, OpenBao reachable, GitLab webservice healthy, in-cluster registry TCP-listening, apps/<name>/ paths exist, template files parse.
  2. Mint PAT — exec into gitlab-toolbox, run gitlab-rails runner to mint a PAT (scopes: api, read_api, read_repository, write_repository, read_registry, write_registry, read_user, sudo), persist to OpenBao at secret/gitlab/bootstrap/admin_token. Survives tofu destroy.
  3. Seed glab auth login — narrow exception to the "no glab in the bootstrap" rule (Rule 1). Populates ~/.config/glab-cli/config.yml so the agent/user can glab api in the same shell.
  4. Group — ensure blueprint-apps top-level group exists (visibility=internal).
  5. Per-project provisioning — for each entry in apps_manifest.yaml (in order; shared-code MUST be first since other projects include: it):
  6. ensure project exists in the group,
  7. clone into apps-local/blueprint-apps/<name>/,
  8. rsync apps/<name>/ → working tree,
  9. render .gitlab-ci.yml from apps/shared-code/templates/,
  10. set CI/CD variables (CI_KUBECONFIG_B64, CI_REGISTRY_IMAGE, CI_HELM_CHART_DIR, CI_INSECURE_REGISTRY),
  11. commit + push (only if anything changed).
  12. Smoke test — trigger one pipeline per project on main, poll until terminal, fail if not success. Timeout: 10 min (matches GitLab Runner's default job timeout for kaniko + helm upgrade).

Failure paths: - Token mint fails → check gitlab-toolbox pod is Running. - Group already exists with different visibility → Group step is idempotent. - Project provisioning fails partway through → re-run resumes from the failed step. - Smoke pipeline times out → check Runner registration + cluster RBAC (namespace-create perm for gitlab-runner SA).

Workflow 4 — The Day-to-Day Inner Loop

Once the cluster + GitLab + Runner + Phase-3 projects exist, the canonical edit loop is:

Hold "Alt" / "Option" to enable pan & zoom 
sequenceDiagram
  participant Dev as Developer
  participant Apps as apps/guestbook/<br/>(canonical source)
  participant Bootstrap as blueprint-phase3
  participant Cloned as apps-local/blueprint-apps/guestbook/
  participant GitLab as GitLab
  participant Runner as GitLab Runner (k8s)
  participant Cluster as kind cluster
  participant Edge as Envoy Gateway / NodePort
  participant Browser as Developer browser

  Dev->>Apps: $EDITOR main.go
  Dev->>Bootstrap: uv run blueprint-phase3
  Bootstrap->>Cloned: git pull --ff-only
  Bootstrap->>Cloned: rsync apps/guestbook/ → working tree
  Bootstrap->>Cloned: render .gitlab-ci.yml (from shared-code/templates/)
  Bootstrap->>Cloned: git commit + push origin main
  Cloned->>GitLab: webhook → push event
  GitLab->>Runner: schedule pipeline job (kubernetes executor)
  Runner->>Runner: helm lint + helm template (validation stage)
  Runner->>Runner: kaniko build → gitlab-registry.gitlab.svc:5000/blueprint-apps/guestbook
  Runner->>Cluster: helm upgrade --install guestbook ./helm-chart
  Cluster->>Edge: Service → Envoy HTTPRoute
  Edge-->>Browser: https://guestbook.local.example.net responds

The whole dev → push → running workload cycle is on the order of 60–120 seconds, dominated by the kaniko build.

The two working surfaces are critical:

  • apps/<name>/ — canonical source, frozen after first push. Edit code here.
  • apps-local/blueprint-apps/<name>/ — gitignored working tree, recreated on every --reset-clones run. Re-rendered every blueprint-phase3 run.

Mis-treating either side creates silent drift. See docs/phase-3.md §12 for the full version of this distinction.

Data Flow

The Day-to-Day Inner Loop data flow, split across the two natural axes.

Control plane — Developer → GitLab

From git push to a job request on the Runner. The control plane ends at the GitLab scheduler; the Runner's own work lives in the next diagram.

Hold "Alt" / "Option" to enable pan & zoom 
flowchart LR
  Dev["Developer<br/>(edit apps/)"] -->|"uv run blueprint-phase3"| P3["blueprint-phase3"]
  P3 -->|"rsync"| AL["apps-local/blueprint-apps/<name>/"]
  P3 -->|"render .gitlab-ci.yml<br/>(from apps/shared-code/templates/)"| AL
  AL -->|"git commit + push"| GL["GitLab<br/>(receiving repo)"]
  GL -->|"webhook → CI pipeline"| Runner["GitLab Runner<br/>(Kubernetes executor)"]

  OB["OpenBao"] -.->|"gitlab-rails runner PAT (Phase 3)"| P3
  OB -.->|"initial_root_password (Phase 2)"| GL

Data plane — Runner → Browser

What the Runner actually does once the job is scheduled: lint, build, push, deploy, expose. The control plane ends here.

Hold "Alt" / "Option" to enable pan & zoom 
flowchart LR
  Runner["GitLab Runner<br/>(Kubernetes executor)"] -->|"read CI_KUBECONFIG_B64<br/>(base64 kubeconfig)"| KC["CI kubeconfig"]
  Runner -->|"helm lint + template"| HC["Helm chart validation"]
  Runner -->|"kaniko build → push"| Reg["gitlab-registry.gitlab.svc:5000"]
  Runner -->|"helm upgrade --install"| Cluster["kind cluster"]
  Cluster -->|"Service → HTTPRoute"| EG["Envoy Gateway<br/>(chart subchart)"]
  EG -->|"TLS terminated"| Browser["Browser<br/>https://app.local.example.net"]

  OB["OpenBao"] -.->|"runner registration token"| Runner

State Management

There are several pieces of mutable state in this system. Each has a single owner; the rule is "exactly one writer, multiple readers":

State Owner Reader(s)
infra/tofu/terraform.tfstate tofu (human runs) The OpenTofu CLI
infra/data/shared/stable/<service>/ CNPG operator (PG), MinIO (/data), OpenBao State (PG), Redis, Gitaly Their consumers
infra/secrets/openbao-init.json bootstrap/phase2/openbao.py OpenBaoClient (read after restart)
infra/secrets/gitlab-runtime-secrets.yaml bootstrap/phase2/persistent_secrets.py Same module on restore
infra/secrets/cnpg-role-passwords.json, redis-password.txt, minio-root-{user,password}.txt respective Phase-2 installers All reverted via --destroy
infra/tls/wildcard/{ca,cert,key}.pem bootstrap/phase2/wildcard_certs.py kind_node_trust.py (binds into containerd)
OpenBao KV v2 (secret/gitlab/...) OpenBaoClient.write(...) All installers via OpenBaoClient.read(...)
GitLab projects (blueprint-apps/*) blueprint-phase3 Runner (CI/CD variables)
apps-local/blueprint-apps/<name>/ blueprint-phase3 Developer (clone loop only)

Error Handling

Idempotency hooks (the foundational rule)

Every installer is designed so re-run == resume from failed step, success == no-op. This isn't a try/except afterthought — installers check kubectl get/helm list for current state before acting. Examples:

  • CloudNativePGInstaller skips helm install if the Cluster already exists; checks kubectl wait --for=condition=Ready cluster/postgresql-cnpg after install.
  • OpenBaoInstaller calls bao operator init only on first install (returns error on second; we catch); bao operator unseal runs only if bao status reports sealed.
  • GitlabInstaller uses helm upgrade --install (atomic) and reads the existing root password from OpenBao to avoid re-minting on re-runs.
  • WildcardCertsInstaller checks for Secret/gitlab-wildcard-tls and only re-mints when missing or expired.

Pipeline error reporting

Phase2Pipeline.run() wraps the whole orchestration in a try/except that:

  1. Logs the failing step number (Phase 2 install failed: ...).
  2. Recommends re-running (bootstrap.py --phase 2 — every step is idempotent).
  3. Maps the failure to a specific installer (see smoke tests in provision-phase-2/SKILL.md).

Phase3Pipeline follows the same pattern.

Pod-UID owned files

When bootstrap --destroy can't rm -rf an infra/data/shared/stable/<service>/ dir because it's owned by a pod UID (e.g. openbao=100, postgres=1001), the bootstrap falls back to a one-shot Docker/Podman privileged container that bind-mounts the parent dir and chmod -R a+rwX && rm -rf the child. Same trick the null_resource.wipe_data provisioner uses on the tofu side. See infra/scripts/bootstrap/cli.py:destroy + phase2/local_path_provisioner.py for the full trap.

Conflicting flags

If tofu apply -var=preserve_stateful_data=true and bootstrap --destroy --preserve-data disagree (one true, one false), the destroy step's null_resource.wipe_data hooks into the kind cluster's bind-mount semantics; the result is a half-wiped state where the cluster is gone but data dirs linger (or vice versa). Both flags must agree or you get a divergent state — see AGENTS.md rule #4 for the full contract.

What's Next