Skip to content

Deep Dive: Phase 2 Pipeline

Phase 2 installs every cluster-resident application on top of the Phase-1 kind cluster: Gateway API CRDs → local-path StorageClass → CNPG → Redis → MinIO → OpenBao → wildcard TLS → GitLab → Runner. It is idempotent and strictly ordered — the order is encoded as 13 methods on Phase2Pipeline and the rationale is documented in this file.

Overview

infra/scripts/bootstrap/phase2/pipeline.py:Phase2Pipeline is the orchestrator. It owns ordering and error reporting but delegates install logic to one installer per step. Every step is wrapped in the same try/except pattern, so a failure on step 7 doesn't wipe steps 1–6; re-running the bootstrap resumes from the failed step.

The 13-step pipeline + 3 supporting installers:

Hold "Alt" / "Option" to enable pan & zoom 
sequenceDiagram
  participant Pipe as Phase2Pipeline
  participant Log as Logger
  participant CRDs as GatewayCRDsInstaller
  participant LPath as LocalPathProvisionerInstaller
  participant SS as StableStorageInstaller
  participant PG as CloudNativePGInstaller
  participant RDS as RedisInstaller
  participant MIN as MinioInstaller
  participant OB as OpenBaoInstaller
  participant WC as WildcardCertsInstaller
  participant PS as PersistentSecretsInstaller
  participant GL as GitlabInstaller
  participant RUN as GitLabRunnerInstaller
  participant KNT as KindNodeTrustInstaller
  participant CDNS as CoreDNSPatch
  participant RDP as RegistryDNSFixInstaller

  Pipe->>Log: [bootstrap] Step 1/13 Pre-flight
  Pipe->>Log: kubectl cluster-info + helm version
  Pipe->>Log: Step 2/13 Install Gateway API CRDs
  Pipe->>CRDs: install()
  Pipe->>Log: Step 3/13 Install local-path StorageClass
  Pipe->>LPath: install()
  Pipe->>Log: Step 4/13 Pre-create stable PV/PVC pairs
  Pipe->>SS: install()
  Pipe->>Log: Step 5/13 Install CloudNativePG + Cluster + role/db
  Pipe->>PG: install()
  Pipe->>Log: Step 6/13 Install Redis (single-node)
  Pipe->>RDS: install()
  Pipe->>Log: Step 7/13 Install MinIO + 11 buckets
  Pipe->>MIN: install()
  Pipe->>Log: Step 8/13 Install OpenBao + init + unseal
  Pipe->>OB: install()
  Pipe->>Log: Step 9/13 Mint wildcard TLS + listener Secrets
  Pipe->>WC: install()
  Pipe->>Log: Step 10/13 Patch CoreDNS (*.local.example.net)
  Pipe->>CDNS: install()
  Pipe->>Log: Step 11/13 Mount wildcard CA into containerd
  Pipe->>KNT: install()
  Pipe->>Log: Step 12/13 Restore chart-managed Secrets + install GitLab + snapshot
  Pipe->>PS: restore()
  Pipe->>GL: install()
  Pipe->>Log: Step 13/13 Patch in-cluster registry /etc/hosts + install Runner
  Pipe->>RDP: install()
  Pipe->>RUN: install()

Key Files

File Purpose
phase2/pipeline.py Phase2Pipeline orchestrator + the 13 step methods.
phase2/catalog.py Phase2Installers dataclass — bundle of every installer the pipeline owns.
phase2/gateway.py GatewayCRDsInstaller — standard Gateway API v1.5.0 + chart-shipped Envoy CRDs (EnvoyProxy, ClientTrafficPolicy, BackendTLSPolicy, TCPRoute).
phase2/local_path_provisioner.py rancher/local-path-provisioner manifest + mark local-path as default SC + configure pathBase = /var/local/shared.
phase2/stable_storage.py Pre-create PV/PVC pairs for PG/Redis/MinIO/OpenBao/Gitaly. Stamps CNPG-specific PVC annotations + ownerReferences[controller=true].
phase2/cloudnative_pg.py CNPG operator helm + Cluster/postgresql-cnpg (single instance, 8Gi) + bootstrap gitlab + openbao PG roles.
phase2/redis.py bitnami/redis chart with architecture=standalone. Snapshot password.
phase2/minio.py minio chart + 11 buckets via in-cluster mc + dual-key gitlab-rails-storage Secret.
phase2/openbao.py openbao chart + bao operator init (writes infra/secrets/openbao-init.json) + bao operator unseal + PG backend wired to the CNPG cluster.
phase2/wildcard_certs.py openssl self-signed CA + wildcard cert + 4 Gateway listener Secrets.
phase2/persistent_secrets.py snapshot + restore chart-managed Secrets (rails/gitaly/kas passwords, initial-root-password). Excludes PG/Redis/MinIO (their own snapshot files).
phase2/gitlab.py GitLab chart 10.1.1: bundled Envoy (gateway-helm) + bundled OpenBao subchart; point at the external PG/Redis/MinIO.
phase2/runner.py GitLab Runner chart 0.71.0: registers against http://gitlab-webservice-default.gitlab.svc:8181.
phase2/secrets.py OpenBaoClient — hvac-backed client + auto port-forward to openbao.openbao.svc:8200.
phase2/coredns_patch.py Pod-side DNS rewrite for *.local.example.net (NodeLocal DNSCache + CoreDNS rewrite).
phase2/kind_node_trust.py Mount the wildcard CA into containerd's per-host certs directories (registry TLS pulls).
phase2/registry_dns_pin.py Patch /etc/hosts inside each kind node so kubelet can reach the registry on registry.local.example.net.
phase2/references/*.yaml committed install-time YAML — cluster-postgresql.yaml, helm-values-{cnpg,redis,minio,openbao,gitlab,runner}.yaml, gateway-api-crds/*.

Step-by-Step: The Why of the Order

The 13-step order is strict and the boot-time invariants depend on it.

Step 1 — Pre-flight

def _step_preflight(self) -> None:
    self.log.info("[bootstrap] Step 1/13  Pre-flight (cluster reachable)")
    self.runner.run(["kubectl", "cluster-info"], check=True)
    self.runner.run(["helm", "version", "--short"], check=True)

Why first: every other step assumes a reachable cluster and a working helm. Failing fast here saves the rest of the pipeline from cascading errors.

Step 2 — Gateway API CRDs

GatewayCRDsInstaller.install() applies:

  • Standard Gateway API v1.5.0 CRDs (Gateway, HTTPRoute, ReferenceGrant, GatewayClass)
  • Chart-shipped Envoy CRDs (EnvoyProxy, ClientTrafficPolicy, BackendTLSPolicy, TCPRoute — experimental)

Why second: chart 10.x sub-installs Envoy Gateway via gateway-helm; that subchart's CRDs MUST exist before the GitLab chart install at step 12. Installing them ourselves (rather than letting the chart do it) means later steps can reference HTTPRoute and EnvoyProxy immediately.

Step 3 — local-path StorageClass

LocalPathProvisionerInstaller.install():

  1. Applies the upstream local-path-storage.yaml (pin: v0.0.30).
  2. Marks local-path as the default StorageClass (storageclass.kubernetes.io/is-default-class: "true").
  3. Patches the config map so pathBase = /var/local/shared — this is the host-side dir bound via kind's extra_mounts in infra/tofu/cluster.tf.

Why third: chart 10.x's global.storageClass: local-path (set in helm-values-gitlab.yaml) requires the SC to exist before any chart-managed PVC is claimed. If you skip this step, GitLab's chart-managed PVCs (gitaly, prometheus, registry, rails/uploads) land on standard (which doesn't exist) and fail.

Step 4 — Stable storage (pre-create PV/PVC pairs)

StableStorageInstaller.install() pre-creates PVs + PVCs for the services we want identity-preserving across tofu destroy && tofu apply:

  • CloudNativePG (PG cluster)
  • Redis
  • MinIO
  • OpenBao (PG backend storage)
  • Gitaly (chart-managed, but we pre-create the hostPath-backed PV)

For CNPG specifically, the installer:

  1. Names the PVC <cluster-name>-<serial> (e.g. postgresql-cnpg-1).
  2. Sets the required annotations: cnpg.io/cluster, cnpg.io/instanceName, cnpg.io/instanceRole=primary, cnpg.io/nodeSerial=1, cnpg.io/pvcRole=main.
  3. Sets ownerReferences[] to contain the Cluster as a controller: true reference after the Cluster is created (see step 5).

Why fourth: the CNPG operator's claim-by-controller selector (.metadata.controller) only works if PVCs are pre-created with the right name + annotations. If you let the chart/Cluster create its own PVC, you lose identity preservation across recreate.

Step 5 — CloudNativePG

CloudNativePGInstaller.install():

  1. Installs the CNPG operator (cloudnative-pg/cloudnative-pg chart).
  2. Waits for the cnpg-controller-manager Deployment.
  3. Applies the Cluster/postgresql-cnpg (single instance, 8Gi PVC, SCRAM-SHA-256 auth).
  4. Waits for Cluster.Ready == True.
  5. Creates the gitlab + openbao PG roles; passwords → infra/secrets/cnpg-role-passwords.json.
  6. Creates the gitlabhq_production + openbao databases.

Why before Redis/MinIO/OpenBao/GitLab: every other service depends on PG (GitLab uses gitlabhq_production; chart-bundled OpenBao uses openbao; standalone OpenBao uses openbao).

Step 6 — Redis

RedisInstaller.install():

  1. helm install redis bitnami/redis with architecture=standalone (no Sentinel, no replicas).
  2. Waits for the master pod.
  3. Snapshots the auto-generated password Secret to infra/secrets/redis-password.txt.

Why here: GitLab's global.redis.host points at redis-master.redis.svc:6379. The chart expects it to exist with a known password.

Step 7 — MinIO

MinioInstaller.install():

  1. helm install minio minio/minio (single-node, no distributed mode).
  2. Creates the 11 GitLab buckets via in-cluster mc (the minio Pod exposes the necessary tools):
  3. lfs, artifacts, uploads, packages, backups, terraform-state, ci-secure-files, pages, dependency-proxy, snippets, plus the internal bucket for the registry backend
  4. Snapshots root user + password to infra/secrets/minio-root-{user,password}.txt.
  5. Creates the dual-key gitlab-rails-storage Opaque Secret:
  6. connection key — Rails-side Fog/AWS provider schema (used by Rails object_store settings)
  7. config key — Docker registry native s3: block (used by the container registry's s3: driver)

Why before GitLab: GitLab's global.appConfig.object_store.*.connection.secret: gitlab-rails-storage must exist with both keys populated. If you let the chart's bundled (formerly) MinIO subchart create the bucket config, the chart-minted password wouldn't match our on-disk data (since we wiped the chart's bundled PG/Redis/MinIO in chart 10.x).

Step 8 — Bootstrap-installed OpenBao

OpenBaoInstaller.install():

  1. helm install openbao openbao/openbao.
  2. Waits for openbao-0 Running+Ready.
  3. On first install: bao operator init -format=json → persists to infra/secrets/openbao-init.json (0600).
  4. bao operator unseal if currently sealed.
  5. Mounts KV v2 at secret/.
  6. Stores PG role creds for the chart-bundled OpenBao subchart (which uses the same external CNPG).

Why before wildcard + GitLab: OpenBao is the secret backend for the rest of Phase 2. gitlab.py reads secret/gitlab/initial_root_password and stores it; runner.py reads secret/gitlab/runner/registration_token. Without OpenBao up, secrets have nowhere to live.

Step 9 — Wildcard TLS

WildcardCertsInstaller.install():

  1. openssl req -newkey rsa:2048 -x509 -sha256 -days 3650 -nodes ... → infra/tls/wildcard/ca.pem + cert.pem + key.pem.
  2. Materialises 4 Secrets in the gitlab namespace:
  3. gitlab-wildcard-tls (cfssl's own wildcard + CA)
  4. registry-tls, kas-tls, minio-tls (alias Secrets pointing at the same cert material)
  5. Re-mounts the CA into containerd's per-host certs directories (/etc/containerd/certs.d/{registry,kas,gitlab,minio}.local.example.net/ca.pem).
  6. Optionally exports the CA to the host (the user adds it to the trust store separately).

Why before the GitLab chart install (step 12): the chart's pre-install Gateway certificateRefs block needs the Secrets to exist with tls.crt + tls.key. Without them, Gateway reports Listeners[].tls.certificateRefs[].name not found.

Why before kind_node_trust (step 11): kubelet's containerd config references the same CA via /etc/containerd/certs.d/<host>/hosts.toml — those files need to point at a CA that's already on disk.

Step 10 — CoreDNS patch (cluster-side wildcard resolution)

CoreDNSPatch.install():

  1. Uses kubectl patch on the coredns ConfigMap to add a rewrite rule for *.local.example.net → the Gateway service ClusterIP.
  2. Restarts the CoreDNS pods.

Why tenth: the cluster's kube-system pods (the Runner, the CI jobs) need to resolve gitlab.local.example.net and registry.local.example.net to the in-cluster Gateway, not to 127.0.0.1. Without this, kubelet's image pull from registry.local.example.net would fail.

Step 11 — kind_node_trust (containerd CA)

KindNodeTrustInstaller.install():

  1. For each kind node, creates /etc/containerd/certs.d/{registry,kas,gitlab,minio}.local.example.net/.
  2. Symlinks ca.pem into each (the mounted source from the host).
  3. Restarts containerd on each node.

Why eleventh: with the wildcard cert materialised (step 9) AND CoreDNS rewriting the wildcard (step 10), kubelet's image-pull TLS verification still needs to trust the CA. The mirror is what containerd's config_path = "/etc/containerd/certs.d" reads.

Step 12 — Persistent secrets restore + GitLab chart install + snapshot

PersistentSecretsInstaller.restore():

  1. Reads infra/secrets/gitlab-runtime-secrets.yaml.
  2. Re-applies every Secret in the snapshot to the gitlab namespace.

Why before GitLab install: chart-managed Services (gitaly, registry) keep their Secret material stable across recreate by reading from this snapshot. If we skip restore, the chart mints fresh secrets whose passwords don't match the on-disk PVs.

GitlabInstaller.install():

  1. helm upgrade --install gitlab charts.gitlab.io/gitlab with helm-values-gitlab.yaml.
  2. Chart bundles:
  3. gateway-helm (Envoy Gateway as managed subchart)
  4. bundled OpenBao subchart (gitlab-openbao) → uses the external CNPG via global.openbao.psql.host
  5. Wires to external PG/Redis/MinIO via:
  6. global.psql.host: postgresql-cnpg-rw.postgresql.svc
  7. global.redis.host: redis-master.redis.svc
  8. appConfig.object_store.*.connection.secret: gitlab-rails-storage
  9. Waits for: gitlab-webservice-default, gitlab-registry, gitlab-kas, gitlab-sidekiq-all-in-1, gitlab-task-runner, gitlab-gitaly-0.
  10. After install, exec into gitlab-toolbox and read initial_root_password + the runner registration token → write to OpenBao at secret/gitlab/initial_root_password + secret/gitlab/runner/registration_token.

Why after restore + OpenBao + wildcard + persistent_secrets: dependencies on every prior step.

PersistentSecretsInstaller.snapshot():

After GitLab is up, snapshot chart-managed Secrets (rails/gitaly/kas passwords) into infra/secrets/gitlab-runtime-secrets.yaml. This snapshot is what step 12-restore uses on the next install.

Step 13 — Registry DNS pin + Runner install

RegistryDNSFixInstaller.install():

Patches /etc/hosts inside each kind node so kubelet's image-pull path resolves registry.local.example.net correctly. Even with the CoreDNS rewrite (step 10), some paths (specifically the registry's internal callers) need the /etc/hosts mirror.

GitLabRunnerInstaller.install():

  1. helm upgrade --install gitlab-runner charts.gitlab.io/gitlab-runner with helm-values-runner.yaml.
  2. gitlabUrl: http://gitlab-webservice-default.gitlab.svc:8181 (in-cluster Service DNS + plain HTTP — TLS terminates at Envoy above).
  3. Kubernetes executor.
  4. Reads registration token from OpenBao at secret/gitlab/runner/registration_token.

Why last: it depends on every other step being up — the runner registers against GitLab, reads CI variables that depend on the registry being up, etc.

Architecture

Hold "Alt" / "Option" to enable pan & zoom 
classDiagram
  class Phase2Pipeline {
    +paths: Paths
    +runner: CommandRunner
    +log: Logger
    +installers: Phase2Installers
    +run() int
    -_step_preflight() void
    -_step_gateway_crds() void
    -_step_local_path() void
    -_step_stable_storage() void
    -_step_cnpg() void
    -_step_redis() void
    -_step_minio() void
    -_step_openbao() void
    -_step_wildcard_certs() void
    -_step_coredns_patch() void
    -_step_kind_node_trust() void
    -_step_persistent_secrets_restore() void
    -_step_gitlab() void
    -_step_registry_dns_pin() void
    -_step_persistent_secrets_snapshot() void
    -_step_runner() void
  }

  class Phase2Installers {
    +crds: GatewayCRDsInstaller
    +local_path: LocalPathProvisionerInstaller
    +stable_storage: StableStorageInstaller
    +cnpg: CloudNativePGInstaller
    +redis: RedisInstaller
    +minio: MinioInstaller
    +openbao: OpenBaoInstaller
    +wildcard_certs: WildcardCertsInstaller
    +persistent_secrets: PersistentSecretsInstaller
    +gitlab: GitlabInstaller
    +runner: GitLabRunnerInstaller
    +kind_node_trust: KindNodeTrustInstaller
    +coredns_patch: CoreDNSPatch
    +registry_dns_pin: RegistryDNSFixInstaller
  }

  class HelmAppInstaller {
    <<abstract>>
    +install() void
    +uninstall() void
  }

  class GatewayCRDsInstaller
  class LocalPathProvisionerInstaller
  class StableStorageInstaller
  class CloudNativePGInstaller
  class RedisInstaller
  class MinioInstaller
  class OpenBaoInstaller
  class WildcardCertsInstaller
  class PersistentSecretsInstaller
  class GitlabInstaller
  class GitLabRunnerInstaller

  Phase2Pipeline --> Phase2Installers
  Phase2Installers "1" --> "*" HelmAppInstaller
  HelmAppInstaller <|-- LocalPathProvisionerInstaller
  HelmAppInstaller <|-- RedisInstaller
  HelmAppInstaller <|-- OpenBaoInstaller
  HelmAppInstaller <|-- GitlabInstaller
  HelmAppInstaller <|-- GitLabRunnerInstaller

Implementation Details

Idempotency by Inspection

Most installers check current state before acting:

# Excerpt from OpenBaoInstaller
def install(self) -> None:
    if self._is_initialised():
        log.info("OpenBao already initialised; skipping init")
    else:
        out = self._run_bao_operator_init()
        self._persist_init(out)
    if self._is_sealed():
        self._unseal()

# Excerpt from CloudNativePGInstaller
def install(self) -> None:
    if self._cluster_exists():
        log.info("Cluster exists; verifying Ready")
        self._wait_ready()
    else:
        self._helm_install()
        self._wait_ready()
    self._create_roles_and_dbs()  # idempotent — IF NOT EXISTS

The Pre-Install Re-Run Rule

The GitlabInstaller is never allowed to re-mint the initial root password on re-runs — that would invalidate existing browser sessions. Instead:

def install(self) -> None:
    if self._root_password_already_in_openbao():
        password = self.openbao.read_gitlab_initial_root_password()
    else:
        password = self._generate_random_password()
        self.openbao.write_gitlab_initial_root_password(password)
        # patch the values.yaml override
        self._set_initial_root_password_in_chart_values(password)
    self._helm_install()

The runner registration token, however, IS refreshed on every re-run (chart-managed; cheap to update).

The Snapshot / Restore Loop

PersistentSecretsInstaller implements a two-phase idiom:

  • restore() (step 12a) — apply every Secret in the snapshot to the namespace before the chart install so existing PVs match.
  • snapshot() (step 12c) — re-read chart-managed Secrets from a fresh cluster after the chart install and re-write the snapshot file.

Skipping restore = chart-minted fresh passwords that don't match the on-disk data → GitLab logs FATAL: password authentication failed for user "gitlab" and never recovers. Restore is therefore mandatory.

How Installers Avoid Hardcoded Versions

# phase2/redis.py
class RedisInstaller(HelmAppInstaller):
    REPO_KEY = "redis"   # ← key in VERSIONS.json:helm_repositories

# VERSIONS.json excerpt:
# "redis": {
#   "name": "redis",
#   "url": "https://charts.bitnami.com/bitnami",
#   "chart": "redis",
#   "chart_version": "19.6.4",
#   "values_overrides": { ... }
# }

versions.py:load_versions() reads VERSIONS.json; versions.chart_version("redis") returns 19.6.4. No installer references 19.6.4 directly. Bumping it = edit JSON + cache the new tarball + re-run.

Smoke Tests (What "green" Looks Like)

Per .agents/skills/provision-phase-2/SKILL.md §3:

Check How to verify
Cluster + helm reachable kubectl cluster-info + helm version --short
5 Envoy gateway pods kubectl -n gitlab get pods -l app.kubernetes.io/name=envoy
Gateway/gitlab Programmed kubectl -n gitlab get gateway gitlab -o jsonpath='{.status.conditions[?(@.type=="Programmed")].status}' → True
local-path is default SC kubectl get sc local-path -o jsonpath='{.metadata.annotations.storageclass\.kubernetes\.io/is-default-class}' → "true"
CloudNativePG cluster Ready kubectl -n postgresql get cluster postgresql-cnpg -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}' → True
Redis reachable kubectl -n redis exec redis-master-0 -- redis-cli PING → PONG
MinIO S3 reachable kubectl -n minio port-forward svc/minio 9001:9001 + mc ls local
OpenBao unsealed bao status | jq .sealed → false
Wildcard CA on disk ls infra/tls/wildcard/{ca.pem,cert.pem,key.pem}
4 listener Secrets exist kubectl -n gitlab get secrets | grep -E 'gitlab-wildcard-tls|registry-tls|kas-tls|minio-tls'
GitLab UI returns 200 curl -kI https://gitlab.local.example.net/-/health → 200 OK
Runner registered Admin → CI/CD → Runners shows 1 runner, online
14 GitLab pods Running kubectl -n gitlab get pods (webservice, sidekiq, kas, gitaly, prometheus, toolbox, registry, postgresql, redis, minio, openbao, shared-secrets, gitlab-runner, …)

Failure Mapping

Each smoke-test failure has a one-shot instruction in the Iteration loop section of the per-phase skill:

Symptom Look at
Gateway not Programmed phase2/kind_node_trust.py + phase2/wildcard_certs.py
CloudNativePG not Ready phase2/cloudnative_pg.py + phase2/stable_storage.py (PVC name + annotations)
CNPG FATAL: password authentication failed for user "gitlab" you skipped persistent_secrets.restore() — see phase2/persistent_secrets.py
Runner never registers phase2/runner.py (token re-read from OpenBao)
https://gitlab.local.example.net 404 check /etc/hosts entry + CA trust store
OpenBao sealed: true after restart run uv run blueprint-secrets unseal (re-runs bao operator unseal)

Conventions (from AGENTS.md)

  • Bootstrap prepares, never applies. The bootstrap never runs tofu apply. It runs helm upgrade --install, kubectl apply, bao operator init, etc. — these are applications, not infrastructure, and the spec allows the bootstrap to drive them.
  • No shell for non-trivial logic. infra/scripts/bootstrap/ is a class-based Python package composed of single-responsibility classes.
  • VERSIONS.json is the single source of truth.
  • Helm charts cached locally. infra/helm-charts/<name>-<version>.tgz — no network round-trip on re-install.
  • Templates are real YAML files on disk. No yaml.dump({...}) of a dict that hardcodes pipeline shape. The chart values file phase2/references/helm-values-gitlab.yaml is a committed file, not a Python literal.
  • Pod-to-GitLab traffic uses Service DNS, port 8181, HTTP. Never https://gitlab.local.example.net from inside a pod.

Potential Improvements

  • A --summary flag that prints the current step number + the next 3 steps (for --check mode without re-running everything).
  • A phase2/references/ schema validator (jsonschema over helm-values-*.yaml) — fails on missing required override keys.
  • Replace the imperative _step_* methods with a declarative table (step_label, installer_method_name, prereq_step_labels), so the pipeline order can be introspected and reordered.
  • Per-step timing in a phase2/last_timings.json so a regression in chart install times is visible.