Deep Dive: Bootstrap Package

infra/scripts/bootstrap/ is the Python package that owns every install step in the blueprint. It is the composition root of the system — every dependency that any installer needs (paths, logger, shell runner, version catalog, OpenBao client, kubeconfig) is wired here.

Overview

The package implements the spec's bootstrap.py requirement as a SOLID module:

• Composition roots (app.py:BootstrapApp, app_phase3.py:Phase3App) wire all dependencies and run a Pipeline orchestrator.
• Single-responsibility installer classes under phase2/<thing>.py and the dedicated Phase-3 modules under phase3/<thing>.py each own exactly one piece of the install.
• Cross-cutting Protocols (Logger, CommandRunner, PortForward) decouple the bootstrap from the side effects (logs, shell, network forwarding), so --dry-run and --check work without conditional code sprinkled everywhere.
• Versions pinned in JSON (VERSIONS.json) — no class hardcodes a version string.

Module Map

infra/scripts/bootstrap/
├── VERSIONS.json                ← single source of truth (versions + helm repos + chart values)
├── __main__.py                  python3 -m bootstrap → cli:main
├── cli.py                       click wrapper → blueprint-bootstrap
├── secrets_cli.py               click wrapper → blueprint-secrets
├── phase3_cli.py                click wrapper → blueprint-phase3
├── app.py                       BootstrapApp (Phase 1+2 composition root)
├── app_phase3.py                Phase3App (Phase 3 composition root)
│
├── app_installer.py             HelmAppInstaller generic + HeadlampInstaller subclass + factory
├── helm_cache.py                HelmChartCache (downloads <name>-<ver>.tgz)
├── tofu.py                      TofuRunner (init + validate + next_steps; NO apply)
├── prereq.py                    Prereq ABCs + Docker/Kubectl/Kind/Helm/Tofu checks
│
├── logger.py                    Logger Protocol + Console/Null
├── shell.py                     CommandRunner Protocol + Subprocess/DryRun
├── port_forward.py              generic 127.0.0.1:<svc>:port helper
├── paths.py                     resolved Paths dataclass (chart dir, secrets, tls, …)
├── versions.py                  load_versions() + tool_pin() + helm_repo()
├── os_detect.py                 OSFamily detection (arch/debian/rhel/darwin)
├── installer.py                 per-family installer Strategy (Arch/Debian/RHEL/Darwin)
├── help_full.py                 short + long click help rendering
│
├── phase2/                      ←── Phase 2 installers (13 steps)
│   ├── pipeline.py              Phase2Pipeline (orchestrator)
│   ├── catalog.py               Phase2Installers dataclass (bundle of every installer)
│   ├── gateway.py               Gateway API CRDs
│   ├── local_path_provisioner.py
│   ├── stable_storage.py        pre-create PV/PVC pairs + CNPG annotations
│   ├── cloudnative_pg.py        CNPG operator + Cluster + role/db bootstrap
│   ├── redis.py                 bitnami/redis single-node
│   ├── minio.py                 MinIO + 11 GitLab buckets + dual-key Secret
│   ├── openbao.py               OpenBao install + init + unseal
│   ├── wildcard_certs.py        self-signed wildcard cert + 4 listener Secrets
│   ├── gitlab.py                GitLab chart (bundles Envoy + bundled OpenBao)
│   ├── runner.py                GitLab Runner registration
│   ├── persistent_secrets.py    snapshot + restore chart-managed Secrets
│   ├── secrets.py               OpenBaoClient (hvac + auto port-forward)
│   ├── kind_node_trust.py       mount wildcard CA into containerd registry hosts
│   ├── coredns_patch.py         CoreDNS rewrite for *.local.example.net inside cluster
│   ├── registry_dns_pin.py      /etc/hosts entry for the registry inside the cluster
│   └── references/              committed YAML/CRD templates
│
└── phase3/                      ←── Phase 3 modules (own GitLab-side provisioning)
    ├── apps_manifest.py         AppSpec + AppsManifest (loader for apps_manifest.yaml)
    ├── apps_manifest.yaml       4-project manifest (shared-code first)
    ├── apps_local.py            AppsLocalFS (apps-local/ working tree manager)
    ├── ci_render.py             per-project .gitlab-ci.yml substitution
    ├── gitlab_client.py         python-gitlab + requests wrapper (no glab!)
    ├── glab_client.py           glab auth login seeder (Rule 1 narrow exception)
    ├── pipeline.py              Phase3Pipeline (6-step orchestrator)
    └── tests/                   pytest tests (incl. test_no_yaml_literals_in_ci_render)

Key Files & Responsibilities

cli.py, secrets_cli.py, phase3_cli.py — Click wrappers

Three entry points, one each:

• blueprint-bootstrap — install (--phase 1 \| 2), check (--check), destroy (--destroy [--preserve-data] [--dry-run] [--yes]), port-forward helper (--port-forward <target>).
• blueprint-secrets — read OpenBao secrets (read <path> <key>), open the UI (ui), generic in-cluster port-forward (port-forward <target> with --list).
• blueprint-phase3 — Phase 3 GitLab provisioning (--check, --destroy, --reset-clones, --project <name>, --no-overwrite-ci).

no_args_is_help=True means uv run blueprint-bootstrap with no args prints the short help (cheat sheet) — prevents running Phase 1 in the wrong directory by accident.

app.py — Phase 1+2 composition root

@dataclass(frozen=True)
class BootstrapApp:
    paths: Paths
    runner: CommandRunner
    log: Logger
    cache: HelmChartCache
    tofu: TofuRunner
    prereq_registry: PrereqRegistry
    installer: ArchInstaller | DebianInstaller | RhelInstaller | DarwinInstaller
    headlamp: HeadlampInstaller
    phase2: Phase2Pipeline

    def run(self) -> int:
        # Phase 1: prereq → helm_cache → tofu init/validate (NO apply) → print [user] next commands
        # Phase 2: phases2.run()

The prepare / apply boundary is encoded here: run() stops after tofu validate and prints the user's commands. There is no apply() method on TofuRunner.

app_phase3.py — Phase 3 composition root

Mirrors app.py but is a separate class. Phase 3 has nothing to do with cluster bootstrap — it provisions GitLab projects and renders CI templates. Same from_argv() factory pattern.

app_installer.py — Generic helm install helper

HelmAppInstaller is the abstract base; HeadlampInstaller is the concrete Phase-1 subclass.

class HelmAppInstaller(ABC):
    REPO_KEY: str    # VERSIONS.json key (e.g. "gitlab", "openbao")
    NAMESPACE: str
    RELEASE: str

    def install(self) -> None:
        # 1. ensure namespace
        # 2. helm repo add + repo update (against infra/helm-charts cache)
        # 3. helm upgrade --install ... with values file
        # 4. wait for rollout

Most Phase-2 installers (OpenBaoInstaller, GitlabInstaller, GitLabRunnerInstaller) extend this base and override pre-/post-install hooks.

paths.py — Resolved filesystem paths

Every installer takes Paths as a constructor arg. Paths resolves:

• bootstrap_dir — the package's absolute parent
• blueprint_dir — the repo root (../ from bootstrap_dir)
• infra_dir — blueprint_dir/infra
• tofu_dir, secrets_dir, tls_dir, data_dir, helm_charts_dir

Paths.from_bootstrap_dir(path) is the factory; all CLIs call it.

logger.py & shell.py — Cross-cutting Protocols

class Logger(Protocol):
    def info(self, msg: str) -> None: ...
    def ok(self, msg: str) -> None: ...
    def warn(self, msg: str) -> None: ...
    def err(self, msg: str) -> None: ...

class CommandRunner(Protocol):
    def run(self, argv: list[str], *, check: bool = True, env: Mapping[str, str] | None = None, cwd: Path | None = None) -> CommandResult: ...

Two implementations each: ConsoleLogger/NullLogger, SubprocessRunner/DryRunRunner. --check swaps the logger for NullLogger; --dry-run swaps the runner for DryRunRunner. Every installer takes them as constructor args — never imports print() or subprocess.run() directly.

versions.py + VERSIONS.json

load_versions(path) returns a dict; tool_pin(name) returns e.g. "v0.27.0"; helm_repo(name) returns the cached chart path. No class hardcodes a version.

Example VERSIONS.json keys: - tools.{docker,kubectl,kind,helm,opentofu,openssl} — per-OS package + version pin - kubernetes.kindest_node_image — pinned to v1.31.0 - helm_repositories.{local-path-provisioner,headlamp,openbao,gitlab,gitlab-runner} — URL + chart + chart_version + values_overrides - cloudnative_pg — operator version, cluster YAML

port_forward.py — Generic cluster-side forwarding

PortForward(runner, kubeconfig_path, namespace, service, remote_port, local_port) is used by:

• OpenBaoClient — openbao.openbao.svc:8200 → 127.0.0.1:8200 (KV v2 + token auth)
• GitlabClient — gitlab-webservice-default.gitlab.svc:8181 → 127.0.0.1:8181 (REST + port-less push)
• cli.py — --port-forward gitlab → NodePort on the chart-managed Envoy Gateway → 127.0.0.1:8443
• secrets_cli.py — port-forward openbao, port-forward gitlab-registry, port-forward minio, etc.

shared_for(...) is a memoised factory: one forward per (cluster, namespace, service, port) tuple, torn down on Ctrl-C.

Architectural Patterns

1. Single-Responsibility Installer + Composition Root

# infra/scripts/bootstrap/app.py
@dataclass
class BootstrapApp:
    paths: Paths
    runner: CommandRunner
    log: Logger
    prereq_registry: PrereqRegistry
    cache: HelmChartCache
    tofu: TofuRunner
    installer: ArchInstaller | DebianInstaller | RhelInstaller | DarwinInstaller
    headlamp: HeadlampInstaller
    phase2: Phase2Pipeline

    @classmethod
    def from_argv(cls, argv: list[str] | None = None) -> "BootstrapApp": ...

    def run(self) -> int:
        ...

Every Phase-2 installer follows the same shape:

@dataclass
class CNPGInstaller:
    paths: Paths
    runner: CommandRunner
    log: Logger
    cache: HelmChartCache
    openbao: OpenBaoClient | None  # wired only when OpenBao is needed

    def install(self) -> None:
        # 1. ensure namespace
        # 2. helm install / kubectl apply
        # 3. wait for Ready
        # 4. (optionally) snapshot creds

    def _snapshot_passwords(self) -> None: ...

The composition root wires it; the installer owns the details.

2. Prepare / Apply Boundary

# infra/scripts/bootstrap/tofu.py
class TofuRunner:
    def init(self) -> None: ...
    def validate(self) -> None: ...
    def next_steps(self) -> list[str]: ...   # prints `tofu apply`, etc.
    # No apply() method.

The bootstrap enforces the rule by omission — there is literally no method to call. The same rule expressed in BootstrapApp.run():

log.info(f"[user] Run: tofu -chdir={self.paths.tofu_dir} apply -auto-approve")
log.info(f"[user] Run: helm install headlamp ...")
# (we never execute these ourselves)

3. Branch by Abstraction (Protocols)

log: Logger = ConsoleLogger() if not args.check else NullLogger()
runner: CommandRunner = DryRunRunner() if args.dry_run else SubprocessRunner()

This is why --dry-run works without conditional code everywhere. Each installer holds runner (a CommandRunner) and log (a Logger) as fields — they have no idea whether they're in dry-run mode.

4. Idempotent Pipelined Orchestrator

class Phase2Pipeline:
    installers: Phase2Installers  # bundle of every installer

    def run(self) -> int:
        try:
            self._step_preflight()
            self._step_gateway_crds()
            self._step_local_path()
            ...
            self._step_runner()
        except Exception as e:
            log.err(f"Phase 2 install failed: {e}")
            log.err("Re-run `bootstrap.py --phase 2`. Every step is idempotent.")
            return 1
        return 0

Each step delegates to one installer; the pipeline owns ordering + error reporting.

Sample Install Sequence

# Trace of what uv run blueprint-bootstrap --phase 2 does internally:

Phase2Pipeline.run()
├── Phase2Pipeline._step_preflight()
│   └── runner.run(["kubectl", "cluster-info"], check=True)
├── Phase2Pipeline._step_gateway_crds()
│   └── installers.crds.install()        # GatewayCRDsInstaller
├── Phase2Pipeline._step_local_path()
│   └── installers.local_path.install()  # LocalPathProvisionerInstaller
├── Phase2Pipeline._step_stable_storage()
│   └── installers.stable_storage.install()
├── Phase2Pipeline._step_cnpg()
│   └── installers.cnpg.install()
│       ├── ensure namespace 'postgresql'
│       ├── helm install cnpg cloudnative-pg-<ver>.tgz
│       ├── wait for cnpg-controller-manager Deployment
│       └── apply Cluster/postgresql-cnpg (8Gi, single instance)
├── Phase2Pipeline._step_redis()
│   └── installers.redis.install()
├── Phase2Pipeline._step_minio()
│   └── installers.minio.install()       # includes bucket creation
├── Phase2Pipeline._step_openbao()
│   └── installers.openbao.install()
│       ├── helm install openbao openbao-<ver>.tgz
│       ├── wait for openbao-0 pod Running+Ready
│       ├── if sealed: bao operator unseal
│       └── on first install: bao operator init → infra/secrets/openbao-init.json
├── Phase2Pipeline._step_wildcard_certs()
│   └── installers.wildcard_certs.install()
│       ├── openssl req -newkey rsa:2048 -x509 ... → infra/tls/wildcard/ca.pem
│       ├── openssl req ... → infra/tls/wildcard/cert.pem
│       └── apply 4 Secrets (gitlab-wildcard-tls, registry-tls, kas-tls, minio-tls)
├── Phase2Pipeline._step_persistent_secrets_restore()
│   └── installers.persistent_secrets.restore()  # gitlab-runtime-secrets.yaml
├── Phase2Pipeline._step_gitlab()
│   └── installers.gitlab.install()
│       ├── helm upgrade --install gitlab gitlab-<ver>.tgz
│       │   # bundled: gateway-helm (Envoy) + bundled OpenBao subchart
│       ├── wait for gitlab-webservice-default + registry + kas
│       └── write secret/gitlab/initial_root_password to OpenBao
├── Phase2Pipeline._step_persistent_secrets_snapshot()
│   └── installers.persistent_secrets.snapshot()
└── Phase2Pipeline._step_runner()
    └── installers.runner.install()
        ├── helm upgrade --install gitlab-runner gitlab-runner-<ver>.tgz
        └── write secret/gitlab/runner/registration_token to OpenBao

Every step is idempotent; each one logs [bootstrap] Step N/13 ... so you can grep the terminal.

Dependencies

Module	Internal	External
app.py	app_installer, helm_cache, installer, logger, os_detect, paths, prereq, shell, tofu, versions, phase2/*	—
phase2/*	.. (parent: app_installer, logger, paths, shell)	sibling .secrets
phase3/*	.., ..phase2.secrets	python-gitlab, requests, git, gitlab-rails runner (via subprocess)
secrets.py (phase2)	..port_forward, ..shell, ..paths, ..logger	hvac

External Python deps live in pyproject.toml:dependencies:

dependencies = [
    "hvac>=2.3.0,<3.0",
    "click>=8.1.7,<9.0",
    "python-gitlab>=5.0.0,<6.0",
    "requests>=2.32.0,<3.0",
    "pyyaml>=6.0.1,<7.0",
]

Testing

• infra/scripts/bootstrap/phase3/tests/ — pytest suite. Includes test_no_yaml_literals_in_ci_render which fails on any yaml.dump literal in ci_render.py (enforces Rule 0: "templates must have their own files").
• Smoke tests live in .agents/skills/provision-phase-{1,2,3}/SKILL.md §3 — they enumerate checkable invariants (e.g. "5 Envoy pods Running", "Gateway Programmed=True", "runner registered in Admin → CI/CD → Runners", "OpenBao has 3 secrets").
• Iteration loop lives in the same SKILL.md §5 — maps a failed smoke test to the exact installer + line range to inspect.

Potential Improvements

• A persistent --json log mode for piping into Loki / Grafana would make long install debugging easier (currently everything is human-readable text with [bootstrap] / [user] prefixes).
• The Phase-2 install order is hard-coded in pipeline.py; a small DSL (list of (step_label, installer_method_name) tuples) would let us describe dependencies instead of embedding the order in method calls.
• The apps_manifest.yaml parser (apps_manifest.py) does a one-shot YAML load. Validating the manifest against a pydantic-style schema would catch malformed entries earlier.
• A --uninstall flag (mirror of --install) on each Phase-2 installer — currently the only inverse is bootstrap --destroy, which is cluster-level. Per-component teardown would simplify iteration when one chart upgrade goes wrong.

For the Phase-2 step-by-step, see deep-dive/Phase 2 Pipeline.md. For Phase 3, see deep-dive/Phase 3 Pipeline.md.