Deep Dive: Phase 3 Pipeline

Phase 3 is the GitLab-side app onboarding workflow. Where Phase 2 stands up the platform (cluster, GitLab, Runner, OpenBao), Phase 3 stands up the projects inside GitLab — the four blueprint-apps/* repos with their CI/CD pipelines, registry images, and CI/CD variables. The contract is "no manual glab step required to rebuild" — Phase 3 survives tofu destroy && tofu apply && bootstrap --phase 2 && bootstrap-phase3 by re-minting the admin PAT on the new cluster.

Overview

infra/scripts/bootstrap/phase3/pipeline.py:Phase3Pipeline is the 6-step orchestrator. Each step delegates to a Phase-3 module:

• phase3/apps_manifest.py — load apps_manifest.yaml (4 entries: shared-code first, guestbook, redis, redis-slave).
• phase3/apps_local.py — AppsLocalFS: manage the apps-local/blueprint-apps/<name>/ working tree (git clone + pull + rm).
• phase3/ci_render.py — render .gitlab-ci.yml from templates via string.Template.safe_substitute. Rule 0: no YAML literals here.
• phase3/gitlab_client.py — wrapper around python-gitlab + requests for the GitLab REST API. Rule 1: no glab in the bootstrap.
• phase3/glab_client.py — narrow exception: seed ~/.config/glab-cli/config.yml via glab auth login so the agent can run glab in the same shell.
• phase3/pipeline.py — the 6-step orchestrator.

The hard contract is the rebuild invariant:

tofu -chdir=infra/tofu destroy -auto-approve
tofu -chdir=infra/tofu apply   -auto-approve
uv run blueprint-bootstrap --phase 2    # recreates OpenBao + GitLab + Runner
uv run blueprint-phase3                 # recreates the 4 GitLab projects + CI

After this four-command sequence, the exact same end state is reproduced: same blueprint-apps group path, same 4 project IDs, same CI/CD variables, same per-project pipeline shape, same registry images, same secret/gitlab/bootstrap/admin_token PAT. There is no manual glab step in between.

Architecture

Hold "Alt" / "Option" to enable pan & zoom

classDiagram
  class Phase3App {
    +paths: Paths
    +gitlab: GitlabClient
    +openbao: OpenBaoClient
    +apps_local: AppsLocalFS
    +manifest: AppsManifest
    +log: Logger
    +check: bool
    +destroy: bool
    +no_overwrite_ci: bool
    +reset_clones: bool
    +run() int
    +from_argv(...) Phase3App
  }

  class Phase3Pipeline {
    +paths: Paths
    +gitlab: GitlabClient
    +openbao: OpenBaoClient
    +apps_local: AppsLocalFS
    +manifest: AppsManifest
    +log: Logger
    +run() Phase3Result
    -_preflight() void
    -_mint_pat_if_missing() void
    -_seed_glab_auth() void
    -_ensure_group() void
    -_provision_projects() void
    -_smoke_test() void
  }

  class AppsManifest {
    +specs: tuple[AppSpec, ...]
    +filter(name) AppsManifest
    +iter() Iterator[AppSpec]
  }

  class AppSpec {
    +name: str
    +src_path: Path
    +chart_path: Path
    +build_image: bool
    +has_unit_tests: bool
    +helm_release: str
    +helm_namespace: str
    +registry_image: str
    +dockerfile_path: Path | None
    +redis_host: str | None
    +redis_tag: str | None
  }

  class GitlabClient {
    +pat: str
    +base_url: str
    +gl: gitlab.Gitlab
    +session: requests.Session
    +get(url) Any
    +post(url, json) Any
    +put(url, json) Any
    +delete(url) Any
  }

  class AppsLocalFS {
    +paths: Paths
    +log: Logger
    +clone_or_pull(name) Path
    +rsync_apps(name, working_tree) void
    +write_rendered_ci(name, content) void
    +commit_and_push(name) bool
    +wipe() void
  }

  class RenderResult {
    +path: Path
    +template: str
    +substitutions: dict[str, str]
  }

  Phase3App --> Phase3Pipeline
  Phase3App --> AppsManifest
  Phase3App --> GitlabClient
  Phase3App --> AppsLocalFS
  Phase3App --> OpenBaoClient
  Phase3Pipeline --> GitlabClient
  Phase3Pipeline --> AppsLocalFS
  Phase3Pipeline --> AppsManifest
  AppsManifest "1" --> "*" AppSpec
  GitlabClient --> OpenBaoClient
  GitlabClient --> PortForward

Key Files

File	Purpose
phase3/pipeline.py	Phase3Pipeline orchestrator + the 6 step methods + Phase3Result.
phase3/apps_manifest.py	AppSpec dataclass + AppsManifest collection + load_apps_manifest(path) loader.
phase3/apps_manifest.yaml	The 4-project manifest (data only — no template content per Rule 0).
phase3/apps_local.py	AppsLocalFS — manages apps-local/blueprint-apps/<name>/ working tree.
phase3/ci_render.py	Substitutes string.Template.safe_substitute over apps/shared-code/templates/*.yml.tpl.
phase3/gitlab_client.py	GitlabClient — wraps python-gitlab + requests. Mints admin PAT via gitlab-rails runner (exec into gitlab-toolbox).
phase3/glab_client.py	seed_glab_auth() — narrow exception to Rule 1. Calls glab auth login --token <PAT> to populate ~/.config/glab-cli/config.yml.
apps/shared-code/templates/*.yml[.tpl]	The per-project CI templates (the one source of truth for pipeline shape).

Step-by-Step

Step 1 — Pre-flight

def _preflight(self) -> None:
    # 1. glab auth (or python-gitlab fallback) → GitLab reachable
    # 2. OpenBao reachable (port-forward 8200 if needed)
    # 3. GitLab webservice healthy (curl /-/health → 200)
    # 4. in-cluster registry TCP-listening (gitlab-registry.gitlab.svc:5000)
    # 5. apps/<name>/ paths exist for each entry in apps_manifest.yaml
    # 6. template files parse (yaml.safe_load)

Fails fast — no GitLab-side mutation unless every invariant holds.

Step 2 — Mint PAT (idempotent; survives tofu destroy)

def _mint_pat_if_missing(self) -> None:
    existing = self.openbao.read("gitlab/bootstrap/admin_token")
    if existing:
        log.info("admin PAT already in OpenBao; reusing")
        self._gitlab.set_pat(existing["token"])
        return

    # Mint a fresh PAT via gitlab-rails runner execed in the toolbox pod
    pat = self._gitlab.mint_admin_pat_via_toolbox(scopes=PAT_SCOPES)
    self.openbao.write(
        "gitlab/bootstrap/admin_token",
        {"token": pat},
    )

mint_admin_pat_via_toolbox does:

1. kubectl -n gitlab exec toolbox -- gitlab-rails runner - <<< " token = User.find_by_username('root').personal_access_tokens.create!( scopes: %w[api read_api read_repository write_repository read_registry write_registry read_user sudo], name: 'blueprint-phase3' ) puts token.token "
2. Parses the PAT from the runner's stdout.

Scopes chosen to cover: project/group CRUD + CI/CD variables + runners + pipelines + admin-mode operations (Phase 3 needs sudo for the gitlab-rails runner exec in some iterations). The mint happens in-cluster so the fresh PAT is scoped to the same root user; on a wipe+recreate, the chart-minted root password changes (chart auto-mints), but the bootstrap mints a new PAT under that fresh root.

Step 3 — Seed glab auth login

def _seed_glab_auth(self) -> None:
    pat = self.openbao.read("gitlab/bootstrap/admin_token")["token"]
    self.glab_client.seed_glab_auth(pat=pat, gitlab_host="gitlab.local.example.net")

This is the narrow exception to Rule 1 ("bootstrap never uses glab"). It exists so the agent/user can run glab api ... in the same shell without re-authenticating.

The relevant glab config (~/.config/glab-cli/config.yml) is only read by the developer shell / agent — bootstrap operations read the OpenBao PAT directly via python-gitlab. Bootstrap never reads the file.

Step 4 — Ensure the blueprint-apps group

def _ensure_group(self) -> None:
    gl = self._gitlab.gl
    try:
        gl.groups.get("blueprint-apps")
        log.info("blueprint-apps group exists; reusing")
    except gitlab.exceptions.GitlabGetError:
        gl.groups.create({"name": "blueprint-apps", "path": "blueprint-apps",
                          "visibility": "internal"})
        log.info("blueprint-apps group created")

visibility=internal so any logged-in user (including the bootstrap's PAT) can read the projects, but they're hidden from anonymous users.

Step 5 — Per-project provisioning

For each AppSpec in the manifest (in order; shared-code first):

def _provision_project(self, spec: AppSpec) -> None:
    name = spec.name

    # 5a. ensure project exists in the group
    self._gitlab.ensure_project(
        name=name,
        namespace="blueprint-apps",
        visibility="internal",
    )

    # 5b. clone (or pull) into apps-local/blueprint-apps/<name>/
    working_tree = self.apps_local.clone_or_pull(
        name=name,
        gitlab_url="https://gitlab.local.example.net/blueprint-apps/<name>.git",
    )

    # 5c. rsync apps/<name>/ → working tree
    self.apps_local.rsync_apps(name=name, src=spec.src_path)

    # 5d. render .gitlab-ci.yml (unless --no-overwrite-ci)
    if not self._no_overwrite_ci or not (working_tree / ".gitlab-ci.yml").exists():
        rendered = render_for(
            spec=spec,
            templates_dir=templates_dir(self.paths.blueprint_dir),
        )
        self.apps_local.write_rendered_ci(name=name, content=rendered.content)

    # 5e. set CI/CD variables
    self._gitlab.set_project_variables(
        name=name,
        variables={
            "CI_KUBECONFIG_B64": base64.b64encode(kubeconfig_bytes).decode(),
            "CI_REGISTRY_IMAGE": spec.registry_image,
            "CI_HELM_CHART_DIR": spec.chart_path,
            "CI_INSECURE_REGISTRY": "1",  # registry is plain HTTP on the cluster-internal Service
        },
    )

    # 5f. commit + push (if anything changed)
    self.apps_local.commit_and_push(name=name)

Why shared-code MUST be first: other projects' .gitlab-ci.yml does include: { project: blueprint-apps/shared-code, ref: main, file: templates/<...> }. If shared-code doesn't exist when guestbook's pipeline runs, every pipeline fails with "remote: project not found".

Why --no-overwrite-ci: lets you hand-edit .gitlab-ci.yml in apps-local/ and have it stick across bootstrap runs. The bootstrap then only re-renders on the next clean-up (blueprint-phase3 --reset-clones).

Why CI_INSECURE_REGISTRY=1: the in-cluster registry's containerd mount is plain HTTP on gitlab-registry.gitlab.svc:5000 (TLS terminates at Envoy Gateway above). The CI job writes /etc/docker/daemon.json with {"insecure-registries": ["127.0.0.1:5000", "gitlab-registry.gitlab.svc:5000"]} and restarts dockerd. Without this, kaniko fails on x509: certificate signed by unknown authority.

Step 6 — Smoke test

def _smoke_test(self) -> None:
    for spec in self.manifest.iter():
        pipeline = self._gitlab.trigger_main_pipeline(name=spec.name)
        log.info(f"triggered pipeline #{pipeline.id} for {spec.name}; polling")
        deadline = time.time() + SMOKE_PIPELINE_TIMEOUT_S  # 600s
        while time.time() < deadline:
            pipeline = self._gitlab.refresh_pipeline(name=spec.name, id=pipeline.id)
            if pipeline.status in ("success", "failed", "canceled", "skipped"):
                break
            time.sleep(SMOKE_POLL_INTERVAL_S)
        if pipeline.status != "success":
            raise RuntimeError(f"smoke pipeline for {spec.name} {pipeline.status}")

10-minute timeout matches GitLab Runner's default job timeout for kaniko + helm upgrade. If anything fails, the bootstrap returns non-zero with the failing pipeline URL.

Implementation Details

apps_manifest.yaml — Data Only

Rule 0 says "templates must have their own files". The apps_manifest.yaml contains only data — names, paths, flags. No template content:

- name: shared-code
  src_path: apps/shared-code
  chart_path: ""
  build_image: false
  has_unit_tests: false
  helm_release: shared-code
  helm_namespace: shared-code
  registry_image: gitlab-registry.gitlab.svc:5000/blueprint-apps/shared-code

- name: guestbook
  src_path: apps/guestbook
  chart_path: helm-chart
  build_image: true
  has_unit_tests: false
  helm_release: guestbook
  helm_namespace: guestbook
  registry_image: gitlab-registry.gitlab.svc:5000/blueprint-apps/guestbook
  dockerfile_path: guestbook-go/Dockerfile

- name: redis
  src_path: apps/redis
  chart_path: helm-chart
  build_image: false       # uses upstream redis:7.4.1
  helm_release: redis-master-demo
  helm_namespace: redis-demo
  registry_image: gitlab-registry.gitlab.svc:5000/blueprint-apps/redis
  redis_host: redis-master.redis-demo.svc.cluster.local
  redis_tag: "7.4.1"

- name: redis-slave
  src_path: apps/redis-slave
  chart_path: helm-chart
  build_image: false
  helm_release: redis-slave-demo
  helm_namespace: redis-slave-demo
  registry_image: gitlab-registry.gitlab.svc:5000/blueprint-apps/redis-slave
  redis_host: redis-replica.redis-slave-demo.svc.cluster.local
  redis_tag: "7.4.1"

Critical flags:

• build_image: true → renderer picks per-project-app.gitlab-ci.yml.tpl (has a build stage with kaniko).
• build_image: false → renderer picks per-project-redis.gitlab-ci.yml.tpl (no build stage; uses upstream image via values.yaml).
• has_unit_tests → reserved for future use (no pytest in any of the shipped apps today).
• helm_release + helm_namespace are deliberately namespaced away from the infrastructure redis release/namespace. Earlier these were both redis, which let the smoke pipeline's helm upgrade --install shadow the bitnami/redis release that backs GitLab. The new names (redis-master-demo, redis-demo) leave the infrastructure release untouched.
• dockerfile_path is relative to CI_PROJECT_DIR (which equals the repo root in GitLab CI). For guestbook, the Go source lives in apps/guestbook/guestbook-go/, so the Dockerfile is at guestbook-go/Dockerfile from the repo root.

ci_render.py — Substitute, Don't Literal-Build

def render_for(spec: AppSpec, templates_dir: Path) -> RenderResult:
    if spec.name == "shared-code":
        template_file = templates_dir / SHARED_CODE_TEMPLATE  # literal .yml
    elif spec.build_image:
        template_file = templates_dir / APP_TEMPLATE           # build stage
    else:
        template_file = templates_dir / REDIS_TEMPLATE          # no build stage

    body = template_file.read_text()
    rendered = string.Template(body).safe_substitute({
        "name":      spec.name,
        "build":     "true" if spec.build_image else "false",
        "tests":     "true" if spec.has_unit_tests else "false",
        "release":   spec.helm_release,
        "namespace": spec.helm_namespace,
        "redis_host": spec.redis_host or "",
        "redis_tag": spec.redis_tag or "",
    })
    return RenderResult(path=template_file, template=body, substitutions={...})

The regression test tests/test_ci_render.py::test_no_yaml_literals_in_ci_render fails on any dedent( or triple-quoted YAML string literal in this module's source. Any new field requires extending the templates and (if the field is conditional) introducing a new template file.

apps_local.py — Working Tree Manager

AppsLocalFS encapsulates the apps-local/blueprint-apps/<name>/ lifecycle:

class AppsLocalFS:
    def clone_or_pull(self, *, name: str, gitlab_url: str) -> Path:
        working_tree = self._working_tree_path(name)
        if not working_tree.exists():
            self._git_clone(name=name, url=gitlab_url, dst=working_tree)
        else:
            self._git_pull_ff_only(working_tree)
        return working_tree

    def rsync_apps(self, *, name: str, src: Path) -> None:
        # rsync -av --delete src/ working_tree/
        ...

    def write_rendered_ci(self, *, name: str, content: str) -> None:
        ...

    def commit_and_push(self, *, name: str) -> bool:
        # git add -A; git commit -m "..."; git push origin main
        ...

The CA path (infra/tls/wildcard/ca.pem) is set on every git invocation via -c http.sslCAInfo=... so the self-signed wildcard is trusted without polluting the user's ~/.gitconfig.

gitlab_client.py — python-gitlab + requests

Two layers:

1. python-gitlab for typed CRUD (projects, groups, variables, runners, pipelines, users).
2. requests for the few endpoints python-gitlab doesn't expose cleanly (repository-files API, runner registration tokens, KAS-specific paths).

The wrapper is stateless — no caching, no auto-retry, no port-forward management (port-forward orchestration lives in pipeline.py). Methods raise gitlab.exceptions.GitlabError or requests.HTTPError.

Auth: lazy fetch from OpenBao at secret/gitlab/bootstrap/admin_token. No hard-coded PAT. A bootstrap --destroy of Phase 2 also wipes the OpenBao init (and the chart-managed Secrets); the next bootstrap --phase 2 + blueprint-phase3 re-inits OpenBao + mints a fresh PAT.

glab_client.py — Auth Seeder (the Narrow Exception)

def seed_glab_auth(*, pat: str, gitlab_host: str) -> None:
    # Narrowly allowed: populates the user's glab config so they can
    # `glab api ...` in the same shell. Bootstrap operations never
    # read this file.
    subprocess.run(
        ["glab", "auth", "login", "--hostname", gitlab_host, "--token", pat],
        check=True,
    )

Justification: glab auth login is the only documented glab command where the alternative (edit ~/.config/glab-cli/config.yml) is worse — every glab version writes a slightly different yaml structure, and editing it by hand is brittle. Every other bootstrap operation uses python-gitlab/requests.

Smoke Tests

Per .agents/skills/provision-phase-3/SKILL.md §3:

Check	How to verify
blueprint-apps group exists	glab api groups/blueprint-apps \| jq .id
4 projects exist	glab api 'groups/blueprint-apps/projects?per_page=10' \| jq '.[] | .path_with_namespace'
Each project has .gitlab-ci.yml	glab api 'projects/:id/repository/files/.gitlab-ci.yml?ref=main' \| jq .content
CI/CD variables set	glab api 'projects/:id/variables' \| jq '.[].key'
Smoke pipelines green	4 pipelines (one per project) with status = success
Workloads reachable	curl https://guestbook.local.example.net → 200

Failure Mapping

Each failure has a one-shot map in provision-phase-3/SKILL.md §5 (Iteration loop):

Symptom	Look at
Token mint fails	gitlab-toolbox pod must be Running. Check kubectl -n gitlab get pods -l app=toolbox.
Group visibility wrong	Re-run (the step is idempotent but won't change visibility on existing groups).
Project creation fails partway	Re-run — Phase 3 resumes from the failed entry.
Smoke pipeline failed for guestbook	Check glab api projects/:id/jobs/:id/trace. Most common: kaniko fails on x509 → check CI_INSECURE_REGISTRY=1 is set on the project.
Smoke pipeline failed for redis	helm chart syntax error → helm template ./helm-chart from apps/redis/helm-chart/.
gitlab-rails runner exec hangs	toolbox pod is unhealthy → check kubectl -n gitlab logs gitlab-toolbox-XYZ.
glab auth login fails	glab CLI not installed → apt install glab / brew install glab.

Conventions (from AGENTS.md, Phase 3-specific)

• All GitLab-side changes are automated by blueprint-phase3. The bootstrap mints a fresh admin PAT via gitlab-rails runner (exec in toolbox) and uses python-gitlab + requests — never glab — for the GitLab API.
• apps/ is committed; apps-local/ is gitignored runtime state. Canonical source lives in this repo; bootstrap clones into apps-local/, rsyncs, renders, commits, pushes.
• The first project in apps_manifest.yaml is shared-code. Other projects' CI include: it via cross-project include.
• *.local.example.net is for humans, not for cluster traffic. Pods MUST use gitlab-webservice-default.gitlab.svc:8181, gitlab-registry.gitlab.svc:5000, etc.
• Pod-to-GitLab traffic uses Service DNS, port 8181, HTTP. Plain HTTP — TLS terminates at Envoy Gateway above.
• glab is for the human/AI agent, not the bootstrap. The one exception is glab auth login (phase3/glab_client.py:seed_glab_auth).

Potential Improvements

• Schema-validate apps_manifest.yaml against a pydantic model at bootstrap startup (early failure on malformed entries).
• Render the per-project CI YAML through a strict jsonschema after substitution (catch $variable typos that .safe_substitute silently swallows).
• Watch the smoke pipeline via glab ci view --live for nicer output (right now polling is silent between status changes).
• Idempotency improvement: when a project already exists with the same variables, skip the set_project_variables call (currently rewrites every time).