` demos¶

This phase turns the demo apps in blueprint/apps/ (guestbook, redis, redis-slave) into first-class GitLab projects, each with a working CI pipeline that builds, tests, and deploys the application into the Phase-1 cluster using its own Helm chart.

It depends on Phase 2 being healthy (GitLab + Runner + OpenBao + MinIO + the in-cluster registry reachable from CI pods). It is application-only work, not infrastructure work — the bootstrap CLI is a Python app and uses whichever Python tool is right for the job (the requests library for raw HTTP, python-gitlab for typed GitLab resource models, subprocess.run for shelling out). The glab CLI tool is for the human or AI agent driving the bootstrap interactively, not for the bootstrap itself.

0. Three architectural rules¶

Rule 0 — Pipeline code lives INSIDE GitLab¶

The previous draft of this plan put the CI template, the per-project .gitlab-ci.yml, the kubeconfig blob, and the apps manifest in infra/scripts/bootstrap/phase3/references/. That approach is rejected.

Pipeline code is application code — it describes how a piece of software is built, tested, and deployed. It belongs in the same repository as the application it builds. The bootstrap repo (k8s-cicd/blueprint/) is infrastructure tooling; it provisions the cluster and the GitLab instance, then steps out of the way. Putting pipeline code there would:

Couple the application lifecycle to the bootstrap toolchain — every pipeline edit would require a commit in the bootstrap repo and a uv run blueprint-phase3 re-run.
Duplicate the spec rule "no bash, use a Python app" — the bootstrap is the Python app; the pipelines are Go/shell because that's what builds Go containers. The "no bash" rule applies to the bootstrap, not to CI job scripts.
Confuse reviewers — blueprint/infra/scripts/bootstrap/phase3/references/ci-template-app-pipeline.yml looks like infrastructure, not application configuration.

The pipeline code that multiple projects share lives in a new top-level apps/shared-code/ directory — itself an apps/ subdirectory, itself a project in GitLab, but not cloned into each per-app repo. The other apps reference it via GitLab's include: mechanism (project, not local path).

Rule 1 — `glab` is for the human/agent, not the bootstrap¶

The glab CLI is great for one-off interactive work and for AI agents iterating on a plan in real time: its subcommands map cleanly to user intent (glab mr create, glab ci view, glab repo clone), it has stable JSON output (-F json), and it handles auth state transparently.

The bootstrap app is not an interactive agent. It's a Python program that runs end-to-end (uv run blueprint-phase3) with structured logging, retries, error handling, and unit tests. For this, glab is the wrong tool: shelling out to a subprocess for every API call adds 50-200 ms of overhead, loses structured exception handling, requires keeping the glab binary in lockstep with the bootstrap's expectations, and makes mocking in tests tricky (you have to mock subprocesses instead of requests adapters).

Therefore: - The bootstrap uses Python — primarily requests for raw GitLab HTTP API calls, with python-gitlab for typed resource models where its abstraction saves boilerplate (project create, variable CRUD, runner registration, etc.). The choice between requests and python-gitlab per call site is a code-review decision, not a hard rule. - glab is used by the human or AI agent driving the bootstrap interactively (e.g. an AI agent verifying a step succeeded with glab api projects/<id> rather than curl, or a developer checking pipeline status with glab ci view). - glab may be invoked from the bootstrap only when the alternative would be re-implementing logic that's already in the CLI (rare — example: glab auth login to seed the local config from an OpenBao-stored PAT, since python-gitlab has no equivalent one-liner). Each such use gets a comment explaining why requests/python-gitlab was not enough.

This is the inverse of the previous draft's rule. The previous draft said "the bootstrap drives glab as a subprocess"; this draft says "the bootstrap drives the HTTP API directly, and the agent may use glab to verify / inspect / debug".

Rule 2 — Source of truth lives in `apps-local/` not `apps/`¶

apps/ is the canonical GitLab-side source — it's the content that will end up in blueprint-apps/<name> after the first blueprint-phase3 run. After that run, the truth moves to GitLab: future edits happen in the GitLab repo (or via PR from a developer's fork), not in apps/.

But the bootstrap needs a mutable, idempotent working copy to rsync / git push from, and that working copy must be re-creatable from a clean checkout (so a fresh git clone k8s-cicd can still run blueprint-phase3 without first running through GitLab).

The solution is a second directory:

blueprint/
├── apps/                              ← CANONICAL — frozen after first
│   │                                    push to GitLab. Re-cloned
│   │                                    into apps-local/ on bootstrap
│   │                                    runs. NOT a working tree.
│   ├── guestbook/
│   ├── redis/
│   ├── redis-slave/
│   └── shared-code/                   ← NEW (Rule 3): the cross-app
│                                        shared pipeline code. Frozen
│                                        alongside the rest of apps/.
├── apps-local/                        ← WORKING TREE — created by
│   │                                    blueprint-phase3, mutated by
│   │                                    the bootstrap, pushed to GitLab.
│   │                                    Not committed to the bootstrap
│   │                                    repo (gitignored).
│   ├── blueprint-apps/                ← top-level group working copy
│   │   ├── guestbook/                 ← `glab repo clone` target +
│   │   │                                 rsync from apps/guestbook/
│   │   ├── redis/
│   │   ├── redis-slave/
│   │   └── shared-code/               ← cloned once, referenced via
│   │                                     `include:` from the other
│   │                                     projects' .gitlab-ci.yml

Lifecycle:

First run (uv run blueprint-phase3 on a fresh checkout):
The bootstrap creates apps-local/blueprint-apps/ and git clones each repo from apps/ into apps-local/blueprint-apps/<name>/.
It then adds the generated CI files (.gitlab-ci.yml + ci/), commits, and git pushes to GitLab.
apps-local/ is added to .gitignore and never committed.
Subsequent runs (uv run blueprint-phase3 again):
The bootstrap detects apps-local/blueprint-apps/ already exists. It does git pull on each sub-repo to pick up any changes the user (or CI) made on the GitLab side (e.g. a hand-edited CI variable, a new branch).
It then rsyncs the delta between apps/<name>/ and apps-local/blueprint-apps/<name>/, produces a new commit if there are changes, and pushes.
uv run blueprint-phase3 --reset-clones (escape hatch):
The bootstrap deletes apps-local/ and starts from a fresh git clone of each GitLab repo. Use this when apps-local/ is corrupted or has diverged irreparably from upstream.

Why this is better than apps/ as a working tree:

apps/ is the distribution copy — what a fresh user clones. It's frozen, deterministic, and matches what the bootstrap will push the first time.
apps-local/ is the bootstrap's runtime working tree — it carries the rsync delta, the generated CI files, the upstream HEAD ref, and the next-commit message. None of that is canonical.
A user reviewing the repo on GitHub can ignore apps-local/ entirely; a user running the bootstrap locally never has to look at apps/ after the first run.

Rule 3 — Cross-app shared pipeline code lives in `apps/shared-code/`¶

apps/shared-code/ is a fourth GitLab project under the blueprint-apps group. Its repo contains the CI code shared by all three apps — the kaniko+helm templates, helper scripts, shared test runners, etc. The other three projects reference it from their .gitlab-ci.yml via GitLab's cross-project include::

# per-project .gitlab-ci.yml
include:
  - project: blueprint-apps/shared-code
    ref: main
    file: /templates/phase3-app-pipeline.yml
  - project: blueprint-apps/shared-code
    ref: main
    file: /templates/phase3-redis-pipeline.yml

The shared-code project is also created and pushed by the bootstrap — it gets the same treatment as guestbook/redis/ redis-slave (project create, source push, CI variables). The bootstrap's apps_manifest.yaml lists it alongside the three app projects, with build_image: false (no Dockerfile) and has_unit_tests: false.

What the bootstrap owns vs. what GitLab owns¶

Concern	Lives in	Pushed by
Cluster, GitLab, Runner install	bootstrap repo	manual + `blueprint-bootstrap`
Group + project provisioning	bootstrap repo	`blueprint-phase3`
kubeconfig discovery	bootstrap repo	`blueprint-phase3`
CI/CD variable injection	bootstrap repo	`blueprint-phase3`
Python GitLab HTTP client code	bootstrap repo	`blueprint-phase3`
`apps/<name>/` source (canonical)	bootstrap repo	committed to the repo
`apps/shared-code/` source	bootstrap repo	committed to the repo
`apps-local/` working tree	bootstrap host	gitignored, runtime only
`.gitlab-ci.yml`	per-project repo	bootstrap writes into apps-local/ and pushes
CI template (`ci/phase3-app-pipeline.yml`)	`apps/shared-code/` repo	bootstrap pushes shared-code project
App source code in GitLab	per-project repo	bootstrap rsyncs from `apps/<name>/` and pushes
Helm chart	per-project repo	already in `apps/<name>/helm-chart/`
CI scripts (any helper code)	`shared-code/` repo	already in `apps/shared-code/`
Test code (pytest, etc.)	per-project repo	already in `apps/<name>/`

Concretely: the bootstrap takes the canonical apps/<name>/ content, drops a clone in apps-local/blueprint-apps/<name>/, generates the per-project .gitlab-ci.yml there, commits, pushes. The shared-code repo provides cross-project templates that the per-project .gitlab-ci.yml references via GitLab include:.

1. Goals and non-goals¶

In scope¶

One GitLab top-level group blueprint-apps containing four projects:
blueprint-apps/guestbook ← apps/guestbook/
blueprint-apps/redis ← apps/redis/
blueprint-apps/redis-slave ← apps/redis-slave/
blueprint-apps/shared-code ← apps/shared-code/ (cross-app shared CI templates + helper scripts)
Each project receives (pushed as a single commit on main):
The full contents of its apps/<name>/ subdir.
A rendered .gitlab-ci.yml at the repo root, produced by reading one of the per-project template files in apps/shared-code/templates/ and substituting $name / $build / $tests / $release / $namespace via bootstrap/phase3/ci_render.py (Python string.Template). The template files are real YAML in the canonical source — NOT Python strings — so they can be hand-edited and git diff'd normally. For the three app projects, the rendered file include:s the shared pipeline templates from the shared-code project via GitLab cross-project include: — not local: includes. The shared-code project's own .gitlab-ci.yml is minimal (it's a library, not a deployable).
Each project gets:
The GitLab-Runner Kubernetes executor already registered by Phase 2 (no new runner install).
CI/CD variables (CI_KUBECONFIG, CI_REGISTRY_IMAGE, etc.) injected by the bootstrap at provision time.
A working CI pipeline per app project with three stages:
build — kaniko-build the application image, push to the in-cluster registry (registry.local.example.net/blueprint-apps/<name>:<sha>).
test — run the project's test suite (pytest for the Go guestbook tests? verify; redis demo's redis-benchmark).
deploy — helm upgrade --install the project's helm-chart/ against the in-cluster kubeconfig, pointed at the freshly-built image tag.
A new CLI entry point blueprint-phase3 (sibling of blueprint-bootstrap / blueprint-secrets), wired into the existing pyproject.toml [project.scripts] table. Lives in its own phase3/ package under infra/scripts/bootstrap/. This is not a new subcommand on blueprint-bootstrap — Phase 3 is its own concern and the CLI split keeps the original bootstrap focused on cluster bootstrap.
A reusable CI template generated by Python and committed to the shared-code project (not the per-app projects). Templates live at apps/shared-code/templates/ and are referenced by per-project .gitlab-ci.yml via GitLab cross-project include:. The bootstrap does not ship a pre-rendered copy of the templates in its own repo.

Out of scope (explicit)¶

Mirror/push from outside GitLab to the in-cluster registry is not added — only CI-built images get pushed. The local apps/<name>/ source is uploaded to GitLab as the repo content, not as a separate docker push.
No GitLab Runner install — Phase 2 already installed and registered the runner against gitlab-webservice-default.gitlab.svc:8181.
No ArgoCD / GitOps — deployment is the pipeline's job.
No enterprise edition features — same constraint as Phase 2.
glab is NOT the bootstrap's GitLab API. The bootstrap is a Python app and talks to GitLab via python-gitlab (typed resource models — preferred for project create, variable CRUD, runner registration) and requests (raw HTTP — preferred for one-off calls and ad-hoc endpoints). glab is the human / AI agent's GitLab CLI; the bootstrap uses it only in narrow exceptions where no Python equivalent exists (e.g. glab auth login).
No CI template files committed in the bootstrap repo. All pipeline code lives in the GitLab projects — the shared-code project for cross-app templates, each app project for its own .gitlab-ci.yml.

2. Source layout — bootstrap side¶

blueprint/
├── apps/                              ← CANONICAL GitLab-side source (frozen
│   │                                    after first push). Committed to the
│   │                                    bootstrap repo; re-distributed on each
│   │                                    `git clone`.
│   ├── guestbook/                     ← guestbook-go/ + helm-chart/
│   ├── redis/                         ← redis YAMLs + helm-chart/
│   ├── redis-slave/                   ← redis-replica YAMLs + helm-chart/
│   └── shared-code/                   ← NEW (Rule 3). Cross-app CI templates
│                                        + helper scripts. Treated as an
│                                        application repo (gets a project in
│                                        GitLab), but has no deployable output
│                                        — it exists to be `include:`d by the
│                                        other three projects.
│       ├── templates/                 ← YAML templates; real files,
│       │   │                            NOT Python strings (Rule 0).
│       │   │                            Pushed to the shared-code
│       │   │                            project by the bootstrap as
│       │   │                            ordinary source.
│       │   ├── phase3-app-pipeline.yml          ← shared kaniko+test+deploy
│       │   │                                      (included by guestbook)
│       │   ├── phase3-redis-pipeline.yml        ← shared test+deploy
│       │   │                                      (included by redis / redis-slave)
│       │   ├── per-project-app.gitlab-ci.yml.tpl       ← template for
│       │   │                                              guestbook-style
│       │   │                                              .gitlab-ci.yml
│       │   │                                              (string.Template
│       │   │                                              variables)
│       │   ├── per-project-redis.gitlab-ci.yml.tpl     ← template for
│       │   │                                              redis-style
│       │   │                                              .gitlab-ci.yml
│       │   └── per-project-shared-code.gitlab-ci.yml   ← literal file
│       │                                                  (no substitution;
│       │                                                  for the
│       │                                                  shared-code
│       │                                                  project itself)
│       ├── scripts/                   ← helper shell/python scripts shared
│       │   │                            by all pipelines
│       │   └── kaniko-build.sh        (example)
│       └── README.md
│
├── apps-local/                        ← WORKING TREE (gitignored, runtime only)
│   │                                    Created by blueprint-phase3 on first
│   │                                    run. Contains a `git clone` of each
│   │                                    GitLab project under the
│   │                                    `blueprint-apps` group. The bootstrap
│   │                                    rsyncs from apps/<name>/ into
│   │                                    apps-local/blueprint-apps/<name>/,
│   │                                    generates the per-project CI files,
│   │                                    commits, and pushes.
│   └── blueprint-apps/
│       ├── guestbook/                 ← `git clone https://…/guestbook`
│       ├── redis/
│       ├── redis-slave/
│       └── shared-code/
│
├── infra/scripts/bootstrap/
│   ├── phase2/                        ← unchanged (Phase 2 install)
│   └── phase3/                        ← NEW (Phase 3 — GitLab provisioning only)
│       ├── __init__.py
│       ├── pipeline.py                ← Phase3Pipeline orchestrator (provisioning steps)
│       ├── catalog.py                 ← Phase3Installers dataclass
│       ├── gitlab_client.py           ← Python GitLab client wrapper.
│       │                                 Uses `python-gitlab` for typed
│       │                                 resource CRUD (projects, groups,
│       │                                 variables, runners) and `requests`
│       │                                 directly for ad-hoc endpoints.
│       │                                 NOT a `glab` wrapper — see Rule 1.
│       ├── gitlab_api.py              ← high-level helpers
│       │                                 (create_group, create_project,
│       │                                 push_repo, set_ci_variable) — all
│       │                                 delegate to gitlab_client.py.
│       ├── apps_manifest.yaml         ← METADATA only: name, src_path,
│       │                                 build_image?, chart_path,
│       │                                 has_unit_tests, helm_release,
│       │                                 helm_namespace, registry_image. NO
│       │                                 template content.
│       ├── apps_local.py              ← AppsLocalFS — manages the
│       │                                 apps-local/ working tree (clone,
│       │                                 pull, rsync, commit, push).
│       ├── ci_render.py               ← reads a per-project template file
│       │                                 from apps/shared-code/templates/
│       │                                 and substitutes AppSpec variables
│       │                                 via string.Template. NO YAML
│       │                                 literals live in this module —
│       │                                 the templates are real files in
│       │                                 the canonical source (Rule 0).
│       ├── kubeconfig.py              ← in-cluster kubeconfig builder
│       │                                 (the Service-account kubeconfig
│       │                                 the CI pods will consume —
│       │                                 written into a per-project CI
│       │                                 variable, NOT into a file in
│       │                                 this repo).
│       ├── registry_auth.py           ← in-cluster registry creds reader
│       └── smoke_test.py              ← end-of-install verifier
│
├── infra/scripts/
│   └── phase3.py                      ← thin shim → delegates to bootstrap/phase3/
│
├── .gitignore                         ← add `apps-local/`
│
├── pyproject.toml                     ← add python-gitlab (or requests) dep,
│                                          plus:
│                                            [project.scripts]
│                                              "blueprint-bootstrap" = "bootstrap.cli:main"
│                                              "blueprint-secrets"   = "bootstrap.secrets_cli:main"
│                                              "blueprint-phase3"    = "bootstrap.phase3_cli:main"

3. Source layout — what lands in GitLab¶

After blueprint-phase3 runs, the four projects look like this:

blueprint-apps/
├── guestbook/                  ← GitLab repo
│   ├── <existing apps/guestbook content>
│   ├── guestbook-go/
│   ├── helm-chart/
│   └── .gitlab-ci.yml           ← generated by ci_render.py
│                                   includes templates from
│                                   shared-code via cross-project include:
│
├── redis/                      ← GitLab repo
│   ├── <existing apps/redis content>
│   ├── helm-chart/
│   └── .gitlab-ci.yml           ← generated; includes phase3-redis-pipeline.yml
│                                   from shared-code
│
├── redis-slave/                ← GitLab repo
│   ├── <existing apps/redis-slave content>
│   ├── helm-chart/
│   └── .gitlab-ci.yml           ← generated; includes phase3-redis-pipeline.yml
│                                   from shared-code
│
└── shared-code/                ← GitLab repo
    ├── templates/
    │   ├── phase3-app-pipeline.yml       ← canonical, hand-editable in GitLab
    │   └── phase3-redis-pipeline.yml     ← canonical, hand-editable in GitLab
    ├── scripts/                ← helper scripts (committed here, not in
    │   │                          bootstrap)
    │   └── kaniko-build.sh     (example)
    └── .gitlab-ci.yml           ← minimal (this project has no build/test/deploy
                                    of its own; it just exposes templates to the
                                    other projects via include:)

The cross-project include: looks like this in the per-project .gitlab-ci.yml:

# blueprint-apps/guestbook/.gitlab-ci.yml
include:
  - project: blueprint-apps/shared-code
    ref: main
    file: /templates/phase3-app-pipeline.yml

The ci/phase3-*.yml files in the shared-code project are canonical and hand-editable in GitLab — they're treated as ordinary application source. If the user later wants to evolve a template, they edit it in the shared-code repo on the GitLab side, commit, push; the other projects pick it up on their next pipeline run via the cross-project include: ref: main.

If the user wants to fork the templates for an experiment, they create a branch in the shared-code repo (ref: my-branch in the per-project .gitlab-ci.yml); no bootstrap changes required.

4. The two project-wide rules, applied¶

Rule 1 — Bootstrap uses Python; `glab` is for the agent¶

bootstrap/phase3/gitlab_client.py is a thin wrapper over python-gitlab + requests. It exposes the operations Phase 3 needs as typed Python methods. Every call is structured (typed exception on 4xx/5xx, retry with backoff on transient failures, JSON response model). Unit tests use unittest.mock or responses to fake HTTP — no subprocess mocks required.

Mapping from operation to library:

Operation	Library	Notes
List groups	`python-gitlab`	`gl.groups.list()`
Create top-level group	`python-gitlab`	`gl.groups.create({...})`
Create project in group	`python-gitlab`	`gl.projects.create({...})`
Update project settings	`python-gitlab`	`project.save()` after mutation
Add CI/CD variable	`python-gitlab`	`project.variables.create({...})`
Add CI/CD variable (file content)	`python-gitlab`	`project.variables.create({"value": b64})`
Read CI/CD variable	`python-gitlab`	`project.variables.get(key)`
Delete CI/CD variable	`python-gitlab`	`project.variables.delete(key)`
List runners	`python-gitlab`	`gl.runners.list()`
List pipeline status	`python-gitlab`	`project.pipelines.list(...)`
Cross-project include sanity check	`requests`	`GET /projects/:id/repository/files/...` — `python-gitlab` doesn't expose the file-blob endpoint as cleanly
Auth bootstrap (`glab auth login`)	`glab` (narrow exception)	python-gitlab doesn't have a one-shot "store this PAT locally and use it for every subsequent call" CLI equivalent. We invoke `glab auth login --token <pat>` once at bootstrap init so the local `~/.config/glab-cli/config.yml` is populated — useful when the user later runs `glab` interactively to debug.

The GitlabClient is parameterised by: - gitlab_host — default https://gitlab.local.example.net:8443 (the kind NodePort-forward target; see AGENTS.md § User-facing access). - auth_token — read from OpenBao at secret/gitlab/bootstrap/admin_token. Phase 3 mints a PAT for root with api, write_repository, write_registry, admin_mode scopes. - kubeconfig — uses the same kubeconfig Phase 2 writes (infra/tofu/kubeconfig).

glab usage from an AI agent or developer is encouraged where it's the right tool — e.g. an agent verifying a step with glab api projects/<id>, or a developer watching pipeline progress with glab ci view -w. The plan does not add glab as a bootstrap dependency; it assumes glab is already on $PATH on the developer's host (the agent / dev toolchain), which is a normal Linux setup.

Rule 2 — New entry point `blueprint-phase3`, not a new subcommand¶

Adding Phase 3 as a subcommand of blueprint-bootstrap would mean every new feature pulls a 13-step Phase-2 installer list along with it. Phase 3 is its own concern.

Changes:

Add a new click-based CLI: infra/scripts/bootstrap/phase3_cli.py → from .app_phase3 import Phase3App; def main(): …. Same two-tier help pattern (ShortOption + --help-full) used by blueprint-bootstrap.

Wire it into pyproject.toml:

[project.scripts]
"blueprint-bootstrap" = "bootstrap.cli:main"
"blueprint-secrets"   = "bootstrap.secrets_cli:main"
"blueprint-phase3"    = "bootstrap.phase3_cli:main"

Add infra/scripts/phase3.py as a thin shim (matches infra/scripts/bootstrap.py) so the old python3 infra/scripts/phase3.py … path keeps working.
CLI flags:
blueprint-phase3 (no args) — short help.
--help-full — long help.
--check — pre-flight (kubeconfig reachable, GitLab up, glab on PATH with auth set, OpenBao reachable).
--destroy — DELETE the four projects + the blueprint-apps top-level group. Cleans up CI-built images from the registry. Mirrors Phase 2's --destroy.
--project <name> — limit to one project.
--no-overwrite-ci — if the receiving repo already has a .gitlab-ci.yml, skip the push for that file (default: overwrite).
--reset-clones — delete apps-local/ and re-clone each GitLab project from scratch. Use when the local working tree has diverged irreparably.

5. Phase 3 step-by-step pipeline (bootstrap side)¶

Phase3Pipeline runs 7 ordered steps. Every step is idempotent; re-runs are safe.

Pre-flight — GitlabClient + OpenBaoClient reachability check; verify the GitLab Runner Deployment is Ready; verify the in-cluster registry Service has a TCP listener. Fail-fast if any of these are down.
Mint a GitLab admin PAT — exec into the gitlab-toolbox pod and run gitlab-rails runner to mint a personal access token for root with api, write_repository, write_registry, admin_mode scopes. Store it in OpenBao at secret/gitlab/bootstrap/admin_token. Use the token directly for python-gitlab / requests calls. (The optional glab auth login --token <…> call is a no-op for the bootstrap, but useful for the developer / agent — see Rule 1 in § 0.)

Create the top-level group blueprint-apps — Python:

gl.groups.create({"path": "blueprint-apps",
                  "name": "Blueprint Apps",
                  "visibility": "internal"})

Ensure apps-local/blueprint-apps/ working tree — delegate to apps_local.py:AppsLocalFS:
If --reset-clones or apps-local/ does not exist: mkdir apps-local/blueprint-apps/, then for each entry in apps_manifest.yaml: git clone <gitlab-url> apps-local/blueprint-apps/<name>/.
Otherwise: for each entry, git -C apps-local/blueprint-apps/<name>/ pull --ff-only origin main.
For each entry in apps_manifest.yaml, in order:

Create the project if it does not exist:

gl.projects.create({"path": spec.name,
                    "namespace_id": group_id,
                    "visibility": "internal",
                    "default_branch": "main",
                    "builds_enabled": True})

(Idempotent: catch the gitlab.exceptions.GitlabCreateError with response_code=400 and message

"has already been
      taken"

; treat as success.)

Rsync the canonical source into the working tree:
```
apps_local.sync_from_canonical(
    src=Path("apps") / spec.name,
    dst=Path("apps-local") / "blueprint-apps" / spec.name,
)
```
sync_from_canonical does an rsync -a --delete --exclude=.git from apps/<name>/ into the working tree, so apps-local/blueprint-apps/<name>/ reflects the canonical content.
Render the per-project .gitlab-ci.yml into the working tree:
```
rendered = ci_render.render_for(spec)
gitlab_ci_dst = Path("apps-local/blueprint-apps") / spec.name / ".gitlab-ci.yml"
gitlab_ci_dst.write_text(rendered)
```
ci_render.render_for(spec) reads one of the per-project template files in apps/shared-code/templates/ and runs Python string.Template.safe_substitute over it — no YAML literals live in ci_render.py itself. The output is a single file: apps-local/blueprint-apps/<name>/.gitlab-ci.yml. The shared pipeline templates live in the shared-code repo's templates/ directory; the per-project file include:s them via cross-project include (see § 6).
Build the kubeconfig secret in-memory by kubeconfig.py (using the in-cluster Service account JWT — NOT a file in the bootstrap repo). The result is a base64-encoded YAML string stored as a CI/CD variable via python-gitlab:
```
project.variables.create({
    "key": "CI_KUBECONFIG_B64",
    "value": base64.b64encode(kubeconfig_yaml).decode(),
    "variable_type": "env",
    "masked": False,
    "protected": False,
})
```
5. Commit + push (if anything changed):
```
git -C apps-local/blueprint-apps/<name>/ add -A
git -C apps-local/blueprint-apps/<name>/ commit -m "Phase 3: sync from apps/<name>"
git -C apps-local/blueprint-apps/<name>/ push origin main
```
The commit-and-push is done by subprocess.run (it's a git operation, not a GitLab API call — see Rule 1). 6. Set the rest of the CI/CD variables via python-gitlab: - CI_REGISTRY_IMAGE — registry.local.example.net/blueprint-apps/<name>. - CI_HELM_CHART_DIR — relative path inside the repo (helm-chart). - CI_INSECURE_REGISTRY — "true".
Smoke-test the pipeline — via python-gitlab:
```
pipeline = project.pipelines.create({"ref": "main"})
```
Then poll project.pipelines.get(pipeline.id) until status is one of success/failed/canceled. Verify:
The pipeline's build job produced an image (via python-gitlab's project.container_registry.repositories.list()).
The pipeline's test job passed.
The pipeline's deploy job finished with helm list -n <ns> showing the release as deployed (the bootstrap runs helm list via subprocess.run).

Repeat for redis, redis-slave, and shared-code (the latter has no build/test/deploy — just verify the pipeline triggers and the lint job passes).

Print post-install handoff — list the project URLs and the user-side commands to view pipelines (which the developer / agent can run with glab ci view).

6. The CI files (committed in GitLab)¶

The shared CI templates live in apps/shared-code/templates/ in the canonical source, get pushed to the shared-code project, and are referenced by the per-app projects via GitLab's cross-project include:.

The per-project .gitlab-ci.yml is generated by reading one of the per-project template files in apps/shared-code/templates/ and substituting $name / $build / $tests / $release / $namespace via Python string.Template (in bootstrap/phase3/ci_render.py). The template files are real YAML in the canonical source — NOT Python strings — so they can be hand-edited and git diff'd normally.

`apps/shared-code/templates/phase3-app-pipeline.yml` (used by `guestbook`)¶

Hand-edited in GitLab; lives in apps/shared-code/templates/ in the canonical source; pushed to the shared-code project by the bootstrap. Changes here propagate to all consuming projects on their next pipeline run.

# blueprint-apps/shared-code/templates/phase3-app-pipeline.yml
# Lives in the shared-code project. Edit there to evolve the
# pipeline; the guestbook project picks up changes via the
# cross-project include: ref: main.
stages: [build, test, deploy]

# ---- build (apps with a Dockerfile) ----
build:image:
  stage: build
  image:
    name: gcr.io/kaniko-project/executor:v1.23.2-debug
    entrypoint: [""]
  script:
    - mkdir -p /kaniko/.docker
    - |
      echo "{\"auths\":{\"${CI_REGISTRY}:${CI_REGISTRY_PORT}\":
        {\"auth\":\"$(printf '%s:%s' "${CI_REGISTRY_USER}"
         "${CI_REGISTRY_PASSWORD}" | base64 -w0)\"}}}" \
        > /kaniko/.docker/config.json
    - >-
      /kaniko/executor
      --context "${CI_PROJECT_DIR}"
      --dockerfile "${CI_PROJECT_DIR}/Dockerfile"
      --destination "${CI_REGISTRY_IMAGE}:${CI_COMMIT_SHORT_SHA}"
      --cache=true
      --insecure-registry="${CI_REGISTRY}"
  rules:
    - if: $CI_APP_BUILD == "true"

# ---- test ----
test:unit:
  stage: test
  image: python:3.13-slim
  before_script:
    - pip install --no-cache-dir -r requirements.txt
  script:
    - pytest -q
  rules:
    - if: $CI_APP_BUILD == "true" && $CI_HAS_UNIT_TESTS == "true"

# ---- deploy (helm) ----
deploy:helm:
  stage: deploy
  image:
    name: alpine/helm:3.16.3
    entrypoint: [""]
  before_script:
    - mkdir -p "${CI_PROJECT_DIR}/.kube"
    - echo "${CI_KUBECONFIG_B64}" | base64 -d > "${CI_PROJECT_DIR}/.kube/config"
    - export KUBECONFIG="${CI_PROJECT_DIR}/.kube/config"
    - helm dependency update "${CI_PROJECT_DIR}/${CI_HELM_CHART_DIR}"
  script:
    - >-
      helm upgrade --install "${CI_HELM_RELEASE}"
      "${CI_PROJECT_DIR}/${CI_HELM_CHART_DIR}"
      --namespace "${CI_HELM_NAMESPACE}" --create-namespace
      --set image.repository="${CI_REGISTRY_IMAGE}"
      --set image.tag="${CI_COMMIT_SHORT_SHA}"
      --wait
  rules:
    - if: $CI_COMMIT_BRANCH == "main"

`ci/phase3-redis-pipeline.yml` (used by `redis`, `redis-slave`)¶

# Auto-generated by blueprint-phase3. Do not hand-edit; re-run
# `blueprint-phase3 --project <name>` to regenerate.
stages: [test, deploy]

# ---- test (redis is upstream image; no build) ----
test:redis-smoke:
  stage: test
  image: redis:7.4.1
  script:
    - redis-cli -h "${CI_REDIS_HOST}" PING
  rules:
    - if: $CI_APP_BUILD != "true"

# ---- deploy (helm) ----
deploy:helm:
  stage: deploy
  image:
    name: alpine/helm:3.16.3
    entrypoint: [""]
  before_script:
    - mkdir -p "${CI_PROJECT_DIR}/.kube"
    - echo "${CI_KUBECONFIG_B64}" | base64 -d > "${CI_PROJECT_DIR}/.kube/config"
    - export KUBECONFIG="${CI_PROJECT_DIR}/.kube/config"
    - helm dependency update "${CI_PROJECT_DIR}/${CI_HELM_CHART_DIR}"
  script:
    - >-
      helm upgrade --install "${CI_HELM_RELEASE}"
      "${CI_PROJECT_DIR}/${CI_HELM_CHART_DIR}"
      --namespace "${CI_HELM_NAMESPACE}" --create-namespace
      --set image.tag="${CI_REDIS_TAG:-7.4.1}"
      --wait
  rules:
    - if: $CI_COMMIT_BRANCH == "main"

Per-project `.gitlab-ci.yml` (rendered)¶

This is a tiny rendered file — rendered, not generated. It comes from one of the per-project template files in apps/shared-code/templates/ (e.g. per-project-app.gitlab-ci.yml.tpl) with $name / $build / $tests / $release / $namespace substituted in by bootstrap/phase3/ci_render.py via Python string.Template.

The rendered file include:s the shared template from the shared-code project via cross-project include — not local: (the template does not live in the per-project repo).

The literal source template file (apps/shared-code/templates/per-project-app.gitlab-ci.yml.tpl) looks like this:

# blueprint-apps/shared-code/templates/per-project-app.gitlab-ci.yml.tpl
# Rendered by blueprint-phase3 ci_render.py into each app project's
# .gitlab-ci.yml. Hand-edit this template to change the per-project
# shape; re-run blueprint-phase3 to re-render every project.

include:
  - project: blueprint-apps/shared-code
    ref: main
    file: /templates/phase3-app-pipeline.yml

variables:
  CI_APP_NAME: "$name"
  CI_APP_BUILD: "$build"
  CI_HAS_UNIT_TESTS: "$tests"
  CI_HELM_RELEASE: "$release"
  CI_HELM_NAMESPACE: "$namespace"

After rendering with spec = AppSpec(name="guestbook", ...), the bootstrap writes this to apps-local/blueprint-apps/guestbook/.gitlab-ci.yml:

# Auto-rendered by blueprint-phase3. Source template:
# apps/shared-code/templates/per-project-app.gitlab-ci.yml.tpl
include:
  - project: blueprint-apps/shared-code
    ref: main
    file: /templates/phase3-app-pipeline.yml

variables:
  CI_APP_NAME: "guestbook"
  CI_APP_BUILD: "true"
  CI_HAS_UNIT_TESTS: "false"
  CI_HELM_RELEASE: "guestbook"
  CI_HELM_NAMESPACE: "guestbook"

The shared-code project's own .gitlab-ci.yml is copied verbatim from apps/shared-code/templates/per-project-shared-code.gitlab-ci.yml (no substitution — it's a literal file in the canonical source). It only runs the lint job that parses the templates with yaml.safe_load:

# blueprint-apps/shared-code/templates/per-project-shared-code.gitlab-ci.yml
# Copied verbatim into the shared-code project's .gitlab-ci.yml
# by blueprint-phase3 (no template substitution).
stages: [lint]

lint:templates:
  stage: lint
  image: python:3.13-slim
  before_script:
    - pip install --no-cache-dir pyyaml
  script:
    - |
      python -c "import yaml; \
        [yaml.safe_load(open(f)) for f in \
         ['templates/phase3-app-pipeline.yml', \
          'templates/phase3-redis-pipeline.yml']]; \
        print('OK')"

7. Bootstrap integration (entry-point wiring)¶

# infra/scripts/bootstrap/phase3_cli.py
import click
from .app_phase3 import Phase3App
from .help_full import ShortOption, help_full_option

@click.command(
    context_settings={"help_option_names": ["-h", "--help"]},
    no_args_is_help=True,
    help=(
        "Phase 3: create GitLab projects for blueprint/apps/, wire "
        ".gitlab-ci.yml, and run the build/test/deploy pipeline against "
        "the Phase-2 cluster. The bootstrap is a Python app and uses "
        "python-gitlab + requests to talk to GitLab — `glab` is the "
        "agent/developer CLI, not used by the bootstrap itself. "
        "Pipeline code (CI templates, kubeconfig) lives IN the "
        "created GitLab projects, not in the bootstrap repo."
    ),
    short_help="Phase 3: create apps/ GitLab projects + run CI pipelines.",
)
@help_full_option()
@click.option("--check", is_flag=True, cls=ShortOption,
              short_help="Pre-flight only (don't mutate GitLab).",
              help="Pre-flight only — verify GitLab reachability, "
                   "Runner Deployment Ready, in-cluster registry "
                   "TCP-listening, glab on PATH. No mutations.")
@click.option("--destroy", is_flag=True, cls=ShortOption,
              short_help="Delete the blueprint-apps group and all 4 projects.",
              help="Delete the blueprint-apps group and all 4 projects. "
                   "Cleans up CI-built images from the in-cluster registry.")
@click.option("--project", default=None, cls=ShortOption,
              short_help="Limit to one project.",
              help="Limit the run to one project (guestbook/redis/"
                   "redis-slave/shared-code). Defaults to all four.")
@click.option("--no-overwrite-ci", is_flag=True, cls=ShortOption,
              short_help="Skip .gitlab-ci.yml push if file already exists.",
              help="Skip the .gitlab-ci.yml push if the file already exists "
                   "in the receiving repo. Default: overwrite.")
@click.option("--reset-clones", is_flag=True, cls=ShortOption,
              short_help="Delete apps-local/ and re-clone from GitLab.",
              help="Delete apps-local/ entirely and re-clone each project "
                   "from GitLab. Use when the local working tree has "
                   "diverged irreparably.")
def main(check, destroy, project, no_overwrite_ci, reset_clones):
    Phase3App.from_argv(
        check=check, destroy=destroy,
        project_filter=project, no_overwrite_ci=no_overwrite_ci,
        reset_clones=reset_clones,
    ).run()

# infra/scripts/bootstrap/app_phase3.py  (composition root)
class Phase3App:
    def __init__(self, gitlab: GitlabClient, openbao: OpenBaoClient,
                 apps_local: AppsLocalFS, log, paths):
        self._gitlab = gitlab
        self._openbao = openbao
        self._apps_local = apps_local
        self._pipeline = Phase3Pipeline(gitlab, openbao, apps_local, log, paths)

    @classmethod
    def from_argv(cls, *, check, destroy, project_filter,
                  no_overwrite_ci, reset_clones):
        paths = Paths()
        log = ConsoleLogger()
        runner = SubprocessRunner()
        openbao = OpenBaoClient(runner, paths, log)         # shared with Phase 2
        gitlab = GitlabClient(paths=paths, openbao=openbao)  # python-gitlab + requests
        apps_local = AppsLocalFS(
            canonical_root=paths.repo_root / "apps",
            working_root=paths.repo_root / "apps-local",
        )
        return cls(gitlab, openbao, apps_local, log, paths)

    def run(self):
        if self._check:
            return self._pipeline.preflight()
        if self._destroy:
            return self._pipeline.destroy()
        return self._pipeline.run(
            project_filter=self._project_filter,
            no_overwrite_ci=self._no_overwrite_ci,
            reset_clones=self._reset_clones,
        )

8. The render function (`bootstrap/phase3/ci_render.py`)¶

The per-project .gitlab-ci.yml is a template, not a Python string. Per Rule 0, templates must live as their own files in the canonical source — so the per-project template files live in apps/shared-code/templates/ alongside the shared pipeline templates.

File layout (canonical source)¶

apps/shared-code/
├── templates/
│   ├── phase3-app-pipeline.yml                ← shared kaniko+test+deploy pipeline (used by guestbook)
│   ├── phase3-redis-pipeline.yml              ← shared test+deploy pipeline (used by redis / redis-slave)
│   ├── per-project-app.gitlab-ci.yml.tpl      ← per-project .gitlab-ci.yml TEMPLATE
│   │                                            (uses $name, $build, $tests, $release, $namespace)
│   ├── per-project-redis.gitlab-ci.yml.tpl    ← per-project .gitlab-ci.yml TEMPLATE for redis
│   ├── per-project-shared-code.gitlab-ci.yml  ← literal file (no substitution needed) for the
│   │                                            shared-code project itself
│   └── variables.yaml                         ← KEY=value map of substitution variables per project
│                                                (read by ci_render.py to populate the templates)
├── scripts/
│   └── (helper scripts committed here)
└── README.md

The *.tpl suffix is a convention (not a GitLab-recognized suffix). Each template uses Python string.Template syntax ($variable or ${variable}); the substitution values come from apps_manifest.yaml via ci_render.py.

The three per-project `.gitlab-ci.yml` templates¶

`apps/shared-code/templates/per-project-app.gitlab-ci.yml.tpl`¶

Used for guestbook (and any other future project that has a Dockerfile to build). The only substitutions are the variables: block at the bottom — the include: is identical across all three app projects.

# blueprint-apps/shared-code/templates/per-project-app.gitlab-ci.yml.tpl
# Rendered by blueprint-phase3 ci_render.py into each app project's
# .gitlab-ci.yml. Hand-edit the per-app project on GitLab if you
# want to add project-local jobs; otherwise edit the shared
# template and re-render.

include:
  - project: blueprint-apps/shared-code
    ref: main
    file: /templates/phase3-app-pipeline.yml

variables:
  CI_APP_NAME: "$name"
  CI_APP_BUILD: "$build"
  CI_HAS_UNIT_TESTS: "$tests"
  CI_HELM_RELEASE: "$release"
  CI_HELM_NAMESPACE: "$namespace"

`apps/shared-code/templates/per-project-redis.gitlab-ci.yml.tpl`¶

Used for redis and redis-slave. Same shape as the app template, just pointing at the redis pipeline.

# blueprint-apps/shared-code/templates/per-project-redis.gitlab-ci.yml.tpl
include:
  - project: blueprint-apps/shared-code
    ref: main
    file: /templates/phase3-redis-pipeline.yml

variables:
  CI_APP_NAME: "$name"
  CI_APP_BUILD: "$build"
  CI_HAS_UNIT_TESTS: "$tests"
  CI_HELM_RELEASE: "$release"
  CI_HELM_NAMESPACE: "$namespace"

`apps/shared-code/templates/per-project-shared-code.gitlab-ci.yml`¶

Used for the shared-code project itself. No substitution — this is a literal file committed verbatim. It only runs the lint job that parses the templates with yaml.safe_load.

# blueprint-apps/shared-code/templates/per-project-shared-code.gitlab-ci.yml
stages: [lint]

lint:templates:
  stage: lint
  image: python:3.13-slim
  before_script:
    - pip install --no-cache-dir pyyaml
  script:
    - |
      python -c "import yaml; \
        [yaml.safe_load(open(f)) for f in \
         ['templates/phase3-app-pipeline.yml', \
          'templates/phase3-redis-pipeline.yml']]; \
        print('OK')"

The render function¶

bootstrap/phase3/ci_render.py is now a thin substitution helper — no YAML literals, no multiline strings. It picks the right template based on the AppSpec and runs string.Template.safe_substitute over it.

# infra/scripts/bootstrap/phase3/ci_render.py

from __future__ import annotations
from dataclasses import dataclass
from pathlib import Path
from string import Template

@dataclass(frozen=True)
class AppSpec:
    name: str             # "guestbook"
    src_path: str         # "apps/guestbook" (relative to repo root)
    chart_path: str       # "helm-chart" (relative to repo root)
    build_image: bool     # True if there's a Dockerfile to build
    has_unit_tests: bool  # True if there's a pytest suite
    helm_release: str     # "guestbook"
    helm_namespace: str   # "guestbook"
    registry_image: str   # "registry.local.example.net/blueprint-apps/guestbook"

# apps_manifest.yaml carries this metadata for each project,
# including the shared-code project (which has build_image=false,
# has_unit_tests=false — it's a library repo, not a deployable).

# Per spec rule "templates must have their own files" — the
# three per-project .gitlab-ci.yml templates live as real YAML
# files in apps/shared-code/templates/, NOT as Python strings
# inside this module. This module only:
#   1. picks the right template file based on the AppSpec
#   2. substitutes $name / $build / $tests / $release / $namespace
#      from the AppSpec into the template content
#   3. returns the rendered string

TEMPLATES_DIR = Path("apps") / "shared-code" / "templates"

# Map from AppSpec.name → template filename. The mapping lives
# in this Python module (a small dict) because it's structural
# logic, not template content. Adding a new app variant means
# adding one entry here AND one .yml.tpl file in TEMPLATES_DIR.
TEMPLATE_FOR = {
    "shared-code": "per-project-shared-code.gitlab-ci.yml",
}


def _template_for(spec: AppSpec) -> Path:
    """Returns the absolute path to the per-project .gitlab-ci.yml
    template for the given project. App projects pick by
    build_image; library projects pick by name."""
    if spec.name in TEMPLATE_FOR:
        return TEMPLATES_DIR / TEMPLATE_FOR[spec.name]

    if spec.build_image:
        return TEMPLATES_DIR / "per-project-app.gitlab-ci.yml.tpl"
    return TEMPLATES_DIR / "per-project-redis.gitlab-ci.yml.tpl"


def _vars(spec: AppSpec) -> dict[str, str]:
    """Substitution variables for $name / $build / $tests /
    $release / $namespace in the template files."""
    return {
        "name": spec.name,
        "build": str(spec.build_image).lower(),
        "tests": str(spec.has_unit_tests).lower(),
        "release": spec.helm_release,
        "namespace": spec.helm_namespace,
    }


def render_for(spec: AppSpec) -> str:
    """Read the per-project template file from disk and substitute
    the AppSpec variables into it. Returns the rendered YAML as
    a string — the caller writes it to apps-local/.../.gitlab-ci.yml.
    """
    template_path = _template_for(spec)
    template_text = template_path.read_text()
    return Template(template_text).safe_substitute(_vars(spec))

The output of render_for(spec) is a single string (the rendered .gitlab-ci.yml content). It is written by Phase3Pipeline into apps-local/blueprint-apps/<name>/.gitlab-ci.yml before the git add step. It never touches the bootstrap repo's working tree — the bootstrap is provisioning the GitLab project, not authoring files in its own tree.

Note: the per-project template files (the *.yml.tpl files in apps/shared-code/templates/) are committed to the bootstrap repo as ordinary YAML source. They are read at runtime by ci_render.py; the bootstrap never writes back to them. This is the practical application of Rule 0 — templates are real files, not Python strings.

9. Tests + verification¶

Per the spec rule "no bash, Python app", Phase 3 tests live under infra/scripts/bootstrap/phase3/tests/ and run with pytest:

test_ci_render.py — ci_render.render_for(spec) produces valid YAML; the cross-project include: resolves; variables.CI_APP_NAME matches the manifest entry. The test asserts the YAML is syntactically valid AND semantically correct by parsing it with yaml.safe_load() and asserting structure. Tests both the app and redis variants, plus the special-case shared-code literal file. The test also asserts that ci_render.py itself contains NO YAML literals — a regex search over the module source for dedent( returns no matches; this is a structural guard against re-introducing inline templates.
test_template_files.py — every file in apps/shared-code/templates/*.yml.tpl parses as valid YAML after a no-op substitution ($name etc. replaced with placeholder strings); every * reference in an include: exists as a sibling file in templates/. This guards against a hand-edit that breaks the cross-project include.
test_apps_manifest.py — apps_manifest.yaml parses; every entry's src_path exists; chart_path exists under src_path; build_image=True entries have a Dockerfile under src_path. The shared-code entry asserts the templates/ and scripts/ directories exist.
test_gitlab_client.py — GitlabClient.create_group / create_project / set_variable are exercised against a mock HTTP server (responses lib); no subprocess mocks required.
test_apps_local.py — AppsLocalFS.clone_all, pull_all, sync_from_canonical, commit_and_push are exercised against a temp directory with a fake remote (a bare git repo created via git init --bare).
test_pipeline.py — dry-run pipeline (no real GitLab or git calls; inject a FakeGitlabClient + a temp AppsLocalFS) — exercises step ordering and idempotency. The render_for output is asserted to match a fixture (so refactors that silently change pipeline shape get caught).

End-to-end smoke (manual, one-time, on the user's host): run uv run blueprint-phase3. After completion, verify with glab api projects/blueprint-apps%2Fguestbook/repository/tree (agent / developer side) that .gitlab-ci.yml is in the guestbook repo and that glab api projects/blueprint-apps%2Fshared-code/repository/tree shows templates/ and scripts/ and .gitlab-ci.yml.

10. Deliverables checklist¶

[ ] apps/shared-code/ directory with:
- templates/phase3-app-pipeline.yml
- templates/phase3-redis-pipeline.yml
- scripts/ (placeholder for helper scripts)
- README.md describing the project
[ ] .gitignore updated to exclude apps-local/.
[ ] bootstrap/phase3/ package (10 files; see § 2 tree).
[ ] bootstrap/phase3_cli.py (entry point) + infra/scripts/phase3.py (shim).
[ ] pyproject.toml script entry + python-gitlab and requests dependencies.
[ ] apps/shared-code/templates/ directory with:
- phase3-app-pipeline.yml (shared kaniko+test+deploy)
- phase3-redis-pipeline.yml (shared test+deploy)
- per-project-app.gitlab-ci.yml.tpl (template with $name, $build, $tests, $release, $namespace)
- per-project-redis.gitlab-ci.yml.tpl (same shape, points at phase3-redis-pipeline.yml)
- per-project-shared-code.gitlab-ci.yml (literal file, no substitution)
- scripts/ (placeholder for helper scripts)
- README.md describing the project
[ ] bootstrap/phase3/apps_manifest.yaml (METADATA only; no template content; lists all 4 projects including shared-code).
[ ] bootstrap/phase3/ci_render.py (reads one of the per-project template files from apps/shared-code/templates/ and substitutes AppSpec variables via Python string.Template. NO YAML literals live in this module — test_ci_render.py enforces this with a regex guard).
[ ] bootstrap/phase3/apps_local.py (AppsLocalFS — manages apps-local/blueprint-apps/ working tree).
[ ] Four GitLab projects under blueprint-apps/:
- guestbook ← apps/guestbook/ content + rendered .gitlab-ci.yml
- redis ← apps/redis/ content + rendered .gitlab-ci.yml
- redis-slave ← apps/redis-slave/ content + rendered .gitlab-ci.yml
- shared-code ← apps/shared-code/ content + copied-verbatim .gitlab-ci.yml (lint only)
[ ] Working build/test/deploy pipelines for guestbook, redis, redis-slave.
[ ] Working lint pipeline for shared-code.
[ ] docs/phase-3.md (this file) committed.
[ ] README updated with the uv run blueprint-phase3 cheat sheet row.
[ ] AGENTS.md updated with the Phase 3 layout table + the three new rules ("bootstrap uses Python, not glab", "Phase 3 has its own entry point", "pipeline code lives IN GitLab" / "templates are real files, not Python strings").
[ ] Tests: test_ci_render, test_template_files, test_apps_manifest, test_gitlab_client, test_apps_local, test_pipeline — all passing under pytest.

11. Open questions to resolve before implementation¶

apps/redis-slave/redis-slave/ subdirectory — does it contain a Dockerfile / Go source, or is the redis:7.4.1 upstream image enough? If there's a custom Go binary that wraps redis (some guestbook demos do this), we may need a build stage here too — which means redis-slave uses the per-project-app.gitlab-ci.yml.tpl template (so the bootstrap picks build_image=True for it in apps_manifest.yaml) instead of the redis variant.
Image repository base URL — the in-cluster registry is reachable in-cluster at registry-webservice-default.gitlab.svc.cluster.local:5000, but from the host it's https://registry.local.example.net. The CI pod uses the in-cluster DNS; the bootstrap CLI uses the host-side URL. Confirm we use the right URL for each side. (CI_REGISTRY env var comes from the runner's config.toml; verify it points at the in-cluster DNS.)
Project visibility — internal is the recommended default (logged-in users only; readable by any logged-in user including the bootstrap CLI). Pick internal.
Deployment namespace — each app lands in its own namespace (guestbook, redis, redis-slave); the CI_HELM_NAMESPACE variable equals the app name.
Phase 3 destroy behaviour — does --destroy helm uninstall the releases before deleting the GitLab projects? Yes — uninstall first, then delete projects, then clean up registry. Avoids orphaned pods on the next run.
Re-render policy — if the user hand-edits .gitlab-ci.yml in a GitLab project, then re-runs blueprint-phase3, does the bootstrap overwrite? Default: yes, overwrite (the bootstrap is the source of truth for pipeline shape). Flag --no-overwrite-ci to skip the .gitlab-ci.yml push if the file already exists in the repo. The shared templates under apps/shared-code/templates/ are always re-pushed (they live in the canonical source as ordinary files; the bootstrap's job is to keep GitLab in sync with apps/shared-code/).

12. Working with the repos after Phase 3¶

Phase 3 leaves you with two distinct working surfaces, and the line between them is the single most important thing to internalise. Mis-treat either side and you end up with silent drift between GitLab and the bootstrap's local state.

blueprint/                                    (this repo, on the host)
├── apps/<name>/                              CANONICAL GitLab-side source
│                                              (frozen after first push;
│                                              edit code here, the
│                                              bootstrap re-pushes it)
│
└── apps-local/blueprint-apps/<name>/         WORKING TREE — one git
                                               clone per GitLab project.
                                               Gitignored; re-creatable
                                               from GitLab with
                                               --reset-clones.

GitLab (https://gitlab.local.example.net/blueprint-apps/)
└── <name>                                    RECEIVING REPO — what
                                               `glab` / `git push` see.
                                               The bootstrap syncs the
                                               canonical source into
                                               here on every Phase 3 run.

12.1 What lives where, and who pushes what¶

Concern	Lives in	Pushed by
App source code (Go, Python, Dockerfile)	`apps/<name>/` (this repo, committed)	`blueprint-phase3` rsync → apps-local → commit → push
Helm chart	`apps/<name>/helm-chart/` (committed)	same path as source
Per-project `.gitlab-ci.yml` (the rendered file)	apps-local working tree + GitLab	`blueprint-phase3` re-renders from `apps/shared-code/templates/per-project-*.tpl`
Shared CI templates (`phase3--pipeline.yml`, `per-project-.tpl`)	`apps/shared-code/` (committed, pushed to `shared-code` project)	`blueprint-phase3` pushes the whole `shared-code/` tree
`apps/local/<name>/.gitlab-ci.yml` (after re-render)	same	same
CI/CD variables (`CI_KUBECONFIG_B64`, `CI_REGISTRY_IMAGE`, …)	GitLab project settings (set as project CI variables)	`blueprint-phase3` upserts on every run
`.gitlab-ci.yml` hand-edits in the GitLab UI	—	These WILL be overwritten on the next `blueprint-phase3` run (use `--no-overwrite-ci` to skip the push for that one file)

12.2 The canonical edit loop¶

The intended day-to-day loop is:

Edit code in apps/<name>/ (this repo). The directory is committed; treat it as the source of truth.
Run uv run blueprint-phase3 to rsync the delta, re-render .gitlab-ci.yml, commit, and push. Re-runs are idempotent — only changed files get a new commit.
Watch the pipeline in the GitLab UI or via glab ci view -p blueprint-apps/<name> --live.

# 1. Edit the canonical source
$EDITOR apps/guestbook/guestbook-go/main.go

# 2. Run the bootstrap
export KUBECONFIG=$PWD/infra/tofu/kubeconfig
uv run blueprint-phase3
# → "synced + pushed guestbook to blueprint-apps/guestbook @ <sha>"

# 3. Watch the pipeline
glab ci view -p blueprint-apps/guestbook --live
# or open https://gitlab.local.example.net/blueprint-apps/guestbook/-/pipelines

12.3 The local-clone loop (apps-local/)¶

Sometimes you need to iterate on a pipeline template or hand-edit a CI file without going through a full blueprint-phase3 run (e.g. debugging a kaniko flag, adding a one-off script for a single branch). The apps-local/ working tree is a regular git clone of each GitLab project — you can push to it directly:

# 1. cd into the per-project working tree
cd apps-local/blueprint-apps/guestbook

# 2. Branch + edit + push like a normal git repo
git checkout -b debug/fix-kaniko-cache
$EDITOR .gitlab-ci.yml
git commit -am "debug: drop cache flag to test cold build"
git push -u origin debug/fix-kaniko-cache

# 3. Watch the branch's pipeline in the UI
# https://gitlab.local.example.net/blueprint-apps/guestbook/-/pipelines?ref=debug/fix-kaniko-cache

The catch: a hand-edited .gitlab-ci.yml in the working tree will be overwritten the next time blueprint-phase3 re-renders. Two ways around this:

Work on a branch in apps-local/. The bootstrap always touches main; your branch stays untouched until you merge it.
Use --no-overwrite-ci when you want the bootstrap to leave a hand-edited .gitlab-ci.yml alone (everything else still gets rsynced + committed). Useful when iterating on a CI shape before folding it back into the shared template.

12.4 Inspecting + debugging pipelines¶

The four projects end up with ids you can use to build URLs. The bootstrap prints the project IDs at the end of a green run, but you can also list them:

# All four projects, with IDs + URLs
glab api 'groups/blueprint-apps/projects?per_page=10' \
  | jq '.[] | {id, path_with_namespace, web_url, last_activity_at}'

For one project:

# List recent pipelines
glab api 'projects/<id>/pipelines?per_page=5' \
  | jq '.[] | {id, status, ref, web_url}'

# Trace a specific job
glab api 'projects/<id>/jobs/<job_id>/trace' | tail -80

# Stream a running pipeline
glab ci view -p blueprint-apps/<name> --live

For the registry (where the build stage pushes images):

# List image tags for a project
glab api 'projects/<id>/registry/repositories' \
  | jq '.[] | {id, path, tag_count}'

# Tags for a specific image
glab api 'projects/<id>/registry/repositories/<repo_id>/tags?per_page=10' \
  | jq '.[] | {name: .name, created_at: .created_at}'

12.5 Common operations cheat-sheet¶

I want to…	Run
Sync the latest `apps/<name>/` to GitLab	`uv run blueprint-phase3`
See the pre-flight only, no mutations	`uv run blueprint-phase3 --check`
Wipe the working tree + re-clone everything	`uv run blueprint-phase3 --reset-clones`
Wipe the 4 GitLab projects (keep the group)	`uv run blueprint-phase3 --destroy`
Limit a run to one project	`uv run blueprint-phase3 --project guestbook`
Leave a hand-edited `.gitlab-ci.yml` alone	`uv run blueprint-phase3 --no-overwrite-ci`
Pull remote changes (web-UI edits, MRs)	`cd apps-local/blueprint-apps/<name> && git pull --ff-only`
Push a hand-edit on a branch	`cd apps-local/blueprint-apps/<name> && git push -u origin <branch>`
Watch the latest pipeline	`glab ci view -p blueprint-apps/<name> --live`
Read a job's full log	`glab api projects/<id>/jobs/<job_id>/trace`
List the group's projects + IDs	`glab api 'groups/blueprint-apps/projects'`
Re-render only (no push)	`cd apps-local/blueprint-apps/<name> && uv run python -m bootstrap.phase3.ci_render /path/to/apps_manifest.yaml` (or just edit + commit manually)

12.6 Trust + auth¶

The bootstrap's glab auth login step (Rule 1's narrow exception) writes the PAT to ~/.config/glab-cli/config.yml. The bootstrap never reads that file — it uses the PAT from OpenBao for its own API calls. But the file is what lets you (and an AI agent) run glab api ... from a shell without re-prompting for a token. If glab api starts returning 401 after a fresh install, run uv run blueprint-phase3 again — the mint step will refresh the PAT in OpenBao and re-seed the local glab-cli/config.yml.

12.7 What NOT to do¶

Don't git push from apps/<name>/ directly. That directory is not a working tree — it's the canonical source the bootstrap owns. Pushing from it bypasses the render + commit step and lands the wrong file shape on GitLab. Always edit apps/<name>/, then run blueprint-phase3.
Don't hand-edit files under apps-local/blueprint-apps/<name>/ on main and expect them to stick. The next blueprint-phase3 run will rsync from apps/<name>/ and overwrite. Use a branch (see §12.3) or fold the change back into the canonical source + the shared templates.
Don't glab project delete or glab variable set outside the bootstrap. Per the Phase-3 rules in AGENTS.md § 4 rule #3, every GitLab-side change is automated by blueprint-phase3. Hand-edits create drift the bootstrap will undo on the next run.
Don't re-push the bootstrap PAT into a different OpenBao path. The pipeline depends on secret/gitlab/bootstrap/admin_token. If you ever wipe OpenBao (uv run blueprint-bootstrap --destroy --yes does this), the next blueprint-phase3 mints a fresh PAT and re-seeds the path automatically.

13. How to provision (for humans + AI agents)¶

The companion skill at .agents/skills/provision-phase-3/SKILL.md is the source of truth for the iteration loop. It follows the same ten-section template as provision-phase-1 / provision-phase-2:

Pre-flight (uv run blueprint-phase3 --check)
Install (uv run blueprint-phase3 — six idempotent steps)
Smoke tests (4 projects exist, CI/CD variables set, registry has one image per project, latest pipeline per project is success, deployed guestbook image tag matches the short-SHA the pipeline stamped)
URLs you can reach after install (the *.local.example.net hostnames from Phase 2 — Phase 3 adds no new ones)
Iteration loop (each step maps to one method on phase3/pipeline.py or one file under apps/shared-code/templates/)
Canonical (known-good) pinned versions
Common pitfalls (frozen — append, don't rewrite)
Rules of thumb (the three ownership rules in § 0 above, restated as "what owns this?" — the bootstrap, the per-project source, or the shared template)
When the install is green
How to undo (blueprint-phase3 --destroy [--reset-clones] wipes the GitLab-side state; the full clean-slate chain is tofu destroy && apply && bootstrap --phase 2 && blueprint-phase3)

If you are hand-running Phase 3, the canonical sequence is:

cd blueprint
export KUBECONFIG=$PWD/infra/tofu/kubeconfig
uv sync                                        # one-time
uv run blueprint-bootstrap --phase 2            # GitLab + Runner + OpenBao (must already be green)
uv run blueprint-phase3 --check                # pre-flight only
uv run blueprint-phase3                        # full install (6 steps)
# Push a commit to trigger a real build:
cd apps-local/blueprint-apps/guestbook
echo "# smoke at $(date -Iseconds)" >> README.md
git commit -am "smoke: touch README"
git push origin main
# Watch from the GitLab UI: https://gitlab.local.example.net/blueprint-apps/guestbook/-/pipelines

If you are handing Phase 3 to an AI agent, paste .agents/skills/provision-phase-3/SKILL.md into the chat (or let the agent auto-load it via its skills directory), then prompt:

Run provision-phase-3 end to end and report when all four smoke pipelines are green. Don't run anything that needs sudo — print the command and ask me to run it. Follow the skill's Smoke tests section before declaring green.

The skill's Iteration loop maps every known failure to the exact file + line range to fix, so the agent has a deterministic recovery path even when something fails.

Phase 3 — GitLab projects + CI/CD pipelines for the apps/ demos¶