Skip to content

Project Overview

What is this Blueprint?

This Blueprint is a reproducible, fully local GitLab + Kubernetes CI/CD stack — a single repo that builds a 5-node kind cluster on the developer's laptop, installs a self-hosted GitLab CE on top, and demonstrates the full inner loop from git push → CI pipeline → running workload on https://<app>.local.example.net.

The delivery model is agentic-driven — the same .agents/skills/ SKILL.md files that explain the install to a human also work as the runbook a Copilot / Cursor / Claude Code agent reads as background context the moment the repo opens.

Core Purpose

  1. Zero-cloud CI/CD: Everything (cluster, GitLab, Runner, Postgres, Redis, MinIO, OpenBao, sample app) runs on the developer machine. No cloud account, no managed Kubernetes, no public DNS — only a /etc/hosts entry the user maintains themselves.
  2. Self-host a real GitLab: GitLab chart 10.x (GitLab 19.x CE), with bundled Envoy Gateway for ingress, a bundled OpenBao subchart for Secrets Manager, and external CloudNativePG / Redis / MinIO charts (chart 10.x dropped the bundled subcharts for those).
  3. End-to-end demo of the inner loop: a git push to a GitLab project triggers a Runner-driven pipeline that runs helm lint, helm template, docker build (BuildKit/kaniko), docker push to the in-cluster registry, and helm upgrade --install against the cluster.
  4. Idempotent bootstrap: every step (host prereqs → cluster → CNPG → Redis → MinIO → OpenBao → GitLab → Runner → Phase-3 projects + CI) is safe to re-run. A full tofu destroy && apply && bootstrap --phase 2 && blueprint-phase3 reproduces the exact end state.
  5. Manifest the rules in code and skills: hard rules (cluster lifecycle is tofu only, no committed secrets, bootstrap prepares never applies, no glab in the bootstrap, *.local.example.net is for humans not pods) are encoded both in AGENTS.md and in the per-phase skills.

Technology Stack

Languages & Runtimes

Layer Tech Where
Bootstrap (host installer) Python ≥3.11, packaged with uv / hatchling infra/scripts/bootstrap/, pyproject.toml
Cluster IaC OpenTofu (tofu) ≥1.6 infra/tofu/*.tf
Sample workload Go (guestbook) apps/guestbook/guestbook-go/
Demo backing services upstream redis:7.4.1 (no custom Dockerfile) apps/redis/, apps/redis-slave/

Frameworks & Tools

  • Kubernetes in Docker: kind v0.27.0, 5 nodes (1 control-plane + 3 gitlab + 1 runner), kindest/node:v1.31.0
  • Helm ≥3.16 — every chart install is helm upgrade --install from a locally cached .tgz
  • Local-path-provisioner v0.0.30 (raw manifest), patched so chart-managed PVCs land under infra/data/shared/
  • CloudNativePG operator + single-instance cluster (8Gi)
  • Redis — bitnami/redis, architecture=standalone
  • MinIO — single-pod minio/minio + 11 GitLab buckets created via in-cluster mc
  • OpenBao (Vault-compatible KV v2) — bootstrapped standalone + chart-bundled gitlab-openbao
  • GitLab CE chart 10.1.1 (chart-bundled Envoy Gateway + chart-bundled OpenBao subchart)
  • GitLab Runner chart 0.71.0 (Kubernetes executor, registers against gitlab-webservice-default.gitlab.svc:8181)
  • cfssl (chart's pre-install Job) mints the self-signed wildcard cert for *.local.example.net

Key Python Dependencies

Package Purpose
hvac ≥2.3,<3 OpenBao HTTP client (KV v2 + token auth)
python-gitlab ≥5.0,<6 Typed GitLab REST API client (Phase 3)
requests ≥2.32 Raw HTTP for endpoints python-gitlab doesn't cover
click ≥8.1,<9 CLI wrappers (blueprint-bootstrap, blueprint-secrets, blueprint-phase3)
pyyaml ≥6.0,<7 YAML parsing for CI templates + apps manifest

Key Features

  • Three CLI entry points (post uv sync):
  • blueprint-bootstrap [--phase 1\|2] [--check] [--dry-run] [--destroy] [--port-forward] [--user]
  • blueprint-secrets read\|ui\|port-forward — auto-port-forwards 127.0.0.1:8200 for OpenBao
  • blueprint-phase3 [--check] [--destroy] [--reset-clones] [--project X] [--no-overwrite-ci]
  • Every installer is a single-responsibility class wired together by composition roots (BootstrapApp, Phase3App) — appending "another installer" means adding one class, one wiring line, and (if user-facing) one CLI option. SOLID over monoliths.
  • Versions pinned in one place (infra/scripts/bootstrap/VERSIONS.json) — no class hardcodes a version string.
  • Helm charts cached locally at infra/helm-charts/<name>-<version>.tgz; re-installs don't need network.
  • No plaintext secrets in git — OpenBao stores day-to-day secrets (secret/gitlab/initial_root_password, secret/gitlab/runner/registration_token, secret/gitlab/bootstrap/admin_token); host-side snapshots land in infra/secrets/ (gitignored, mode 0600).
  • Templates are real YAML files on disk — apps/shared-code/templates/*.yml.tpl is the per-project CI source. phase3/ci_render.py does string.Template.safe_substitute over them. The rule "templates must have their own files" has a regression test.
  • AGENTS.md frontmatter rule: skill description: fields are single-line quoted YAML (no folded continuations).
  • Runbook-as-skill: three .agents/skills/provision-phase-{1,2,3}/SKILL.md files following the canonical 10-section template (Pre-flight → Install → Smoke tests → URLs → Iteration loop → Pinned versions → Common pitfalls → Rules of thumb → When install is green → How to undo).

Project Structure

blueprint/
├── AGENTS.md                          # hard rules + layout map (humans + agents)
├── index.md                          # quick-start + post-install cheat-sheet
├── pyproject.toml                     # uv project → blueprint-bootstrap + blueprint-secrets + blueprint-phase3
├── uv.lock                            # committed for reproducibility
├── apps/                              # CANONICAL GitLab-side source (frozen after first push)
│   ├── shared-code/                   # cross-app CI templates + helper scripts
│   ├── guestbook/                     # demo: Go app + helm chart + Dockerfile
│   ├── redis/                         # demo: redis master + helm chart (no Dockerfile → upstream)
│   └── redis-slave/                   # demo: redis slave workload + helm chart
├── apps-local/                        # WORKING TREE (gitignored; one git clone per GitLab project)
├── docs/
│   ├── prereqs.md                     # OS matrix + hardware floor + ports + DNS
│   ├── phase-1.md                     # cluster bring-up runbook
│   ├── phase-2.md                     # GitLab + Runner + OpenBao contributor guide
│   ├── phase-3.md                     # GitLab-side app onboarding + per-project CI contract
│   ├── secrets.md                     # full secret inventory + flow diagram
│   └── smart-docs/                    # this documentation (smart-docs C4 output)
├── .agents/skills/
│   ├── provision-phase-1/SKILL.md
│   ├── provision-phase-2/SKILL.md
│   └── provision-phase-3/SKILL.md
└── infra/
    ├── helm-charts/                   # locally cached chart tarballs
    ├── scripts/
    │   ├── bootstrap.py               # thin shim → bootstrap package
    │   ├── phase3.py                  # thin shim → phase3_cli.py
    │   └── bootstrap/                 # class-based package (SOLID)
    │       ├── VERSIONS.json          # SINGLE SOURCE OF TRUTH for all versions
    │       ├── cli.py                 # blueprint-bootstrap entry point (click)
    │       ├── secrets_cli.py         # blueprint-secrets entry point (click)
    │       ├── phase3_cli.py          # blueprint-phase3 entry point (click)
    │       ├── app.py                 # composition root: BootstrapApp (Phase 1+2)
    │       ├── app_phase3.py          # composition root: Phase3App
    │       ├── app_installer.py       # HelmAppInstaller + HeadlampInstaller
    │       ├── helm_cache.py          # downloads charts to infra/helm-charts/
    │       ├── tofu.py                # tofu init/validate/next_steps (no apply)
    │       ├── prereq.py              # prereq ABCs (Docker/Kubectl/Kind/Helm/Tofu)
    │       ├── versions.py            # VERSIONS dict loader + tool_pin()
    │       ├── paths.py               # resolved filesystem paths
    │       ├── logger.py              # Logger protocol + Console/Null
    │       ├── shell.py               # CommandRunner protocol + Subprocess/DryRun
    │       ├── os_detect.py           # apt/dnf/pacman/brew branching
    │       ├── installer.py           # OSFamily → package manager Strategy
    │       ├── port_forward.py        # generic 127.0.0.1:<svc>:port helper
    │       ├── phase2/                # Phase 2 installers (see Architecture doc)
    │       └── phase3/                # Phase 3 modules (see Architecture doc)
    ├── data/                          # hostPath source for the kind cluster (gitignored)
    ├── secrets/                       # gitignored, mode 0600 — see docs/secrets.md
    └── tofu/                          # OpenTofu IaC for the kind cluster

Getting Started

# 1. Get the code
git clone https://github.com/bruj0/k8s-cicd-gitlab.git
cd k8s-cicd-gitlab/blueprint

# 2. Install Python deps into .venv/ (uv.lock committed → reproducible)
uv sync

# 3. Phase 1 prep: prereqs + tofu init + helm chart cache. Prints next commands.
uv run blueprint-bootstrap --phase 1

# 4. YOU apply — the bootstrap never applies infra.
tofu -chdir=infra/tofu apply -auto-approve

# 5. Phase 2 install — Gateway CRDs + CNPG + Redis + MinIO + OpenBao + GitLab + Runner.
#    End-to-end takes ~10 min on a beefy laptop.
uv run blueprint-bootstrap --phase 2

# 6. Post-install host-side steps (trust CA, /etc/hosts, OpenBao login).

# 7. Phase 3 — provision the 4 GitLab projects + CI pipelines.
uv run blueprint-phase3

Or hand the whole thing to an AI agent — paste the three per-phase SKILL.md files into the first message and prompt "Run provision-phase-1, then provision-phase-2, then provision-phase-3. Don't run anything that needs sudo; print and ask. Follow each skill's Smoke tests section before declaring green." The agent reads each skill's Pre-flight → Install → Smoke tests → Iteration loop in order.

For the deep architecture, see 2. Architecture Overview.md. For how a git push becomes a running workload, see 3. Workflow Overview.md.

Architecture Summary

Three discrete layers, all contained inside one repo:

  1. Infrastructure (IaC): infra/tofu/ — kind_cluster.cicd resource provisions a 5-node cluster with shared hostPath mounts (infra/data/shared/) for stable PVCs. null_resource.wipe_data destroy provisioner guarantees full wipe on tofu destroy (paired with the bootstrap's --destroy).
  2. Application stack (Phase 2): infra/scripts/bootstrap/phase2/ — Gateway API CRDs → local-path SC → CNPG → Redis → MinIO → OpenBao → wildcard TLS → GitLab (with bundled Envoy + bundled OpenBao subchart) → GitLab Runner. Orchestrated by Phase2Pipeline (13 ordered, idempotent steps).
  3. GitLab-side app onboarding (Phase 3): infra/scripts/bootstrap/phase3/ — mints admin PAT via gitlab-rails runner → seeds glab auth login → creates blueprint-apps group → for each entry in apps_manifest.yaml (shared-code, guestbook, redis, redis-slave): creates project, clones into apps-local/, rsyncs from apps/, renders .gitlab-ci.yml, sets CI/CD variables, commits, pushes, triggers a smoke pipeline.

Phase 3 survives tofu destroy && apply && bootstrap --phase 2 by re-minting the PAT — no manual glab step required to rebuild. See 2. Architecture Overview.md § Phase 3 invariants for the contract.