Deep Dive: Tofu IaC Layer¶

infra/tofu/ is the only thing allowed to create or delete the kind cluster. The bootstrap itself never runs tofu apply. The bootstrap also never runs kind create cluster, never docker rm -f kind-cicd-*, never kubectl delete namespace "to tidy up". If the cluster is in a weird state, the only safe fix is tofu state rm <orphan-resource> — never hand-delete state.

This file explains what the IaC layer owns, why, and the contracts other layers depend on.

Overview¶

infra/tofu/ has exactly one job: lifecycle the 5-node kind cluster that hosts every other component. The files are intentionally small:

infra/tofu/
├── providers.tf                 # provider pins (kind ~> 0.11, helm ~> 3.0, local ~> 2.5, null ~> 3.2)
├── variables.tf                 # cluster_name, kubernetes_version, node_shapes, kubeconfig_path, data_root, domain, preserve_stateful_data
├── locals.tf                    # resolved absolute paths + kind-style node specs
├── cluster.tf                   # kind_cluster.cicd + kubeconfig rewrite + smoke test + null_resource.wipe_data
├── outputs.tf                   # kubeconfig_path, ca_*, wildcard_*, phase_ready
├── providers.tf
├── tofu.tfvars.example          # committed; copy to tofu.tfvars (gitignored) for local overrides
├── tofu.tfvars                  # gitignored, real local overrides
├── .terraform/                  # generated by tofu init
└── .terraform.lock.hcl          # dependency lock (committed)

The state lives in terraform.tfstate (gitignored) and is the single source of truth for "does the cluster exist?". tofu state list should be empty after a successful destroy.

Key Files¶

File	Purpose
`providers.tf`	`kind ~> 0.11`, `helm ~> 3.0`, `local ~> 2.5`, `null ~> 3.2`.
`variables.tf`	`cluster_name`, `kubernetes_version`, `node_shapes`, `kubeconfig_path`, `data_root`, `domain`, `preserve_stateful_data`.
`locals.tf`	Resolves absolute paths + node shapes into kind-style node specs.
`cluster.tf`	`kind_cluster.cicd` + per-node hostPath bind mounts + containerd config patches (registry TLS trust) + `null_resource.wipe_data` destroy provisioner.
`outputs.tf`	`kubeconfig_path`, `ca_`, `wildcard_`, `phase_ready`.
`tofu.tfvars.example`	Committed template; copy to `tofu.tfvars` for local overrides.

Architecture¶

Hold "Alt" / "Option" to enable pan & zoom

classDiagram
  class kind_cluster {
    +name: string
    +node_image: string
    +wait_for_ready: bool
    +kubeconfig_path: string
    +kind_config: KindConfig
  }

  class KindConfig {
    +api_version: string
    +kind: string
    +containerd_config_patches: list
    +nodes: list[NodeConfig]
  }

  class NodeConfig {
    +role: string
    +extra_mounts: list[Mount]
  }

  class Mount {
    +host_path: string
    +container_path: string
    +propagation: string
    +read_only: bool
  }

  class null_resource.wipe_data {
    +triggers: map
    +provisioner(local-exec): mkdir -p + kubectl cordon + kubectl exec rm -rf
  }

  class local_file.kubeconfig_merged
  class local_file.ca_pem

  kind_cluster "1" --> "5" NodeConfig
  NodeConfig "1" --> "*" Mount
  kind_cluster ..> null_resource.wipe_data : depends_on

The Cluster Spec¶

cluster.tf:kind_cluster.cicd provisions:

resource "kind_cluster" "cicd" {
  name            = var.cluster_name                  # default "cicd"
  node_image      = "kindest/node:${var.kubernetes_version}"
  wait_for_ready  = true
  kubeconfig_path = abspath(var.kubeconfig_path)     # infra/tofu/kubeconfig

  kind_config {
    kind        = "Cluster"
    api_version = "kind.x-k8s.io/v1alpha4"

    containerd_config_patches = [
      <<-EOT
        [plugins."io.containerd.grpc.v1.cri".registry]
          config_path = "/etc/containerd/certs.d"
      EOT
    ]

    dynamic "node" {
      for_each = local.nodes    # 1 cp + 3 gitlab + 1 runner
      content {
        role = node.value.role_kind
        extra_mounts { /* see below */ }
      }
    }
  }
}

Why 5 nodes (1 cp + 3 gitlab + 1 runner)?¶

1 control-plane @ 4Gi / 2 CPU — runs the kube-apiserver, etcd, CoreDNS, the local-path provisioner, and the CloudNativePG operator. 4Gi is enough.
3 gitlab workers @ 8Gi / 4 CPU — GitLab chart 10.x (GitLab 19.x) deploys Cloud Native architecture with separate webservice, sidekiq, kas, gitaly, prometheus, plus the chart-bundled OpenBao subchart. 4Gi per worker is below GitLab's minimum reference architecture; 8Gi + 4 CPU is the smallest viable shape. 3 workers (instead of 1) so pods can spread across nodes.
1 runner worker @ 8Gi / 4 CPU — pinned separately so the CI workload class doesn't evict webservice/sidekiq pods under load.

Total advisory memory: 4 + 8 + 8 + 8 + 8 = 36Gi. Total advisory CPU: 2 + 4 + 4 + 4 + 4 = 18. The host must be able to back both — docs/prereqs.md § Hardware floor says 24 GB free RAM, 4 cores minimum for the kind spec to fit comfortably; the 5-node default bumps the floor for the GitLab-class workloads.

Override via tofu.tfvars (node_shapes = [...]) to fit a smaller host.

Why `extra_mounts.propagation = Bidirectional` (default)¶

extra_mounts {
  host_path      = abspath("${var.data_root}/shared")
  container_path = "/var/local/shared"
  propagation    = var.preserve_stateful_data ? "HostToContainer" : "Bidirectional"
}

Bidirectional mode means container umounts propagate to the host as delete events — i.e. rm -rf inside a container drops the host file. This is what we want for the destructive default (preserve_stateful_data = false): kind's container umount is treated as "delete everything under this mount", so the chart-managed PVC teardown script (local-path-config teardown rm -rf) sweep propagates back to the host.

The preserve flag flips this to HostToContainer (default) so the host bind-source isn't propagated through on container umount — useful for users who recreate the cluster but want to reuse the on-disk PG / Redis / MinIO data.

The two flags must agree between tofu apply and bootstrap --destroy or the next install sees a divergent contract.

Containerd config patches¶

containerd_config_patches = [
  <<-EOT
    [plugins."io.containerd.grpc.v1.cri".registry]
      config_path = "/etc/containerd/certs.d"
  EOT
]

Tells containerd to consult /etc/containerd/certs.d for registry-specific TLS overrides. The per-host directories (registry.local.example.net, gitlab.local.example.net, ...) are populated by the bootstrap's phase2/kind_node_trust.py via a node_lifecycle_pre_start hook. Without the patch + the bootstrap writing the cert symlinks, kubelet fails to pull from the in-cluster registry with x509: certificate signed by unknown authority.

Why kind `extraPortMappings` is intentionally NOT used¶

"We use kubectl port-forward rather than host-port mappings."

This is the NodePort data-plane choice in action (see Architecture Overview § Decision 1). The kind cluster's data-plane is NodePort-type; the host accesses it via kubectl port-forward driven by blueprint-secrets port-forward (or blueprint-bootstrap --port-forward).

The Wipe Provisioner¶

resource "null_resource" "wipe_data" {
  count = var.preserve_stateful_data ? 0 : 1

  triggers = {
    cluster_id = kind_cluster.cicd.id
  }

  provisioner "local-exec" {
    when        = destroy
    command     = <<-EOT
      set -euo pipefail
      if [[ -d "${abspath(var.data_root)}/shared" ]]; then
        docker run --rm \
          -v "${abspath(var.data_root)}/shared:/data:rw" \
          --privileged \
          alpine:3.20 \
          sh -c "find /data -mindepth 1 -delete || true"
      fi
    EOT
  }
}

tofu destroy runs this provisioner as part of teardown. It bind-mounts the host data root into a one-shot Alpine privileged container that does find … -delete. This handles the case where the host can't unlink files owned by a pod UID (openbao=100, postgres=1001).

The cluster's depends_on = [kind_cluster.cicd] ensures the data wipe happens after kind itself has been torn down (so the host bind isn't mounted from inside a container at the time we delete).

count = var.preserve_stateful_data ? 0 : 1 makes the whole provisioner a no-op when the preserve flag is set.

Variables¶

variable "preserve_stateful_data" {
  description = <<-EOT
    Bidirectional-mode flag for the host-side stateful data:
      true  → `tofu destroy` leaves infra/data/shared/* intact (legacy
              2026-06 contract; preserve_stateful_data + mv teardown).
      false → `tofu destroy` is a full reset (default 2026-07+).
              Paired with `--destroy` (no --preserve-data) on the
              bootstrap side.
    Set via `tofu -chdir=infra/tofu apply -var=preserve_stateful_data=true`.
    The bootstrap CLI mirrors this with `bootstrap --destroy --preserve-data`.
  EOT
  type    = bool
  default = false
}

The default flipped from true to false in 2026-07. The change was paired across three coupled knobs (kind extra_mounts.propagation, the null_resource.wipe_data provisioner, and the local-path provisioner's teardown script). The trap is that flipping one without the others produces a divergent state — see AGENTS.md rule #4 for the full contract.

Outputs¶

infra/tofu/outputs.tf:

output "phase_ready" { value = "1 (cluster up); run uv run blueprint-bootstrap --phase 2 to install the stack." }

The bootstrap prints these on success. The kubeconfig_path output is what bootstrap --phase 2 reads via Paths.from_bootstrap_dir() to set KUBECONFIG.

Implementation Details¶

Bidirectional mode coupling¶

When preserve_stateful_data = false:

Kind extra_mounts.propagation: Bidirectional — host sees container umounts as deletes.
null_resource.wipe_data is created (count = 1) — runs on tofu destroy.
Local-path provisioner's teardown: rm -rf — chart-managed PVC dirs are nuked on PVC delete.

When preserve_stateful_data = true:

Kind extra_mounts.propagation: HostToContainer (default) — host ignores container umounts.
null_resource.wipe_data is absent (count = 0).
Local-path provisioner's teardown: mv — chart-managed PVC dirs are renamed to .preserved-*, for manual cleanup later.

The trap is changing the flag on a running cluster without re-applying — tofu apply -var=preserve_stateful_data=true alone flips the variable, but the kind cluster's mounts were already declared Bidirectional at apply time. The user needs to tofu destroy && tofu apply together to actually pivot the mode.

State management¶

terraform.tfstate is the source of truth for "does the cluster exist?". tofu state list should be empty after a successful destroy.

If tofu state list is non-empty when docker ps | grep kind- is empty, tofu thinks the cluster is up but it isn't — the only safe fix is tofu state rm <orphan-resource> (per resource). Don't hand-delete state — if tofu still thinks a resource is up, the next apply will try to use it.

Provider lock¶

.terraform.lock.hcl is committed for reproducibility. kind ~> 0.11, helm ~> 3.0, local ~> 2.5, null ~> 3.2. New providers = edit providers.tf + run tofu init -upgrade + commit the lock.

Variables Reference¶

Name	Default	Description
`cluster_name`	`"cicd"`	Passed to `kind create cluster --name`. Container names + kubeconfig path derive from it.
`kubernetes_version`	`"v1.31.0"`	kindest/node image tag. Defaults to the upstream LTS recommended for kind 0.27+.
`domain`	`"local.example.net"`	Base DNS domain for `*.wildcard` the chart mints. Read by Phase 2 `gitlab` installer for `global.hosts.domain`. Not used by Phase 1.
`kubeconfig_path`	`"./kubeconfig"`	Where to write the merged kubeconfig (side-by-side; doesn't touch `~/.kube/config`).
`data_root`	`"../data"`	Host directory whose `shared/` sub-path is bind-mounted onto every node at `/var/local/shared`.
`node_shapes`	5 nodes (1 cp + 3 gitlab + 1 runner)	Ordered list of `{ name, role, memory, cpu }`.
`preserve_stateful_data`	`false` (2026-07+)	Bidirectional-mode flag. Paired with `bootstrap --destroy --preserve-data`.

Conventions (from AGENTS.md)¶

The bootstrap never runs OpenTofu. It is run manually. There is no apply() method on TofuRunner. The bootstrap exposes init(), validate(), next_steps() — that's it.
The only way to create or delete the cluster is tofu. No kind create cluster, no docker rm -f kind-cicd-*, no kubectl delete ns.
Destroy is destructive (2026-07+). The default wipes everything. --preserve-data opts back into the 2026-06 contract.
NodePort, not LoadBalancer. Chart 10.x's bundled Envoy Gateway defaults the data-plane Service to LoadBalancer; kind has no provisioner, so the Service never settles. The bootstrap owns the NodePort pinning (helm-values-gitlab.yaml:gatewayApiResources.envoy.proxySpec.provider.kubernetes.envoyService.type: NodePort).

Potential Improvements¶

Auto-tofu init -upgrade on cache miss (currently the bootstrap calls tofu init once + assumes the lock is committed; new contributors see "no such provider" until they read AGENTS.md § 1).
Output the phase_ready message with the kubeconfig path + domain preview (currently a static string).
Surface the preserve_stateful_data flag explicitly in the help (tofu plan -var-file tofu.tfvars.example diff highlighting the mount propagation change).
A tofu output -json wrapper script that re-renders the bootstrap's "next steps" block in --check mode (saves users a cat outputs.tf).

For the bootstrap that consumes these variables and outputs, see Bootstrap Package.md and Phase 2 Pipeline.md.