Delivery Pipeline: From a Branch to Production

How a change travels on this instance: branch, remote check, preview, landing on the merged result, next, the release PR, main, the image and the deploy. The commands for each step, what each check proves, how concurrent branches stay safe, and where the time goes.

Last updated: November 14, 2026

On this page

Every change to this instance (release 0.0.3) travels the same road:

branch ─► remote check (affected plan) ─► preview ─► land (full plan on the merged result)
       ─► next ─► release PR ─► main ─► image ─► deploy

Work lands on next. main moves only through a release PR that promotes a proven next commit unchanged, and production runs an image built from main, pinned by digest. Each step below names its command, what it proves, and what it does not.

The steps

1. Branch

Cut a branch from next. Write a changelog fragment for anything the changelog should mention (changelog.d/<id>.toml, see Changelog fragments). Never edit CHANGELOG.md by hand.

2. Remote check: the affected plan

just forge-check            # checks HEAD on the forge (remote build/check env on GKE)
just forge-status           # what the forge is running
just forge-logs <job>       # follow one job

The forge is a Kubernetes Job on the GKE cluster with a persistent, pre-built cache of next. The check runs scripts/gate.sh --changed against the merge-base with next: formatting, lint, documentation, source gates and the tests of every crate the branch affects, with passes cached per test package.

Proves: the branch, as committed, passes every gate for the crates it touches. Does not prove: anything about the result of merging it with whatever else landed on next meanwhile. That is the landing check.

3. Preview

just preview-up            # a running instance of HEAD in its own namespace
just preview-open <id>     # port-forward and open it
just preview-list
just preview-down <id>

A preview is the branch's binary on the production image, with its own database and seed data, in namespace preview-<id>. Previews expire after 24 hours (just preview-gc). Proves: the change boots, migrates a fresh schema and serves its pages. It is for looking, not a gate.

4. Land: the full plan on the merged result

just land <PR>                       # check, merge, prove, release, ship
just land <PR> --no-ship             # stop after the release
just land <PR> --no-release          # stop after the merge into next
just land <PR> --dry-run             # print every command; change nothing

just land runs scripts/forge/land.sh, one timed, resumable step at a time:

Step	What happens	What it proves
check	the forge merges the PR onto the current `next` and runs the full gate on that merged tree	the exact tree `next` is about to become passes everything
merge	`gh pr merge --merge`, only if `next` is still the commit the check merged onto; otherwise check again (at most 3 times)	`next` never receives an untested tree
gate proof	the check's result names the merged tree; when the merge commit has that tree, it is the release proof and nothing runs twice. Otherwise a `refresh` job runs the full gate on `next`	`scripts/check-gates-green.sh` accepts the release candidate
release	`just release <version>` from a clean worktree at `origin/next`	`main` receives the proven commit unchanged
ship	`just ship` from a clean worktree at `origin/main`	production runs that commit's image
record	one line in `docs/build/evidence/deliveries.jsonl`	the delivery clock has every timestamp

Re-running just land <PR> after a failure continues from the first step not done: it reads GitHub, git and the deploy ledger, not a local state file. Shipping is refused between 01:45 and 04:45 UTC, the nightly retention window, unless --force-window.

Until the forge runs, just land --proof cloudbuild uses the interim Cloud Build proof instead: successful gates and tests builds tagged with the PR head, whose branch must already contain next.

5. `next` → release PR → `main`

just release <version> (run by just land, or on its own) verifies the gate proof for the next commit, pushes the frozen ref promote/<version>/<main>/<sha>, opens the release PR, re-verifies that the merge tree equals the proven tree and merges it. The version must already be on next: bump it with scripts/sync-release-version.sh and record the schema rung with just schema-baseline in a PR of their own.

6. Image and deploy

just ship builds the committed main head in Cloud Build, resolves the image tags to digests in Artifact Registry and deploys by digest with Helm. The chart's pre-upgrade hook runs the migrations with the new binary while the old pods still serve. Each deploy appends one line to docs/build/evidence/deploys.jsonl with its phases in seconds.

How concurrent branches stay safe

Serial landing on the merged result. Only one landing check holds the cache lane at a time, and a merge happens only if next has not moved since its check. Two branches that pass alone but break together are caught at the second landing, never on next.
Expand-only migrations. A schema change adds to the base schema and is additive within a release line, so the old pods keep working while the new binary migrates, and a rollback of the image does not need a rollback of the schema.
Changelog fragments. One file per change in changelog.d/, assembled at release, so no two branches edit the same lines of CHANGELOG.md.
One deploy at a time. just ship deploys the head of main, which holds every landed change before it; the deploy ledger records which commit is live.

Changelog fragments

[root]
section = "Added"
area = "Tooling"
text = "one line in the changelog register."

section is one of Breaking, Added, Changed, Fixed, Removed. An optional [[crate]] entry (name, section, text) names a crate of the root workspace and is written as a **<crate>:** bullet. just changelog-check validates every fragment; scripts/changelog-assemble.py --version X.Y.Z --date YYYY-MM-DD --dry-run prints the release block. Format: changelog.d/README.md.

Where time goes

Measured on 2026-11-14, before the forge

These are that day's measurements on Cloud Build, the path the forge replaces.

Stage	Time
Cloud Build `gates` run	4m50s and 6m35s
Cloud Build `tests` run	30m36s: compile 8m12s, unit 35 s (1,424 tests), integration 6m48s (737), contract 13m40s (117)
Queue before each build	41–57 s
Landing commit → merged on `main`	41m37s, of which 30m36s was the tests run
`just release`	20 s
Image build (warm)	5.5–7.6 min
Deploy (from the deploy ledger)	typically 400–540 s
Fresh worktree to buildable	not reached in 18 minutes

On that day the gates ran twice for one change: once on the branch head and again (6m35s) on the merge commit, because the Cloud Build proof is keyed by commit, not by tree. The forge's tree-keyed landing result removes the second run.

Measured on the forge

Stage	Time
Forge check, affected plan (warm cache)	pending first run
Forge landing check, full plan on the merged result	pending first run
Forge refresh after a merge	pending first run
Queue and pod start	pending first run
Preview up	pending first run
`just land`: check → merged on `next`	pending first run
`just land`: merged → on `main`	pending first run
`just land`: on `main` → live	pending first run
First commit → production (median)	pending first run

Reading the clock

just delivery-clock --pr <N>              # one landed change, as JSON + a table
just delivery-clock --since 2026-11-01    # every PR merged into next since then
just delivery-clock --since 2026-11-01 --append   # backfill deliveries.jsonl
just flow-report                          # target/flow-report/flow-report.md

scripts/delivery-clock.py rebuilds the record of any landed change from the sources of truth (git, the PR, Cloud Build runs, forge results and the deploy ledger), so the clock does not depend on just land having been used. A source it cannot read leaves its fields empty, never guessed. Each record splits first commit → production into five stages: edit → PR, PR → merge, merge → release, release → image, image → live.

scripts/flow-report.py summarises a window: lead time from landed to released, push → green on the gate builds, rework, time lost in cancelled runs, the files every branch edits (collisions), the blast radius per commit, gate runs by mode, and first commit → production with the median of each stage.

System health

The console page System health (/admin/system/health, console roles, under Platform beside Data lake) is the permanent, data-backed form of the clock. Every number on it comes from three tables the instance keeps — pipeline_runs, pipeline_run_phases and delivery_changes — and none is typed in by hand. The window is ?range=7d|30d|90d (30 days by default); a change belongs to the window by its merge, a run by its finish. Medians and p90s are computed by Postgres (percentile_cont).

How records arrive. The app's pods cannot read the CI bucket, so the machine that ran a step pushes its record, the same way the deploy ledger is loaded:

just pipeline-sync https://<instance>              # forge results, Cloud Build runs, deliveries.jsonl
just pipeline-sync https://<instance> --dry-run    # print the records, post nothing
just pipeline-sync https://<instance> --since 2026-11-01

scripts/forge/pipeline-sync.sh builds the records with scripts/pipeline_records.py and posts them to POST /api/public/admin/pipeline/runs and …/pipeline/changes (admin only, an admin console session cookie as for just deploy-ledger-sync). Both are upserts on a natural key — forge:<job-id> / cloudbuild:<build-id> for a run, the PR number for a change — so re-running is harmless. With PIPELINE_SYNC_URL set, just forge-check and the other forge commands push each job as it finishes, with the queue and scheduling time only the submitter measures. Deploys stay in deploy_runs (just deploy-ledger-sync).

Panel	What it shows	Source
Changes landed	PRs merged into `next` in the window	`delivery_changes.merged_at`
First commit to production	median and p90 of `first_commit_to_production_s` (deployed changes only)	`delivery_changes`
Push to verdict	median total seconds of forge `check` runs	`pipeline_runs` (kind `check`)
Check pass rate	passing `check` runs over all `check` runs	`pipeline_runs`
Waiting share	median of queue seconds over total seconds, per run	`pipeline_runs` (runs that recorded a queue)
Warm cache	forge runs that started on a warm cache	`pipeline_runs.cache_state`
Where a delivery's time goes	median seconds per stage: edit → PR, PR → merge, merge → release, release → image, image → live	`delivery_changes.*_s`
Recent deliveries	the last ten changes, each a waterfall of its stages, with the runs recorded against its PR; open a run for its phases as spans	`delivery_changes`, `pipeline_runs`, `pipeline_run_phases`
Run duration per day	daily median total per kind, forge kinds split cold / warm; the axis is the days that had runs	`pipeline_runs`
Recent runs	kind, ref or PR, result, queue / run / total, cache, selection, when; a failed run links its log when it is an `https://` URL (Cloud Build), and shows a `gs://` path as text (forge)	`pipeline_runs`
Slowest and failing phases	phases ranked by p90 seconds, and the phases that failed most	`pipeline_run_phases`

Queue is created to started, run is started to finished, total is both. For a forge job "queue" is the lane wait plus pod scheduling plus sidecar start, as the submitter's timing line reports them; a forge run synced later from GCS alone has no queue figure and is left out of the waiting share.