Plan: Migration Gate + Verify-Only Boot

On this page

Status

Step Description Status

M0

Docs first: this plan page + ADR-063 + nav entries + epic &78 with children #1305/#1306/#1307/#1308/#1276/#1275/#1309 (+ the #1310 privilege-boundary follow-up, outside the epic). No code changes.

Done (2026-08-02) — this MR

M1

craig-db schema state machine: pure classify (records in, up-only embedded set) → SchemaState with deterministic precedence; verify_schema (read-only, typed refusals) + apply_by_verdict (gate consumer); proptests + scratch-DB matrix; schema_compat_floor create+seed migrations ×8. Closes #1306.

Done (2026-08-02) — this MR

M2

craig-api: phased bootstrap split (bootstrap_data_plane → verify → bootstrap_control_plane); migrate argv mode with typed MigrateModeError, logging-only telemetry, minimal single connection; validate-migration-constraints floor lint. Closes #1307.

Done (2026-08-02) — this MR

MF

xtask ONE typed service registry — fixes cmd/migrate.rs DATABASES silently omitting craig_composition; completeness test with explicit exclusions. Micro-MR, ordered before M3a. Closes #1308.

Done (2026-08-02) — this MR

M3a

Fleet gates, GATE-FIRST: ×8 mains gain the migrate-mode line; boot swaps raw run_migrationsapply_by_verdict (boot still applies when Behind); compose gates ×8 via YAML anchors sharing image:+build:; cargo xtask migrate apply; structural compose invariant + callable-only-from lint; deployment-guide first pass. MR Relates to #1276 (not final).

Done (2026-08-02) — this MR

M3b

Verify-only flip: ×8 boot → verify_schema + floor check; serving binaries never apply DDL; rollback-boundary test (runtime N−1 migrators), upgrade-in-place test, Postgres-only proof ×8, automated poisoned-gate test; CHANGELOG pre-1.0 breaking entry. Closes #1276.

Done (2026-08-03) — this MR

M4

Template pre-warm: explicit 8-base registry wired into cargo xtask test + validate’s nextest phase; lifecycle-lock semantics; cold bounds; parity ×4 + concurrency legs. After M1 only. Closes #1275.

Done (2026-08-03) — this MR

M5

Close-out docs sweep (developer-guide, shared-crates, local-dev, deployment-guide, craig-bootstrap module doc) + plan Status finalization + archive + epic close + Plan Completion Audit. Closes #1309.

Done (2026-08-03) — this MR (deployment-guide + craig-bootstrap slots were already corrected in M3a/M3b; this MR carries developer-guide, shared-crates, local-dev + the archive move). Interleaved fix during M4: #1311 (AdvisoryLease release race, !1168 — surfaced by the M4 battery, fixed on its own branch)

Epic: &78
Issues: #1305 (M0), #1306 (M1), #1307 (M2), #1308 (MF), #1276 (M3a+M3b core), #1275 (M4), #1309 (M5); follow-up #1310 (privilege boundary, outside the epic)
Branches: feature/1276-m0-adr-063, feature/1276-m1-schema-state, feature/1276-m2-migrate-mode, feature/1276-mf-service-registry, feature/1276-m3a-fleet-gates, feature/1276-m3b-verify-only, feature/1275-m4-template-prewarm, feature/1276-m5-closeout
ADR: ADR-063

Context

Every stateful service boots through craig_api::bootstrap() (crates/craig-api/src/bootstrap.rs:148-197) and then applies its embedded migrations inline from its boot orchestrator (craig_db::run_migrations, crates/craig-db/src/lib.rs:281-284 — the #1153 detached-session variant; 8 call sites, e.g. cases src/lib.rs:122, composition src/main.rs:87) before the only TcpListener::bind. Three verified defects, ranked:

  1. Rollback is bricked TODAY. sqlx’s Migrator::run hard-errors with VersionMissing when the database contains versions the binary does not embed — an old binary against a newer database refuses to start. There is no deploy-and-roll-back story at all.

  2. Boot convoy (latent). Migrations run before any health surface exists; replicas convoy on sqlx’s session advisory lock and kill-loop under orchestrator probes.

  3. #1275 (observed). The same defect on the test plane: lazy template migration under 8-way nextest contention — waiters die at the 120 s kill.

The v1 design of this plan was externally reviewed (2026-08-02, ~40 findings, 6 release blockers) and redesigned. The headline flaw: v1’s gate ran run_migrations unconditionally, so a rolled-back (older) gate image would itself VersionMissing against the newer database — the gate re-bricked the rollback it existed to restore. v2 fixes this with one shared schema state machine consumed by BOTH modes, so the old gate no-ops and the old app boots.

Scope

In scope:

  • ONE schema state machine in craig-db (classify / verify_schema / apply_by_verdict).

  • A migrate argv mode on every stateful service binary (same image, one-shot, minimal).

  • Compose migration gates ×8 + cargo xtask migrate apply; staged M3a→M3b transition.

  • The schema_compat_floor table ×8 + the destructive-migration lint.

  • The #1275 template pre-warm (explicit registry, wired into the real entrypoints).

  • The xtask service-registry fix (craig_composition omission).

  • Docs: ADR-063, deployment-guide gate section, close-out sweep.

Out of scope:

  • A DB privilege boundary (migration-owner vs runtime roles) — specced as follow-up #1310; ADR-063 records the honesty note.

  • Quorum/k8s manifests — the deployment guide documents the required shape (parallelism-1 Job or deploy hook); CRAIG ships compose as the reference orchestration.

  • CI-related work (#1189 externally blocked).

Design — the core

The schema state machine (total relation; deterministic precedence)

Input: applied records (version, checksum, success) read with a plain SELECT from _sqlx_migrations — only SQLSTATE 42P01 ("relation does not exist") maps to Behind-from-zero; every other error stays a DB error — plus the embedded set filtered to up-migrations only (sqlx skips down-files when applying, sqlx-core 0.8.6 migrator.rs:169-171 — the classifier filters identically or reversible migrations misclassify).

With extras = applied ∖ embedded and missing = embedded ∖ applied, precedence (first match wins; the embedded set is nonempty by construction — all 8 services embed migrations — so max(embedded) is always defined):

# Condition State

1

any applied row success = false

Dirty { version }

2

any common version with differing checksum

ChecksumMismatch { version }

3

extras ≠ ∅ AND (missing ≠ ∅ OR any extra ≤ max(embedded)) — forked, interleaved, or doubly-incomparable history

Diverged { unknown }

4

missing ≠ ∅ (reachable only when applied ⊊ embedded)

Behind { missing }

5

extras ≠ ∅ (reachable only when embedded ⊊ applied AND all extras strictly > max(embedded))

Ahead { extra }

6

otherwise — sets equal

Exact

Row 3 is deliberately wide: embedded {1,3} vs applied {1,2} is Diverged (interleaved unknown inside the embedded range), and the mirror embedded {1,2} vs applied {1,3} is likewise Diverged (doubly-incomparable — a naive "embedded ∖ applied ≠ ∅ → Behind" rule would classify it Behind and the gate’s apply would then VersionMissing on the unknown 3; sqlx validates EVERY applied migration against the embedded set before applying, sqlx-core 0.8.6 migrator.rs:160-161). The safety consequence: Behind is reachable ONLY when applied ⊊ embedded, so the gate’s apply is VersionMissing-free by construction.

The two consumers

State Gate (migrate mode) Serving boot (verify)

Exact

no-op, exit 0 (log verdict)

boot

Ahead

no-op, exit 0 ← restores rollback

boot + warn! IF the compat floor passes

Behind

apply via run_migrations (safe by construction: Behind ⇒ applied ⊊ embedded ⇒ sqlx cannot VersionMissing)

refuse

Diverged / Dirty / ChecksumMismatch

refuse, exit ≠ 0

refuse

Compatibility floor (direction ≠ compatibility)

Ahead-tolerance alone would let an M-era binary boot against a schema whose CONTRACT migrations dropped structures it uses — the repo already contains an authorized drop (cases 20260802180000_drop_idempotency_responses.sql; CONTRIBUTING.adoc:65 permits removal after deprecation). Fix: a schema_compat_floor table (single row, min_required_version BIGINT, seeded 0 by a new migration ×8, landing in M1). Contract (destructive) migrations MUST bump the floor to their own version in the same migration file; verify-Ahead refuses when floor > max(embedded) ("this binary predates the compatibility floor — roll forward"). Enforcement: the validate-migration-constraints lint flags DROP/destructive statements lacking a same-file floor bump (M2). Rollback support is thereby explicit: to the newest release whose embedded max ≥ the floor.

Refusal ergonomics

SchemaVerifyError / gate refusals are typed thiserror, deployment-NEUTRAL base message ("run this service image’s migration command against this database") + an optional caller-supplied hint (gate_hint: Option<&str> — compose callers pass docker compose up craig-<svc>-migrate; k8s docs pass the Job name). Style contract per ADR-048’s BootVerifyError (boot_verify.rs:40-41). Exit contract: typed error → anyhow chain in main → exit 1 (fleet convention; ADR-063 records that the typed message IS the operator surface — refusal precedes bind, so no health endpoint exists to differentiate on).

Design by unit

M1 — craig-db: the state machine (+ floor migrations ×8)

Files: crates/craig-db/src/schema_state.rs (new module) + lib.rs re-exports; services/*/migrations/<ts>_schema_compat_floor.sql ×8.

  • Pure classify(applied: &[AppliedRecord], embedded: &[EmbeddedMigration]) → SchemaState (records in, not sets — success is part of the input); async wrappers verify_schema(&self, migrator, floor_check, gate_hint) and the gate-side apply_by_verdict(&self, migrator) implementing the two-consumer table.

  • Read-only proof obligations (tests): verify creates NOTHING on a missing table; succeeds under a SELECT-only role; acquires no advisory lock; non-42P01 errors preserved.

  • Proptests (mandatory — universally-quantified comparator): precedence totality (every (applied, embedded) pair yields exactly one state), Ahead ⇒ extras > max(embedded), round-trip Behind→apply→Exact.

  • Scratch-DB matrix legs (harness precedent crates/craig-db/tests/migration_session.rs): exact / absent table / strict subset (Behind) / strict future suffix (Ahead) / incomparable (Diverged both ways) / interleaved unknown (Diverged) / dirty+mismatch precedence / query-failure passthrough / down-migration filtering.

  • Migrations ×8: schema_compat_floor create+seed (same timestamp per service; after each service’s current max version).

M2 — craig-api: phased bootstrap + migrate mode + floor lint

Files: crates/craig-api/src/bootstrap.rs (the split — bootstrap() lives HERE, not in the craig-bootstrap crate, which is authz-extension plumbing), crates/craig-api/src/lib.rs (migrate mode beside print_openapi_if_requested, lib.rs:293-306), xtask/src/cmd/validate.rs (migration-constraints lint).

  • Bootstrap split (v1 verified schema too late — JWKS + MQ init preceded it and could mask the refusal): bootstrap() splits into bootstrap_data_plane(prefix) → {settings, telemetry, db} and bootstrap_control_plane(…​) → {auth, MQ, …}. Services call: data_plane → verify_schema → control_plane. (Mechanical — the fn already does DB at step 4, auth at 7, MQ at 8; bootstrap.rs:148-197.)

  • Migrate mode: pub async fn run_migrate_mode_if_requested(env_var: &str, service: &str, migrator: &Migrator) → Result<bool, MigrateModeError> — typed thiserror (no anyhow at a library pub boundary; PrintOpenapiError precedent, lib.rs:266-276), variants for argv / env / telemetry-init / connect / migrate errors. Semantics:

    • args_os() exact-shape check: [binary, "migrate"] ONLY — extras/conflicts rejected; --print-openapi precedence as built: mains call run_migrate_mode_if_requested FIRST (it does no I/O on the not-this-mode path, so a lone --print-openapi falls through to the untouched openapi mode; migrate --print-openapi together = typed conflict); non-Unicode argv is an error, not a panic.

    • logging-ONLY telemetry (no OTLP exporter path), reading the documented env (RUST_LOG) — the "Postgres-only" claim is scoped honestly: one REQUIRED env var (CRAIG_<SVC>__DATABASE_URL), standard logging envs optional.

    • a MINIMAL single-connection path — as built (recorded deviation from the earlier PgConnection::connect sketch): a 1-connection DbPool::connect_with, NOT the serving-pool helper (connect_database installs pool metrics + a background probe — wrong for a one-shot; its real signature takes four scalars, bootstrap.rs:314-339). Rationale: apply_by_verdict / the #1153 detached-session run_migrations are the tested apply paths; a parallel raw-connection path would duplicate them. Intent kept: no metrics, no probe, one connection.

    • then apply_by_verdict (M1) — the no-op/apply/refuse table above.

  • Floor lint: extend validate-migration-constraints — a destructive statement (DROP TABLE/COLUMN, ALTER … DROP) without a same-file schema_compat_floor bump = error.

MF — xtask: ONE typed service registry (ordered before M3a)

Files: xtask/src/registry.rs (new), xtask/src/dsn.rs, xtask/src/cmd/migrate.rs.

Pre-MF state (fixed by this unit): the correct 8-service inventory lived at xtask/src/dsn.rs:28 (STATEFUL_SERVICES) while cmd/migrate.rs:97-105’s `DATABASES duplicated it MINUS craig_composition (real bug: snapshot/rollback silently skipped a DB). Own fix: issue (#1308), own micro-MR: one typed registry (service, db_name, env_prefix, migrations_dir, gate_name); migrate.rs + dsn.rs consumers derive from it; completeness test pins it against the compose DB inventory with EXPLICIT exclusions (craig_intake — DB exists, intentionally unused per ADR-017; optional IdP databases).

M3a — fleet gates, GATE-FIRST (boot keeps applying, now verdict-driven)

Files: the 8 boot orchestrators (services/*/src/main.rs, except cases + security whose boot lives in src/lib.rs), root docker-compose.yml, xtask/src/cmd/migrate.rs, xtask/src/devstack_guard.rs:474 (reason string), docs/modules/ROOT/pages/deployment-guide.adoc.

Staged transition (v1 deleted boot-apply in the same MR the gates arrived — external deployments on a behind DB would brick; and it blew the 500-LOC rule):

  • ×8 boot orchestrators: the migrate-mode first line + boot swaps raw run_migrationsapply_by_verdict (boot still applies when Behind — the no-op fallback; this alone already un-bricks rollback fleet-wide via Ahead-tolerance). As built: the two non-serving process modes are dispatched through ONE craig-api helper, run_process_modes_if_requested(service, &migrator, &openapi) — the ordering contract (migrate first, openapi second) and the CRAIG_<SVC>__DATABASE_URL derivation live in one place instead of 8 main-copies (also what keeps the fattest boot orchestrators under the B2 function-LOC budget — clippy’s 40-line lint is file-allowed for these bootstrap fns; B2’s 100-line counter is the one that binds).

  • Compose gates ×8 in root docker-compose.yml, YAML anchors sharing image: + build: with the app service (image-only gates could run a STALE :local image and exit 0 silently — anchors + the xtask --build path close that; production uses immutable tags/digests, documented). restart: "no"; depends_on: postgres: service_healthy; env = the one DATABASE_URL var. App services gain service_completed_successfully edges. Note: compose gates are NOT keyless as a class — cases/seed still mount field-key.env in the same compose model; the devstack always has the key via xtask provisioning.

  • cargo xtask migrate apply [--service <name>]: registry-driven; validates names before starting anything; up -d --build locally; bounded polling with container logs on timeout + a kill of the timed-out one-shot (never silently left running); partial-success contract: report per-service, exit ≠ 0 if any failed (additive migrations make partial application safe to retry — recorded in ADR-063 + runbook).

  • Structural compose invariant test (xtask test over docker-compose.yml; as built: yaml-rust2, not serde_yaml — serde_yaml is archived upstream and yaml-rust2 is already in-tree transitively via the config crate; rationale recorded in xtask/Cargo.toml): every registry service has exactly ONE gate with matching image anchor + DB URL + a service_completed_successfully edge on the app; plus a validate lint: run_migrations callable only from craig-db + the migrate-mode helper (so that post-M3b no serving path applies DDL).

  • deployment-guide first pass: gate section + budget arithmetic (gates ×8 + old + surge replica worst case replaces the "boot pool is cold" row at :716/:738-741).

M3b — verify-only flip (after the gates are proven in-graph)

Files: the 8 boot orchestrators (as M3a), CHANGELOG.adoc, new tests.

  • ×8 boot: apply_by_verdictverify_schema (+ floor check). Serving binaries now never apply DDL. CHANGELOG (pre-1.0 breaking): external deployments MUST run the gate (the compose graph carries it automatically; k8s = parallelism-1 Job or deploy hook — an init container is NOT equivalent: per-pod, recreates the convoy; ADR-063 says so). As built: the composition lives in ONE craig-api fn, bootstrap_verified(prefix, service_name, &migrator) — data plane → verify_schema (FloorCheck::Enforce, gate hint derived from the service name) → control plane, so the verify-precedes-JWKS/MQ ordering is structural rather than an 8-copy convention; mains are ONE call. BootstrapError gains the SchemaVerify variant. The isolation lint now also forbids .apply_by_verdict( outside craig-db + the migrate-mode helper (no speculative bootstrap allowance — bootstrap verifies, never applies). The poisoned-gate pin asserts compose’s actual diagnostic (didn’t complete successfully, with the older dependency failed to start wording also accepted).

  • Rollback boundary test (a verifier-tolerance test alone is insufficient): via RUNTIME migrators (Migrator::new(dir), already used in-repo at template_db.rs:291) — apply the FULL set to a scratch DB, then run the N−1 gate (runtime migrator over a truncated dir copy = the old image’s set) → must no-op exit-0 (Ahead), then the N−1 verify → must boot-verdict. Old gate + old app against the newer DB, executable in-battery.

  • Upgrade-in-place test: an existing seeded volume upgraded THROUGH the gates (no wipe/reseed) with data preserved — the real operator path; reseed is not the upgrade story.

  • Postgres-only proof ×8: every service binary’s migrate run with Keycloak/RabbitMQ/ field-key/unrelated settings ABSENT (compose run --no-deps or env-stripped exec): covers missing/invalid URL, exact argv, Ahead no-op, Diverged refusal, concurrent double invocation (advisory-lock serialized, both exit 0).

  • Automated poisoned-gate test (dependent gets "dependency failed to start") — a regression test, not manual MR evidence.

M4 — template pre-warm (#1275)

Files: crates/craig-test-lib/src/template_db.rs (registry export), xtask/src/cmd/test.rs:14, xtask/src/cmd/validate.rs (phase_devstack_and_tests).

  • Explicit registry (no grep heuristics — a grep matches docs/definitions/negative tests). The 8 warmable bases: craig_cases_enctest, craig_security_authztest, craig_security_archtest, craig_security_dlqdedup, craig_security_bkfill, craig_security_conflict, craig_financial_1179, craig_rules_archtest — mapped to the 4 unique migration dirs (cases, security, financial, rules); a registry-completeness check fails on an unregistered ensure_template call site. As built, completeness is RUNTIME-ENFORCED (stronger than the unit-test sketch): ensure_template itself refuses a base not in the exported WARMABLE_BASES registry with the registration instruction, so a new call site fails its own first run — grep-free by construction.

  • Wire into the REAL entrypoints: cargo xtask test (cmd/test.rs:14 — the nextest path) and validate’s phase_devstack_and_tests before the nextest stage. (NOT e2e — that runs Playwright and reseeds AFTER any warm.)

  • Lifecycle semantics: runs under the existing xtask devstack lifecycle lock (no nested locking when called from validate); requires ports reconciled; no down-stack mutation.

  • Bounds: cold total + per-base timeout with diagnostics (the observed failure IS cold creation under contention — warm-state ≤2 s is the secondary bar); the 5 security bases share one migration set — warm once, template-copy where the mechanics allow, else document the 5× cost. As built: 3 min total bound + 1 min per-base warn with per-base timings printed; the security siblings DO template-copy (CREATE … TEMPLATE from the group leader, then ensure’s fast verify) — each unique set applies exactly once.

  • Parity test per unique dir (runtime vs embedded fingerprint ×4, template_db.rs:66-85) + a concurrent post-warm ensure_template + clone-latency leg.

M5 — close-out

Docs sweep riding no feature — developer-guide.adoc:124-125 (boot-migration teaching)
:143 ("seven stateful services"), shared-crates.adoc:143 (run_migrations-only), local-dev.adoc:226 (seven snapshot DBs), deployment-guide.adoc:684 (omits composition, lists stateless intake), craig-bootstrap lib.rs:33-37 module doc; plan Status finalization, epic close, Plan Completion Audit, archive + nav move. Docs otherwise ride their feature units (M1 API docs with M1, pre-warm docs with M4).

Unit DAG and ledger

M0 (docs/ADR/epic) → M1 (craig-db) → M2 (craig-api) → M3a (fleet, gate-first) → M3b (verify-only) → M5
                                MF (registry) ──────────┘         M4 (#1275) — after M1 only
Unit Branch Closes Type Weight ~LOC

M0

feature/1276-m0-adr-063

#1305

docs

2

~650 docs (named docs exception)

M1

feature/1276-m1-schema-state

#1306

feat

5

~500

M2

feature/1276-m2-migrate-mode

#1307

feat

3

~450

MF

feature/1276-mf-service-registry

#1308

fix

1

~150

M3a

feature/1276-m3a-fleet-gates

#1276 (Relates to — not final)

feat

5

~500

M3b

feature/1276-m3b-verify-only

#1276 (Closes)

feat

3

~400

M4

feature/1275-m4-template-prewarm

#1275

fix

3

~450

M5

feature/1276-m5-closeout

#1309

docs

1

~250 docs

Epic &78 "Migration gate + verify-only boot" (Plan::MIGRATION-GATE): children = #1305/#1306/#1307/#1308/#1309 + #1276 (repurposed as the M3a+M3b core — scope comment, the &75/#1181 precedent) + #1275 (M4). Follow-up filed at M0: #1310 (migration-owner vs runtime DB roles — the privilege boundary ADR-063 honestly does not claim).

Verification

  • Full battery + e2e on every MR (standing gates: fmt / clippy / budgets --fail-on-regression / axis / plan-lint / check-docs; J1–J8 subagent review; battery push runs ALONE). Post-M3a AND post-M3b: devstack reseed (the compose graph changed), then the upgrade-in-place test guards the non-reseed path.

  • The schema matrix (M1), rollback boundary + Postgres-only ×8 + poisoned-gate
    upgrade-in-place (M3b), and the pre-warm cold-bound (M4) are the release-gating evidence, each named in its MR.

Documentation Updates

  • ADR-063 — lands at M0

  • deployment-guide gate section + budget arithmetic — M3a; CHANGELOG breaking — M3b

  • CHANGELOG.adoc == Unreleased per code unit

  • Close-out sweep (developer-guide / shared-crates / local-dev / deployment-guide / craig-bootstrap module doc) — M5

Edit this page · latest