Plan: Migration Gate + Verify-Only Boot
On this page
- Status
- Context
- Scope
- Design — the core
- Design by unit
- M1 — craig-db: the state machine (+ floor migrations ×8)
- M2 — craig-api: phased bootstrap + migrate mode + floor lint
- MF — xtask: ONE typed service registry (ordered before M3a)
- M3a — fleet gates, GATE-FIRST (boot keeps applying, now verdict-driven)
- M3b — verify-only flip (after the gates are proven in-graph)
- M4 — template pre-warm (#1275)
- M5 — close-out
- Unit DAG and ledger
- Verification
- Documentation Updates
Status
| Step | Description | Status |
|---|---|---|
M0 |
Docs first: this plan page + ADR-063 + nav entries + epic &78 with children #1305/#1306/#1307/#1308/#1276/#1275/#1309 (+ the #1310 privilege-boundary follow-up, outside the epic). No code changes. |
Done (2026-08-02) — this MR |
M1 |
craig-db schema state machine: pure |
Done (2026-08-02) — this MR |
M2 |
craig-api: phased bootstrap split ( |
Done (2026-08-02) — this MR |
MF |
xtask ONE typed service registry — fixes |
Done (2026-08-02) — this MR |
M3a |
Fleet gates, GATE-FIRST: ×8 mains gain the migrate-mode line; boot swaps raw |
Done (2026-08-02) — this MR |
M3b |
Verify-only flip: ×8 boot → |
Done (2026-08-03) — this MR |
M4 |
Template pre-warm: explicit 8-base registry wired into |
Done (2026-08-03) — this MR |
M5 |
Close-out docs sweep (developer-guide, shared-crates, local-dev, deployment-guide, craig-bootstrap module doc) + plan Status finalization + archive + epic close + Plan Completion Audit. Closes #1309. |
Done (2026-08-03) — this MR (deployment-guide + craig-bootstrap slots were already corrected in M3a/M3b; this MR carries developer-guide, shared-crates, local-dev + the archive move). Interleaved fix during M4: #1311 (AdvisoryLease release race, !1168 — surfaced by the M4 battery, fixed on its own branch) |
Epic: &78
Issues: #1305 (M0), #1306 (M1), #1307 (M2), #1308 (MF), #1276 (M3a+M3b core), #1275 (M4), #1309 (M5); follow-up #1310 (privilege boundary, outside the epic)
Branches: feature/1276-m0-adr-063, feature/1276-m1-schema-state, feature/1276-m2-migrate-mode, feature/1276-mf-service-registry, feature/1276-m3a-fleet-gates, feature/1276-m3b-verify-only, feature/1275-m4-template-prewarm, feature/1276-m5-closeout
ADR: ADR-063
Context
Every stateful service boots through craig_api::bootstrap()
(crates/craig-api/src/bootstrap.rs:148-197) and then applies its embedded migrations
inline from its boot orchestrator (craig_db::run_migrations,
crates/craig-db/src/lib.rs:281-284 — the #1153 detached-session variant; 8 call sites,
e.g. cases src/lib.rs:122, composition src/main.rs:87) before the only
TcpListener::bind. Three verified defects, ranked:
-
Rollback is bricked TODAY. sqlx’s
Migrator::runhard-errors withVersionMissingwhen the database contains versions the binary does not embed — an old binary against a newer database refuses to start. There is no deploy-and-roll-back story at all. -
Boot convoy (latent). Migrations run before any health surface exists; replicas convoy on sqlx’s session advisory lock and kill-loop under orchestrator probes.
-
#1275 (observed). The same defect on the test plane: lazy template migration under 8-way nextest contention — waiters die at the 120 s kill.
The v1 design of this plan was externally reviewed (2026-08-02, ~40 findings, 6 release
blockers) and redesigned. The headline flaw: v1’s gate ran run_migrations unconditionally,
so a rolled-back (older) gate image would itself VersionMissing against the newer database —
the gate re-bricked the rollback it existed to restore. v2 fixes this with one shared
schema state machine consumed by BOTH modes, so the old gate no-ops and the old app boots.
Scope
In scope:
-
ONE schema state machine in craig-db (
classify/verify_schema/apply_by_verdict). -
A
migrateargv mode on every stateful service binary (same image, one-shot, minimal). -
Compose migration gates ×8 +
cargo xtask migrate apply; staged M3a→M3b transition. -
The
schema_compat_floortable ×8 + the destructive-migration lint. -
The #1275 template pre-warm (explicit registry, wired into the real entrypoints).
-
The xtask service-registry fix (
craig_compositionomission). -
Docs: ADR-063, deployment-guide gate section, close-out sweep.
Out of scope:
-
A DB privilege boundary (migration-owner vs runtime roles) — specced as follow-up #1310; ADR-063 records the honesty note.
-
Quorum/k8s manifests — the deployment guide documents the required shape (parallelism-1 Job or deploy hook); CRAIG ships compose as the reference orchestration.
-
CI-related work (#1189 externally blocked).
Design — the core
The schema state machine (total relation; deterministic precedence)
Input: applied records (version, checksum, success) read with a plain SELECT from
_sqlx_migrations — only SQLSTATE 42P01 ("relation does not exist") maps to
Behind-from-zero; every other error stays a DB error — plus the embedded set filtered to
up-migrations only (sqlx skips down-files when applying, sqlx-core 0.8.6
migrator.rs:169-171 — the classifier filters identically or reversible migrations
misclassify).
With extras = applied ∖ embedded and missing = embedded ∖ applied, precedence (first
match wins; the embedded set is nonempty by construction — all 8 services embed
migrations — so max(embedded) is always defined):
| # | Condition | State |
|---|---|---|
1 |
any applied row |
|
2 |
any common version with differing checksum |
|
3 |
|
|
4 |
|
|
5 |
|
|
6 |
otherwise — sets equal |
|
Row 3 is deliberately wide: embedded {1,3} vs applied {1,2} is Diverged (interleaved
unknown inside the embedded range), and the mirror embedded {1,2} vs applied {1,3} is
likewise Diverged (doubly-incomparable — a naive "embedded ∖ applied ≠ ∅ → Behind" rule
would classify it Behind and the gate’s apply would then VersionMissing on the unknown
3; sqlx validates EVERY applied migration against the embedded set before applying,
sqlx-core 0.8.6 migrator.rs:160-161). The safety consequence: Behind is reachable ONLY
when applied ⊊ embedded, so the gate’s apply is VersionMissing-free by construction.
The two consumers
| State | Gate (migrate mode) |
Serving boot (verify) |
|---|---|---|
|
no-op, exit 0 (log verdict) |
boot |
|
no-op, exit 0 ← restores rollback |
boot + |
|
apply via |
refuse |
|
refuse, exit ≠ 0 |
refuse |
Compatibility floor (direction ≠ compatibility)
Ahead-tolerance alone would let an M-era binary boot against a schema whose CONTRACT
migrations dropped structures it uses — the repo already contains an authorized drop
(cases 20260802180000_drop_idempotency_responses.sql; CONTRIBUTING.adoc:65 permits removal
after deprecation). Fix: a schema_compat_floor table (single row,
min_required_version BIGINT, seeded 0 by a new migration ×8, landing in M1).
Contract (destructive) migrations MUST bump the floor to their own version in the same
migration file; verify-Ahead refuses when floor > max(embedded) ("this binary predates
the compatibility floor — roll forward"). Enforcement: the validate-migration-constraints
lint flags DROP/destructive statements lacking a same-file floor bump (M2). Rollback support
is thereby explicit: to the newest release whose embedded max ≥ the floor.
Refusal ergonomics
SchemaVerifyError / gate refusals are typed thiserror, deployment-NEUTRAL base message
("run this service image’s migration command against this database") + an optional
caller-supplied hint (gate_hint: Option<&str> — compose callers pass
docker compose up craig-<svc>-migrate; k8s docs pass the Job name). Style contract per
ADR-048’s BootVerifyError (boot_verify.rs:40-41). Exit contract: typed error → anyhow
chain in main → exit 1 (fleet convention; ADR-063 records that the typed message IS the
operator surface — refusal precedes bind, so no health endpoint exists to differentiate on).
Design by unit
M1 — craig-db: the state machine (+ floor migrations ×8)
Files: crates/craig-db/src/schema_state.rs (new module) + lib.rs re-exports;
services/*/migrations/<ts>_schema_compat_floor.sql ×8.
-
Pure
classify(applied: &[AppliedRecord], embedded: &[EmbeddedMigration]) → SchemaState(records in, not sets —successis part of the input); async wrappersverify_schema(&self, migrator, floor_check, gate_hint)and the gate-sideapply_by_verdict(&self, migrator)implementing the two-consumer table. -
Read-only proof obligations (tests): verify creates NOTHING on a missing table; succeeds under a SELECT-only role; acquires no advisory lock; non-42P01 errors preserved.
-
Proptests (mandatory — universally-quantified comparator): precedence totality (every
(applied, embedded)pair yields exactly one state), Ahead ⇒ extras > max(embedded), round-trip Behind→apply→Exact. -
Scratch-DB matrix legs (harness precedent
crates/craig-db/tests/migration_session.rs): exact / absent table / strict subset (Behind) / strict future suffix (Ahead) / incomparable (Diverged both ways) / interleaved unknown (Diverged) / dirty+mismatch precedence / query-failure passthrough / down-migration filtering. -
Migrations ×8:
schema_compat_floorcreate+seed (same timestamp per service; after each service’s current max version).
M2 — craig-api: phased bootstrap + migrate mode + floor lint
Files: crates/craig-api/src/bootstrap.rs (the split — bootstrap() lives HERE, not in
the craig-bootstrap crate, which is authz-extension plumbing), crates/craig-api/src/lib.rs
(migrate mode beside print_openapi_if_requested, lib.rs:293-306),
xtask/src/cmd/validate.rs (migration-constraints lint).
-
Bootstrap split (v1 verified schema too late — JWKS + MQ init preceded it and could mask the refusal):
bootstrap()splits intobootstrap_data_plane(prefix)→ {settings, telemetry, db} andbootstrap_control_plane(…)→ {auth, MQ, …}. Services call: data_plane →verify_schema→ control_plane. (Mechanical — the fn already does DB at step 4, auth at 7, MQ at 8;bootstrap.rs:148-197.) -
Migrate mode:
pub async fn run_migrate_mode_if_requested(env_var: &str, service: &str, migrator: &Migrator) → Result<bool, MigrateModeError>— typed thiserror (no anyhow at a library pub boundary;PrintOpenapiErrorprecedent, lib.rs:266-276), variants for argv / env / telemetry-init / connect / migrate errors. Semantics:-
args_os()exact-shape check:[binary, "migrate"]ONLY — extras/conflicts rejected;--print-openapiprecedence as built: mains callrun_migrate_mode_if_requestedFIRST (it does no I/O on the not-this-mode path, so a lone--print-openapifalls through to the untouched openapi mode;migrate --print-openapitogether = typed conflict); non-Unicode argv is an error, not a panic. -
logging-ONLY telemetry (no OTLP exporter path), reading the documented env (
RUST_LOG) — the "Postgres-only" claim is scoped honestly: one REQUIRED env var (CRAIG_<SVC>__DATABASE_URL), standard logging envs optional. -
a MINIMAL single-connection path — as built (recorded deviation from the earlier
PgConnection::connectsketch): a 1-connectionDbPool::connect_with, NOT the serving-pool helper (connect_databaseinstalls pool metrics + a background probe — wrong for a one-shot; its real signature takes four scalars,bootstrap.rs:314-339). Rationale:apply_by_verdict/ the #1153 detached-sessionrun_migrationsare the tested apply paths; a parallel raw-connection path would duplicate them. Intent kept: no metrics, no probe, one connection. -
then
apply_by_verdict(M1) — the no-op/apply/refuse table above.
-
-
Floor lint: extend
validate-migration-constraints— a destructive statement (DROP TABLE/COLUMN, ALTER … DROP) without a same-fileschema_compat_floorbump = error.
MF — xtask: ONE typed service registry (ordered before M3a)
Files: xtask/src/registry.rs (new), xtask/src/dsn.rs, xtask/src/cmd/migrate.rs.
Pre-MF state (fixed by this unit): the correct 8-service inventory lived at
xtask/src/dsn.rs:28 (STATEFUL_SERVICES) while cmd/migrate.rs:97-105’s `DATABASES
duplicated it MINUS craig_composition (real bug: snapshot/rollback silently skipped a
DB). Own fix: issue (#1308), own micro-MR: one typed registry
(service, db_name, env_prefix, migrations_dir, gate_name); migrate.rs + dsn.rs
consumers derive from it; completeness test pins it against the compose DB inventory with
EXPLICIT exclusions (craig_intake — DB exists, intentionally unused per ADR-017; optional
IdP databases).
M3a — fleet gates, GATE-FIRST (boot keeps applying, now verdict-driven)
Files: the 8 boot orchestrators (services/*/src/main.rs, except cases + security whose
boot lives in src/lib.rs), root docker-compose.yml, xtask/src/cmd/migrate.rs,
xtask/src/devstack_guard.rs:474 (reason string),
docs/modules/ROOT/pages/deployment-guide.adoc.
Staged transition (v1 deleted boot-apply in the same MR the gates arrived — external deployments on a behind DB would brick; and it blew the 500-LOC rule):
-
×8 boot orchestrators: the migrate-mode first line + boot swaps raw
run_migrations→apply_by_verdict(boot still applies when Behind — the no-op fallback; this alone already un-bricks rollback fleet-wide via Ahead-tolerance). As built: the two non-serving process modes are dispatched through ONE craig-api helper,run_process_modes_if_requested(service, &migrator, &openapi)— the ordering contract (migrate first, openapi second) and theCRAIG_<SVC>__DATABASE_URLderivation live in one place instead of 8 main-copies (also what keeps the fattest boot orchestrators under the B2 function-LOC budget — clippy’s 40-line lint is file-allowed for these bootstrap fns; B2’s 100-line counter is the one that binds). -
Compose gates ×8 in root
docker-compose.yml, YAML anchors sharingimage:+build:with the app service (image-only gates could run a STALE:localimage and exit 0 silently — anchors + the xtask--buildpath close that; production uses immutable tags/digests, documented).restart: "no";depends_on: postgres: service_healthy; env = the one DATABASE_URL var. App services gainservice_completed_successfullyedges. Note: compose gates are NOT keyless as a class — cases/seed still mountfield-key.envin the same compose model; the devstack always has the key via xtask provisioning. -
cargo xtask migrate apply [--service <name>]: registry-driven; validates names before starting anything;up -d --buildlocally; bounded polling with container logs on timeout + a kill of the timed-out one-shot (never silently left running); partial-success contract: report per-service, exit ≠ 0 if any failed (additive migrations make partial application safe to retry — recorded in ADR-063 + runbook). -
Structural compose invariant test (xtask test over
docker-compose.yml; as built: yaml-rust2, not serde_yaml — serde_yaml is archived upstream and yaml-rust2 is already in-tree transitively via theconfigcrate; rationale recorded inxtask/Cargo.toml): every registry service has exactly ONE gate with matching image anchor + DB URL + aservice_completed_successfullyedge on the app; plus a validate lint:run_migrationscallable only from craig-db + the migrate-mode helper (so that post-M3b no serving path applies DDL). -
deployment-guide first pass: gate section + budget arithmetic (gates ×8 + old + surge replica worst case replaces the "boot pool is cold" row at :716/:738-741).
M3b — verify-only flip (after the gates are proven in-graph)
Files: the 8 boot orchestrators (as M3a), CHANGELOG.adoc, new tests.
-
×8 boot:
apply_by_verdict→verify_schema(+ floor check). Serving binaries now never apply DDL. CHANGELOG (pre-1.0 breaking): external deployments MUST run the gate (the compose graph carries it automatically; k8s = parallelism-1 Job or deploy hook — an init container is NOT equivalent: per-pod, recreates the convoy; ADR-063 says so). As built: the composition lives in ONE craig-api fn,bootstrap_verified(prefix, service_name, &migrator)— data plane →verify_schema(FloorCheck::Enforce, gate hint derived from the service name) → control plane, so the verify-precedes-JWKS/MQ ordering is structural rather than an 8-copy convention; mains are ONE call.BootstrapErrorgains theSchemaVerifyvariant. The isolation lint now also forbids.apply_by_verdict(outside craig-db + the migrate-mode helper (no speculative bootstrap allowance — bootstrap verifies, never applies). The poisoned-gate pin asserts compose’s actual diagnostic (didn’t complete successfully, with the olderdependency failed to startwording also accepted). -
Rollback boundary test (a verifier-tolerance test alone is insufficient): via RUNTIME migrators (
Migrator::new(dir), already used in-repo attemplate_db.rs:291) — apply the FULL set to a scratch DB, then run the N−1 gate (runtime migrator over a truncated dir copy = the old image’s set) → must no-op exit-0 (Ahead), then the N−1 verify → must boot-verdict. Old gate + old app against the newer DB, executable in-battery. -
Upgrade-in-place test: an existing seeded volume upgraded THROUGH the gates (no wipe/reseed) with data preserved — the real operator path; reseed is not the upgrade story.
-
Postgres-only proof ×8: every service binary’s
migraterun with Keycloak/RabbitMQ/ field-key/unrelated settings ABSENT (composerun --no-depsor env-stripped exec): covers missing/invalid URL, exact argv, Ahead no-op, Diverged refusal, concurrent double invocation (advisory-lock serialized, both exit 0). -
Automated poisoned-gate test (dependent gets "dependency failed to start") — a regression test, not manual MR evidence.
M4 — template pre-warm (#1275)
Files: crates/craig-test-lib/src/template_db.rs (registry export),
xtask/src/cmd/test.rs:14, xtask/src/cmd/validate.rs (phase_devstack_and_tests).
-
Explicit registry (no grep heuristics — a grep matches docs/definitions/negative tests). The 8 warmable bases:
craig_cases_enctest,craig_security_authztest,craig_security_archtest,craig_security_dlqdedup,craig_security_bkfill,craig_security_conflict,craig_financial_1179,craig_rules_archtest— mapped to the 4 unique migration dirs (cases, security, financial, rules); a registry-completeness check fails on an unregisteredensure_templatecall site. As built, completeness is RUNTIME-ENFORCED (stronger than the unit-test sketch):ensure_templateitself refuses a base not in the exportedWARMABLE_BASESregistry with the registration instruction, so a new call site fails its own first run — grep-free by construction. -
Wire into the REAL entrypoints:
cargo xtask test(cmd/test.rs:14— the nextest path) and validate’sphase_devstack_and_testsbefore the nextest stage. (NOT e2e — that runs Playwright and reseeds AFTER any warm.) -
Lifecycle semantics: runs under the existing xtask devstack lifecycle lock (no nested locking when called from validate); requires ports reconciled; no down-stack mutation.
-
Bounds: cold total + per-base timeout with diagnostics (the observed failure IS cold creation under contention — warm-state ≤2 s is the secondary bar); the 5 security bases share one migration set — warm once, template-copy where the mechanics allow, else document the 5× cost. As built: 3 min total bound + 1 min per-base warn with per-base timings printed; the security siblings DO template-copy (
CREATE … TEMPLATEfrom the group leader, then ensure’s fast verify) — each unique set applies exactly once. -
Parity test per unique dir (runtime vs embedded fingerprint ×4,
template_db.rs:66-85) + a concurrent post-warmensure_template+ clone-latency leg.
M5 — close-out
Docs sweep riding no feature — developer-guide.adoc:124-125 (boot-migration teaching)
:143 ("seven stateful services"), shared-crates.adoc:143 (run_migrations-only),
local-dev.adoc:226 (seven snapshot DBs), deployment-guide.adoc:684 (omits composition,
lists stateless intake), craig-bootstrap lib.rs:33-37 module doc; plan Status
finalization, epic close, Plan Completion Audit, archive + nav move. Docs otherwise ride
their feature units (M1 API docs with M1, pre-warm docs with M4).
Unit DAG and ledger
M0 (docs/ADR/epic) → M1 (craig-db) → M2 (craig-api) → M3a (fleet, gate-first) → M3b (verify-only) → M5
MF (registry) ──────────┘ M4 (#1275) — after M1 only
| Unit | Branch | Closes | Type | Weight | ~LOC |
|---|---|---|---|---|---|
M0 |
|
#1305 |
docs |
2 |
~650 docs (named docs exception) |
M1 |
|
#1306 |
feat |
5 |
~500 |
M2 |
|
#1307 |
feat |
3 |
~450 |
MF |
|
#1308 |
fix |
1 |
~150 |
M3a |
|
#1276 ( |
feat |
5 |
~500 |
M3b |
|
#1276 (Closes) |
feat |
3 |
~400 |
M4 |
|
#1275 |
fix |
3 |
~450 |
M5 |
|
#1309 |
docs |
1 |
~250 docs |
Epic &78 "Migration gate + verify-only boot" (Plan::MIGRATION-GATE): children =
#1305/#1306/#1307/#1308/#1309 + #1276 (repurposed as the M3a+M3b core — scope comment, the
&75/#1181 precedent) + #1275 (M4). Follow-up filed at M0: #1310 (migration-owner vs runtime
DB roles — the privilege boundary ADR-063 honestly does not claim).
Verification
-
Full battery + e2e on every MR (standing gates: fmt / clippy / budgets
--fail-on-regression/ axis / plan-lint / check-docs; J1–J8 subagent review; battery push runs ALONE). Post-M3a AND post-M3b: devstack reseed (the compose graph changed), then the upgrade-in-place test guards the non-reseed path. -
The schema matrix (M1), rollback boundary + Postgres-only ×8 + poisoned-gate
upgrade-in-place (M3b), and the pre-warm cold-bound (M4) are the release-gating evidence, each named in its MR.
Documentation Updates
-
ADR-063 — lands at M0
-
deployment-guide gate section + budget arithmetic — M3a; CHANGELOG breaking — M3b
-
CHANGELOG.adoc== Unreleasedper code unit -
Close-out sweep (developer-guide / shared-crates / local-dev / deployment-guide / craig-bootstrap module doc) — M5