ADR-048: Field-Encryption Key Lifecycle — Devstack/CI Storage, Seeding, Lineage & Activation

D2 concurrency — no advisory lock. The per-DB _seed_marker uses a shell pre-check (seed.sh)
ON_ERROR_STOP on every psql + deterministic-PK fail-loud, not the "advisory lock + post-lock recheck" §D2 sketched. All 34 seeded tables carry an explicit deterministic id UUID PRIMARY KEY and each per-DB file is one atomic BEGIN…COMMIT, so a concurrent/second seeder hits a duplicate-key that ON_ERROR_STOP turns into a loud non-zero exit rolling back the whole transaction — safe by failing, never a silent partial commit. The lock only bought graceful-skip-under-race at the cost of a server-side ERROR logged in all six DBs on every re-run + fragile stderr-sentinel parsing.

D2 marker placement — wrapped, not embedded. seed_marker::wrap_sql_phase injects the marker into the already-rendered transaction (in write_output), so the byte-identity guard (which calls render_*_sql directly) is untouched — no re-bless. The identity’s content_digest is over the plaintext (pre- encryption) renders and stays so under C5; the key enters only via the KCV component.

D1 recreation test — read-only. Automated as a SHOW data_directory assertion (the PG18 path only holds if the named volume is mounted at the parent); the complementary "no stray anonymous parent" docker inspect check is a one-time transition step in Local Development rather than a force-recreating test (which would disrupt a concurrent devstack).

Completion gate split out (CI deferred). The D6/§Consequences "`ON_ERROR_STOP` floor + completion gate ship together" pairing is delivered for local dev by verify_seed on every xtask bring path plus a new verify_seed in auto_refresh (dev refresh). The declarative compose service_completed_successfully gate + the garage-init compose one-shot + CI seed-exit fail-loud are deferred to a follow-up C4-gate (CI is currently unreliable and out of scope). The C4 table’s other items (lineage INSERT, kcv()/canary wiring, D5 provisioning, the verify-seed skeleton) are NOT in C4-floor — they remain in C5/C4d/C7 per the decomposition.

Windows/non-unix strategy (D5 was silent). On cfg(unix) the generated devstack/field-key.env is hardened to 0600 with a parent-dir fsync; on other platforms it is a best-effort atomic write plus a loud "permissions not hardened — use WSL2, or supply the key yourself" warning (user-chosen 2026-07-12). Publication is a temp file + persist_noclobber (no-clobber rename), which is symlink-safe on write and race-safe against a lockless validate→`ensure_ready`; a literal O_NOFOLLOW open is moot because C4d only writes (the read path is C7). An existing file is validated + reused (never re-keyed); loose perms are repaired; a symlink/non-regular target is refused.

CI variable-first + verify-seed deferred out of C4d. CI never runs the local generate-once tooling (--skip-devstack + raw docker compose up), so the D5 CI variable-first delivery would be dead code + a second CI signal — it rides with the deferred C4-gate. The verify-seed subcommand is deferred to C6: the existing verify_seed reads only {{.ExitCode}}, so a still-running one-shot false-succeeds; a correct exposure (exited-success / poll-with-timeout + the lifecycle lock) belongs with C6’s verification harness rather than a placeholder over a known-broken primitive.

D2 identity, realized. write_output’s `_seed_identity content digest is over the plaintext renders (cases with a sentinel lineage row) and compute_identity folds in the real FieldEncryptor::kcv() when keyed (KCV_SENTINEL keyless). The randomized ciphertext (random nonce) + the real crypto_key_lineage row are written to the cases output file only — never the identity — so the identity is stable for a fixed (data, key) yet key-bound (a key change flips it → reseed). The local seed_marker::FORMAT_VERSION dup was dropped in favour of craig_crypto::FORMAT_VERSION_V1.

Lineage matches the as-built migration. The seeder’s INSERT INTO crypto_key_lineage supplies (kcv, kcv_version, canary, envelope_version, algo) — including kcv_version, which the §D3 prose above omits; the migration + its constraint tests are authoritative.

Fail-closed load (D4). CRAIG_SEED__ENCRYPTION_MODE accepts only absent (⇒ optional), optional, required; any other value is a hard error. A malformed/wrong-length key is fatal in both modes (only an absent key is tolerated, and only in optional mode) — deliberately unlike cases' load_encryptor, which swallows a malformed key under optional.

D7. CASES_SHA256 re-blessed once (the ssn_hmac column + the sentinel lineage row shifted the keyless cases bytes; the other five renders are byte-identical); proven by a round-trip proptest, a semantic encrypted-render test (decrypts every covered value from the real render), a plaintext-leak test, and a real-binary key/mode matrix.

Expected-mode oracle, not auto-detect. cargo xtask verify-seed --expect {keyless,keyed} asserts the intended mode. The §D6 text below implies reading the stored state and checking it; the as-built makes the caller declare the mode, because an auto-detecting gate is circular — a silent C7 key-mount failure emits plaintext + the sentinel KCV, which an auto gate passes vacuously. C6 wires --expect keyless (validate devstack phase + the non-blocking pentest CI backstop); C7 flips both to --expect keyed.

Completion poll fixed (the C4d-deferred concern, realized). devstack::verify_seed now polls {{.State}}|{{.ExitCode}} through a checked compose capture + a pure classify_seed, replacing the {{.ExitCode}}-only read that misclassified a still-running one-shot (exit 0) as success and a dead daemon as Ok(""). Both callers (bring_up, auto_refresh) inherit it. Since #1420 an absent container (the exited one-shot pruned by routine docker container prune) is no longer a hard error: the poll falls back to the durable _seed_marker sql-phase row (§D2, "marker present ⟺ committed") and refuses only when container AND marker are both absent — a genuinely unseeded stack.

At-rest coverage via a throwaway DB, not a lift to craig-test-lib. The in-process keyed router runs each test against a per-test craig_cases_enctest_<uuid> database (created + migrated + dropped on Drop), returning its FieldEncryptor — isolating the random key from the shared DB (an isolation improvement over C2’s shared-DB + row-cleanup). Lifting the harness to craig-test-lib was rejected (it would drag five service crates into the shared test lib). An API-created-report test proves the 4 TEXT
3 JSONB report columns encrypt at rest (the seed emits no reports); a decrypt-through-service round-trip checks seeded persons/referrals through the live API against the craig_seed::generate oracle.

Hardened key loader. xtask::field_key::load_field_key (used by the keyed branch; reused by provisioning) rejects symlink/non-regular/loose-perms, requires exactly one key assignment, and is TOCTOU-safe via a stat→open→fstat (dev, ino) inode re-check — the O_NOFOLLOW intent (D5) achieved without a libc/rustix dependency, and now on the read path C4d deferred.

from_env boundary + sentinel home. Two gaps (valid-base64-wrong-length, non-Unicode) added to C5’s real-binary key/mode matrix (absent/valid/malformed already covered). KCV_SENTINEL moved to its canonical home in craig-crypto (value unchanged ⇒ byte-identity stable).

Negative test → C7. The restart-under-wrong-key ⇒ refuse-boot negative test moved to #987 (its boot verifier must exist first); the #986 AC moved with it, so Closes #986 strands nothing.

Field-helper anchors. api/encryption/field.rs (cited in §Context, and in §D6’s pass-through rationale) is deleted: the fail-closed read/write matrix now lives in the registry-driven craig_search::row walk, uniformly typed as SearchError (stored ciphertext with no key → CiphertextWithoutKey, still a hard 500), per the ADR-020 amendment of the same date. §D6’s point survives unchanged — the walk still passes legacy plaintext through on a keyed decrypt (pinned by a craig-search row test), so a decrypt-through GET remains non-proof and the exhaustive raw-DB assertions stay necessary.

§Covered columns is now registry-derived. The covered set is declared once in the craig-cases-fields registries and consumed via craig_search::encrypted_fields() by the service write/decrypt paths, the seeder, and verify-seed — the §Covered-columns prose below stays accurate as a snapshot, but the registries (not this prose) are normative. encrypt_field/ encrypt_jsonb_value are retired; JSONB fields keep the same {"v": "<ct>"} envelope, now emitted by the walk.

The SSN blind index is per-field domain-separated. ssn_hmac moved off the single global BLIND_INDEX_INFO_V1 domain onto hmac_domain("ssn", 1, …) (BLIND_INDEX_SCHEME_V1), derived through the one shared FieldSpec::blind_index at every site (service write, planner query, seeder, verify-seed). Pre-1.0 breaking for keyed data: a keyed deployment reseeds (devstack down -v); C7’s activation reseed subsumes this, and D3’s KCV domain (craig-crypto/kcv/v1) remains disjoint.

The D3 boot matrix, realized in craig_cases::boot_verify with two refinements. (1) Scheme metadata (kcv_version/envelope_version/algo) is checked BEFORE the KCV, so a format/scheme migration refuses with its own error rather than masquerading as a wrong key (KCVs across schemes are incomparable). (2) The "no CGEF-bearing rows" test in the lineage-absent branch is a registry-driven scan (the same craig-cases-fields registries every boundary consults, all five entities): a SQL LIKE prefilter on the new craig_crypto::ENVELOPE_B64_PREFIX (the base64 encoding of the magic’s first three bytes — a necessary condition, pinned to ENVELOPE_MAGIC by test) bounds the fetch, and each candidate is confirmed with the exact structural FieldEncryptor::is_encrypted, so plaintext can never refuse boot. The verifier runs after migrations and before the authz engine/workers/listener. A sentinel lineage row under a keyed boot refuses (as the seeder’s canary-sentinel doc anticipated); a real-KCV row under a keyless boot refuses (hoisting ADR-020’s request-time 500s to a startup refusal).

The D3 host-KCV→wipe guard is a dedicated marker, not a file hash. .devstack/field-key.kcv records the provisioned key’s KCV (never raw key material; the file stays excluded from infra.sha256 per §D5/C4d); absence or mismatch escalates to VolumeAction::Wipe + reseed — the same mechanism a modified migration uses, riding auto_refresh’s bounced (wipe) path, which stays on the safe side of the #1005 marker-ordering bug. Absence-⇒-wipe is what makes the first post-C7 `dev start self-activating (§D4’s coordinated recreation with no manual step).

§D6 keyed flip + the CI half. Both verify-seed call sites now assert --expect keyed. The CI backstop (and every raw-compose CI devstack job) fail-closes on the missing env_file until the deferred C4-gate delivers the §D5 CI variable-first key — red-but-allow_failure, named here rather than hidden. The #986-deferred restart-under-wrong-key negative landed as the boot_verify integration matrix over the C6 throwaway-DB harness (key-A lineage + key-B encryptor ≡ the restart, which a fresh empty stack cannot exercise).

Completion gate. craig-web gates on craig-seed: service_completed_successfully, and the seed gates on a new in-graph garage-init compose one-shot (devstack/garage-init/, alpine
curl/jq against the Garage admin API on garage:3903 — endpoints verified against the live v2.2.0 instance). Any docker compose up — dev or the raw CI jobs — now fails loud on a bad seed ("didn’t complete successfully"), and the seed’s attachment upload can never race an uninitialized bucket: proven live, the attachments marker lands on the FIRST fresh start (pre-gate it needed a second run). The duplicated bootstrap wrappers are retired: xtask’s init_garage (bring_up
auto_refresh) and devstack/ci/garage-init.sh; devstack/ci/wait-healthy.sh is superseded by docker compose up -d --wait --wait-timeout 900 in the three raw-compose CI jobs (the success-direction --wait over completed one-shots verified live).

§D5 CI variable-first key. The three CI jobs materialize devstack/field-key.env from the masked + protected CRAIG_FIELD_ENCRYPTION_KEY project variable via devstack/ci/write-field-key.sh — unset ⇒ hard error naming the fix (never a keyless CI devstack). Protected means a manual perf-stress run on an unprotected ref hard-errors by design. The variable itself is maintainer-provisioned (openssl rand -base64 32; rotate freely — every CI devstack seeds from scratch).

Deliberate-break proof (AC1). A broken seed (non-numeric CRAIG_FAMILIES failing the seeder’s arg parse) makes a raw docker compose up -d --wait exit non-zero with craig-web never started — the gate, exercised for real.

Identity split. --extra-fingerprint now folds only tests/fixtures/ + seed.sh; the rules phase carries its own RULES_FP (sha256 over rulesets/) in the craig_rules rules marker. A ruleset edit flips only the rules marker — the SQL-phase identity (and its wipe posture) is untouched, so a version bump lands on dev refresh/dev start with no data wipe.

Convergent, version-aware load. Per fixture file: GET by-name → absent ⇒ POST create; present at the fixture version ⇒ skip; any other version ⇒ PUT update (JDM recompiled, engine cache swapped, ruleset.changed notified — the SQL fallback did none of these; it is deleted, and craig-seed’s compose gate on a healthy craig-rules means an unreachable API is a real failure). A fixture-vs-DB (name, version) receipt runs after the load; the marker is written only on full convergence, and any failure is fatal (the C4 gate then refuses the stack) — an incomplete authz surface must never present as debuggable-only-via-403s.

Root cause. is_valid_rule_set_name (810) banned , rejecting the system’s own derived authz names ({jurisdiction}-authz-{resource_type}); the charset now admits interior (no AMQP topic semantics — ./*/ stay banned), so every ruleset loads through the validated API path.

New shared-crate + schema surface: FieldEncryptor::kcv() + a fixed canary constant in craig-crypto (C4); a crypto_key_lineage migration (C4); the seeder-emitted lineage INSERT (C5); the craig-cases boot verifier (C7); a verify-seed xtask subcommand (skeleton C4, exhaustive assertions C6/C7). No change to ADR-020’s runtime field-helper semantics.

Encryption-covered seed surface is persons.ssn_last_four (+ a new ssn_hmac) and referrals.reporter_first/last/phone only — the seeder has no reports table, so report TEXT/JSONB encryption is exercised by API-created-report tests (D6), not the seed.

Child re-decomposition (the epic plan carries the authoritative table; summary):

C4/C5 ordering. The ON_ERROR_STOP floor — which turns today’s silent duplicate-key rollback into a hard failure — ships in the same child (C4) as the skip-if-seeded marker that prevents the collision, so no C4→C5 window breaks main.

Pre-1.0 devstack break: first pull after D1 needs a one-time cargo xtask dev clean (documented in C8).

Deferred, named-not-hidden: envelope-encryption / wrapped-DEK rotation, the compose-secrets: production loader, and the full production key lifecycle all move to the secrets-management program; the lineage table and delivery path are shaped so those are additive, not rewrites.

Follow-ups filed (not epic children): the marker-before-reseed ordering bug (devstack_guard.rs:700↔:713); the CRAIG_FAMILIES divergence (compose default 12, docker-compose.yml:883; seed.sh default 9, :16; craig-seed default 9, main.rs:75; the families9 byte-identity fixture); the no-AAD cell-swap limitation (threat-model program); and the seed’s non-fatal attachment/rules-API branches (a seed-robustness gap D6’s floor does not cover).

ADR-020 — fail-closed field-helper semantics + the CGEF ciphertext envelope (the primitive this ADR builds the lifecycle around).

Epic &66 plan — the child decomposition this ADR re-scopes.

#1003 — the .dockerignore build-context secret-leak fix that anticipates devstack/field-key.env.

#982–#988 (C2–C8) — the epic children re-decomposed in Consequences.