Plan: Transactional Idempotency Claims — Inbox + HTTP Ingress

On this page

Status

Unit Description Status

U0

ADR-062 + this plan page + nav; api-idempotency.adoc archived

Done (2026-07-31) — !1122

A1

craig-mq transactional attempt machine (#1053; closes #1209)

Done (2026-07-31) — !1123

A2

Convert placement + exchange inbox handlers

Done (2026-07-31) — !1124

A3

Convert security (audit + ledger atomic)

Done (2026-07-31) — !1125

A4

Convert financial (money path; S2S bound; timeout proof)

Done (2026-07-31) — !1126

A5

Convert rules (tx through engine.evaluate audit)

Done (2026-07-31) — !1127

A6

Convert reporting (one tx/envelope) + delete legacy API

Done (2026-07-31) — !1128

B1a

request_claims module + IntentV1 + migrations ×8

Done (2026-07-31) — !1129

B1b

Claims battery (proptests, collect-all helper, synthetic endpoint)

Done (2026-07-31) — !1130

B2

Delete the idempotency middleware; install claims retention (#1194)

Done (2026-07-31) — !1131

SU1

upload_attempts state machine + reconciler (craig-store)

Done (2026-08-02) — !1146

SU2

upload_attempts migrations ×4 + target columns

Done (2026-08-02) — !1147

SU3

cases attachments + court-order document on attempts

Done (2026-08-02) — !1148

SU4

exchange ICPC attachments on attempts

Done (2026-08-02) — !1149

SU5

placement home documents reserve-first + first reconciler worker

Done (2026-08-02) — !1150 merged b72e1c39; #1251 closed

SU6

reporting deterministic exports on attempts

Done (2026-08-02) — !1151 merged 533c159c; #1252 closed

SU7

upload-attempt race matrix

Done (2026-08-02) — !1152 merged 14eacc1e; #1253 closed

SU8

retire AttachmentScanner entity targets

Done (2026-08-02) — !1153 merged 79ddd9cb; #1254 closed

Bc1

cases claims: persons/referrals/allegations/investigations/safety

Done (2026-08-01) — !1132

Bc2

cases claims: case/household/plans/tasks/contacts/court-orders

Done (2026-08-01) — !1134

Bc3a

cases claims: reports family + create_report intent hash

Done (2026-08-01) — !1135

Bc3b

intake client-held id protocol + SDK submit fields

Done (2026-08-01) — !1136

Bp1

placement claims: six creates

Done (2026-08-01) — !1139

Be1

exchange claims: five converts + home-study UNIQUE

Done (2026-08-01) — !1140

Bs1

security claims: six creates + app-minted id refactor

Done (2026-08-01) — !1141

Bs2

one-time-secret pair columns (issue/rotate key)

Done (2026-08-01) — !1142

Bf1

financial claims: four creates + sweep-preview claim

Done (2026-08-01) — !1143

Br1

reporting claims: AFCARS/NCANDS generates

Done (2026-08-01) — !1144

Bru1

rules claims: rule-set create + import wrapper

Done (2026-08-02) — !1145

G1

cases replay guards (SSN lifecycle + CAS/IDF/SUP sweep)

Done (2026-08-02) — !1155 merged 7efb032a; #1266 closed

G2

placement + exchange replay guards

Done (2026-08-02) — !1156 merged 9e429adb; #1267 closed

G3

security/reporting/rules/financial replay guards

Done (2026-08-02) — !1157 merged 67aa1e7c; #1268 closed

G4

convert_report auto-link resumable via outbox consumer

Done (2026-08-02) — !1158 merged 7b7dabcf; #1269 closed

BF

DROP idempotency_responses ×8 + close-out gates

Done (2026-08-02) — !1159 merged f0340e78; #1270 closed; anchors #1178 + #1182 closed with track-completion comments

UF

Program aggregation (CLAUDE.md, scorecard, known-issues, archive)

Done (2026-08-02) — this aggregation/archive MR

Epic: &77 (child of &74)
Anchors: #1178 (Track A) · #1182 (Track B) — close manually at track completion
Unit issues: U0 #1239 · A1 #1053 · A2 #1240 · A3 #1241 · A4 #1242 · A5 #1243 · A6 #1244 · B1a #1245 · B1b #1246 · B2 #1194 · SU1 #1247 · SU2 #1248 · SU3 #1249 · SU4 #1250 · SU5 #1251 · SU6 #1252 · SU7 #1253 · SU8 #1254 · Bc1 #1255 · Bc2 #1256 · Bc3a #1257 · Bc3b #1258 · Bp1 #1259 · Be1 #1260 · Bs1 #1261 · Bs2 #1262 · Bf1 #1263 · Br1 #1264 · Bru1 #1265 · G1 #1266 · G2 #1267 · G3 #1268 · G4 #1269 · BF #1270 · UF #1271
Follow-ups: #1272 (typed rejection mapper) · #1273 (payload_hash NOT NULL) · #1274 (orphan-blob janitor)
ADR: ADR-062

Context

Two platform primitives promised more than they delivered:

  • #1178craig_mq::inbox::handle_idempotently (crates/craig-mq/src/inbox.rs:153) commits its claim, the handler’s effects, and processed_at as three separate autocommit statements, with an unlocked redelivery read. Real semantics: at-least-once with idempotency delegated per handler — which reporting (quality issues: fresh UUIDv7, no natural key) and security (plain audit insert) lack.

  • #1182 — the HTTP idempotency middleware (crates/craig-api/src/idempotency/) is a lease-based, unfenced 30 s claim, carrying the #1194 500-window — and is opt-in with ZERO senders (BFF, CLI, SDKs, e2e, k6 never send idempotency-key).

  • The 2026-07-31 handler-by-handler audit (two independent contextless explorers; verdicts in §R) found 30 non-create routes with stale-replay hazards (including two narrow/mild variants), 4 event-dup-only routes, one TOCTOU race (update_icpc_request), and `convert_report’s post-commit auto-link retry-LOST.

F8 (ADR-055) is the proven model being generalized: client key + canonical hash + partial unique index + claim inside the ONE domain tx; replay re-derives from domain state.

User steers (2026-07-31, fixed): #1178 = transaction-aware handler API; #1182 = "the architecturally correct answer — no shortcuts, no fig leafs, no backwards compat pre 1.0".

Plan review history: v1 externally blocked (~35 findings) → v2 (all folded) → four-lens contextless workflow (28 findings) → v2.1 → second external block (six approvability conditions) → v3 approved 2026-07-31.

Doctrine and horizons

Atomic effects + bounded-horizon deduplication + convergent external effects. Never unqualified "exactly-once".

Boundary Guarantee Horizon

MQ inbox

atomic claim+effects+stamp; hard cap over counted attempts

31 d

HTTP convert-class

atomic claim+effects; replay = effects-dedup (responses re-derived)

30 d knob

One-time secrets, create_report

per-entity pair columns

entity lifetime

§U uploads/exports

atomic claim+attempt; puts converge by digest+generation

31 d attempts

Intake→SHINES (all SHINES-mode submissions)

at-least-once (no receiving identity; ADR-042 boundary)

§G-hardened CAS routes

SQL-guarded state machine + gated events

EXEMPT/LEASED (§R)

per-route characterization (drain-by-design, arbiter-converged)

Track A design — the transactional inbox

A-1 Single-transaction attempt, savepoint accounting

One dedicated pooled connection per attempt; the whole attempt inside the #1203 contain task:

acquire conn → BEGIN            (claim/lock-wait under the DEFAULT 30 s statement timeout)
  claim: INSERT event_inbox (envelope_id, source_service, event_type, payload_hash,
         attempt_started_at) VALUES … ON CONFLICT (envelope_id) DO NOTHING RETURNING received_at
  ├─ claimed → SET LOCAL statement_timeout='5s'        (only AFTER the lock is held)
  │    SAVEPOINT attempt
  │    caught = catch_unwind(AssertUnwindSafe(async { handler(envelope, &mut tx).await }))
  │            under tokio ATTEMPT_DEADLINE = 15 s
  │    ├─ Ok(Ok) → RELEASE; UPDATE processed_at = now(),
  │    │           payload_hash = COALESCE(payload_hash, $h); COMMIT
  │    └─ Ok(Err) | caught panic | deadline → ROLLBACK TO SAVEPOINT;
  │         UPDATE error_count = error_count+1, last_error = $e; COMMIT
  │         if error_count == 5 → DLX surface (A-3) → stamp failed_at (WHERE failed_at IS NULL)
  └─ conflict → SELECT … FOR UPDATE:
       (source_service, event_type, payload_hash [skip if stored NULL]) ≠ computed
                                  → COMMIT; collision path (A-4) — canonical row NEVER touched
       processed_at / failed_at   → COMMIT; dedup
       error_count >= 5           → COMMIT FIRST, then re-surface OUTSIDE any tx
       else retry                 → SET LOCAL 5s; UPDATE attempt_started_at; SAVEPOINT; handler
  • The claim row precedes the savepoint ⇒ survives handler rollback ⇒ failure accounting commits in the SAME tx under the held lock ⇒ hard cap over counted attempts. Cap statement (used everywhere): at most 5 COUNTED attempts; crash-class failures (process crash, connection loss, task abort, tier-2 panics, accounting-commit failure) are uncounted and can add attempts — the cap bounds counted work, broker/replay cadence bounds rate.

  • Timeout hierarchy: claim lock-wait = default 30 s (no SET LOCAL before the claim); handler statements = SET LOCAL 5 s after locking; tokio deadline 15 s. Worst winner hold ≈ 15 + ≤5 drain + accounting ms ≈ ~21 s < the 30 s waiter budget (waiters ≈ replica count; consume loops are serial per queue). Deadline path: timeout → drop handler future (clean drop → rollback to savepoint → accounting InboxError::AttemptTimeout, COUNTED; Drop panic → tier 2). The slow-winner test drives a pg_sleep STATEMENT.

  • Panic tiers: Tier 1 COUNTED — construction + poll panics (the handler CALL sits inside the caught async block). AssertUnwindSafe honesty: the savepoint discards DB state; non-DB captured state carries the SAME exposure the existing #1203 boundary already accepts (subscriber.rs:1861). Tier 2 UNCOUNTED crash-class — drop-of-future + error-Display panics escape to the task boundary → whole-tx rollback.

  • Identity = (source_service, event_type, payload_hash); payload_hash = ONE shared fn sha256(serde_json::to_vec(&envelope.payload)) (stable — publishes round-trip the same outbox JSONB). Additive migration ×8; NULL legacy rows skip the hash leg (backfilled at the next stamp); post-horizon NOT NULL = #1273.

  • Handler: for<'t> FnOnce(EventEnvelope, &'t mut Transaction<'static, Postgres>) → BoxFuture<'t, Result<(), E>> + Send + 'static; futures = { workspace = true } added to craig-mq. The old fn is renamed handle_idempotently_at_least_once at its 6 call sites in A1 (loud rename, not a compat surface) and deleted with its last caller in A6.

  • Cutover barrier: A2–A6 deploy stop-then-start per service (compose Recreate; single-replica devstack is the only pre-1.0 deployment) — old-autocommit and new-locking handlers never run concurrently. Recorded in the deployment guide with a k8s Recreate note.

A-2 Failure & attempt-consumption table

Failure point Counted? Mechanism

Pre-claim db error

No

nack; broker paces

Waiter’s claim statement-times-out (30 s)

No

pre-claim class; winner’s ≈21 s worst hold makes it rare

Handler Err / caught panic (construction or poll)

Yes

savepoint rollback + same-tx accounting

ATTEMPT_DEADLINE or SET LOCAL 5 s statement kill

Yes

InboxError::AttemptTimeout handler-Err path

Drop-of-future / Display-render panic

No (crash-class)

escapes to the task boundary; whole-tx rollback

Stamp/COMMIT error (unknown outcome)

No

claim row arbitrates on redelivery

Accounting COMMIT error

No

row state arbitrates

DLX surface fails / unroutable

n/a

requeue-durable (A-3); failed_at NOT stamped

Process crash mid-attempt

No

whole-tx rollback; one increment lost

A-3 DLX surface — mandatory, deterministic, requeue-durable

  • publish_dlx → mandatory + basic.return verdict (the capture-publisher shape, subscriber.rs:1689). The wrapper is built ENTIRELY from durable row state: occurred_at := stored received_at (publisher.rs:45 change), retry_count := error_count, last_error := stored; envelope = delivery bytes ⇒ repeat/cross-replica surfaces are BYTE-IDENTICAL ⇒ the security consumer’s full-JSON equivalence absorbs them. Token unchanged: dlxcap:{envelope_id}:{consumer_queue}:{received_at}. Publish with NO tx open (an AMQP confirm await must never hold a row lock); the cross-replica double-surface race is accepted and absorbed.

  • Unroutable/failed surface ⇒ nack requeue: TRUE (a typed SurfaceFailed outcome bypasses the 2-strike reject): the source-queue copy IS the durable envelope; the row stays error_count=5, failed_at NULL; every redelivery retries the surface via the ≥5 branch; the beyond-grace watchdog alerts. The missing-binding devstack test asserts failed_at-NULL + watchdog + eventual replay success — NOT a broker-DLX landing (the broker path shares the same exchange/key and would drop it too).

A-4 Collision path (never mutates the canonical row)

event_inbox_collisions (envelope_id, observed_source, observed_type, observed_payload_hash, first_seen, occurrences) ×8 same-timestamp migration; deterministic token dlxcol:{envelope_id}:{consumer_queue}:{observed_payload_hash}; a collision-typed DLX record; the colliding delivery nacks requeue:false; a watchdog over the table. The canonical event’s processed/failed state and token are NEVER touched or reused.

A-5 Retry policy (folds #1053; fixes #1209)

Pacing = broker 2-strike + ADR-022 §D4 admin replay (the #1197 parking tier paces the DLQ consumer only). Budget = error_count across cycles, terminal at 5 after a CONFIRMED surface. Operator escape (runbook, conditional — never a raw delete): DELETE FROM event_inbox WHERE envelope_id = $1 AND failed_at IS NOT NULL + audit note + replay step. inbox_unprocessed_beyond_grace.sql SQL fixed (excludes failed_at IS NOT NULL), the tolerating test replaced — this IS #1209, closed by A1 (justified same-MR: the invariant’s meaning is defined by A1’s semantics). The admin-replay at-least-once contract now rests on exactly the handler atomicity Track A ships — closure recorded in ADR-062.

Track B design — HTTP claims

B-1 request_claims + per-endpoint IntentV1

Schema (×8 DB-owning services, same-timestamp copies):

CREATE TABLE request_claims (
    client_request_id uuid PRIMARY KEY,
    claim_scope       text        NOT NULL CHECK (claim_scope <> ''),
    intent_hash       text        NOT NULL CHECK (intent_hash ~ '^v1:[0-9a-f]{64}$'),
    entity_kind       text        NOT NULL CHECK (entity_kind <> ''),
    entity_id         uuid        NOT NULL,
    created_by_sub    uuid        NOT NULL,   -- EFFECTIVE actor
    created_at        timestamptz NOT NULL DEFAULT now()
);
CREATE INDEX request_claims_created_at_idx ON request_claims (created_at);
  • Module crates/craig-api/src/request_claims.rs; newtypes ClientRequestId(Uuid), ClaimScope(&'static str), IntentHash(String), EntityKind(&'static str).

  • IntentV1 = per-endpoint versioned immutable projection structs — never raw DTO serialization; DTO evolution adds IntentV2 + dual-decode across the horizon. Envelope: {v, scope, jurisdiction, actor, path: [(name, value)], body: &XxxIntentV1}"v1:" + hex(sha256(serde_json::to_vec(…))); zero serde_json::Value. Normalization pinned in B1a: UUIDs lowercase-hyphenated; strings byte-as-submitted (no unicode normalization, documented); money rescale(2); None ⇒ null; no default injection.

  • actor = parse_sub(claims.acting_worker().sub) — the effective actor (BFF impersonation ⇒ the worker), reusing the fleet’s typed-400 invalid-sub path (security partners.rs:473 precedent).

  • claim_first = the FIRST statement of the domain tx, ONE locking statement: INSERT … ON CONFLICT (client_request_id) DO UPDATE SET client_request_id = EXCLUDED.client_request_id RETURNING * (row lock + existing row in one statement — no SELECT window for the pruner). Classification NARROW: scope/actor/entity_kind/intent mismatch ⇒ 409 IDEMPOTENCY_CONFLICT (bodies identical — no existence leak); Replay { entity_id } only when ALL match. precheck = an unlocked fast-path optimization before volatile deps, never the guard. Replay re-runs CURRENT entity authorization, then re-derives via the endpoint’s own detail projection; entity vanished ⇒ typed 409 REPLAY_ENTITY_GONE. Replay returns each endpoint’s OWN success shape (send_exchange ⇒ 202).

  • App-minted ids refactor floor: store create fns gain an explicit UUIDv7 id param + a tx-capable executor (security’s six DB-default gen_random_uuid() INSERTs; cases contacts.rs:20 / placement foster_homes.rs:45 pool-bound mints).

  • Missing client_request_id ⇒ stock axum 422 (JSON) / 400 (multipart); the typed-rejection mapper is #1272. Response statuses otherwise unchanged.

  • Retention: CRAIGREQUEST_CLAIMSWINDOW_DAYS (platform prefix; default 30; 0 ⇒ Disabled); hourly bounded-batch (5 k) advisory-locked pruner request-claims-retention in the vacated worker slot (crates/craig-api/src/lib.rs:442); a beyond-grace watchdog replaces the deleted idempotency_expired_not_cleaned invariant. Composition keeps table + worker (wiring/exact-set uniformity; empty table, no-op prunes). F8 pair columns grandfathered.

  • Client-ID lifecycle: browser — minted at form render, hidden field, the SUBMITTED id re-emitted on validation-error rerenders (never re-minted); CLI — auto-mint per invocation + --request-id for scripted retries (bare rerun = new operation, documented); SDK — parameter with a held-per-builder default; k6/seed — per-iteration mint (documented non-goal).

B-2 Special classes

  • One-time secrets (issue_key, rotate_key): pair columns on partner_api_keys (create_request_id UUID, create_request_hash TEXT with the versioned CHECK) + a partial UNIQUE (create_request_id) WHERE create_request_id IS NOT NULL (the F8 backstop — without it two concurrent same-id issues mint two live keys). Rotate inserts the NEW claimed row FIRST, then conditionally revokes the old (WHERE revoked_at IS NULL) — a response-lost retry finds the claim before any already-revoked refusal. Replay ⇒ typed 409 ONE_TIME_SECRET_ALREADY_ISSUED naming the key id, never the secret; LIFETIME horizon; revoke + reissue runbook; a concurrent second-issue test is REQUIRED. rotate_key gains a 1-field JSON body (breaking).

  • create_report: grandfathered LIFETIME received_request_id + new reports.received_request_hash TEXT NULL (legacy NULL ⇒ compare skipped); IntentV1 = the SubmitReportRequest STRIPPED of captcha_token + website (substantive raw_submission fields INCLUDED; received_ip_hash excluded); intake strips + hashes BEFORE forwarding.

  • Intake edge protocol: the client mints and HOLDS the id (report-form.js form state; SDK helpers take/hold it; partner retries re-sign with a fresh JTI — the JWS covers the id); the handlers STOP minting per request (public.rs:292 / partner.rs:215 — the confirmed double-create bug); the forwarder passes the client-held id verbatim; the honeypot fake id = uuid v5 of the submitted id (stable across retries); the intake multipart proxy (attachments.rs:59) gains forwarding of the new text fields. SHINES-mode horizon: ALL SHINES-bound submissions stay at-least-once (no receiving identity).

  • safety_assessment: the rules-side audit row is the same by-design-per-call class as direct evaluate (documented); the cases-side insert carries the claim.

  • rules import_rule_set: an existing-resource command — breaking wrapper {client_request_id, content}; entity = the path’s ruleset id.

  • sweep preview → claims (audit-confirmed stranded previewed rows on response loss). Replay semantics AS BUILT (Bf1 #1263, deviating from the issue AC’s literal "same run+token"): the run row stores only the execute token’s SHA-256 digest (#1096 — "return to the caller, then forget"), so a replay CANNOT re-expose the original plaintext without a security regression. Instead, a verified replay of a still-previewed, in-window run ROTATES the token (fresh mint, digest swapped under a CAS on state + the 15-minute window — never extending it) and returns the SAME run: original pinned sets, original as-of, fresh token. The lost original is invalidated — harmless in the only scenario that reaches this path (the retryer never saw it). A run that has moved on (executed / expired) answers a typed 409 sweep-run-state; execute stays unchanged.

  • ADRs: supersession-only — accepted ADRs untouched; ADR-062 §Supersessions names the superseded sections.

§U — Generation-fenced upload/export attempts

Immutable upload_attempts rows (id = client_request_id) + generation-fenced target pointers. Digest computed BEFORE claim; object keys derived from (attempt_id, g{generation}); ALL writes Store::put_create (store.rs:121); AlreadyExists ⇒ read-back digest verify. Entity rows are NEVER created pending — the scanner-reaps-entity-under- live-claim defect is unrepresentable.

CREATE TABLE upload_attempts (
    id                     UUID PRIMARY KEY,          -- = client_request_id
    target_kind            TEXT NOT NULL CHECK (target_kind IN (/* per service */)),
    target_id              UUID NOT NULL,             -- polymorphic; no FK
    status                 TEXT NOT NULL DEFAULT 'pending'
        CHECK (status IN ('pending','stored','promoted','superseded','failed')),
    attempt_generation     BIGINT NOT NULL DEFAULT 0, -- fencing token (send-jobs idiom)
    declared_sha256        CHAR(64) NOT NULL CHECK (declared_sha256 ~ '^[0-9a-f]{64}$'),
    size_bytes             BIGINT NOT NULL,
    file_name              TEXT NOT NULL,
    content_type           TEXT NOT NULL,
    intent                 JSONB NOT NULL,            -- typed per kind via sqlx::types::Json<T>
    object_key             TEXT NOT NULL,
    abandoned_object_keys  JSONB NOT NULL DEFAULT '[]',
    failure_reason         TEXT
        CHECK (failure_reason IN ('reaped_no_blob','digest_mismatch','target_gone')),
    result_id              UUID,
    created_by             UUID NOT NULL,
    created_at             TIMESTAMPTZ NOT NULL DEFAULT now(),
    updated_at             TIMESTAMPTZ NOT NULL DEFAULT now(),
    CHECK ((status = 'failed') = (failure_reason IS NOT NULL))  -- failed↔reason paired (SU2;
                                                                -- makes the decision table's
                                                                -- Failed+NULL arm unrepresentable)
);
-- + three partial reconciler indexes (SU2): (updated_at) WHERE status IN
-- ('pending','stored') — the overdue pick; (updated_at) WHERE
-- abandoned_object_keys <> '[]' — the sweep pick; (updated_at) WHERE status IN
-- ('promoted','superseded','failed') — the terminal prune.
  • States pending→stored→promoted[→superseded], pending→failed(reaped_no_blob | digest_mismatch | target_gone), stored→failed(target_gone) (finalize-time discovery — the target vanished after the blob landed; the finalize tx rolls back whole, then the failing CAS + event land in their own tx), failed(reaped_no_blob | digest_mismatch)→pending re-open with generation++ and key rotation (a late zombie put fills only an abandoned g{n} key — swept idempotently while listed; a put landing AFTER its sweep is an untracked, unpromotable, harmless orphan). Every transition is a status+generation CAS; module crates/craig-store/src/attempts.rs owns the exact statements + sha256_hex + key derivation ({kind_prefix}/{target_path}/{attempt_id}/g{n}/{safe_name}; never name a variable gen — edition-2024).

  • One shared finalizer per target kind (an #[async_trait] AttemptFinalizer) runs identically from the handler and the reconciler: finalize tx = attempt CAS stored→promoted + entity INSERT (append kinds: contact/report/icpc/home attachments) or fenced pointer flip (replace kinds: court_orders.current_attempt_id
    (created_at,id)-ordered winner — the pointer never NULLs, never regresses; exports flip current_export_attempt_id/export_generation) + the uploaded event — events atomic with READINESS on every path. The tx1 events + compensation paths in today’s four handlers are deleted.

  • Replay decision table keyed on the tombstone row (31 d retention ≥ every horizon): equal intent ⇒ drive remaining steps / fetch result (promoted ⇒ 200 with the entity; superseded ⇒ 200 with the newer current doc); different ⇒ 422 pre-write (the loser touches NOTHING); failed(reaped/mismatch) ⇒ re-open CAS; failed(target_gone) ⇒ 404; past-prune ⇒ documented fresh-attempt boundary; foreign created_by ⇒ 403.

  • UploadAttemptReconciler (1 h grace, 5 min interval, batch 100) replaces the AttachmentScanner semantics: digest-verified promotion (fixes promote-on-bare-exists()), reap CAS + failure event in ONE tx, abandoned-key sweeps, terminal-row pruning; scanner entity targets retire after a drain window (SU8).

  • Placement home_documents inverts blob-first and placement gains its FIRST reconciler worker (worker exact-set update). Exports: generated_on pinned in intent at first claim; bytes/filename never use now(); retry across a date change reproduces the byte-identical artifact; source drift ⇒ 409 export-source-changed (a new id ⇒ new generation supersedes; the old artifact is retained — federal evidence).

  • Multipart wire: uploads gain REQUIRED text fields client_request_id + file_sha256 (breaking ×5, CHANGELOG’d; there is NO existing JSON metadata part — the shared helper collects file + flat text fields, crates/craig-api/src/multipart.rs:43); exports take an optional JSON-body id. The court-order variant has NO attachment entity — tx1 is a fenced pending-flip on the court_orders row, entity_id = the court-order id.

  • Crash matrix + the six mandated race tests: scanner/put race (both orders), wrong existing digest, two concurrent court uploads, export retry across date/source change, late-put-after-reap, duplicate suppression.

  • Residue: devstack Garage put_create — VERIFIED in SU1 (2026-08-02, wire probe + the committed garage_put_create canary): Garage IGNORES If-None-Match: * PERMANENTLY BY DESIGN (CRDT, no consensus — upstream known-issue, not roadmap), so the prescribed fallback is IN the substrate: store_blob_create prechecks via read-back digest verify before put_create, and the residual precheck→put window is convergent by construction (every legitimate writer of a key passed the declared-digest gate against the same immutable claim row — the horizon table’s "puts converge by digest+generation" IS the correctness anchor; the conditional put is defense-in-depth on honoring backends). Remaining residues: post-31 d id reuse; failure-event latency = grace; the entity-delete orphan-blob window is PRE-EXISTING (#1274).

§G — Replay-guard hardening (audited hazards fixed in-program)

Fix patterns: CAS = status/version/If-Match predicate in the UPDATE; IDF = IS DISTINCT FROM no-op gate; SUP = stage events only on the guarded/returned row; FWW = first-write-wins COALESCE. Each fix ≈ 5–20 LOC + a replay-pin test.

  • G1 cases (as built): update_person (IDF row-change gate over the plaintext columns + the two DETERMINISTIC hmac siblings — ciphertext is randomized and cannot IDF; verification reset only when ssn_full_hmac IS DISTINCT FROM, superseding ADR-051’s every-write-resets — named in ADR-062 §Supersessions; SUP; the zero-row no-op returns the locked row), attest_ssn (AND ssn_verified=false; zero-row disambiguates under the row lock: 404 / MISSING_SSN / STALE_ATTESTATION / same-digest-verified ⇒ 200 no-op / moved-concurrently ⇒ 409), revoke_ssn_verification (AND ssn_verified=true + SUP), clear_ssn (value-bound widened to ssn IS NOT NULL OR ssn_last_four IS NOT NULL so the remedy still clears a standalone last-four; SUP), update_allegation (version CAS in the body; version BIGINT column + response field), update_investigation (status pin ALWAYS armed so the closure gate is exact; event only on a real transition; IDF over the non-status fields; zero-row ⇒ 409 unless the current row already embodies every requested field ⇒ 200 no-op), update_case (status AND assigned-worker pins always armed; case.closed/case.updated only on an actual status change; assignment event exact under the pin), update_case_plan / update_task (IDF + SUP; update_task additionally gains the fleet-standard AND active = true its delete already had — soft-deleted tasks were mutable), update_contact / update_court_order (AND active=true + REQUIRED If-Match version CAS — missing 400, stale 409, deleted 404; version bumped every write and exposed in responses).

  • G2 placement + exchange: update_foster_home (version CAS via REQUIRED If-Match, not the originally-drafted expected-license-status pin — the named kill-target is the active→suspended→active ABA cycle, and a client-carried expected-STATUS re-matches after the cycle so it cannot block the replay re-suspension; only the monotonic foster_homes.version counter distinguishes the replay, and same→same transitions are already 400 by matrix so the plain retry was never the live hazard; G1 contacts/court-orders wire semantics: missing 400 / stale 409 / bump every write), update_training (IDF
    the fleet-standard AND active=true its delete already had — soft-deleted training was mutable) / update_kinship_option / update_education / update_health (IDF no-op gates; education/health additionally SUP — their unconditional education.updated/health.updated staging was 2 of the audit’s 5 event-dups; kinship/training stage no events), exchange update_partner (version CAS via REQUIRED If-Match — kills the deactivated-partner resurrection; delete_partner’s deactivation ALSO bumps `version, else a pre-delete If-Match would still match post-delete and resurrect through the window), delete_partner (AND active=true; double-DELETE now 404 per the fleet convention — previously 204 forever re-stamping updated_at), update_agreement (approved_by/ approved_by_name/approved_at all FWW COALESCE(col,$) — one approval stamp, first wins; always-armed AND status=$expected CAS; the store fn’s fetch_one 500-on-missing bug becomes a clean 404 in passing), update_icpc_request (AND status=$expected — closes the TOCTOU; the staged icpc.status_changed from_status is now proven by the pin, SUP).

  • G3 security + reporting + rules + financial (as designed from the route audit; the generic "status/version CAS" resolves per route): update_review (always-armed status pin from the validated pre-read + the missing AND active = true — a soft-deleted review was still mutable — + fetch_optional: a lost race was a 500), update_nist_control (status pin + fetch_optional 500 fix), update_admin_unit / update_detection_rule (version CAS via REQUIRED If-Match — no status column exists and a stale replay clobbers reference data / re-enables a disabled detection rule; API/CLI-only surfaces), security update_partner (version CAS via REQUIRED body version — the G1 allegations pattern because a craig-web form drives it; suspension cycles are ABA-vulnerable exactly like exchange partners, so a status pin cannot block the stale re-suspension), update_major_change (the event gates on the actual false→true flip read UNDER the update tx — the old gate re-staged security.major_change.reported on every repeated {reported_to_acf: true} PUT), resolve_issue (AND resolved = false; an already-resolved replay converges to 200 returning the ORIGINAL resolver — attribution is first-write-wins), AFCARS/NCANDS review/approve ×4 + transmit ×2 (always-armed AND status = $expected from the validated pre-read — closes the approved→reviewed pre-read-clobber TOCTOU; transmit events stage only on the guarded row, pinned by an N-concurrent-transmits exactly-one-event invariant test), update_rate (AND active = true — a tombstoned rate was still mutable — + version CAS via REQUIRED If-Match; new rate_tables.version), submit_claim (AND status = 'draft' + SUP — concurrent double-submit double-published financial.claim_submitted; pinned by the payments-suite 8-way-race shape), update_rule_set (revision CAS via REQUIRED If-Match — the trigger-bumped rule_sets.revision mints a fresh token for every write, so a stale replay was LEGITIMIZED into the JDM live cache; the import route arms the same store predicate from a locked in-tx read). Version columns ride one migration per service (security ×3: partners / admin_unit_registry / detection_rules; financial ×1: rate_tables); reporting and rules need none (status-only / revision exists).

  • G4 convert_report resumability (as designed from the wiring recon): auto-link
    rollup move to a NEW craig-cases consumer of its own case.report_converted on the Track A substrate; the post-commit pipeline call is deleted. (Audit: conversion.rs:82-186 — a response-lost retry short-circuits and the winner dying post-commit skips the pipeline forever.) Mechanics: the event payload gains seed_case_id (the consumer cannot otherwise reproduce the request’s household seed); queue craig-cases.events bound to case.report_converted via the placement A2 subscriber shape (handle_idempotently + tx_handler, Critical events-subscriber worker) — the #1202 broker credential for craig-cases widens to cover craig-cases\.events (definitions.json + the broker_permissions fleet pin + the deployment guide’s rabbitmqctl forms, in lockstep per the amendment). Auth: the consumer mints the CASES SERVICE PRINCIPAL bearer (br.service_token, the financial A4 S2S-inside-handler pattern, rules calls bounded inside the 15 s attempt deadline) replacing the borrowed user header. Atomicity split, stated honestly: per-link inserts keep their OWN #799 link+audit transactions and converge by the report_persons UNIQUE (replay skips); the ROLLUP event alone stages on the claim transaction — exactly-once by the inbox claim. Semantics change (pre-1.0, CHANGELOG’d): convert’s 200 now means "referral committed, links pending"; the synchronous-link battery tests move to bounded polling via the rollup completion oracle; a raw-envelope redelivery leg (the financial placement_ended_void shape) pins the inbox dedup; a degraded pass (rank failures) refuses consumption so transient faults become counted retries. The #799 fault leg became the DEGRADED-PASS leg end-to-end: its outbox trigger self-disarms on read-committed visibility of the claim row’s error_count >= 1 (attempt 1 cannot see its own uncommitted claim; the immediate strike-1 requeue redelivery can), pinning refusal-counted-once
    convergence + the per-attempt link/audit atomic pair in one deterministic arc — a PERSISTENT fault cannot converge by design (post-strike-2 resumption is the ADR-022 §D4 admin replay, operator-paced).

  • Audit-proven CONVERGENT routes receive characterization tests pinning their guards (grouped into the G MRs).

§R — Route disposition ledger (binding; 156 routes)

Every mutating route carries exactly one class; the classes sum to the per-service totals (40 CONVERT + 7 §U + 2 SECRET + 1 LIFETIME + 16 CONFORMING + 7 EXEMPT + 2 LEASED + 7 FORWARD + 38 §G-FIXED + 36 PROVEN = 156). Verdicts for the non-create routes come from the two 2026-07-31 handler-by-handler audits (every handler + store fn + event staging read; guard citations in the audit columns below). Per-class actors: direct user / BFF-lifted actor via acting_worker() / service principal (federal_export, jws-replay-check, intake forwards) / partner (API key + JWS).

CONVERT — 40 (claim + IntentV1 + replay): cases 14 — create_person, create_referral, create_allegation, create_investigation, submit_safety_assessment, create_case, add_household_member, create_case_plan, create_task, create_contact, create_court_order, create_follow_up, record_decision, create_person_link (natural-key backstop retained); placement 6 — create_foster_home, record_training, create_placement, create_kinship_option, create_education, create_health; exchange 5 — create_partner, create_agreement, create_icpc_request, submit_home_study (+ DB UNIQUE fix), send_exchange (202 replay); financial 5 — create_adjustment, create_rate, create_import_batch, generate_claim, preview_review_sweep; security 6 — create_review, create_nist_control, create_major_change, create_admin_unit, create_detection_rule, create_partner; reporting 2 — generate_afcars, generate_ncands; rules 2 — create_rule_set (backstop retained), import_rule_set (wrapper).

§U-CLASS — 7: cases contact attachments, report attachments, court-order document (replace-variant); placement home_documents (order-normalized); exchange icpc attachments; reporting export_afcars, ncands_export.

ONE-TIME-SECRET — 2: issue_key, rotate_key (lifetime pair columns). LIFETIME — 1: create_report (received_request_id + new hash compare).

CONFORMING — 16: financial create_subsidy_agreement, subsidy transitions, activate, terms, reviews, completion, reschedule, generate_subsidy_payments, sweep execute, import stage/abort/finalize (12); cases jws-replay-check; security register_signer_key; keyring register + replay-check. (convert_report REMOVED from conforming — §G4.)

EXEMPT — 7: calculate_payment (persists nothing), batch_lookup, federal_export (each retry IS another PII read-audit export — noted for audit consumers), test_connectivity, partners::verify, rules evaluate (per-call audit row by design), detection run (audit: per-alert uq_security_alerts_standing_rule arbiter proven — the losing tx rolls back including its staged event; lease absence is a cost, not a correctness, issue).

LEASED/DRAIN — 2: archive run, purge (audit: per-row exactly-once proven — objects-first
count-mismatch-rollback prune tx / WHERE purged_at IS NULL stamps with success-gated events; a response-lost retry legitimately advances the drain — runbook note).

FORWARD — 7: all intake routes (client-held id protocol; the cases-side claim is the arbiter; SHINES-mode = at-least-once).

§G-FIXED — 38: the 30 stale-replay hazards (incl. the narrow update_task/ update_placement and mild delete_partner variants) + 4 event-dup routes + the update_icpc_request TOCTOU + convert_report (§G4), each named in §G with its fix. Audit citations (hazard → cited gap): update_person store/persons.rs:98-122 (COALESCE, verification-triple reset on same-value SSN); attest_ssn :174 (no ssn_verified=false conjunct — revoke-then-replay resurrects); revoke_ssn :187-204 + clear_ssn :210-231 (unguarded + dup audit events); update_allegation store/referrals.rs:192; update_investigation store/investigations.rs:92 (event refires on closed-row edits); update_case store/cases.rs:96 (case.closed on any closed-row edit; stale assignment event); update_case_plan store/case_plans.rs:91; update_task :252 (narrow: completed_at-only bodies); update_contact store/contacts.rs:88 (no active=true); update_court_order store/court_orders.rs:131; update_foster_home store/foster_homes.rs:161 + license cycle transitions.rs:29-40; update_training :324; update_placement (narrow: non-status fields, store/placements.rs:163); update_kinship store/kinship.rs:107; update_education store/education.rs:125; update_health store/health.rs:126; exchange update_partner store/partners.rs:96 (resurrection); exchange delete_partner api/partners.rs:424 (unguarded soft-deactivate); update_agreement store/agreements.rs:92 (approved_at re-stamp); update_icpc_request store/icpc.rs:86 (TOCTOU); security update_review/nist/major_change/admin_unit/detection_rule/partner (stores: reviews.rs:128, nist.rs:166, changes.rs, admin_units.rs:112, detection_rules.rs:78, partners.rs:156); resolve_issue store/issues.rs:124; AFCARS review/approve store/afcars.rs:125 + NCANDS store/ncands.rs:127-153; transmit ×2 (event per matched write); update_rate store/rates.rs:152; submit_claim store/claims.rs:78; update_major_change (event per call); update_rule_set store.rs:188 (stale-JDM cache reinstall + dup invalidation events).

CONVERGENT-PROVEN — 36 (characterization-tested in §G MRs): payments approve, issue, clear (store/payments.rs:174-201 CAS + success-gated events); approve_adjustment (locked validate); delete_adjustment; delete_rate; delete_review; delete_nist_control; delete_major_change; delete_admin_unit; delete_detection_rule; acknowledge_alert (WHERE NOT acknowledged + gated event); delete_partner (security, status<>'deleted'); revoke_key (revoked_at IS NULL + gated event); approve_signer_key; revoke_signer_key (status-guarded); delete_rule_set (active=true, 404 before events); keyring approve; keyring revoke (Pending/non-Revoked file-store CAS); approve_case_plan (draft-only + parent lock + rollback-on-miss); delete_task; delete_contact; delete_attachment (contact); delete_report_attachment; delete_court_order; delete_person_link (hard delete + rollback); delete_training; delete_kinship_option; delete_education; delete_health; delete_home_document; delete_agreement (draft-only hard delete); retry_transaction (status='failed' + fresh correlation_id by design); composition override; composition user-delta (RFC 7232 ETag CAS); admin events replay (idempotent window reset; delivery contract = consumer dedup). Per-service tally: cases 7, placement 5, exchange 2, financial 6, security 10, rules 1, composition 2, keyring 2, shared 1 = 36. Boundary notes: abort_import_batch is counted in CONFORMING (the import family), and update_icpc_request receives the §G2 CAS so it is counted in §G-FIXED — neither appears here. Known caveat carried by the delete family: post-DB blob deletion windows orphan blobs on crash — pre-existing, #1274.

Per-service totals (reproducible): cases 40, composition 2, exchange 13, financial 26, intake 7, keyring 4, placement 18, reporting 11, rules 5, security 29, shared replay 1 = 156.

Client map (per-service surfaces every conversion slice updates): cases → CLI cmd/{case,person,referral,investigation,intake,plan}.rs; BFF routes/cases/*
routes/intake/; test-lib clients/cases/; k6 smoke/load/stress/soak + services/cases.js
person-suggestions-bench; seed.sh:326; the intake forwarder. placement → CLI cmd/{placement,home,kinship,health,education}.rs; BFF routes/placement/; test-lib; k6 ×4
services/placement.js. exchange → CLI cmd/{exchange,agreement,icpc}.rs; BFF routes/exchange/
; test-lib; k6 ×4 + services/exchange.js. financial → CLI cmd/{financial,adjustment,claim}.rs; BFF routes/financial/; test-lib; xtask import-subsidy-history. security → CLI cmd/{partner,security,nist,changes,admin_unit,detection,archive,alerts}.rs; BFF routes/security/partners.rs; test-lib. reporting → CLI cmd/reporting.rs; BFF routes/reporting/; test-lib (+ exports). rules → CLI cmd/rules.rs; BFF routes/rules.rs; seed.sh:139-161; test-lib. intake → SDKs ×3 + static JS + e2e intake specs + k6.

Units & DAG

~35 MRs; branches feature/1178- / feature/1182- (docs units docs/*); each ≤500 changed LOC except six named exceptions (U0 docs; A1 ≈60 % tests; A4 tx-threading; B1b test-only; B2 net-negative deletion; SU1/SU3/Bc3b one-contract families). Sequencing:

U0 ─→ A1 ─→ {A2, A3, A4, A5} ─→ A6 ─→ close #1178          (stop-then-start deploys)
U0 ─→ B1a ─→ B1b ─→ {B2, Bc1, Bc2, Bc3a→Bc3b, Bp1, Be1, Bs1, Bs2, Bf1, Br1, Bru1, G1, G2, G3}
B1a ─→ SU1 ─→ SU2 ─→ {SU3, SU4, SU5, SU6} ─→ SU7 ─→ SU8
A1 ─→ G4  (needs the substrate; cases gains an events consumer)
ALL conversions + SU + G ─→ BF (gated: every service on post-B2 binaries) ─→ close #1182 ─→ UF

Unit contents, estimates, and per-unit ACs live in the epic &77 child issues (#1239–#1271
#1053/#1194); this table is the authoritative scope reference. BF’s DROP of idempotency_responses ×8 is an explicit steer-authorized deviation from CONTRIBUTING’s two-step "later release" wording (single-deploy devstack; recorded in the MR + CHANGELOG Removed, gated on deploy evidence).

Migrations

Migration Services Unit

event_inbox + payload_hash; event_inbox_collisions

×8

A1

request_claims

×8

B1a

upload_attempts + target columns

cases, exchange, placement, reporting

SU2

allegations / contacts / court_orders version BIGINT NOT NULL DEFAULT 1

cases

G1

foster_homes / exchange_partners version BIGINT NOT NULL DEFAULT 1

placement, exchange

G2

partners / admin_unit_registry / detection_rules + rate_tables version BIGINT NOT NULL DEFAULT 1

security, financial

G3

partner_api_keys pair columns + partial UNIQUE

security

Bs2

reports.received_request_hash

cases

Bc3a

icpc_home_studies UNIQUE

exchange

Be1

idempotency_responses DROP

×8

BF

payload_hash NOT NULL (post-horizon)

×8

#1273

Test matrix

Devstack suites run cargo nextest run -p <crate> --profile integration --run-ignored=all; JUnit under test-results/integration/; receipts copied aside + uploaded for RED/GREEN claims. #[ignore = "requires devstack"], no guard fns (the guard-vs-F-024 contradiction is standing — escalated claude-quickstart#27; this plan follows the executable lint).

  • A1: claim+effects+stamp atomicity; deferred CONSTRAINT TRIGGER … INITIALLY DEFERRED commit-failure (built fresh — the #804 row trigger fires at statement time, the wrong point); many-concurrent-all-failing ⇒ the cap stays hard at 5; concurrent duplicate ⇒ one effect; sync-construction panic COUNTED; NULL-hash legacy row ⇒ dedup, not poison; collision path (canonical row untouched, own token); unroutable ⇒ failed_at-NULL + watchdog + replay-after-binding-restore; pg_sleep slow winner ⇒ the duplicate dedups; real-broker ladder (terminal at 5, 6th dedups, byte-identical re-surface absorbed); accounting-commit failure ⇒ not counted; timestamp progression; RED receipt vs the old fn.

  • A2–A6: per-service truth (#804 rollback-unclaims; security two-writes-atomic; financial S2S-timeout rollback proof + billing e2e; rules audit-in-substrate-tx; reporting zero-partial-issues).

  • B1b: intent proptests (key order, path, actor, jurisdiction, unicode); barrier collect-all N-same-key ⇒ 1 claim + 1 row + N×200; forced claim rollback; REPLAY_ENTITY_GONE; scope/actor/entity_kind mismatch leak-free; pruner-race single-statement pin.

  • Per conversion slice: EVERY route instantiated — concurrent create ⇒ one row; replay ⇒ the endpoint’s own success shape; mismatch 409; missing field 422/400 by body shape; commit-then-response-loss replay; same-key-different-path 409; actor separation.

  • §U (SU7): the six mandated race scenarios.

  • §G: per-route replay pins (each fixed hazard ⇒ replay converges + no dup event + the audit’s named interleaving reproduced) + characterization tests for all 35 PROVEN routes
    drain-by-design pins for archive run/purge.

  • Bs2: concurrent second-issue; secret non-reexposure; rotate response-loss.

  • Bc3a/b: fresh-captcha retry replays; JTI re-sign; honeypot stable fake id; multipart proxy field forwarding.

Delivery

Per unit (the standing per-unit discipline): implement → fmt / clippy -D warnings / targeted nextest / quality-budgets / cargo xtask api-docs + check-docs + Antora render on doc-touching units / coverage on craig-mq + craig-api units → full + ignored test commands with JUnit receipts → RED-or-recorded-reason with receipts (A1, B1b, B2) → fresh J1–J8 subagent over the staged diff, findings remediated pre-commit → token-gated commit → backgrounded battery push (tree untouched) → git ls-remote verify → MR (Closes its own unit issue; Relates to the track anchor; URL reported) → merge per the standing green-local directive → merged-state verify → J-record note → closing comment (impl + merge SHAs, AC walk) → epic checkbox + the Status row above → branch delete + remote prune. Close-out: manual #1178/#1182 closes with program comments; epic &77 completion record (PUT description); &74 ticks; Plan Completion Audit; archive this page.

Dispositions

Item Disposition

#1053 / #1209

A1 (unit issue + justified same-MR close)

#1194

B2’s unit issue

#1214

Out of scope (ordering ≠ atomicity) — ADR-062 non-goals

convert_report auto-link loss

Fixed in-program (§G4)

30 stale-replay hazards + 4 event-dups + the ICPC TOCTOU

Fixed in-program (§G1–G3)

Orphan-blob delete windows (pre-existing)

#1274

Typed rejection mapper

#1272

payload_hash NOT NULL

#1273

plans/api-idempotency.adoc

Archived historical-Done with a supersession banner (U0)

Edit this page · latest