Plan: Transactional Idempotency Claims — Inbox + HTTP Ingress
On this page
- Status
- Context
- Doctrine and horizons
- Track A design — the transactional inbox
- Track B design — HTTP claims
- §U — Generation-fenced upload/export attempts
- §G — Replay-guard hardening (audited hazards fixed in-program)
- §R — Route disposition ledger (binding; 156 routes)
- Units & DAG
- Migrations
- Test matrix
- Delivery
- Dispositions
Status
| Unit | Description | Status |
|---|---|---|
U0 |
ADR-062 + this plan page + nav; api-idempotency.adoc archived |
Done (2026-07-31) — !1122 |
A1 |
craig-mq transactional attempt machine (#1053; closes #1209) |
Done (2026-07-31) — !1123 |
A2 |
Convert placement + exchange inbox handlers |
Done (2026-07-31) — !1124 |
A3 |
Convert security (audit + ledger atomic) |
Done (2026-07-31) — !1125 |
A4 |
Convert financial (money path; S2S bound; timeout proof) |
Done (2026-07-31) — !1126 |
A5 |
Convert rules (tx through engine.evaluate audit) |
Done (2026-07-31) — !1127 |
A6 |
Convert reporting (one tx/envelope) + delete legacy API |
Done (2026-07-31) — !1128 |
B1a |
request_claims module + IntentV1 + migrations ×8 |
Done (2026-07-31) — !1129 |
B1b |
Claims battery (proptests, collect-all helper, synthetic endpoint) |
Done (2026-07-31) — !1130 |
B2 |
Delete the idempotency middleware; install claims retention (#1194) |
Done (2026-07-31) — !1131 |
SU1 |
upload_attempts state machine + reconciler (craig-store) |
Done (2026-08-02) — !1146 |
SU2 |
upload_attempts migrations ×4 + target columns |
Done (2026-08-02) — !1147 |
SU3 |
cases attachments + court-order document on attempts |
Done (2026-08-02) — !1148 |
SU4 |
exchange ICPC attachments on attempts |
Done (2026-08-02) — !1149 |
SU5 |
placement home documents reserve-first + first reconciler worker |
Done (2026-08-02) — !1150 merged b72e1c39; #1251 closed |
SU6 |
reporting deterministic exports on attempts |
Done (2026-08-02) — !1151 merged 533c159c; #1252 closed |
SU7 |
upload-attempt race matrix |
Done (2026-08-02) — !1152 merged 14eacc1e; #1253 closed |
SU8 |
retire AttachmentScanner entity targets |
Done (2026-08-02) — !1153 merged 79ddd9cb; #1254 closed |
Bc1 |
cases claims: persons/referrals/allegations/investigations/safety |
Done (2026-08-01) — !1132 |
Bc2 |
cases claims: case/household/plans/tasks/contacts/court-orders |
Done (2026-08-01) — !1134 |
Bc3a |
cases claims: reports family + create_report intent hash |
Done (2026-08-01) — !1135 |
Bc3b |
intake client-held id protocol + SDK submit fields |
Done (2026-08-01) — !1136 |
Bp1 |
placement claims: six creates |
Done (2026-08-01) — !1139 |
Be1 |
exchange claims: five converts + home-study UNIQUE |
Done (2026-08-01) — !1140 |
Bs1 |
security claims: six creates + app-minted id refactor |
Done (2026-08-01) — !1141 |
Bs2 |
one-time-secret pair columns (issue/rotate key) |
Done (2026-08-01) — !1142 |
Bf1 |
financial claims: four creates + sweep-preview claim |
Done (2026-08-01) — !1143 |
Br1 |
reporting claims: AFCARS/NCANDS generates |
Done (2026-08-01) — !1144 |
Bru1 |
rules claims: rule-set create + import wrapper |
Done (2026-08-02) — !1145 |
G1 |
cases replay guards (SSN lifecycle + CAS/IDF/SUP sweep) |
Done (2026-08-02) — !1155 merged 7efb032a; #1266 closed |
G2 |
placement + exchange replay guards |
Done (2026-08-02) — !1156 merged 9e429adb; #1267 closed |
G3 |
security/reporting/rules/financial replay guards |
Done (2026-08-02) — !1157 merged 67aa1e7c; #1268 closed |
G4 |
convert_report auto-link resumable via outbox consumer |
Done (2026-08-02) — !1158 merged 7b7dabcf; #1269 closed |
BF |
DROP idempotency_responses ×8 + close-out gates |
Done (2026-08-02) — !1159 merged f0340e78; #1270 closed; anchors #1178 + #1182 closed with track-completion comments |
UF |
Program aggregation (CLAUDE.md, scorecard, known-issues, archive) |
Done (2026-08-02) — this aggregation/archive MR |
Epic: &77 (child of &74)
Anchors: #1178 (Track A) · #1182 (Track B) — close manually at track completion
Unit issues: U0 #1239 · A1 #1053 · A2 #1240 · A3 #1241 · A4 #1242 · A5 #1243 · A6 #1244 ·
B1a #1245 · B1b #1246 · B2 #1194 · SU1 #1247 · SU2 #1248 · SU3 #1249 · SU4 #1250 · SU5 #1251 ·
SU6 #1252 · SU7 #1253 · SU8 #1254 · Bc1 #1255 · Bc2 #1256 · Bc3a #1257 · Bc3b #1258 ·
Bp1 #1259 · Be1 #1260 · Bs1 #1261 · Bs2 #1262 · Bf1 #1263 · Br1 #1264 · Bru1 #1265 ·
G1 #1266 · G2 #1267 · G3 #1268 · G4 #1269 · BF #1270 · UF #1271
Follow-ups: #1272 (typed rejection mapper) · #1273 (payload_hash NOT NULL) · #1274
(orphan-blob janitor)
ADR: ADR-062
Context
Two platform primitives promised more than they delivered:
-
#1178 —
craig_mq::inbox::handle_idempotently(crates/craig-mq/src/inbox.rs:153) commits its claim, the handler’s effects, andprocessed_atas three separate autocommit statements, with an unlocked redelivery read. Real semantics: at-least-once with idempotency delegated per handler — which reporting (quality issues: fresh UUIDv7, no natural key) and security (plain audit insert) lack. -
#1182 — the HTTP idempotency middleware (
crates/craig-api/src/idempotency/) is a lease-based, unfenced 30 s claim, carrying the #1194 500-window — and is opt-in with ZERO senders (BFF, CLI, SDKs, e2e, k6 never sendidempotency-key). -
The 2026-07-31 handler-by-handler audit (two independent contextless explorers; verdicts in §R) found 30 non-create routes with stale-replay hazards (including two narrow/mild variants), 4 event-dup-only routes, one TOCTOU race (
update_icpc_request), and `convert_report’s post-commit auto-link retry-LOST.
F8 (ADR-055) is the proven model being generalized: client key + canonical hash + partial unique index + claim inside the ONE domain tx; replay re-derives from domain state.
User steers (2026-07-31, fixed): #1178 = transaction-aware handler API; #1182 = "the architecturally correct answer — no shortcuts, no fig leafs, no backwards compat pre 1.0".
Plan review history: v1 externally blocked (~35 findings) → v2 (all folded) → four-lens contextless workflow (28 findings) → v2.1 → second external block (six approvability conditions) → v3 approved 2026-07-31.
Doctrine and horizons
Atomic effects + bounded-horizon deduplication + convergent external effects. Never unqualified "exactly-once".
| Boundary | Guarantee | Horizon |
|---|---|---|
MQ inbox |
atomic claim+effects+stamp; hard cap over counted attempts |
31 d |
HTTP convert-class |
atomic claim+effects; replay = effects-dedup (responses re-derived) |
30 d knob |
One-time secrets, create_report |
per-entity pair columns |
entity lifetime |
§U uploads/exports |
atomic claim+attempt; puts converge by digest+generation |
31 d attempts |
Intake→SHINES (all SHINES-mode submissions) |
at-least-once (no receiving identity; ADR-042 boundary) |
— |
§G-hardened CAS routes |
SQL-guarded state machine + gated events |
— |
EXEMPT/LEASED (§R) |
per-route characterization (drain-by-design, arbiter-converged) |
— |
Track A design — the transactional inbox
A-1 Single-transaction attempt, savepoint accounting
One dedicated pooled connection per attempt; the whole attempt inside the #1203 contain
task:
acquire conn → BEGIN (claim/lock-wait under the DEFAULT 30 s statement timeout)
claim: INSERT event_inbox (envelope_id, source_service, event_type, payload_hash,
attempt_started_at) VALUES … ON CONFLICT (envelope_id) DO NOTHING RETURNING received_at
├─ claimed → SET LOCAL statement_timeout='5s' (only AFTER the lock is held)
│ SAVEPOINT attempt
│ caught = catch_unwind(AssertUnwindSafe(async { handler(envelope, &mut tx).await }))
│ under tokio ATTEMPT_DEADLINE = 15 s
│ ├─ Ok(Ok) → RELEASE; UPDATE processed_at = now(),
│ │ payload_hash = COALESCE(payload_hash, $h); COMMIT
│ └─ Ok(Err) | caught panic | deadline → ROLLBACK TO SAVEPOINT;
│ UPDATE error_count = error_count+1, last_error = $e; COMMIT
│ if error_count == 5 → DLX surface (A-3) → stamp failed_at (WHERE failed_at IS NULL)
└─ conflict → SELECT … FOR UPDATE:
(source_service, event_type, payload_hash [skip if stored NULL]) ≠ computed
→ COMMIT; collision path (A-4) — canonical row NEVER touched
processed_at / failed_at → COMMIT; dedup
error_count >= 5 → COMMIT FIRST, then re-surface OUTSIDE any tx
else retry → SET LOCAL 5s; UPDATE attempt_started_at; SAVEPOINT; handler
-
The claim row precedes the savepoint ⇒ survives handler rollback ⇒ failure accounting commits in the SAME tx under the held lock ⇒ hard cap over counted attempts. Cap statement (used everywhere): at most 5 COUNTED attempts; crash-class failures (process crash, connection loss, task abort, tier-2 panics, accounting-commit failure) are uncounted and can add attempts — the cap bounds counted work, broker/replay cadence bounds rate.
-
Timeout hierarchy: claim lock-wait = default 30 s (no SET LOCAL before the claim); handler statements = SET LOCAL 5 s after locking; tokio deadline 15 s. Worst winner hold ≈ 15 + ≤5 drain + accounting ms ≈ ~21 s < the 30 s waiter budget (waiters ≈ replica count; consume loops are serial per queue). Deadline path: timeout → drop handler future (clean drop → rollback to savepoint → accounting
InboxError::AttemptTimeout, COUNTED; Drop panic → tier 2). The slow-winner test drives apg_sleepSTATEMENT. -
Panic tiers: Tier 1 COUNTED — construction + poll panics (the handler CALL sits inside the caught async block). AssertUnwindSafe honesty: the savepoint discards DB state; non-DB captured state carries the SAME exposure the existing #1203 boundary already accepts (
subscriber.rs:1861). Tier 2 UNCOUNTED crash-class — drop-of-future + error-Display panics escape to the task boundary → whole-tx rollback. -
Identity =
(source_service, event_type, payload_hash);payload_hash= ONE shared fnsha256(serde_json::to_vec(&envelope.payload))(stable — publishes round-trip the same outbox JSONB). Additive migration ×8; NULL legacy rows skip the hash leg (backfilled at the next stamp); post-horizon NOT NULL = #1273. -
Handler:
for<'t> FnOnce(EventEnvelope, &'t mut Transaction<'static, Postgres>) → BoxFuture<'t, Result<(), E>> + Send + 'static;futures = { workspace = true }added to craig-mq. The old fn is renamedhandle_idempotently_at_least_onceat its 6 call sites in A1 (loud rename, not a compat surface) and deleted with its last caller in A6. -
Cutover barrier: A2–A6 deploy stop-then-start per service (compose Recreate; single-replica devstack is the only pre-1.0 deployment) — old-autocommit and new-locking handlers never run concurrently. Recorded in the deployment guide with a k8s Recreate note.
A-2 Failure & attempt-consumption table
| Failure point | Counted? | Mechanism |
|---|---|---|
Pre-claim db error |
No |
nack; broker paces |
Waiter’s claim statement-times-out (30 s) |
No |
pre-claim class; winner’s ≈21 s worst hold makes it rare |
Handler Err / caught panic (construction or poll) |
Yes |
savepoint rollback + same-tx accounting |
ATTEMPT_DEADLINE or SET LOCAL 5 s statement kill |
Yes |
|
Drop-of-future / Display-render panic |
No (crash-class) |
escapes to the task boundary; whole-tx rollback |
Stamp/COMMIT error (unknown outcome) |
No |
claim row arbitrates on redelivery |
Accounting COMMIT error |
No |
row state arbitrates |
DLX surface fails / unroutable |
n/a |
requeue-durable (A-3); failed_at NOT stamped |
Process crash mid-attempt |
No |
whole-tx rollback; one increment lost |
A-3 DLX surface — mandatory, deterministic, requeue-durable
-
publish_dlx→ mandatory + basic.return verdict (the capture-publisher shape,subscriber.rs:1689). The wrapper is built ENTIRELY from durable row state:occurred_at := stored received_at(publisher.rs:45change),retry_count := error_count,last_error := stored; envelope = delivery bytes ⇒ repeat/cross-replica surfaces are BYTE-IDENTICAL ⇒ the security consumer’s full-JSON equivalence absorbs them. Token unchanged:dlxcap:{envelope_id}:{consumer_queue}:{received_at}. Publish with NO tx open (an AMQP confirm await must never hold a row lock); the cross-replica double-surface race is accepted and absorbed. -
Unroutable/failed surface ⇒ nack
requeue: TRUE(a typedSurfaceFailedoutcome bypasses the 2-strike reject): the source-queue copy IS the durable envelope; the row stayserror_count=5, failed_at NULL; every redelivery retries the surface via the ≥5 branch; the beyond-grace watchdog alerts. The missing-binding devstack test asserts failed_at-NULL + watchdog + eventual replay success — NOT a broker-DLX landing (the broker path shares the same exchange/key and would drop it too).
A-4 Collision path (never mutates the canonical row)
event_inbox_collisions (envelope_id, observed_source, observed_type, observed_payload_hash,
first_seen, occurrences) ×8 same-timestamp migration; deterministic token
dlxcol:{envelope_id}:{consumer_queue}:{observed_payload_hash}; a collision-typed DLX record;
the colliding delivery nacks requeue:false; a watchdog over the table. The canonical event’s
processed/failed state and token are NEVER touched or reused.
A-5 Retry policy (folds #1053; fixes #1209)
Pacing = broker 2-strike + ADR-022 §D4 admin replay (the #1197 parking tier paces the DLQ
consumer only). Budget = error_count across cycles, terminal at 5 after a CONFIRMED surface.
Operator escape (runbook, conditional — never a raw delete): DELETE FROM event_inbox WHERE
envelope_id = $1 AND failed_at IS NOT NULL + audit note + replay step.
inbox_unprocessed_beyond_grace.sql SQL fixed (excludes failed_at IS NOT NULL), the
tolerating test replaced — this IS #1209, closed by A1 (justified same-MR: the invariant’s
meaning is defined by A1’s semantics). The admin-replay at-least-once contract now rests on
exactly the handler atomicity Track A ships — closure recorded in ADR-062.
Track B design — HTTP claims
B-1 request_claims + per-endpoint IntentV1
Schema (×8 DB-owning services, same-timestamp copies):
CREATE TABLE request_claims (
client_request_id uuid PRIMARY KEY,
claim_scope text NOT NULL CHECK (claim_scope <> ''),
intent_hash text NOT NULL CHECK (intent_hash ~ '^v1:[0-9a-f]{64}$'),
entity_kind text NOT NULL CHECK (entity_kind <> ''),
entity_id uuid NOT NULL,
created_by_sub uuid NOT NULL, -- EFFECTIVE actor
created_at timestamptz NOT NULL DEFAULT now()
);
CREATE INDEX request_claims_created_at_idx ON request_claims (created_at);
-
Module
crates/craig-api/src/request_claims.rs; newtypesClientRequestId(Uuid),ClaimScope(&'static str),IntentHash(String),EntityKind(&'static str). -
IntentV1 = per-endpoint versioned immutable projection structs — never raw DTO serialization; DTO evolution adds IntentV2 + dual-decode across the horizon. Envelope:
{v, scope, jurisdiction, actor, path: [(name, value)], body: &XxxIntentV1}→"v1:" + hex(sha256(serde_json::to_vec(…))); zeroserde_json::Value. Normalization pinned in B1a: UUIDs lowercase-hyphenated; strings byte-as-submitted (no unicode normalization, documented); money rescale(2);None⇒ null; no default injection. -
actor = parse_sub(claims.acting_worker().sub)— the effective actor (BFF impersonation ⇒ the worker), reusing the fleet’s typed-400 invalid-sub path (securitypartners.rs:473precedent). -
claim_first= the FIRST statement of the domain tx, ONE locking statement:INSERT … ON CONFLICT (client_request_id) DO UPDATE SET client_request_id = EXCLUDED.client_request_id RETURNING *(row lock + existing row in one statement — no SELECT window for the pruner). Classification NARROW: scope/actor/entity_kind/intent mismatch ⇒ 409IDEMPOTENCY_CONFLICT(bodies identical — no existence leak);Replay { entity_id }only when ALL match.precheck= an unlocked fast-path optimization before volatile deps, never the guard. Replay re-runs CURRENT entity authorization, then re-derives via the endpoint’s own detail projection; entity vanished ⇒ typed 409REPLAY_ENTITY_GONE. Replay returns each endpoint’s OWN success shape (send_exchange⇒ 202). -
App-minted ids refactor floor: store create fns gain an explicit UUIDv7 id param + a tx-capable executor (security’s six DB-default
gen_random_uuid()INSERTs; casescontacts.rs:20/ placementfoster_homes.rs:45pool-bound mints). -
Missing
client_request_id⇒ stock axum 422 (JSON) / 400 (multipart); the typed-rejection mapper is #1272. Response statuses otherwise unchanged. -
Retention:
CRAIGREQUEST_CLAIMSWINDOW_DAYS(platform prefix; default 30; 0 ⇒Disabled); hourly bounded-batch (5 k) advisory-locked prunerrequest-claims-retentionin the vacated worker slot (crates/craig-api/src/lib.rs:442); a beyond-grace watchdog replaces the deletedidempotency_expired_not_cleanedinvariant. Composition keeps table + worker (wiring/exact-set uniformity; empty table, no-op prunes). F8 pair columns grandfathered. -
Client-ID lifecycle: browser — minted at form render, hidden field, the SUBMITTED id re-emitted on validation-error rerenders (never re-minted); CLI — auto-mint per invocation +
--request-idfor scripted retries (bare rerun = new operation, documented); SDK — parameter with a held-per-builder default; k6/seed — per-iteration mint (documented non-goal).
B-2 Special classes
-
One-time secrets (
issue_key,rotate_key): pair columns onpartner_api_keys(create_request_id UUID,create_request_hash TEXTwith the versioned CHECK) + a partial UNIQUE(create_request_id) WHERE create_request_id IS NOT NULL(the F8 backstop — without it two concurrent same-id issues mint two live keys). Rotate inserts the NEW claimed row FIRST, then conditionally revokes the old (WHERE revoked_at IS NULL) — a response-lost retry finds the claim before any already-revoked refusal. Replay ⇒ typed 409ONE_TIME_SECRET_ALREADY_ISSUEDnaming the key id, never the secret; LIFETIME horizon; revoke + reissue runbook; a concurrent second-issue test is REQUIRED.rotate_keygains a 1-field JSON body (breaking). -
create_report: grandfathered LIFETIME
received_request_id+ newreports.received_request_hash TEXT NULL(legacy NULL ⇒ compare skipped); IntentV1 = theSubmitReportRequestSTRIPPED ofcaptcha_token+website(substantive raw_submission fields INCLUDED;received_ip_hashexcluded); intake strips + hashes BEFORE forwarding. -
Intake edge protocol: the client mints and HOLDS the id (
report-form.jsform state; SDK helpers take/hold it; partner retries re-sign with a fresh JTI — the JWS covers the id); the handlers STOP minting per request (public.rs:292/partner.rs:215— the confirmed double-create bug); the forwarder passes the client-held id verbatim; the honeypot fake id = uuid v5 of the submitted id (stable across retries); the intake multipart proxy (attachments.rs:59) gains forwarding of the new text fields. SHINES-mode horizon: ALL SHINES-bound submissions stay at-least-once (no receiving identity). -
safety_assessment: the rules-side audit row is the same by-design-per-call class as direct evaluate (documented); the cases-side insert carries the claim.
-
rules import_rule_set: an existing-resource command — breaking wrapper
{client_request_id, content}; entity = the path’s ruleset id. -
sweep preview → claims (audit-confirmed stranded
previewedrows on response loss). Replay semantics AS BUILT (Bf1 #1263, deviating from the issue AC’s literal "same run+token"): the run row stores only the execute token’s SHA-256 digest (#1096 — "return to the caller, then forget"), so a replay CANNOT re-expose the original plaintext without a security regression. Instead, a verified replay of a still-previewed, in-window run ROTATES the token (fresh mint, digest swapped under a CAS on state + the 15-minute window — never extending it) and returns the SAME run: original pinned sets, original as-of, fresh token. The lost original is invalidated — harmless in the only scenario that reaches this path (the retryer never saw it). A run that has moved on (executed / expired) answers a typed 409sweep-run-state; execute stays unchanged. -
ADRs: supersession-only — accepted ADRs untouched; ADR-062 §Supersessions names the superseded sections.
§U — Generation-fenced upload/export attempts
Immutable upload_attempts rows (id = client_request_id) + generation-fenced target
pointers. Digest computed BEFORE claim; object keys derived from (attempt_id,
g{generation}); ALL writes Store::put_create (store.rs:121); AlreadyExists ⇒ read-back
digest verify. Entity rows are NEVER created pending — the scanner-reaps-entity-under-
live-claim defect is unrepresentable.
CREATE TABLE upload_attempts (
id UUID PRIMARY KEY, -- = client_request_id
target_kind TEXT NOT NULL CHECK (target_kind IN (/* per service */)),
target_id UUID NOT NULL, -- polymorphic; no FK
status TEXT NOT NULL DEFAULT 'pending'
CHECK (status IN ('pending','stored','promoted','superseded','failed')),
attempt_generation BIGINT NOT NULL DEFAULT 0, -- fencing token (send-jobs idiom)
declared_sha256 CHAR(64) NOT NULL CHECK (declared_sha256 ~ '^[0-9a-f]{64}$'),
size_bytes BIGINT NOT NULL,
file_name TEXT NOT NULL,
content_type TEXT NOT NULL,
intent JSONB NOT NULL, -- typed per kind via sqlx::types::Json<T>
object_key TEXT NOT NULL,
abandoned_object_keys JSONB NOT NULL DEFAULT '[]',
failure_reason TEXT
CHECK (failure_reason IN ('reaped_no_blob','digest_mismatch','target_gone')),
result_id UUID,
created_by UUID NOT NULL,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT now(),
CHECK ((status = 'failed') = (failure_reason IS NOT NULL)) -- failed↔reason paired (SU2;
-- makes the decision table's
-- Failed+NULL arm unrepresentable)
);
-- + three partial reconciler indexes (SU2): (updated_at) WHERE status IN
-- ('pending','stored') — the overdue pick; (updated_at) WHERE
-- abandoned_object_keys <> '[]' — the sweep pick; (updated_at) WHERE status IN
-- ('promoted','superseded','failed') — the terminal prune.
-
States
pending→stored→promoted[→superseded],pending→failed(reaped_no_blob | digest_mismatch | target_gone),stored→failed(target_gone)(finalize-time discovery — the target vanished after the blob landed; the finalize tx rolls back whole, then the failing CAS + event land in their own tx),failed(reaped_no_blob | digest_mismatch)→pendingre-open with generation++ and key rotation (a late zombie put fills only an abandonedg{n}key — swept idempotently while listed; a put landing AFTER its sweep is an untracked, unpromotable, harmless orphan). Every transition is a status+generation CAS; modulecrates/craig-store/src/attempts.rsowns the exact statements +sha256_hex+ key derivation ({kind_prefix}/{target_path}/{attempt_id}/g{n}/{safe_name}; never name a variablegen— edition-2024). -
One shared finalizer per target kind (an
#[async_trait] AttemptFinalizer) runs identically from the handler and the reconciler: finalize tx = attempt CASstored→promoted+ entity INSERT (append kinds: contact/report/icpc/home attachments) or fenced pointer flip (replace kinds:court_orders.current_attempt_id
(created_at,id)-ordered winner — the pointer never NULLs, never regresses; exports flipcurrent_export_attempt_id/export_generation) + the uploaded event — events atomic with READINESS on every path. The tx1 events + compensation paths in today’s four handlers are deleted. -
Replay decision table keyed on the tombstone row (31 d retention ≥ every horizon): equal intent ⇒ drive remaining steps / fetch result (
promoted⇒ 200 with the entity;superseded⇒ 200 with the newer current doc); different ⇒ 422 pre-write (the loser touches NOTHING);failed(reaped/mismatch)⇒ re-open CAS;failed(target_gone)⇒ 404; past-prune ⇒ documented fresh-attempt boundary; foreigncreated_by⇒ 403. -
UploadAttemptReconciler(1 h grace, 5 min interval, batch 100) replaces the AttachmentScanner semantics: digest-verified promotion (fixes promote-on-bare-exists()), reap CAS + failure event in ONE tx, abandoned-key sweeps, terminal-row pruning; scanner entity targets retire after a drain window (SU8). -
Placement
home_documentsinverts blob-first and placement gains its FIRST reconciler worker (worker exact-set update). Exports:generated_onpinned in intent at first claim; bytes/filename never usenow(); retry across a date change reproduces the byte-identical artifact; source drift ⇒ 409export-source-changed(a new id ⇒ new generation supersedes; the old artifact is retained — federal evidence). -
Multipart wire: uploads gain REQUIRED text fields
client_request_id+file_sha256(breaking ×5, CHANGELOG’d; there is NO existing JSON metadata part — the shared helper collectsfile+ flat text fields,crates/craig-api/src/multipart.rs:43); exports take an optional JSON-body id. The court-order variant has NO attachment entity — tx1 is a fenced pending-flip on thecourt_ordersrow,entity_id= the court-order id. -
Crash matrix + the six mandated race tests: scanner/put race (both orders), wrong existing digest, two concurrent court uploads, export retry across date/source change, late-put-after-reap, duplicate suppression.
-
Residue: devstack Garage
put_create— VERIFIED in SU1 (2026-08-02, wire probe + the committedgarage_put_createcanary): Garage IGNORESIf-None-Match: *PERMANENTLY BY DESIGN (CRDT, no consensus — upstream known-issue, not roadmap), so the prescribed fallback is IN the substrate:store_blob_createprechecks via read-back digest verify beforeput_create, and the residual precheck→put window is convergent by construction (every legitimate writer of a key passed the declared-digest gate against the same immutable claim row — the horizon table’s "puts converge by digest+generation" IS the correctness anchor; the conditional put is defense-in-depth on honoring backends). Remaining residues: post-31 d id reuse; failure-event latency = grace; the entity-delete orphan-blob window is PRE-EXISTING (#1274).
§G — Replay-guard hardening (audited hazards fixed in-program)
Fix patterns: CAS = status/version/If-Match predicate in the UPDATE; IDF =
IS DISTINCT FROM no-op gate; SUP = stage events only on the guarded/returned row;
FWW = first-write-wins COALESCE. Each fix ≈ 5–20 LOC + a replay-pin test.
-
G1 cases (as built):
update_person(IDF row-change gate over the plaintext columns + the two DETERMINISTIC hmac siblings — ciphertext is randomized and cannot IDF; verification reset only whenssn_full_hmac IS DISTINCT FROM, superseding ADR-051’s every-write-resets — named in ADR-062 §Supersessions; SUP; the zero-row no-op returns the locked row),attest_ssn(AND ssn_verified=false; zero-row disambiguates under the row lock: 404 /MISSING_SSN/STALE_ATTESTATION/ same-digest-verified ⇒ 200 no-op / moved-concurrently ⇒ 409),revoke_ssn_verification(AND ssn_verified=true+ SUP),clear_ssn(value-bound widened tossn IS NOT NULL OR ssn_last_four IS NOT NULLso the remedy still clears a standalone last-four; SUP),update_allegation(version CAS in the body;version BIGINTcolumn + response field),update_investigation(status pin ALWAYS armed so the closure gate is exact; event only on a real transition; IDF over the non-status fields; zero-row ⇒ 409 unless the current row already embodies every requested field ⇒ 200 no-op),update_case(status AND assigned-worker pins always armed;case.closed/case.updatedonly on an actual status change; assignment event exact under the pin),update_case_plan/update_task(IDF + SUP;update_taskadditionally gains the fleet-standardAND active = trueits delete already had — soft-deleted tasks were mutable),update_contact/update_court_order(AND active=true+ REQUIREDIf-Matchversion CAS — missing 400, stale 409, deleted 404; version bumped every write and exposed in responses). -
G2 placement + exchange:
update_foster_home(version CAS via REQUIREDIf-Match, not the originally-drafted expected-license-status pin — the named kill-target is theactive→suspended→activeABA cycle, and a client-carried expected-STATUS re-matches after the cycle so it cannot block the replay re-suspension; only the monotonicfoster_homes.versioncounter distinguishes the replay, and same→same transitions are already 400 by matrix so the plain retry was never the live hazard; G1 contacts/court-orders wire semantics: missing 400 / stale 409 / bump every write),update_training(IDF
the fleet-standardAND active=trueits delete already had — soft-deleted training was mutable) /update_kinship_option/update_education/update_health(IDF no-op gates; education/health additionally SUP — their unconditionaleducation.updated/health.updatedstaging was 2 of the audit’s 5 event-dups; kinship/training stage no events), exchangeupdate_partner(version CAS via REQUIREDIf-Match— kills the deactivated-partner resurrection;delete_partner’s deactivation ALSO bumps `version, else a pre-deleteIf-Matchwould still match post-delete and resurrect through the window),delete_partner(AND active=true; double-DELETE now 404 per the fleet convention — previously 204 forever re-stampingupdated_at),update_agreement(approved_by/approved_by_name/approved_atall FWWCOALESCE(col,$)— one approval stamp, first wins; always-armedAND status=$expectedCAS; the store fn’sfetch_one500-on-missing bug becomes a clean 404 in passing),update_icpc_request(AND status=$expected— closes the TOCTOU; the stagedicpc.status_changedfrom_statusis now proven by the pin, SUP). -
G3 security + reporting + rules + financial (as designed from the route audit; the generic "status/version CAS" resolves per route):
update_review(always-armed status pin from the validated pre-read + the missingAND active = true— a soft-deleted review was still mutable — +fetch_optional: a lost race was a 500),update_nist_control(status pin +fetch_optional500 fix),update_admin_unit/update_detection_rule(version CAS via REQUIREDIf-Match— no status column exists and a stale replay clobbers reference data / re-enables a disabled detection rule; API/CLI-only surfaces), securityupdate_partner(version CAS via REQUIRED bodyversion— the G1 allegations pattern because a craig-web form drives it; suspension cycles are ABA-vulnerable exactly like exchange partners, so a status pin cannot block the stale re-suspension),update_major_change(the event gates on the actual false→true flip read UNDER the update tx — the old gate re-stagedsecurity.major_change.reportedon every repeated{reported_to_acf: true}PUT),resolve_issue(AND resolved = false; an already-resolved replay converges to 200 returning the ORIGINAL resolver — attribution is first-write-wins), AFCARS/NCANDS review/approve ×4 + transmit ×2 (always-armedAND status = $expectedfrom the validated pre-read — closes the approved→reviewed pre-read-clobber TOCTOU; transmit events stage only on the guarded row, pinned by an N-concurrent-transmits exactly-one-event invariant test),update_rate(AND active = true— a tombstoned rate was still mutable — + version CAS via REQUIREDIf-Match; newrate_tables.version),submit_claim(AND status = 'draft'+ SUP — concurrent double-submit double-publishedfinancial.claim_submitted; pinned by the payments-suite 8-way-race shape),update_rule_set(revision CAS via REQUIREDIf-Match— the trigger-bumpedrule_sets.revisionmints a fresh token for every write, so a stale replay was LEGITIMIZED into the JDM live cache; the import route arms the same store predicate from a locked in-tx read). Version columns ride one migration per service (security ×3: partners / admin_unit_registry / detection_rules; financial ×1: rate_tables); reporting and rules need none (status-only / revision exists). -
G4 convert_report resumability (as designed from the wiring recon): auto-link
rollup move to a NEW craig-cases consumer of its owncase.report_convertedon the Track A substrate; the post-commit pipeline call is deleted. (Audit:conversion.rs:82-186— a response-lost retry short-circuits and the winner dying post-commit skips the pipeline forever.) Mechanics: the event payload gainsseed_case_id(the consumer cannot otherwise reproduce the request’s household seed); queuecraig-cases.eventsbound tocase.report_convertedvia the placement A2 subscriber shape (handle_idempotently+tx_handler, Criticalevents-subscriberworker) — the #1202 broker credential for craig-cases widens to covercraig-cases\.events(definitions.json + thebroker_permissionsfleet pin + the deployment guide’s rabbitmqctl forms, in lockstep per the amendment). Auth: the consumer mints the CASES SERVICE PRINCIPAL bearer (br.service_token, the financial A4 S2S-inside-handler pattern, rules calls bounded inside the 15 s attempt deadline) replacing the borrowed user header. Atomicity split, stated honestly: per-link inserts keep their OWN #799 link+audit transactions and converge by thereport_personsUNIQUE (replay skips); the ROLLUP event alone stages on the claim transaction — exactly-once by the inbox claim. Semantics change (pre-1.0, CHANGELOG’d): convert’s 200 now means "referral committed, links pending"; the synchronous-link battery tests move to bounded polling via the rollup completion oracle; a raw-envelope redelivery leg (the financial placement_ended_void shape) pins the inbox dedup; a degraded pass (rank failures) refuses consumption so transient faults become counted retries. The #799 fault leg became the DEGRADED-PASS leg end-to-end: its outbox trigger self-disarms on read-committed visibility of the claim row’serror_count >= 1(attempt 1 cannot see its own uncommitted claim; the immediate strike-1 requeue redelivery can), pinning refusal-counted-once
convergence + the per-attempt link/audit atomic pair in one deterministic arc — a PERSISTENT fault cannot converge by design (post-strike-2 resumption is the ADR-022 §D4 admin replay, operator-paced). -
Audit-proven CONVERGENT routes receive characterization tests pinning their guards (grouped into the G MRs).
§R — Route disposition ledger (binding; 156 routes)
Every mutating route carries exactly one class; the classes sum to the per-service totals
(40 CONVERT + 7 §U + 2 SECRET + 1 LIFETIME + 16 CONFORMING + 7 EXEMPT + 2 LEASED + 7 FORWARD
+ 38 §G-FIXED + 36 PROVEN = 156). Verdicts for the non-create routes come from the two
2026-07-31 handler-by-handler audits (every handler + store fn + event staging read;
guard citations in the audit columns below). Per-class actors: direct user / BFF-lifted actor
via acting_worker() / service principal (federal_export, jws-replay-check, intake forwards)
/ partner (API key + JWS).
CONVERT — 40 (claim + IntentV1 + replay): cases 14 — create_person, create_referral, create_allegation, create_investigation, submit_safety_assessment, create_case, add_household_member, create_case_plan, create_task, create_contact, create_court_order, create_follow_up, record_decision, create_person_link (natural-key backstop retained); placement 6 — create_foster_home, record_training, create_placement, create_kinship_option, create_education, create_health; exchange 5 — create_partner, create_agreement, create_icpc_request, submit_home_study (+ DB UNIQUE fix), send_exchange (202 replay); financial 5 — create_adjustment, create_rate, create_import_batch, generate_claim, preview_review_sweep; security 6 — create_review, create_nist_control, create_major_change, create_admin_unit, create_detection_rule, create_partner; reporting 2 — generate_afcars, generate_ncands; rules 2 — create_rule_set (backstop retained), import_rule_set (wrapper).
§U-CLASS — 7: cases contact attachments, report attachments, court-order document (replace-variant); placement home_documents (order-normalized); exchange icpc attachments; reporting export_afcars, ncands_export.
ONE-TIME-SECRET — 2: issue_key, rotate_key (lifetime pair columns). LIFETIME — 1: create_report (received_request_id + new hash compare).
CONFORMING — 16: financial create_subsidy_agreement, subsidy transitions, activate, terms,
reviews, completion, reschedule, generate_subsidy_payments, sweep execute, import
stage/abort/finalize (12); cases jws-replay-check; security register_signer_key; keyring
register + replay-check. (convert_report REMOVED from conforming — §G4.)
EXEMPT — 7: calculate_payment (persists nothing), batch_lookup, federal_export (each retry
IS another PII read-audit export — noted for audit consumers), test_connectivity,
partners::verify, rules evaluate (per-call audit row by design), detection run (audit:
per-alert uq_security_alerts_standing_rule arbiter proven — the losing tx rolls back
including its staged event; lease absence is a cost, not a correctness, issue).
LEASED/DRAIN — 2: archive run, purge (audit: per-row exactly-once proven — objects-first
count-mismatch-rollback prune tx / WHERE purged_at IS NULL stamps with success-gated
events; a response-lost retry legitimately advances the drain — runbook note).
FORWARD — 7: all intake routes (client-held id protocol; the cases-side claim is the arbiter; SHINES-mode = at-least-once).
§G-FIXED — 38: the 30 stale-replay hazards (incl. the narrow update_task/
update_placement and mild delete_partner variants) + 4 event-dup routes + the
update_icpc_request TOCTOU + convert_report (§G4), each named in §G with its fix. Audit
citations (hazard → cited gap): update_person store/persons.rs:98-122 (COALESCE,
verification-triple reset on same-value SSN); attest_ssn :174 (no ssn_verified=false
conjunct — revoke-then-replay resurrects); revoke_ssn :187-204 + clear_ssn :210-231
(unguarded + dup audit events); update_allegation store/referrals.rs:192;
update_investigation store/investigations.rs:92 (event refires on closed-row edits);
update_case store/cases.rs:96 (case.closed on any closed-row edit; stale assignment
event); update_case_plan store/case_plans.rs:91; update_task :252 (narrow:
completed_at-only bodies); update_contact store/contacts.rs:88 (no active=true);
update_court_order store/court_orders.rs:131;
update_foster_home store/foster_homes.rs:161 + license cycle transitions.rs:29-40;
update_training :324; update_placement (narrow: non-status fields, store/placements.rs:163);
update_kinship store/kinship.rs:107; update_education
store/education.rs:125; update_health store/health.rs:126; exchange update_partner
store/partners.rs:96 (resurrection); exchange delete_partner api/partners.rs:424
(unguarded soft-deactivate); update_agreement store/agreements.rs:92 (approved_at
re-stamp); update_icpc_request store/icpc.rs:86 (TOCTOU); security
update_review/nist/major_change/admin_unit/detection_rule/partner (stores: reviews.rs:128,
nist.rs:166, changes.rs, admin_units.rs:112, detection_rules.rs:78, partners.rs:156);
resolve_issue store/issues.rs:124; AFCARS review/approve store/afcars.rs:125 + NCANDS
store/ncands.rs:127-153; transmit ×2 (event per matched write); update_rate
store/rates.rs:152; submit_claim store/claims.rs:78; update_major_change (event per
call); update_rule_set store.rs:188 (stale-JDM cache reinstall + dup invalidation events).
CONVERGENT-PROVEN — 36 (characterization-tested in §G MRs): payments approve, issue, clear
(store/payments.rs:174-201 CAS + success-gated events); approve_adjustment (locked
validate); delete_adjustment; delete_rate; delete_review; delete_nist_control;
delete_major_change; delete_admin_unit; delete_detection_rule; acknowledge_alert
(WHERE NOT acknowledged + gated event); delete_partner (security, status<>'deleted');
revoke_key (revoked_at IS NULL + gated event); approve_signer_key; revoke_signer_key
(status-guarded); delete_rule_set (active=true, 404 before events); keyring approve;
keyring revoke (Pending/non-Revoked file-store CAS); approve_case_plan (draft-only + parent
lock + rollback-on-miss); delete_task; delete_contact; delete_attachment (contact);
delete_report_attachment; delete_court_order; delete_person_link (hard delete + rollback);
delete_training; delete_kinship_option; delete_education; delete_health; delete_home_document;
delete_agreement (draft-only hard delete); retry_transaction (status='failed' + fresh
correlation_id by design); composition override; composition user-delta (RFC 7232 ETag CAS);
admin events replay (idempotent window reset; delivery contract = consumer dedup).
Per-service tally: cases 7, placement 5, exchange 2, financial 6, security 10, rules 1,
composition 2, keyring 2, shared 1 = 36. Boundary notes: abort_import_batch is counted in
CONFORMING (the import family), and update_icpc_request receives the §G2 CAS so it is
counted in §G-FIXED — neither appears here. Known caveat carried by the delete family:
post-DB blob deletion windows orphan blobs on crash — pre-existing, #1274.
Per-service totals (reproducible): cases 40, composition 2, exchange 13, financial 26, intake 7, keyring 4, placement 18, reporting 11, rules 5, security 29, shared replay 1 = 156.
Client map (per-service surfaces every conversion slice updates): cases → CLI
cmd/{case,person,referral,investigation,intake,plan}.rs; BFF routes/cases/*
routes/intake/; test-lib clients/cases/; k6 smoke/load/stress/soak + services/cases.js
person-suggestions-bench; seed.sh:326; the intake forwarder. placement → CLI
cmd/{placement,home,kinship,health,education}.rs; BFF routes/placement/; test-lib; k6 ×4
services/placement.js. exchange → CLI cmd/{exchange,agreement,icpc}.rs; BFF
routes/exchange/; test-lib; k6 ×4 + services/exchange.js. financial → CLI
cmd/{financial,adjustment,claim}.rs; BFF routes/financial/; test-lib; xtask
import-subsidy-history. security → CLI
cmd/{partner,security,nist,changes,admin_unit,detection,archive,alerts}.rs; BFF
routes/security/partners.rs; test-lib. reporting → CLI cmd/reporting.rs; BFF
routes/reporting/; test-lib (+ exports). rules → CLI cmd/rules.rs; BFF routes/rules.rs;
seed.sh:139-161; test-lib. intake → SDKs ×3 + static JS + e2e intake specs + k6.
Units & DAG
~35 MRs; branches feature/1178- / feature/1182- (docs units docs/*); each ≤500
changed LOC except six named exceptions (U0 docs; A1 ≈60 % tests; A4 tx-threading; B1b
test-only; B2 net-negative deletion; SU1/SU3/Bc3b one-contract families). Sequencing:
U0 ─→ A1 ─→ {A2, A3, A4, A5} ─→ A6 ─→ close #1178 (stop-then-start deploys)
U0 ─→ B1a ─→ B1b ─→ {B2, Bc1, Bc2, Bc3a→Bc3b, Bp1, Be1, Bs1, Bs2, Bf1, Br1, Bru1, G1, G2, G3}
B1a ─→ SU1 ─→ SU2 ─→ {SU3, SU4, SU5, SU6} ─→ SU7 ─→ SU8
A1 ─→ G4 (needs the substrate; cases gains an events consumer)
ALL conversions + SU + G ─→ BF (gated: every service on post-B2 binaries) ─→ close #1182 ─→ UF
Unit contents, estimates, and per-unit ACs live in the epic &77 child issues (#1239–#1271
#1053/#1194); this table is the authoritative scope reference. BF’s DROP of
idempotency_responses ×8 is an explicit steer-authorized deviation from CONTRIBUTING’s
two-step "later release" wording (single-deploy devstack; recorded in the MR + CHANGELOG
Removed, gated on deploy evidence).
Migrations
| Migration | Services | Unit |
|---|---|---|
event_inbox + payload_hash; event_inbox_collisions |
×8 |
A1 |
request_claims |
×8 |
B1a |
upload_attempts + target columns |
cases, exchange, placement, reporting |
SU2 |
allegations / contacts / court_orders |
cases |
G1 |
foster_homes / exchange_partners |
placement, exchange |
G2 |
partners / admin_unit_registry / detection_rules + rate_tables |
security, financial |
G3 |
partner_api_keys pair columns + partial UNIQUE |
security |
Bs2 |
reports.received_request_hash |
cases |
Bc3a |
icpc_home_studies UNIQUE |
exchange |
Be1 |
idempotency_responses DROP |
×8 |
BF |
payload_hash NOT NULL (post-horizon) |
×8 |
#1273 |
Test matrix
Devstack suites run cargo nextest run -p <crate> --profile integration --run-ignored=all;
JUnit under test-results/integration/; receipts copied aside + uploaded for RED/GREEN
claims. #[ignore = "requires devstack"], no guard fns (the guard-vs-F-024 contradiction is
standing — escalated claude-quickstart#27; this plan follows the executable lint).
-
A1: claim+effects+stamp atomicity; deferred
CONSTRAINT TRIGGER … INITIALLY DEFERREDcommit-failure (built fresh — the #804 row trigger fires at statement time, the wrong point); many-concurrent-all-failing ⇒ the cap stays hard at 5; concurrent duplicate ⇒ one effect; sync-construction panic COUNTED; NULL-hash legacy row ⇒ dedup, not poison; collision path (canonical row untouched, own token); unroutable ⇒ failed_at-NULL + watchdog + replay-after-binding-restore;pg_sleepslow winner ⇒ the duplicate dedups; real-broker ladder (terminal at 5, 6th dedups, byte-identical re-surface absorbed); accounting-commit failure ⇒ not counted; timestamp progression; RED receipt vs the old fn. -
A2–A6: per-service truth (#804 rollback-unclaims; security two-writes-atomic; financial S2S-timeout rollback proof + billing e2e; rules audit-in-substrate-tx; reporting zero-partial-issues).
-
B1b: intent proptests (key order, path, actor, jurisdiction, unicode); barrier collect-all N-same-key ⇒ 1 claim + 1 row + N×200; forced claim rollback; REPLAY_ENTITY_GONE; scope/actor/entity_kind mismatch leak-free; pruner-race single-statement pin.
-
Per conversion slice: EVERY route instantiated — concurrent create ⇒ one row; replay ⇒ the endpoint’s own success shape; mismatch 409; missing field 422/400 by body shape; commit-then-response-loss replay; same-key-different-path 409; actor separation.
-
§U (SU7): the six mandated race scenarios.
-
§G: per-route replay pins (each fixed hazard ⇒ replay converges + no dup event + the audit’s named interleaving reproduced) + characterization tests for all 35 PROVEN routes
drain-by-design pins for archive run/purge. -
Bs2: concurrent second-issue; secret non-reexposure; rotate response-loss.
-
Bc3a/b: fresh-captcha retry replays; JTI re-sign; honeypot stable fake id; multipart proxy field forwarding.
Delivery
Per unit (the standing per-unit discipline): implement → fmt / clippy -D warnings / targeted
nextest / quality-budgets / cargo xtask api-docs + check-docs + Antora render on
doc-touching units / coverage on craig-mq + craig-api units → full + ignored test commands
with JUnit receipts → RED-or-recorded-reason with receipts (A1, B1b, B2) → fresh J1–J8
subagent over the staged diff, findings remediated pre-commit → token-gated commit →
backgrounded battery push (tree untouched) → git ls-remote verify → MR (Closes its own unit
issue; Relates to the track anchor; URL reported) → merge per the standing green-local
directive → merged-state verify → J-record note → closing comment (impl + merge SHAs, AC
walk) → epic checkbox + the Status row above → branch delete + remote prune. Close-out:
manual #1178/#1182 closes with program comments; epic &77 completion record (PUT
description); &74 ticks; Plan Completion Audit; archive this page.
Dispositions
| Item | Disposition |
|---|---|
#1053 / #1209 |
A1 (unit issue + justified same-MR close) |
#1194 |
B2’s unit issue |
#1214 |
Out of scope (ordering ≠ atomicity) — ADR-062 non-goals |
convert_report auto-link loss |
Fixed in-program (§G4) |
30 stale-replay hazards + 4 event-dups + the ICPC TOCTOU |
Fixed in-program (§G1–G3) |
Orphan-blob delete windows (pre-existing) |
#1274 |
Typed rejection mapper |
#1272 |
payload_hash NOT NULL |
#1273 |
plans/api-idempotency.adoc |
Archived historical-Done with a supersession banner (U0) |