ADR-062: Transactional Idempotency Claims — Atomic Effects at Both Ingress Boundaries

ADR-022 §D3.2 (the inbox three-statement claim/stamp shape and its at-least-once caveats) is superseded by this document’s single- transaction attempt machine; ADR-022 §D5 (the persistent idempotency middleware) is superseded in full — the middleware is deleted, and no stored-response replay survives anywhere in the platform. ADR-022’s outbox, retention, occurrence-token, and admin-replay decisions stand unchanged; its §D4 admin-replay at-least-once contract now rests on the handler atomicity this ADR provides, closing the loop its amendments left open.

ADR-055's F8 client-keyed idempotency is NOT superseded — it is grandfathered as a conforming specialization of this doctrine (its pair columns are an in-transaction claim with domain provenance) and becomes the platform-wide pattern’s reference implementation.

ADR-059's re-route of the retry-ladder question to #1053 is resolved here: pacing belongs to the broker and the parking tier; the inbox owns the total counted-attempt budget and the terminal state.

ADR-051's "every ssn write resets verification — same-value included" policy is superseded by §G1 (#1266): the reset now keys on the DETERMINISTIC digest actually changing (ssn_full_hmac IS DISTINCT FROM the incoming sibling). ADR-051 chose the unconditional reset to keep re-entry from re-verifying; under this program’s replay doctrine the same rule made a RETRIED identical PUT destroy a verification granted in between — precisely the stale-replay hazard §G exists to kill. The hmac gate preserves ADR-051’s real intent (a different SSN can never stay verified) while making the identical replay a no-op; the value-bound attestation, revoke, and clear writes gain the matching state predicates (ssn_verified = false / = true / SSN-data-on-file), so their replays re-stamp and re-audit nothing. All other ADR-051 decisions stand.

The event inbox (craig_mq::inbox::handle_idempotently) commits its claim, the handler’s domain effects, and the processed_at stamp as three separate autocommit statements. A crash between handler success and the stamp re-runs a committed handler; concurrent deliveries re-run it in parallel past an unlocked read. Real semantics were at-least-once with idempotency delegated to each handler — and a handler-by-handler audit (2026-07-31, both audit reports transplanted into the plan’s §R ledger) found consumers with no fallback idempotency at all, plus 30 non-create HTTP routes whose "convergent" guards live only in pre-read validators (or nowhere), and 4 routes that stage a fresh outbox envelope per replay.

The HTTP idempotency middleware held a 30-second lease with no ownership fencing (#1182), a 500-window between logical expiry and physical cleanup (#1194) — and was opt-in with zero senders: no client this repository owns (BFF, CLI, SDKs, e2e, k6) ever sent the idempotency-key header. It protected nothing while carrying two open defects.

Lease fencing the existing middleware (claim generations, heartbeats, owner columns): narrows the double-execution window without closing it — the claim and the effects still commit separately, which is the disease itself. Rejected.

Stored-response replay (middleware-style, inside the tx): re-introduces body caps, header projection, staleness, and finalize fencing. F8’s re-derive-from-domain-state model needs none of it. Rejected everywhere, including uploads.

Per-entity pair columns fleet-wide instead of request_claims: ~35 bespoke migrations and store changes for no correctness gain; retained only where the horizon must be entity lifetime (one-time secrets, create_report). Rejected as the general mechanism.

A transaction-aware handler trait (RPITIT) instead of the boxed-future closure: saves one box per event at the cost of a public trait surface under deny(missing_docs). Rejected.

Persisting terminal envelopes in the inbox for unroutable DLX surfaces: the source-queue copy already IS durable if the delivery is requeued instead of rejected. Rejected in favor of requeue-durability.

A forensics-only attempt counter with no cap: unbounded poison-replay ping-pong with no terminal state. Rejected (#1053 resolution).

Pre-1.0 breaking wire changes, CHANGELOG’d per unit: mandatory client_request_id on every convert-class route; two required multipart text fields on upload routes; a wrapper body on import_rule_set and a JSON body on rotate_key; the middleware’s Idempotency-Key header, replay-marker headers, and 422-mismatch status are gone (mismatch is 409).

The inbox handler contract changes from Fn(EventEnvelope) to FnOnce(EventEnvelope, &mut Transaction); every consumer service converts under a stop-then-start deployment barrier (no mixed-binary window).

30 audited stale-replay-hazard routes (plus 4 event-dup routes and the ICPC TOCTOU) gain SQL status/version predicates and success-gated event staging in-program (§G) — including the SSN attestation lifecycle, federal report chains, and live ruleset content.

The dedup horizons are explicit operator knobs (CRAIGREQUEST_CLAIMSWINDOW_DAYS, inbox retention) with named post-horizon behavior; the records-retention schedule remains DFCS-blocked (#1073) and these knobs do not pre-empt it.

Non-goals: event ordering (#1214 stays open); SHINES-bound intake submissions remain at-least-once (no receiving identity — the ADR-042 boundary); load-tooling dedup.

#1304 (2026-08-02) — consumer pool isolation. The §G4 cases events consumer is the fleet’s only inbox handler that makes NESTED pool acquisitions: the substrate’s claim connection is held for the whole attempt while the auto-link pass opens the report read and the per-link #799 own-transactions from the same pool. Sharing the 10-connection service pool made battery-load starvation reproducible (HTTP PoolTimedOut 500s
acquire convoys past the 120 s test kill + rollup-oracle misses, first seen 2026-08-02 — the contention is a feedback loop, since a failed nested acquire counts a rank_failures refusal and the redelivery adds load), and would let a retry-looping consumer eat HTTP request capacity in production. Rule, binding fleet-wide: an inbox handler that acquires connections beyond its claim connection MUST run on a dedicated pool; handlers that work purely on the claim transaction stay on the service pool. As built in cases: EVENTS_CONSUMER_POOL_CONNS = 3 (1 claim + 1 sequential nested acquisition + headroom), sessions tagged application_name=craig-cases-events, the wiring pinned live by the battery via pg_stat_activity.

1034 (2026-08-04) — the rule generalizes to VOLUME-based starvation. craig-security’s audit subscriber works purely on its claim transaction, but it is the system’s highest-volume consumer (it ingests every event every service emits): under full-battery load its attempt cadence alone competed the 10-connection service pool away from HTTP handlers — the 2026-07-14 transient 500 cluster with flaky healthz, invisible in isolation. Amended rule: a dedicated pool is ALSO required when a consumer’s sustained volume can saturate the shared pool, nested acquisitions or not; the claim-tx-only stay-on-the-service-pool default applies to ordinary-volume consumers. As built in security: one 3-connection pool (application_name=craig-security-events) feeds BOTH the audit and DLQ subscribers (each attempt holds one claim connection; 2 structural
headroom); the audit leg is battery-pinned via pg_stat_activity and the DLQ leg rides the same pool object by construction.

#1326 (2026-08-04) — the rule generalizes to ALL co-located domain workers. craig-exchange stacked FOUR worker classes on its 10-connection HTTP pool — the events subscriber (one claim connection per in-flight delivery), the ExchangeSendWorker (up to 8 semaphore-bounded dispatches, each a brief post-adapter finalize tx), and the upload-attempt reconciler, beside the outbox drain. No single class saturates the pool, but under battery-peak pressure (a 7.7 s INSERT stall) every held connection stretches and the AGGREGATE starves HTTP: icpc uploads and transaction GETs 500’d on PoolTimedOut while the acquire probe reported session exhaustion (2026-08-04). Amended rule: a service’s long-running domain workers ride dedicated pools sized structural-requirement-plus-headroom; the shared service pool is for request handlers (and the outbox drain, whose failed pass is retried next tick and 500s no one). As built in exchange: craig-exchange-events (3), craig-exchange-send (3 — finalize txs open AFTER the adapter send, so holds are brief and the pool bounds DB concurrency below the dispatch semaphore), craig-exchange-reconciler (2); events + send are battery-pinned via pg_stat_activity (the reconciler’s 5-minute first pass puts its tag outside any test budget; it rides the same constructor).

#1363 (2026-08-08) — cheap losers: same-key claim racers no longer hold blocked connections. The original claim_first made N same-key racers serialize ON THE WINNER’S UNCOMMITTED CLAIM ROW: each loser’s INSERT … ON CONFLICT waited inside its own open domain transaction, pinning a pooled connection for the winner’s whole transaction duration — an 8-way duplicate volley held 8 of the 10 default connections, and under load the acquire timeout surfaced as raw 500s that broke the all-succeed contract (battery-observed on POST /v1/security/admin/units). A bounded in-transaction lock wait was REJECTED: any in-tx shed fires exactly when the winner is slow, turning contract-guaranteed 200s into refusals. As built: claim_first opens with a pg_try_advisory_xact_lock prelude (a dedicated 2-arg classid — a keyspace disjoint from every 1-arg hashtext fleet user — keyed on a fold of the client_request_id; cross-id collisions cost one spurious retry, never correctness). The winner proceeds — among prelude-running writers its INSERT can then only meet COMMITTED rows, so classification never blocks (two recorded residuals degrade to the OLD blocking behavior, never to corruption: a mixed-version rolling deploy where an old binary claims without the lock, and an uncommitted pruner DELETE briefly blocking the arbiter). A concurrent racer receives an INTERNAL contended sentinel; its transaction aborts in ~milliseconds (zero effects ran — claim_first is contractually the first statement) and AppState::execute_claimed_tx (the claim-bearing replacement entry for execute_within_tx; AsyncFnMut, so the closure re-runs with per-attempt clones) retries the whole domain closure after a POOL-FREE exponential backoff (25 ms doubling to a 400 ms cap, ~8 s budget). The re-run classifies against the committed winner (Replay — all-succeed holds, however slow the winner) or wins the lock itself if the winner rolled back. Recorded bound (§H-class guarantee): an N-way same-key volley sustains ONE held connection (the winner’s) plus millisecond-scale probes; budget exhaustion and pool-acquire timeouts on the transaction runners surface as typed 503 (SERVICE_UNAVAILABLE), never raw 500s. Pool RESIZING is explicitly not the fix — it moves the exhaustion cliff without unbinding loser holds from winner latency.