ADR-022: Event Durability + Idempotency (Outbox, Inbox, DLQ, Replay, Idempotent Middleware)

On this page

Status

Accepted (2026-04-28). DLQ wiring (Step 4 of the implementation plan) shipped first; outbox/inbox/replay/idempotency middleware follow in Steps 8-13. Implementation tracked in Platform Stabilization plan § Steps 4, 8, 9, 10, 12, 13.

Retention policy (load-bearing) — corrected + ENFORCED by #965/#1129 (ADR-058)

  • event_outbox: 30 days, published rows only (published_at IS NOT NULL, windowed on created_at). Pending rows are retained indefinitely: they are the only durable evidence of an undelivered notification (there is no outbox retry cap or DLQ), and the outbox_unpublished_beyond_grace invariant is their operator signal. The pre-#965 "both published and NULL rows" wording here and in the (sqlx-checksummed, immutable) migration comments is SUPERSEDED by this section + ADR-058.

  • event_inbox: 31 days — strictly greater than the outbox window. The old "matching/equal" wording was wrong at the boundary: the two tables are pruned asynchronously by different service databases, so equality lets a replayed outbox row arrive after its dedup record was pruned. The precise invariant is inbox ≥ outbox + (max prune cadence + clock skew); the 1-day margin dominates both, and ServiceSettings::load() REJECTS inbox < outbox + 1. Window changes move fleet-wide only.

  • idempotency_responses: 24 hours TTL (response cache, not durability).

Enforcement: the hourly, jittered, advisory-locked prune inside every OutboxWorker (craig-mq::retention; batched, cycle-capped, 0 disables per table) — since #965/#1129, ADR-058 carries the full design. The admin replay endpoint mechanically refuses ranges older than the outbox window (a replay past the dedup horizon would re-execute consumer side effects); replay inside the window remains the deliberate operational signal, and application-layer idempotency must hold across it.

Polling-vs-LISTEN/NOTIFY abandonment criterion

The outbox worker uses 1s polling in v1. Upgrade to Postgres LISTEN/NOTIFY when either of these triggers fires for sustained 24h windows:

  • outbox.depth p95 > 500

  • outbox publish lag p95 > 5s

Without an explicit metric threshold "revisit later" becomes a permanent TODO.

Multi-valued audience for cross-service token reuse prevention

This ADR covers durability invariants only. Audience semantics live in ADR-021; included here as a pointer since both ADRs are part of the same platform-stabilization wave.

Related: ADR-003 (RabbitMQ topic exchange topology), ADR-014 (shared client patterns).

Context

The external review (2026-04-28) and four internal audit subagents identified a cluster of related platform-invariant gaps in the eventing layer:

  1. Events not transactional with the DB write. Every service-side events.rs follows the pattern:

    if let Err(e) = publisher.publish(&envelope).await {
    tracing::warn!(error = %e, "failed to publish case.referral_created");
}

    DB write commits; publish fails; downstream services never see the event. Two systems silently inconsistent.

  2. Consumer-side dedup not implemented. The wildcard audit subscriber on craig-security and the rule-evaluation subscriber on craig-rules will both re-process duplicate envelopes if a producer retries. Some side-effects are idempotent at the application layer (e.g., audit log inserts with unique constraints); others are not (rule evaluations writing to rule_evaluations).

  3. DLQ pretense. crates/craig-mq/src/subscriber.rs:132-138 logs "sending to DLQ" via nack with requeue=false, but subscriber.rs:88-90 declares queues with FieldTable::default() — no x-dead-letter-exchange arg, no craig.dlx exchange bound. Failed messages just vanish on broker default config.

  4. No replay mechanism. If a downstream consumer’s logic was wrong and the messages were already consumed (or the broker was offline), there’s no way to surface the events again for reprocessing. This is exactly the disaster-recovery scenario where durable events matter most.

  5. Idempotency middleware brittle. crates/craig-api/src/idempotency.rs uses an in-memory Arc<DashMap> keyed by claims.sub:idempotency_key. Doesn’t survive restart. Doesn’t span instances. Plus a real bug at line 89-101: if response body exceeds 2 MiB, the middleware silently empties the response. Not safe for offline- worker retries that may span hours.

These are bundled into a single ADR because the architectural decisions are interconnected: transactional outbox needs an at-least-once-delivery contract, which needs a real DLQ and consumer-side idempotency, which needs replay to recover from upstream-bug scenarios, which needs persistent idempotency middleware on the API to prevent accidental double-writes during replay. Splitting into 5 ADRs would invite contradictory design choices.

Decision

Five interlocking patterns:

D1. Transactional outbox (publisher side)

Per-service event_outbox table:

CREATE TABLE event_outbox (
    id              UUID PRIMARY KEY DEFAULT uuidv7(),
    aggregate_id    UUID,
    event_type      TEXT NOT NULL,
    envelope        JSONB NOT NULL,
    created_at      TIMESTAMPTZ NOT NULL DEFAULT now(),
    published_at    TIMESTAMPTZ,
    attempts        INT NOT NULL DEFAULT 0,
    last_error      TEXT,
    CONSTRAINT event_outbox_published_check
        CHECK (published_at IS NULL OR published_at >= created_at)
);
CREATE INDEX idx_event_outbox_pending
    ON event_outbox (created_at)
    WHERE published_at IS NULL;

Retention: 30 days. Replay window is 30 days; older rows cleaned up.

Publisher::publish_in_tx(&mut Transaction, &EventEnvelope) → Result<(), sqlx::Error> stages the envelope to the outbox in the caller’s transaction. If the domain write rolls back, the staged envelope rolls back with it. Atomic.

crates/craig-mq/src/outbox.rs (new): outbox worker per service. Polls every 1s, selects up to 100 pending rows, publishes via Publisher::publish(), marks published_at = now(). On Rabbit failure: increment attempts, capture last_error, back off (exponential, capped at 60s).

Polling-vs-LISTEN/NOTIFY

V1 uses Postgres polling. Upgrade to PG LISTEN/NOTIFY (with the polling worker as a fallback) when either trigger fires:

  • outbox.depth p95 > 500 sustained over a 24h window

  • sustained outbox lag p95 > 5s

Either trigger files a follow-on plan.

D2. Idempotent inbox (consumer side)

Per-service event_inbox table:

CREATE TABLE event_inbox (
    envelope_id     UUID PRIMARY KEY,
    source_service  TEXT NOT NULL,
    event_type      TEXT NOT NULL,
    received_at     TIMESTAMPTZ NOT NULL DEFAULT now(),
    processed_at    TIMESTAMPTZ
);
CREATE INDEX idx_event_inbox_received ON event_inbox (received_at);

Retention: 30 days, matching outbox. If a replay older than 30 days bypasses dedup, that’s a deliberate operational signal — fix the application-layer idempotency, don’t extend dedup.

crates/craig-mq/src/inbox.rs (new) provides handle_idempotently(db, envelope, handler): atomic INSERT + ON CONFLICT DO NOTHING; if the envelope is already in the inbox, skip the handler. UPDATE processed_at after handler success.

Subscriber wrappers adopt this in Step 9.

D3. DLX/DLQ wiring

  • New durable topic exchange craig.dlx.

  • Every consumer queue declared with x-dead-letter-exchange = craig.dlx and x-dead-letter-routing-key = dlq.<queue-name>.

  • New DLQ consumer in craig-security: queue craig-security.dlq bound to craig.dlx with routing key dlq.#. Handler writes to a new dead_letter_audit table.

  • Threshold alert: count > 10 within a 1-hour window for any single event_type emits a security.dlq.threshold_exceeded event — the per-event_type AUDIT record of the burst (an audit_log row via the wildcard subscriber). The operator-visible alerting mechanism is the shipped Prometheus rule set over the #1199 dlq_* metrics (deployment guide §DLQ alerting recommendations; the choice is recorded in §Amendment #1205). #1156 pinned the emission to the threshold crossing (the arrival moving that event_type’s count from ≤ 10 to 11) — a sustained breach re-emits nothing until the sliding window drops back under and crosses again; the per-arrival dead letter recorded warn stays the sustained signal.

D4. Admin event replay

Per service:

POST /v1/<service>/admin/events/replay?from=<rfc3339>&to=<rfc3339>&event_type=<str>&dry_run=<bool>
Authorization: Bearer <admin-token-with-{service}.admin scope>

Implementation: a single UPDATE event_outbox SET published_at = NULL, attempts = 0, last_error = NULL WHERE created_at BETWEEN $from AND $to, narrowed by AND event_type = $event_type when the optional event_type scope is supplied. Returns the matched-row count and the re-staged-row count (0 on dry_run), echoing the window + scope for audit readability; the call is audit-logged via a tracing::warn!. The outbox worker re-publishes the re-staged rows on its next tick. Inbox dedup suppresses duplicate side-effects for redeliveries within the 30-day retention window on a best-effort, at-least-once basis — it is NOT exactly-once: the handler invocation and the processed_at stamp are separate commits, so a crash in that window re-runs a handler that already succeeded (true exactly-once is the inbox redesign, #1178). The event_type scope lets an operator re-drive only the stream a downstream bug affected instead of the whole window.

Amendment (2026-07-07, #795): the optional event_type scope was added and this paragraph corrected to the as-built. The earlier sketch referenced a replay_count column and returned row ids; neither was adopted — the replay_count open question below was declined, and replay forensics come from the audit-logged tracing::warn! rather than a per-row counter.

Amendment (2026-07-27, #1180): the dedup-guarantee sentence was corrected from "ensures no duplicate side-effects" to the as-built best-effort at-least-once boundary. The inbox opens no transaction — handler invocation and the processed_at stamp are separate commits — so a crash between them re-runs a succeeded handler; exactly-once is the #1178 redesign. Docs/comment-only; no behavioural change (matching crates/craig-mq/src/inbox.rs).

D5. Persistent idempotency middleware

Per-service idempotency_responses table (as-built: the base 20260430182558 migration plus the platform-stab-2 §D1 atomic-claim migration 20260505151229, which renamed the HTTP-status column to status_code and added the claim state machine):

CREATE TABLE idempotency_responses (
    cache_key        TEXT PRIMARY KEY,
    method           TEXT NOT NULL,
    path             TEXT NOT NULL,          -- descriptive only; lookups key on cache_key
    user_sub         UUID NOT NULL,
    body_fingerprint TEXT,                   -- nullable until finalize
    status           TEXT NOT NULL DEFAULT 'succeeded'
        CHECK (status IN ('processing', 'succeeded', 'failed')),
    status_code      SMALLINT,               -- nullable until finalize
    headers          JSONB,                  -- nullable until finalize
    body             BYTEA,
    body_too_large   BOOLEAN NOT NULL DEFAULT false,
    started_at       TIMESTAMPTZ NOT NULL DEFAULT now(),
    finished_at      TIMESTAMPTZ,
    claim_expires_at TIMESTAMPTZ NOT NULL DEFAULT now() + INTERVAL '30 seconds',
    created_at       TIMESTAMPTZ NOT NULL DEFAULT now(),
    expires_at       TIMESTAMPTZ NOT NULL DEFAULT now() + INTERVAL '24 hours'
);
CREATE INDEX idx_idempotency_expires ON idempotency_responses (expires_at);
CREATE INDEX idx_idempotency_responses_processing
    ON idempotency_responses (claim_expires_at) WHERE status = 'processing';

  • cache_key = sha256(method || ":" || path || ":" || query || ":" || user_sub || ":" || idempotency_key) (#1184: query is the raw request query string, "" when absent, hashed VERBATIM. It participates so two same-key POSTs to the same path with different query strings can never collide — the old formula omitted it, replaying the wrong cached response, or 422 on a body mismatch. Semantically-equivalent-but-reordered queries hash differently, so a client must send a stable query for a given key; a deliberate simplicity choice over full query canonicalization. The DB path column still stores the bare path — descriptive metadata only, never used for lookup.)

  • body_fingerprint = sha256(request_body_bytes) — same key + different body returns 422

  • Request body cap: 2 MiB hard. Above 2 MiB → 413.

  • Cached response body cap: 1 MiB. Above 1 MiB → store status + headers + body_too_large = true, body NULL. A replay serves the too-large sentinel response — NOTHING re-executes (the same key never re-runs the handler, and re-sending under a NEW key would duplicate the mutation; the safe client recovery contract is #1182’s open question). The LIVE winner always receives its full body — see the #1183 amendment. Asymmetric on purpose — caching multi-megabyte responses balloons the table.

  • Cleanup: DELETE FROM idempotency_responses WHERE expires_at < now() every 1 hour.

Cross-instance: the table is shared across service instances connected to the same DB, so two replicas behind a load balancer dedupe correctly.

Amendment — #1128 (2026-07-25): claim-lease drain + publisher confirms (epic &71)

The 2026-07-25 performance pass reworked the outbox drain:

  • Claim-lease, publishes outside the lock window. The original drain held FOR UPDATE row locks + a pool connection + an open transaction across every AMQP publish in the batch — a slow broker pinned locks and xmin for up to batch x publish-latency. Rows are now CLAIMED in a short tx (FOR UPDATE SKIP LOCKED + a claimed_at stamp; migration 20260726040000_outbox_claim_lease.sql x8 services), published with no locks held, then stamped. A crash between claim and stamp heals by lease expiry (60s) — at-least-once, which this ADR already accepted; the inbox dedup remains the consumer-side boundary. A failed publish releases the claim immediately.

  • No throughput ceiling. The worker drains until a pass comes back short, so bursts clear within the tick instead of trickling at BATCH x poll-rate (~100/s before). LISTEN/NOTIFY remains the latency upgrade path.

  • Publisher confirms enabled (bootstrap confirm_select): published_at now means "broker ACKed", not "written to socket"; a broker Nack maps to a typed publish error and the row stays pending. The per-publish confirm RTT sits outside any lock window.

Amendment — #1179 (2026-07-28): the arbiter-backed idempotent consumer (first instance)

The inbox is best-effort at-least-once (§D3.2 as corrected by #1180; exactly-once is the #1178 redesign), so "handlers must be idempotent" is load-bearing — and until #1179 no money-path consumer actually was. placement.activated → per-diem payment is now the reference shape for consumer-side idempotency under this ADR:

  • a DB uniqueness arbiter on a natural key derived from immutable event data (payments_one_perdiem_per_placement_period; lifetime semantics — see the ADR-053 #1179 amendment for why voided must not free that key);

  • the insert arbitrates FIRST (ON CONFLICT … DO NOTHING, fetch_optional), and EVERY side-effect — including the same-tx outbox stage — runs only on a real insert (staged-iff-inserted, the same discipline as the payment-lifecycle CAS);

  • an early probe on immutable payload data absorbs replays before any dependency lookup, so a post-success replay during an outage resolves Ok instead of erroring into the retry ladder/DLQ;

  • a reload-and-compare on the absorb path (the subsidy generator’s persist_expected_row precedent) distinguishes benign replays (structured info) from derivation drift (detection-only structured error).

Consumers with replayable side-effects should follow this shape rather than rely on the inbox alone.

Amendment — #1183 (2026-07-28): body integrity + the post-2xx terminal rule (§D5)

The middleware previously substituted empty/synthetic bodies for successful responses: an over-cap 2xx was emptied by the capped to_bytes and PERSISTED as a canonical succeeded row with an empty body (2xx/empty live and on every replay until expiry — the documented sentinel only materialized at exactly cap+1 bytes), and a finalize_succeeded DB error served 2xx/empty while logging "returning live response". As-built after #1183:

  • The post-2xx terminal rule. Once the handler returns 2xx its side effects are committed, so a delivery failure after that (over-cap body, mid-stream body error, client cancellation) finalizes the succeeded + body_too_large SENTINEL — never failed, which would make the slot re-claimable and invite a Path-D re-execution of committed work. Residue: if the sentinel finalize ITSELF fails the row stays processing and claim-expiry recovery can still re-execute — closing that needs the #1182 claim generation (and the #1194 expiry-window fix).

  • The live winner always receives its full body. Capture is frame-level (http_body_util::BodyStream); the crossing frame is split zero-copy AT the cap, and an over-cap response streams prefix + remainder + rest through to the client while the sentinel commits. Middleware buffering is bounded by the cap. Deltas: an over-cap winner without an explicit Content-Length switches to chunked framing on HTTP/1.1 (HTTP/2 unaffected); the sentinel row commits BEFORE the tail finishes streaming, so a concurrent loser can replay the sentinel while the winner is still downloading (benign — that is what a sentinel is for); trailer frames survive the LIVE path on both arms but are never cached (a replay carries no trailers).

  • Mid-stream 2xx body errors finalize the sentinel and re-emit the error after the buffered prefix — the response STREAM terminates exactly as with no middleware; the client is never handed a silently-truncated "complete" body.

  • Client cancellation (request future dropped post-2xx) fires a drop-guard that spawns the sentinel finalize — the row cannot linger processing on our account. The guard’s write is unfenced like every finalizer until #1182 (a stale guard can overwrite a reclaimed slot with the sentinel — an availability loss, never a fabricated success).

  • Typed finalize outcomes. finalize_succeeded borrows the body (CachedBody::Full/TooLarge; an over-cap Full is stored fail-safe as the sentinel) and both finalizers return Applied / Lost (zero rows — the slot vanished), with a transport error logged as outcome-UNKNOWN (the UPDATE may or may not have committed). The caller keeps its own bytes and serves them regardless of DB outcome.

  • Non-2xx responses pass through untouched (Response::from_parts, no buffering) — oversized error bodies are no longer truncated, and response extensions + HTTP version now survive on every path.

  • Operational constraint: response bodies must NOT lazily hold pooled DB connections (the over-cap path awaits the finalize while the handler’s tail is un-polled; no craig handler body does — materialized Json only).

  • Observability: structured events idempotency.overflow, idempotency.body_stream_error, idempotency.cancel_guard_fired, idempotency.finalize_lost, idempotency.finalize_unknown. Known discontinuity: http_observability completes its latency/inflight metrics when the response OBJECT returns, so over-cap tail streaming, tail errors, and cancellations are invisible to those metrics (documented, not instrumented here).

  • Mixed-fleet honesty: the change is schema-compatible (no migration), but an UN-upgraded replica keeps writing poisoned rows until drained — the behavioral guarantee starts when the last old replica exits. Pre-existing poisoned rows (succeeded, empty body, body_too_large = false) are indistinguishable from legitimately-empty successes and are NOT repaired; they age out at expires_at
    hourly cleanup (mind the #1194 expiry-window race).

Amendment — #1185 (2026-07-28): send-jobs claim lease + generation fencing

exchange_send_jobs (the partner HTTP send queue) had been left on the pre-#1128 claim pattern: no claim timestamp, so its restart-recovery sweep keyed on next_attempt_at — SCHEDULED ELIGIBILITY, untouched at claim — and a job claimed out of a >60s-overdue backlog looked "stuck" instantly; a peer (re)boot re-claimed it mid-send and two workers POSTed the same transmission. The terminal finalizers UPDATEd by bare id, so the racing workers then fought over sent/failed and both staged exchange.sent/exchange.failed. As-built:

  • The #1128 lease, ported — claimed_at stamped at claim; recovery folds into the claim WHERE (claimed_at IS NULL OR claimed_at < now() - CLAIM_LEASE), so lease expiry re-admits rows on the ordinary 1s poll — recovery is CONTINUOUS and the bootstrap-only sweep is deleted. CLAIM_LEASE_SECS = 90 — 3× the verified 30s per-send transport cap (per-adapter send_timeout + the shared-client backstop), compile-time floor-asserted; the outbox’s 60s is too tight for a full partner send.

  • PLUS the generation CAS the outbox does not need — the outbox’s unguarded stamp_published is safe because a double publish is inbox-deduped; a double partner POST is NOT (only craig_correlation_id, adapter-dependent), and send-jobs finalize writes three-way state (job + transaction + events). Every terminal/retry transition therefore carries AND status = 'in_flight' AND claim_generation = $n and reports typed Applied/Stale; on Stale the worker rolls back — no transaction stamp, no events. Conflicting-event emission is closed structurally (events stage only after Applied, same tx). Even the previously status-guarded retry needed the generation: the old guard passed precisely when a NEWER claim held the row in_flight.

  • Transaction-side defense-in-depth — update_transaction_status refuses a terminal target on an already-terminal row (a stale worker can no longer flip success ↔ failed); the guarded failed → pending retry UPDATE remains the only legal terminal exit. Note the transition-spec drift left annotated, not refactored: can_transaction_transition models Failed → Retry → Pending, but production goes failed → pending directly and Retry never reaches the DB; the lease reclaim is expiry WITHIN in_flight, not a status transition.

  • At-least-once, stated: a crash after the partner accepted but before finalize re-sends after lease expiry — a DUPLICATE (identical, never conflicting) exchange.sent can follow; craig_correlation_id is the partner-side dedup boundary and inbox dedup the consumer-side one. Known limitation carried over from the old sweep: a poison job that kills workers mid-send re-claims indefinitely (MAX_ATTEMPTS counts adapter errors only) — a claim-side cap would strand rows, so none was added.

  • Mixed-fleet deploys must drain old replicas first: an old binary both double-POSTs mid-send and carries the UNFENCED bare-id finalize that stamps over new-generation claims. One-time, pre-1.0.

Amendment — #1196 (2026-07-28): consumer-queue identity on the app dead-letter path (epic &75 C1)

The two producers of craig.dlx routing keys disagreed on what original_queue meant, giving one failure two forensic identities:

  • The broker path (nack-without-requeue) routes under the queue’s x-dead-letter-routing-key = dlq.<consumer-queue> — e.g. dlq.craig-financial.events.

  • The app path (inbox retry cap, surface_to_dlx) passed the producer’s envelope.source_service to publish_dlx — e.g. dlq.craig-cases for the very same consumer failure. Worse, two DISTINCT consumer failures of one fan-out event collapsed to the same producer key.

Root cause: the consumer-queue name was dropped at the subscriber’s handler(envelope) boundary and was simply not in scope inside handle_idempotently — source_service was a stand-in for missing information, not a decision.

The fix threads identity, not signatures: handle_idempotently and surface_to_dlx gain a consumer_queue: &str parameter; each service’s subscribe closure passes its queue-name literal (already in scope at every call site). The subscriber’s Fn(EventEnvelope) bound is untouched. The app path now routes dlq.<consumer-queue> and records the consumer queue in the _dlx.original_queue wrapper field, exactly matching the broker path; source_service remains in the wrapper (and the audit row) as producer provenance only.

Pre-1.0 breaking, no data migration: historical dead_letter_audit rows written by the app path carry producer-keyed original_queue values (craig-<producer> rather than craig-<consumer>.events); they are documented as such, not rewritten (the table is ADR-058 audit-class — append-only). This amendment is C1 of epic &75, the prerequisite for a coherent per-occurrence dead-letter identity (C2/#1181); the DLQ consumer’s own log-and-ack forensic-loss defect is C3/#1197.

Amendment — #1181 (2026-07-29): per-occurrence dead-letter idempotency (epic &75 C2)

A DLQ-consumer ack failure redelivers the dead letter; pre-C2 the redelivery wrote a second identical dead_letter_audit row, inflating the #1156 per-hour count and spuriously tripping the exact-crossing threshold. Idempotency is a property of an occurrence: the audit table stays ADR-058 append-only and gains a nullable occurrence_token + occurred_at with a PARTIAL unique arbiter (WHERE occurrence_token IS NOT NULL; ON CONFLICT … DO NOTHING) — zero row rewrites, zero backfill; historical and tokenless rows keep NULL (NULLs distinct — they each record).

Token derivation (craig-mq, at the consume boundary; the handler receives a typed DeadLetterDelivery):

  • App path: surface_to_dlx derives dlxcap:{envelope_id}:{queue}:{inbox received_at epoch} from durable inbox state that exists BEFORE any publish — a retried surface after a lost publisher confirm re-derives the SAME token and the arbiter absorbs the duplicate (per-call minting was itself a double-record bug). The token + surface time travel in the versioned _dlx wrapper (v: 1).

  • Broker path: xdeath:{envelope_id}:{queue}:{reason}:{count}:{time} from the LATEST x-death entry only — {queue, reason} is RabbitMQ’s documented compression key; a latest-entry queue mismatch records tokenless (never attribute another queue’s history); strict UTF-8; occurred_at is trusted only for count == 1 (a repeat death inherits the FIRST death’s timestamp — receipt time is the approximation).

  • Precedence: valid x-death OUTRANKS the wrapper — a replayed app-wrapped envelope the broker dead-letters again is a NEW occurrence.

  • OccurrenceToken is a bounded validated newtype (prefix grammar, ≤ 256 bytes, ASCII-graphic): malformed/hostile input degrades to tokenless capture, never to a failing INSERT the DLQ loop would ack-and-lose.

Suppression is classified, never assumed (R7): on arbiter suppression the canonical row’s immutable facts (envelope id, queue, event type, envelope JSON) must ALL match for a phantom rollback; a DIVERGENT claim of the same token is re-inserted TOKENLESS in the same tx — append-only quarantine, loudly logged, never absorbed, alert-inert on its own turn.

Threshold window re-key (R5/R8): the #1156 count now keys on LEAST(COALESCE(occurred_at, dlq_received_at), dlq_received_at) (occurrence time when known, clamped to receipt — a future-dated occurrence cannot lurk into the window later), served by the matching expression index (EXPLAIN-pinned); the crossing check runs ONLY when the arriving row is in-window, with the gate evaluated in SQL on the same clock and operator as the count (an app-clock skew band could otherwise re-fire a crossed alert). An outage-recovery backlog is recorded but can no longer masquerade as a current-hour burst. Archive aging deliberately stays on dlq_received_at (receipt governs retention; occurrence governs only the window).

Accepted bounds (also on #1181’s superseding-AC record):

  • Dual-path residue: if the DLX publish itself keeps failing, the broker eventually dead-letters the original delivery too — one terminal failure can record both a dlxcap: and an xdeath: row (correlated by envelope
    queue). Structural unification is #1197’s single-authoritative-path scope.

  • Same-second fresh-history replay collision (AMQP 1-second timestamps).

  • Dedup memory = the audit hot window: archival deletes the row and the partial index forgets the token; a post-archive redelivery re-records (pinned).

  • Tokenless (raw/foreign) publishes keep pre-C2 duplicate-row behavior — occurrence idempotency is claimed for CRAIG-produced dead letters only.

  • Trust boundary: a write-credentialed publisher can forge well-formed tokens (predictable grammar). Forging an EXISTING token with divergent content lands in quarantine (recorded); forging it with identical content is a no-op by definition. Note the bound holds per-credential: a write-only publisher cannot consume/ack the DLQ. #1202 (re-pointed from #1198 by the C4 scope decision) closed the credential side: per-service least-privilege accounts replace the shared overprivileged credential, so consume/ack-suppression is confined to craig-security alone, AND topology-suppression is closed — no service can delete/redeclare craig.dlx (exchanges are operator-owned; configure on a shared exchange is denied fleet-wide). The remaining residual is routed injection: any write-credentialed service can still publish a forged dlq.* record to craig.dlx (topic permissions are the future hardening, recorded in ADR-003 Amendment #1202).

Pre-1.0 breaking, no shims: subscribe_dlq handlers take DeadLetterDelivery (both consumers updated); publish_dlx takes the caller-derived token; deploy is single-replica stop-the-world (an old replica records tokenless rows under old window semantics — drain first).

Consequences

  • All ~42 service-side events::publish_* callsites take &mut Transaction instead of &Publisher. Type system enforces the migration.

  • Every domain handler that emits events restructures from tx.commit().await?; events::publish_X(&publisher, …​).await to events::publish_X(&mut tx, …​).await?; tx.commit().await?;

  • RabbitMQ admin UI grows two new exchanges (craig.dlx queue family) and per-service inbox/outbox state in Postgres. Operators see additional gauges (outbox depth, inbox dedup hit rate) once OpenTelemetry instrumentation is wired in (deferred to a follow-on plan; metrics emit hooks are added by Step 8).

  • DLQ consumer audit table grows over time; standard ops practice is to manually triage every alert and either re-publish the envelope (admin replay endpoint covers this) or mark it as known-bad.

  • Idempotency middleware now requires a DB connection. Slightly higher cold-path latency on first idempotent write (~5ms for the cache lookup); cache hits are still fast.

  • No data migration. Existing events that have already published ride through; new events use the new path immediately.

Open questions

  1. Outbox-worker tuning. 1s polling cadence for v1. Abandonment criterion to LISTEN/NOTIFY is documented above; tune after first benchmark.

  2. DLQ alert threshold. count > 10 / event_type / hour is a starting position. Tune after first real broker traffic.

  3. Inbox cleanup vs. retention. 30 days matches outbox. If an offline worker uploads events older than 30 days during a sync, the inbox doesn’t dedup them. Acceptable — the application layer must be idempotent for this case (and it generally is for our event schemas). But document explicitly.

  4. Idempotency response body 1 MiB cap. Conservative; revisit if the body_too_large sentinel surfaces as a UX problem (i.e., real responses regularly exceed 1 MiB).

  5. replay_count column on outbox. Add or skip? Adding lets us forensic-audit "how many replays has this envelope seen." Original recommendation: add (cheap), use for the Step 16 audit story. Resolution (2026-07-07, #795): declined. The column was never added; replay forensics come from the audit-logged tracing::warn! the replay handler emits (acting admin sub, window, event_type scope, matched/re-staged counts), which covers the audit story without a per-row counter. See the §D4 amendment.

Alternatives considered

A. Status quo (tracing::warn! on publish failure, no inbox, no DLQ, in-memory idempotency)

Rejected. External review explicitly flagged each of the 5 components as a P0/P1 gap.

B. Postgres LISTEN/NOTIFY instead of polling

Rejected for v1; revisit per the abandonment criterion. LISTEN/NOTIFY needs a long- lived listener connection per service which complicates the connection-pool story; the polling approach is one extra thread per service and trivially correct. The threshold check is a simple "did we cross the inflection point yet" question.

C. Kafka or NATS instead of RabbitMQ

Rejected. ADR-003 picks RabbitMQ; switching brokers is a different ADR for a different day. The outbox/inbox patterns are broker-agnostic; the same code shape works against Kafka if we ever switch.

D. CDC (Debezium / pg_logical) for outbox publication

Rejected for v1. Operationally heavier than polling; CDC has its own offset and checkpoint state to manage. Polling is simpler and matches the "boring" stabilization posture of the parent plan.

E. Redis for idempotency middleware backing

Rejected. CRAIG has no Redis dependency today (per ADR-013, web sessions live in encrypted cookies for exactly this reason). Adding Redis for one middleware is a new operational dependency; Postgres works and is already there.

F. Message-level dedup in RabbitMQ (deduplication plugin)

Rejected. Plugin-level dedup is broker-specific and operationally fragile; the inbox- table approach is portable, version-controlled, and queryable.

Amendment — #1197 (2026-07-29): the DLQ consumer’s log-and-ack posture is retired (epic &75 C3)

§D3’s DLQ consumer no longer acks unconditionally: handler failures are classified by the handler (Result<(), DlqError<E>> — TransientRetry/TransientPark/Permanent) and the session parks, retries in-session, or quarantines per ADR-059, acking ONLY after a committed audit row or a client-confirmed capture publish; malformed envelopes quarantine with their raw bytes instead of ack-and-drop; every ack/nack failure in BOTH consume sessions tears the session down instead of warn-and-continue (the pre-#1197 posture silently leaked prefetch credit until the consumer stalled). subscribe_dlq gains a validated DlqRetryConfig and the typed handler contract (pre-1.0 breaking; all call sites updated, no shims).

The #1181 amendment’s accepted bound 1 is re-routed: "structural unification is #1197’s single-authoritative-path scope" → producer-side dual-ingest unification now rides #1053 (whose acceptance criteria carry it explicitly — #1053 owns the retry-ladder redesign the dual ingest is a product of). C3’s disposition tier is the single authoritative CONSUMER-side path; the dual dlxcap:/xdeath: ingest bound itself is unchanged until #1053 lands.

Amendment — #1199 (2026-07-29): threshold-alert consumer honesty + the pre-C2 false-alert disclosure (epic &75 C5)

Consumer honesty. §D3’s "emits a security.dlq.threshold_exceeded event for ops dashboards" was aspirational: the event’s ONLY consumer anywhere in the tree is craig-security’s own wildcard (#) audit subscriber, which writes an audit_log row. No dashboard, pager, or operator channel consumes it. The gap was tracked as #1205 (operator-visible routing — with the #1199 dlq_* Prometheus metrics landed, alerting rules over dlq_queue_depth were the candidate mechanism; resolved — see §Amendment #1205 below). Until #1205 landed, the operator-facing signals were the #1199 gauges/counters and the error!-per-quarantine log line — triage flows in the DLQ triage runbook.

Pre-C2 false-alert disclosure (non-retractable by design). Before the #1181 occurrence dedup landed, a DLQ-consumer ack failure redelivered the dead letter and the redelivery wrote a second identical dead_letter_audit row — inflating the per-hour count and able to trip the exact-crossing threshold spuriously. Any such false count==11 alert left exactly two residues: an event_outbox row (pruned at the ~30 d transport window) and an audit_log row (90 d hot, then archived) — no security_alerts row is created on this path, so there is nothing to acknowledge, clear, or retract, and audit-class rows are immutable by ADR-058 rule. This is a DISCLOSURE, not a reconciliation: operators reading pre-2026-07-29 threshold alerts must treat the counts as possibly redelivery-inflated. Recorded here and in the runbook’s threshold-alert-interpretation section.

Amendment — #1205 (2026-08-09): operator-visible threshold routing via shipped Prometheus alerting rules

Resolves the §Amendment #1199 consumer-honesty gap. The chosen mechanism is option 1 of the two the issue named: a shipped set of Prometheus alerting-rule recommendations over the #1199 dlq_* metrics, landed in the deployment guide (§DLQ alerting recommendations) — NOT an event consumer routing to an operator channel. Rationale, recorded per the acceptance criteria:

  • An event consumer routing to "an operator channel" embeds a paging-vendor / transport decision (email, webhook, PagerDuty, …) this tree cannot make for its deployments — exactly the class of choice CRAIG leaves to operators everywhere else (transport security, broker TLS, retention schedules).

  • The #1199 instruments already exist and deployments already scrape /metrics; alerting rules are configuration on infrastructure every deployment runs, adding zero runtime surface, zero new consumers, and zero new failure modes to the DLQ path itself.

  • Honesty bound, disclosed in the shipped rules: dlq_outcomes_total carries no event_type label, so the metrics analogue of the §D3 crossing (increase(dlq_outcomes_total{outcome="recorded"}[1h]) > 10) is the AGGREGATE rate — an APPROXIMATE analogue, usually more sensitive (it sums across types) but able to stay silent when the event fires: the event’s per-type window also counts divergent-quarantine audit rows the outcome="recorded" series excludes, and the counter undercounts across a craig-security restart or a crash between the DB commit and the increment. The security.dlq.threshold_exceeded event remains the authoritative per-event_type audit record for triage; §D3’s wording is resolved accordingly (no aspirational "for ops dashboards" claim remains).

The DLQ triage runbook’s threshold-alert-interpretation section now points at the shipped rules as the paging mechanism and keeps the event as the triage record. ADR-059’s routed-out note is updated in place.

Edit this page · latest