ADR-059: Bounded DLQ Retry — Error Taxonomy, Parking Replayer, Quarantine, Settlement Teardown

A delivery is acked ONLY after a committed audit row or a client-confirmed durable publish. Every hop in this design is a client publish with publisher confirms + mandatory
persistent delivery mode, acked only on Confirmation::Ack with no returned message — there is no broker-managed dead-letter hop anywhere in the retry path.

Bounds, named: dedup is exactly-once per occurrence token while its canonical row is in the audit hot window (post-archive redelivery re-records — the C2 R9 bound); the dual producer paths can still mint two tokens for one terminal failure (the C2 R1b bound — producer-side unification is #1053’s scope); tokenless (raw/foreign) deliveries have no automatic dedup and gain more re-execution paths under teardown redelivery (disclosed, rate-bounded by D5’s cross-cycle backoff); operator policies (message-ttl, expires, max-length*, destructive overflow) on craig* queues can silently destroy any queue-resident copy — an operator convention the deployment guide prohibits, not runtime-enforceable with AMQP-only credentials.

craig_mq::Disposition { TransientRetry, TransientPark, Permanent }
DlqError<E> { disposition, source } (the InboxError<E> generic precedent — no type erasure). subscribe_dlq handlers return Result<(), DlqError<E>>; the HANDLER — which sees the concrete error — chooses the disposition; craig-mq never downcasts or string-matches the source. The consumer’s classification (craig-security): Permanent ONLY for provably message-specific failures (SQLSTATE class 22 data exceptions, class 23 constraint violations, serializing the message’s own envelope); class 40 → TransientRetry (retry-helps); pool/connection/io/TLS/protocol + classes 08/57/53 → TransientPark (the outage short-circuit — in-session retries buy nothing during a hard outage); class 42 + unknown SQLSTATEs + decode/schema errors → TransientPark (systemic drift must never quarantine a whole backlog); the missing-canonical-row invariant bail → TransientRetry (a real race with the ADR-058 archive pruner — a retry records the occurrence). Fail-safe default: TransientPark.

Divergent-content token collisions stay HANDLER-internal (the C2 R7 tokenless DB-quarantine row) — the quarantine queue of this ADR and the tokenless DB-quarantine row of ADR-022 are different mechanisms with different purposes.

Ok → ack. TransientRetry → up to handler_attempts TOTAL invocations in-session (retry_delay(n) = 100 ms · n + jitter(0..=100 ms), checked arithmetic, sleeps racing shutdown) → exhausted → PARK. TransientPark → PARK immediately. Permanent → QUARANTINE. Malformed body (undeserializable) → QUARANTINE with reason malformed and the raw bytes base64-embedded — never silent-dropped (this closes the C6 gap; run_session’s undeserializable branch is unchanged — the broker dead-letters it here). A delivery whose routing key lacks the canonical `dlq. prefix is never parked (its republish would be unroutable — the DLQ binds dlq.#): QUARANTINE, reason noncanonical_route. A carried park_count >= park_cap → QUARANTINE, reason park_cap_exhausted. Any publish failure, confirm outcome other than a clean Ack, or mandatory return → NO ack → SessionEnd::SettlementFailed.

{queue}.parking: durable, classic, NO TTL, NO DLX arguments — a plain holding queue (published to via the default exchange; the return routing key lives in the capture wrapper, not the transport). subscribe_dlq spawns a SECOND supervised consumer on it — own session/connection, prefetch 1, sharing a per-subscription CHILD of the shutdown token (#1232: service shutdown propagates parent→child; the consumer arm cancels the child on its abnormal end so the joined task’s death is visible to handle supervision): receive a parked message → due = parked_at + parking_ttl → sleep until due (racing shutdown; FIFO + a uniform TTL means due times are monotonic, so head-blocking IS the delay mechanism) → republish to craig.dlx under the ORIGINAL dlq.* routing key (confirmed + mandatory + persistent) → ack the parked delivery only on a clean confirm. A broker crash mid-sleep redelivers the parked message and the due time is recomputed — safe. The client republish carries no x-death header (the broker interplay for client-carried history is undocumented — never relied on), so the return leg derives identity purely from the capture wrapper.

lapin settlement resolves Result<bool>: Ok(false) — nothing was actually sent (killed or already-settled acker) — is a settlement FAILURE; success is Ok(true) only. ALL ack/nack failures in BOTH run_session and run_dlq_session end the session with SessionEnd::SettlementFailed; the supervisor DROPS the old session before rebuilding (a healthy-connection settlement failure must not leave the old consumer holding unacked deliveries while a new one builds), and the broker requeues the unacked prefetch window. Backoff carries across consecutive SettlementFailed ends and resets ONLY on a non-settlement end — a settle-one-fail-one flapper still doubles to the 30 s cap. Teardown logs at error!. Disclosures: teardown redeliveries arrive redelivered = true and consume the 2-strike ladder’s first strike (a DLX-rate inflation after settlement incidents — not loss); a persistent settlement failure is an acceptable-loud bounded loop (acks fail by channel state, not message content). After a confirmed publish, the corresponding ack runs un-raced against shutdown but bounded (5 s) — a hang tears down; the unacked original redelivers.

JSON body, the _dlx precedent — no AMQP-header novelty. Outer EventEnvelope mirrors the original’s id/source_service/event_type (so a fall-through recording keeps threshold semantics; the malformed variant uses a fresh v7 id, source_service = the DLQ queue name, event_type = "dlq.malformed"); payload:

(the malformed variant carries original_body_b64 instead of original_envelope). The ORIGINAL occurrence token travels verbatim — no new grammar prefix; dedup identity survives the park hop, and a parked return that was already recorded absorbs as an exact C2 R7 phantom. Size safety: last_error truncated to 2 KiB (+ sha256 of the full text); if the serialized capture would exceed MAX_CAPTURE_BYTES (12 MiB, under RabbitMQ 4.x’s 16 MiB default max_message_size), the embedded body is truncated to a prefix + sha256 + truncated: true — a capture can never become unpublishable and wedge the settlement loop. Unwrap happens in build_dead_letter_delivery BEFORE the handler; validation mirrors the C2 R6/R2 rules (v == 1, original_queue agreement, token grammar when present). park_count parses as any valid u32 — the cap applies as DISPOSITION POLICY, never as wrapper validity (a cap decrease or mixed-replica caps must not turn legitimate parked messages into forged fall-throughs). An invalid wrapper falls through to normal derivation and records as the wrapper envelope — forensically intact, bounding forgery amplification.

{queue}.quarantine: durable, classic, no TTL, no DLX, no consumer. It holds child-welfare forensic content OUTSIDE the ADR-058 retention machinery — named, not silent: ownership is the operator’s (drained per the deployment guide’s manual-ack replay procedure — consume, validate the capture, republish to craig.dlx confirmed+mandatory, ONLY THEN ack; malformed captures are excluded from replay and drain to file); broker disk alarms are the backstop; every quarantine publish logs at error! (envelope id, reason, queue) alongside the #1199 metrics — the log line carries the envelope id the counter cannot. The parking/quarantine depth gauges this section named as a launch requirement of C5 (#1199) landed there (dlq_queue_depth, see the Amendment below and the DLQ triage runbook). Duplicate copies (ack-fail after a confirmed publish → redelivery → second copy) are a designed outcome; tokenless/malformed captures have NO reliable automatic dedup — each capture carries a fresh capture_id for operator correlation.

DlqRetryConfig { parking_ttl, park_cap, handler_attempts, depth_sample_interval } — named-field + Default (production: 10 min / 12 / 3 total invocations / 30 s sampling → ~2 h outage ride-through before quarantine; while parked an occurrence has no DB row — since #1199 the dlq_queue_depth{queue="….parking"} gauge is its visibility). Validated at subscribe_dlq (handler_attempts ∈ 1..=10, parking_ttl ∈ 1 s..=24 h, park_cap ∈ 1..=100, depth_sample_interval ∈ 1 s..=1 h) with a typed DlqConfigError; the subscribe error becomes SubscribeDlqError { Config, Amqp }.

Parked returns carry the ORIGINAL occurred_at: returns within the 1-hour window (short outages — the common case at the 10-minute default TTL) still count toward the #1156 threshold and can fire the alert; only returns delayed past the window miss permanently — R5 working as designed (a stale backlog must not masquerade as a current burst). The #1199 dlq_queue_depth gauges are the compensating control for outage-masked bursts (landed — see the Amendment).