DLQ Triage Runbook (#1199 / ADR-059)

On this page

How to read and act on the dead-letter pipeline’s signals. Mechanism detail lives in ADR-059 (dispositions, parking replayer, quarantine) and ADR-022 (the audit consumer + threshold alert); broker-side topology and the manual replay procedure live in the deployment guide (§ "Parking and quarantine queues (#1197)").

Signals overview

Every backend service serves Prometheus text at GET /metrics. Instruments are live only when OTEL_EXPORTER_OTLP_ENDPOINT is set on the service — without it /metrics returns an EMPTY 200 (the meter provider is never installed; see the deployment guide’s observability section). The DLQ family metrics come from craig-security (the fleet’s one DLQ consumer, port 8007). In the devstack, docker compose --profile observability up starts Prometheus/Grafana/Jaeger, but you must still set the endpoint env var on the service to light the instruments.

Signal Type Meaning

dlq_queue_depth{queue}

gauge

READY (undelivered) messages in a DLQ-family queue at the last sample — three series: craig-security.dlq (the backlog awaiting the audit handler), ….dlq.parking (waiting out a park TTL), ….dlq.quarantine (terminal until drained). READY excludes unacked in-flight deliveries: quarantine is exact (no consumer), parking undercounts by ≤ 1 (the prefetch-1 replayer holds the head through its TTL sleep), the DLQ itself by ≤ 16 (prefetch).

dlq_depth_last_sample_timestamp{queue}

gauge

Unix seconds of the queue’s last successful depth sample. Check this before trusting a depth value: a frozen depth with an advancing timestamp is a real plateau; frozen both means the sampler is down and the depth is stale, not current.

dlq_captures_total{tier, reason}

counter

Confirmed capture publishes into parking/quarantine, by reason (transient, park_cap_exhausted, permanent, malformed, noncanonical_route). Counts publishes, not distinct messages — an ack failure after a confirmed publish redelivers and double-counts by design. Rates tell you what the handler is deciding; the depth gauges are the stock truth.

dlq_outcomes_total{outcome}

counter

The audit handler’s recording decisions: recorded (new dead_letter_audit row), phantom_absorbed (identical redelivery, no new row), divergent_quarantined (same occurrence token, different content — recorded tokenless to the DB quarantine path).

dlq_handler_panics_total{path}

counter

Handler PANICS converted to errors by the #1203 task boundary, by consumer path (events/dlq/inbox). Any nonzero rate is a code defect, not an outage — this is the disambiguator when dlq_captures_total{reason="transient"} climbs (panics park under the same wire reason as infrastructure trouble).

"quarantining dead-letter delivery" at error! / "parking …" at warn!

log

The per-capture log lines — they carry the envelope id the counters cannot. Every quarantine is an error!.

security.dlq.threshold_exceeded

event

The #1156 crossing alert — see § Threshold-alert interpretation (and its consumer honesty caveat) below.

Backlog growth (craig-security.dlq rising)

A rising DLQ depth means dead letters are arriving faster than the audit handler settles them — or the handler is failing.

  1. Check dlq_outcomes_total rates: a healthy handler shows recorded (plus phantom_absorbed after teardown redeliveries) climbing with the inflow. Flat outcomes with a rising depth = the handler is erroring; check craig-security logs.

  2. Check dlq_captures_total{tier="parking"}: a climbing transient rate means EITHER the handler classifies its failures as infrastructure trouble (DB outage classes — see ADR-059 §D2) and the backlog is riding out the outage in parking, OR the handler is PANICKING (#1203 parks converted panics under the same wire reason). Check dlq_handler_panics_total to tell them apart: zero → outage posture, ride it out (§ Capacity guidance); nonzero → a code defect — the error! log lines carry the envelope id and the bounded panic text.

  3. Repeated "… torn down on settlement failure" error! lines mean broker settlement is failing — the session tears down, redelivers, and backs off to the 30 s cap. Persistent settlement failure is a broker/network incident, not a handler bug.

  4. The upstream cause of dead letters is the ORIGINAL consumer’s 2-strike nack (per-arrival "dead letter recorded" warns name the source queue) — triage the failing consumer service in parallel; the DLQ tier only preserves the evidence.

Parking drain

Parking is self-draining: the replayer returns each capture to the DLQ after parking_ttl (default 10 min), the handler re-runs, and on success the occurrence records and the depth falls. A parked occurrence has no DB row and no threshold visibility — the parking depth gauge is its only footprint (ADR-059 §D8).

  • Depth sawtooths at the TTL cadence during an outage: normal — each failed return re-parks with park_count + 1.

  • After park_cap cycles (default 12, ≈ 2 h total) a capture escalates to quarantine with reason park_cap_exhausted — watch that reason’s rate as the "outage outlasted the budget" tell. A DETERMINISTICALLY panicking handler follows the same road (#1203): parked as transient with last_error = "handler panicked: …", re-invoked once per TTL cycle, quarantined at the cap — dlq_handler_panics_total{path="dlq"} fires on every cycle, so the defect is visible long before the ~2 h escalation.

  • Do NOT consume from {queue}.parking manually — the replayer owns it (prefetch 1, head-blocking sleep). Draining it by hand races the replayer and can only reorder returns.

  • Lowering park_cap in config applies as DISPOSITION POLICY to already-parked captures on their next return (a carried count ≥ the new cap quarantines) — no redeclare, no restart hazard. parking_ttl lives in config too (not queue arguments): the replayer computes each capture’s due time at consume, so a changed TTL applies to every capture it picks up next — only a head already mid-sleep keeps its previously computed due until redelivery or restart.

Quarantine handling

Quarantine ({queue}.quarantine) is terminal until an operator drains it. It holds child-welfare forensic content OUTSIDE the ADR-058 retention machinery: nothing expires it, nothing consumes it, broker disk alarms are the backstop (ADR-059 §D7).

  • Every quarantine publish logs at error! with the envelope id and reason. Non-zero quarantine depth deserves same-day attention; a growing one is an incident.

  • Reasons and what they mean: permanent (message-specific failure — data exception, constraint violation: fix is data-level), park_cap_exhausted (an outage outlasted the parking budget: replay after recovery), malformed (not a parseable envelope — raw bytes are base64-embedded in the capture), noncanonical_route (arrived under a non-dlq.* key — a republish would be unroutable).

  • Drain tooling (#1206): cargo xtask quarantine-drain <queue> --yes automates the procedure — per capture it validates payload._park, republishes returnable captures bytes-unchanged to craig.dlx under the carried original_routing_key (confirmed
    mandatory + persistent, acked ONLY on a clean Ack), and drains malformed/unvalidatable captures to a timestamped NDJSON file (fsynced before their ack). Any unclean confirm nacks-requeue and STOPS the drain — the capture stays in place. Credentials: devstack defaults to the craig-test identity; production passes --amqp-url with an operator-scoped account (the service identities deliberately cannot consume quarantine, #1202).

  • Manual fallback: the deployment guide’s § "Parking and quarantine queues (#1197)" — consume with manual ack, validate the _park capture, republish to craig.dlx under the carried original_routing_key (confirmed + mandatory + persistent), and ack ONLY after a clean confirm. Malformed captures are excluded from replay (no route home) — drain them to a file and ack after the write is flushed.

  • Duplicate copies are a designed outcome (ack failure after a confirmed publish → redelivery → second copy). Tokened duplicates replay as phantoms/divergents downstream — a re-run of the drain over duplicates is SAFE by the same mechanism (documented, not "fixed"); tokenless/malformed captures have NO automatic dedup — correlate by the embedded capture_id.

Threshold-alert interpretation

security.dlq.threshold_exceeded fires on the exact crossing — the arrival moving one event_type’s 1-hour count from 10 to 11 (#1156). A sustained breach re-emits nothing until the window drops under and crosses again; the per-arrival "dead letter recorded" warn is the sustained signal.

  • What pages, what records (#1205): the event’s only consumer is craig-security’s own wildcard audit subscriber — it IS the per-event_type audit_log record of the burst, and that is its whole job. Paging is the shipped Prometheus rule set over the #1199 dlq_* metrics (deployment guide §DLQ alerting recommendations; choice recorded in ADR-022 §Amendment #1205). The rules' burst analogue (increase(dlq_outcomes_total{outcome="recorded"}[1h]) > 10) is the AGGREGATE rate — approximate by design: usually more sensitive (it sums across types), but it can stay silent when the event fires (the event’s per-type window also counts divergent-quarantine rows that series excludes; the counter undercounts across a restart). Whichever signal arrives first, come HERE: the audit_log rows (and the threshold event’s payload: event_type, count) carry the per-type attribution the metric cannot.

  • Window semantics with parking (ADR-059 §D9): parked returns carry the ORIGINAL occurred_at. Returns within the 1-hour window still count and can fire the alert; returns delayed past the window never do — by design (a stale backlog must not masquerade as a current burst). During an outage longer than the window, the threshold can stay silent while parking fills: the depth gauges are the compensating control — watch them, not the alert, during outages.

  • Pre-C2 false-alert disclosure: threshold alerts emitted before the #1181 occurrence dedup landed (2026-07-29) may have been inflated by ack-failure redelivery duplicates — a phantom redelivery wrote a second identical row and could trip the crossing spuriously. The only residues of such an alert are an event_outbox row (~30 d) and an audit_log row (90 d, then archived); no security_alerts row exists on this path, so there is nothing to acknowledge or retract, and audit-class rows are immutable. Treat pre-2026-07-29 threshold-alert counts as possibly inflated — disclosure, not reconciliation (ADR-022 §Amendment #1199).

Capacity guidance

  • Defaults: parking_ttl 10 min × park_cap 12 ≈ 2 h of outage ride-through per capture before quarantine, with up to handler_attempts (3) in-session invocations before the first park. Sized for the common short-outage case; a maintenance window longer than ~2 h WILL escalate the backlog to quarantine — plan to replay it afterwards (that is a bulk manual-ack replay today, #1206 for tooling).

  • Every capture hop is confirmed + mandatory + persistent — parking/quarantine content survives broker restarts (durable classic queues). What it does NOT survive: destructive operator policies. Never apply message-ttl, expires, max-length, max-length-bytes, or destructive overflow to craig* queues (the deployment guide’s MUST-NOTs — expires would delete the consumer-less quarantine queue outright).

  • depth_sample_interval (default 30 s, bounds 1 s..=1 h) trades gauge freshness against broker chatter — three passive declares per tick per DLQ family. The parking-visibility gap during an outage is bounded by it: keep it well under parking_ttl.

  • Broker disk is the quarantine’s real ceiling: RabbitMQ’s disk alarm blocks ALL publishers fleet-wide when it trips, so a neglected quarantine eventually becomes a fleet incident. Drain quarantines promptly; do not let them ride.

Edit this page · latest