DLQ Forensic Durability & Idempotency (epic &75)

On this page

Status

Child Description Status

C1 (#1196)

Thread real consumer-queue identity to the dead-letter path (handle_idempotently/surface_to_dlx/publish_dlx); all consumer call sites updated

Done (2026-07-28) — MR !1098 merged (f45fd445); RED-proofed routing test + broker-path pin; per-service EVENTS_QUEUE const ties subscribe + inbox uses (J-review hardening)

C2 (#1181)

Per-occurrence dead-letter idempotency: occurrence_token + partial unique index + ON CONFLICT DO NOTHING (row-additive migration), threshold fed by the idempotent log; token-derivation fork RESOLVED (derived tokens — see § C2 design below)

Done (2026-07-29) — MR !1100 merged (2cc7116a); RED-proofed twice (U1a stash pin, U1b 8/13→13/13); both code units J-reviewed, findings remediated pre-commit

C3 (#1197)

Bounded DLQ retry with durable capture: MQ error taxonomy (TransientRetry/TransientPark/Permanent — divergent-content collisions stay handler-internal per the C2 R7 as-built), the client-confirmed parking replayer + quarantine queues, settlement-failure session teardown; folds in the malformed-envelope capture. New ADR-059 — see § C3 design below

Done (2026-07-29) — MR !1103 merged (4a65933e); 17-test live matrix incl. the carried-identity dedup pin; all three code units J-reviewed, findings remediated pre-commit

C4 (#1198)

Pin queue-type & delivery-limit policy (ADR-003 amendment) + mechanical enforcement — see § C4 design below

Done (2026-07-29) — MR !1101 merged (9ad8f702); 7-test live matrix (2 RED, 5 honest pins); mq-topology lint; both code units J-reviewed, findings remediated pre-commit

C5 (#1199)

DLQ observability (backlog/parking/quarantine depth gauges, capture + outcome counters), triage runbook, pre-C2 false-alert disclosure — see § C5 design below

Done (2026-07-29) — MR !1104 merged (42b9d153); the §D7 launch-requirement gauges + outcome counters landed with the triage runbook and the ADR-022 disclosure amendment

Addendum (#1203)

Handler-panic containment: per-delivery task boundary (contain) on all three handler paths, panic → error taxonomy (events nack ladder / DLQ TransientPark / inbox stamped claim + retry cap), dlq_handler_panics_total{path} — see § Addendum design (#1203) below

Done (2026-07-30) — MR !1105 merged (f3d4922e); three observed behavioral REDs (events/DLQ/inbox), all four units J-reviewed, findings remediated pre-commit; inbox-invariant scoping honesty routed to #1209

Addendum (#1202)

Per-service broker credentials + least-privilege permissions + operator-owned exchange topology (closes the configure-includes-delete forensic-suppression hole) — see § Addendum design (#1202) below

Done (2026-07-30) — MR !1106 merged (ba88360d); four units + the F-054 pre-push fix, five J-reviews, two observed REDs; follow-ups #1210/#1211

Epic: &75 (child of &74)
Issues: #1196 (C1), #1181 (C2), #1197 (C3), #1198 (C4), #1199 (C5) — C2–C5 carry Plan::DLQ-FORENSIC; addenda #1202 (broker permission separation, from the C4 scope decision) and #1203 (handler-panic supervisor death, from the C4 design review); routed out of C5: #1205 (operator alert routing), #1206 (quarantine drain/replay tooling — the declined optional scope), #1207 (opentelemetry-observability plan-page reconcile)
Review state: v1 single-MR plan REJECTED by external stop-ship review (2026-07-28); every finding verified against source in six Explore passes; epic decomposition + the per-occurrence-token core approved 2026-07-28. C1 is detailed here (ships with this plan); C2–C5 get their own just-in-time plans, each through plan review (C2 v2 approved 2026-07-29 after a ~20-finding review; C4 v2 approved 2026-07-29 after a ~28-finding review; C3 v2 approved 2026-07-29 after a 5-stop-ship review that replaced the broker TTL+DLX return hop with the client-driven parking replayer; C5 approved 2026-07-29 first-pass after the internal adversarial review).

Context — why #1181 became an epic

The original issue asked for (W1f) fixing the DLQ consumer’s log-and-ack (forensic loss on DB outage) and (W1g) a (envelope_id, original_queue) dedup key on dead_letter_audit. Five verified facts make that un-implementable as written (anchors at 4b9e3c35):

  1. dead_letter_audit is ADR-058 audit-class — "Never hard-deleted while unarchived" (ADR-058 §Audit class); the only sanctioned delete is the archive-then-prune engine. A dedup-backfill DELETE is illegal, and the engine cannot reconcile individual duplicates. A UNIQUE … ON CONFLICT DO NOTHING no-op insert, however, is compatible with the insert-only invariant.

  2. The proposed key is incoherent. The broker DLX path stamps dlq.<consumer-queue> (subscriber.rs:124); the inbox retry-cap path stamps the producer source_service (inbox.rs:317 → publisher.rs:125). Same failure, different key; two distinct consumer failures of one fan-out event collapse to one key on the app path. The real consumer-queue name is dropped at the handler(envelope) boundary (subscriber.rs:455).

  3. A unique (envelope_id, queue) key erases evidence. The #1156 threshold counts raw rows per event_type/hour; a phantom ack-failure redelivery and a genuine later re-dead-lettering carry identical (envelope_id, queue) — the key cannot distinguish them, so it would suppress real occurrences.

  4. W1f needs real machinery. A count-cap-then-ack re-creates the loss; unbounded requeue head-of-line blocks and silently depends on classic-queue-no-delivery-limit semantics that are enforced nowhere (no x-queue-type, no policy, no ADR-003 commitment). Ack/nack failures today warn-and-continue, leaking prefetch credit (PREFETCH 16) until the consumer stalls.

  5. False alerts are non-retractable. A pre-fix inflated count==11 alert leaves only an event_outbox row + an audit_log row — no security_alerts row, no retraction path. Disclosure (C5), not reconciliation.

The core decision (approved)

Idempotency is a property of an occurrence, keyed on a per-occurrence token minted at dead-lettering time — stable across redelivery, distinct across genuine re-occurrences. dead_letter_audit stays the single append-only source of truth: nullable occurrence_token
partial unique index (WHERE occurrence_token IS NOT NULL), insert via ON CONFLICT (occurrence_token) DO NOTHING RETURNING → Option (None = phantom redelivery → short-circuit *before the advisory lock/count/alert, making the threshold structurally immune to redelivery). Zero backfill/UPDATE/DELETE; pre-existing rows keep NULL tokens (historical). Threshold counting stays on the log — no projection table. Token derivation (application- controlled dead-lettering vs broker x-death-derived) is resolved in C2’s plan + ADR.

C1 design (this MR)

# Decision Substance

D1

Queue identity threaded, no Fn signature change

handle_idempotently(db, publisher, consumer_queue, envelope, handler) and surface_to_dlx(…, consumer_queue, …) gain the parameter; publish_dlx already takes original_queue — the app path now passes the real consumer queue instead of envelope.source_service, producing dlq.<consumer-queue> routing identical in shape to the broker path’s x-dead-letter-routing-key. Service closures close over their queue-name literal (exactly as they close over db/publisher) — the subscriber’s Fn(EventEnvelope) bound is untouched.

D2

All consumer call sites updated in the same diff

Every handle_idempotently caller (the per-service event subscribers) passes its queue literal. No compat shim, no defaulted parameter (pre-1.0 hard fork).

D3

source_service demoted to metadata

It remains in the _dlx payload wrapper (producer provenance) and in the audit row’s source_service column; it is no longer used as queue identity anywhere.

D4

Docs

ADR-022 amendment (queue-identity threading + the incoherence it retires); shared-crates.adoc handle_idempotently signature; CHANGELOG (pre-1.0 breaking: original_queue semantics in dead_letter_audit rows produced by the app path change from producer-service to consumer-queue; no data migration — historical rows documented as producer-keyed).

C1 tests

  • Live-broker (devstack) RED-provable: drive an envelope through handle_idempotently to the retry cap → assert the dead_letter_audit row lands with original_queue = "<consumer>.events" (RED today: lands as the producer source_service).

  • Broker-path regression pin: a nack-without-requeue dead-letter records the consumer queue (already true — pinned so C1 can’t regress it).

  • Unit: publish_dlx wrapper still carries source_service as provenance metadata.

C2 design (#1181) — approved v2, 2026-07-29

The token-derivation fork is RESOLVED as derived tokens, no topology change (the epic’s tentatively-preferred application-controlled-DLX rework proved unnecessary; no ADR-003 amendment). v1 of this design was rejected by external review (~20 findings, 10 stop-ship); the v2 rules below carry those remediations (R-numbers cross-reference the review record on MR !1100, merged 2cc7116a, and the ADR-022 amendment).

Occurrence identity

Source Rule

App path (publish_dlx)

Token derived DETERMINISTICALLY from durable inbox state in surface_to_dlx: dlxcap:{envelope_id}:{consumer_queue}:{inbox received_at epoch} — stable across confirm-lost publish retries (R1: no per-call minting; the double-mint was the bug). occurred_at = surface time, carried in the _dlx wrapper with a v: 1 discriminator.

Broker path (x-death)

xdeath:{envelope_id}:{queue}:{reason}:{count}:{time} — {queue, reason} is RabbitMQ’s official compression key (R3). The LATEST x-death entry must match the routing-derived queue, else tokenless (no fallback attribution). occurred_at = the x-death time for count == 1 only; count > 1 reuses the FIRST-death timestamp, so receipt-time is the documented approximation (R4). Queue/reason bytes must be strict UTF-8.

Precedence

Valid x-death FIRST, then valid wrapper, else tokenless (R2 — a replayed wrapped envelope re-dead-lettered by the broker must record as a NEW occurrence).

Validation

OccurrenceToken newtype: prefix ∈ {dlxcap:, xdeath:}, non-empty, ≤ 256 bytes, ASCII-printable-no-whitespace; invalid input records TOKENLESS, never a failing insert (R6).

Conflict handling

ON CONFLICT DO NOTHING on the partial unique index; on suppression, the canonical row is fetched and compared on immutable facts — exact match = phantom (rollback + warn + Ok); divergence = append-only quarantine (re-insert tokenless + error!; R7).

Threshold window

LEAST(COALESCE(occurred_at, dlq_received_at), dlq_received_at) (clock-skew clamp, R8); the crossing check runs only when the NEW row’s effective time is in the current hour (R5 — a stale insert can neither fire nor re-fire the alert).

Accepted bounds (also in #1181’s superseding-AC language + the ADR amendment)

  • Dual-path residue: repeated DLX-publish failure can broker-dead-letter the original delivery too — one terminal failure may record both a dlxcap: and an xdeath: row (correlated by envelope + queue). Producer-side structural unification re-routed to #1053 by the C3 design decision (2026-07-29 — #1053 owns the retry-ladder redesign the dual ingest depends on; C3’s disposition tier is the single authoritative CONSUMER-side path).

  • Same-second fresh-history replay collision (AMQP 1-second timestamps) — real limitation, disclosed (R4).

  • Dedup memory = the audit hot window; post-archive redelivery re-records (R9).

  • Tokenless (raw/foreign) publishes keep the pre-C2 duplicate-row behavior — occurrence idempotency is claimed for CRAIG-produced dead letters only (R9).

  • Token forgery by a write-only-credentialed publisher is bounded by R6+R7: a forged token cannot suppress a divergent genuine record (quarantine); suppressing an identical record is a no-op. Broker permission separation routed to #1202 (originally #1198; re-routed by the C4 scope decision, 2026-07-29).

C4 design (#1198) — approved v2, 2026-07-29

v1 was rejected by external review (~28 findings, 8 blocking); the v2 decisions below carry those remediations (the full review record lands as a note on the C4 MR).

The contract

Decision Substance

Queue type

ALL CRAIG-declared queues are classic with explicit x-queue-type — the production choke-point (prepare_queue, all three subscribe variants) AND the in-tree test helpers (EventCollector, DlxCollector), so the claim is true tree-wide. Quorum migration is a named future ADR gate (post-clustering; revisits the C2 x-death count grammar, at-least-once dead-lettering, and limit posture together).

Delivery limit

Delivery limits are a quorum-queue feature; classic queues do not implement them. Type enforcement transitively neutralizes delivery-limit policies — the machine-checked invariant is the queue type; the no-limit posture follows from it. Residually an operator convention (devstack ships no policies; the deployment guide prohibits delivery-limit / queue-type-affecting policies) — services hold AMQP creds only, so a runtime policy gate is not possible and is not claimed.

What explicit classic buys

(1) Immunity to vhost/node default_queue_type at fresh declare — the real hole (a DQT of quorum today would mint quorum queues with delivery-limit=20; the no-DLX DLQ would drop forensic records on breach). (2) Independence from broker-version type-injection behavior (4.x resolves + compares the type on every declare, so conflict-406 largely pre-exists). (3) Auditable intent in stored args.

Enforcement layers

A declare-time: explicit arg; eager first session ⇒ conflict 406 fails boot. A′ runtime: the reconnect path classifies AMQP 406 → error! (generic topology-precondition message — exchanges are declared before queues, so 406 is not type-specific) with the failing operation’s context; retry-forever kept. B the blocking mq-topology xtask lint — a crate-boundary convention gate (raw declarations only in craig-mq src / test surfaces); it does not prove args, queue_args + its full-table unit pins do. C devstack DQT pins: rabbitmq.conf default_queue_type = classic
quorum_queue.property_equivalence.relaxed_checks_on_redeclaration = false, and the vhost / metadata DQT in definitions.json (canonical exported nested-metadata shape; import never overwrites existing broker state — devstack broker is volume-less, drift cured by recreate; live remedy rabbitmqctl update_vhost_metadata).

Accepted residuals

Exclusive-variant explicit type is hygiene, not enforcement (uuid-named, quorum structurally impossible). Handler-panic supervisor death — RETIRED by #1203 (task-boundary containment on all three handler paths; ADR-003 residual note updated). Pre-3.13-born in-place-upgraded queues may 406 at boot (documented remedy). Policies remain an operator convention. Permission separation — #1202.

C3 design (#1197) — approved v2, 2026-07-29

v1 was rejected by external review (5 stop-ships + ~25 findings). The load-bearing reversal: the broker TTL+DLX return hop is GONE — classic-queue dead-lettering is best effort (ADR-003 reserves at-least-once dead-lettering for the quorum gate), so v1’s "no loss ever" was false at the expiry hop. v2 is the client-driven parking replayer. Full normative design: ADR-059.

Decision Substance

Safety invariant

A delivery is acked ONLY after a committed audit row or a client-confirmed durable publish (confirms + mandatory + persistent on every hop — no broker-managed dead-letter hop in the retry path). Bounds named in ADR-059 §D1: hot-window-scoped dedup (R9), dual-path tokens (R1b → #1053), tokenless no-dedup, operator-policy failure domain.

Taxonomy

Disposition { TransientRetry, TransientPark, Permanent } + DlqError<E> — the handler picks (it sees the concrete error; craig-mq never downcasts). Divergent stays handler-internal (C2 R7). Security’s SQLSTATE map: Permanent ONLY for message-specific classes (22/23
envelope-serialize); 40 → retry; 42/unknown/decode → TransientPark (systemic drift must not quarantine a backlog); the missing-canonical bail → retry (the ADR-058 pruner race).

Parking replayer

{queue}.parking — plain durable classic (no TTL/DLX args → no redeclare-406 hazard, TTL freely changeable). A second supervised consumer (prefetch 1) sleeps each parked message to parked_at + parking_ttl, then republishes to craig.dlx under the ORIGINAL dlq.* key (confirmed + mandatory) and acks only on a clean confirm.

Quarantine

{queue}.quarantine — durable classic, no consumer; permanent/malformed/noncanonical-route/ cap-exhausted captures. Governance in ADR-059 §D7 (operator-drained, error!-per-publish as interim telemetry, C5 gauges = launch requirement, duplicate-copy + no-tokenless-dedup disclosures).

Capture schema

ONE _park wrapper for park + quarantine (v, capture_id, park_count, parked_at, reason, truncated last_error, verbatim occurrence_token, original_routing_key, original_envelope or original_body_b64; 12 MiB size-safe truncation). park_count parses as any u32 — the cap is disposition policy, never wrapper validity.

Settlement

SessionEnd::SettlementFailed for every ack/nack failure in BOTH sessions (Ok(false) = no settlement sent = failure); old session dropped before rebuild; backoff carries across consecutive settlement ends, resets only on non-settlement ends; teardown at error!.

C5 design (#1199) — approved 2026-07-29

One MR (feat(mq): DLQ observability + triage runbook + false-alert disclosure (#1199)), units U0–U3. No new ADR: the instruments follow the established otel+stub-twin metrics pattern (three existing metrics modules, none ADR’d); ADR-059 §D7/§D8/§D9 flip to as-built and ADR-022 gains the disclosure amendment.

Instruments

New craig_mq::metrics::dlq (meter craig-mq-dlq, otel + stub twins). Gauges, sync and push-updated (the craig-common db_pool_acquire_wait_seconds precedent — no callback registry): dlq_queue_depth{queue} over the three family queues (the DLQ itself / .parking / .quarantine) + dlq_depth_last_sample_timestamp{queue} (a dead sampler must not silently freeze depth values). Counters: dlq_captures_total{tier, reason} at ALL three capture sites (capture_failed_delivery, quarantine_malformed, the replayer’s unroutable branch), incremented only after the confirmed publish; dlq_outcomes_total{outcome} via the typed DeadLetterOutcome (recorded / phantom_absorbed / divergent_quarantined), recorded by craig-security’s handler through the re-export. A settlement-teardowns counter was reviewed and CUT (not AC’d; the error! teardown lines stay the loud-loop signal).

Depth sampler

Third arm of subscribe_dlq’s join: own connection (one-connection-per-concern), fresh channel per cycle, passive `queue_declare × 3 → message_count, sample-then-sleep on the new validated DlqRetryConfig.depth_sample_interval (default 30 s, bounds 1 s..=1 h); any error = warn + lazy reconnect next tick (no supervisor machinery — read-only task). AMQP declare-ok.message-count counts READY messages only: quarantine exact (no consumer), parking undercounts ≤ 1 (the prefetch-1 replayer holds the head unacked through its TTL sleep), the DLQ ≤ 16 (prefetch) — stated in instrument descriptions + the runbook.

Runbook

New operations/dlq-triage.adoc (nav § Operations): signals overview, backlog growth, parking drain, quarantine handling (xref the deployment-guide manual-ack replay procedure), threshold-alert interpretation, capacity guidance (10 min × 12 ≈ 2 h ride-through), metrics reference incl. the /metrics empty-200 tier caveat (OTEL_EXPORTER_OTLP_ENDPOINT gates the meter provider) and the counter-vs-gauge honesty (counters can double-count under ack-fail redelivery; the depth gauge is the stock truth).

Disclosure + honesty (ADR-022 amendment)

Pre-C2 inflated count==11 alerts: residue = an event_outbox row (~30 d) + an audit_log row (90 d then archived), NO security_alerts row — nothing to acknowledge or clear; non-retractable by design (audit-class rows are immutable). Disclosure, not reconciliation. §D3’s "for ops dashboards" is aspirational — the event’s only consumer is craig-security’s own wildcard audit subscriber (an audit_log row); the paging gap is #1205, cited from the runbook + amendment.

Tests

Prometheus-bridge asserts on registry.gather() (the craig-common install_prometheus_bridge precedent; nextest’s process-per-test model makes the global meter provider safe). The live scrape test parks TWO messages (READY excludes the replayer’s unacked head) + quarantines one; the security outcome-counter test is the genuine behavioral RED (stash services/craig-security/src against the committed U1 tree — instruments exist, security never increments). Config-bounds + reason-label-mapping units.

Addendum design (#1203) — approved v2, 2026-07-29

Status: Done (2026-07-30) — MR !1105 merged (f3d4922e); v1 rejected by external review (5 stop-ships), v2 approved and shipped as four J-reviewed units with three observed behavioral REDs.

Handler panics currently unwind the supervisor task on all three handler paths (events consumer; the DLQ session — where the panic kills consumer + replayer + sampler through the tokio::join!; the inbox — where an unstamped claim trips the high-severity inbox_unprocessed_beyond_grace invariant and re-runs the handler below INBOX_MAX_RETRIES forever). The v2 mechanism is a real per-delivery task boundary (v1’s futures-lite::catch_unwind was rejected: it guards only the poll — a panicking future destructor or handler Display still unwinds): one crate-private contain(fut) → Result<T, String> helper spawns the handler future in its own immediately-awaited tokio task with error-Display rendering normalized INSIDE, so sync construction, poll, Display, and destructor panics all resolve to an inspectable JoinError (payload message bounded at 512 bytes; non-string payloads dropped under a guard). Events path: panic → correlated error! + the existing 2-strike nack ladder. DLQ path: panic → TransientPark immediately (no in-session re-invoke; parks under the wire-compatible reason="transient"; a deliberate error! + the new dlq_handler_panics_total{path} counter disambiguate code defects from outages — a distinct ParkReason::Panicked is declined as schema churn). Inbox path: an inner boundary stamps the claim row (error_count/last_error) so the retry cap engages and the at-cap DLX surface carries the panic text; new InboxError::HandlerPanicked(String). Pre-1.0 breaking: Fut: 'static on subscribe/subscribe_exclusive, the new inbox variant — all call sites to be updated in the same diff, no shims. ADR-003’s crash-path residual retires at the docs unit; ADR-059 Consequences flip to as-built (converted panics compose as TransientPark; a deterministic panic quarantines at the park cap as park_cap_exhausted).

Addendum design (#1202) — approved v2, 2026-07-30

Status: Done (2026-07-30) — MR !1106 merged (ba88360d); v1 rejected by external review (4 stop-ships + ~20 findings), v2 approved after the exchange-ownership steer and shipped as four J-reviewed units + the F-054 pre-push fix, with two observed REDs.

Today one craig user (administrator tag, wildcard grants) is shared by every service and test, and .env.example ships that identity for seven services (omitting composition’s block entirely). v1 kept service-side exchange declares and therefore had to grant every service configure on both shared exchanges — but configure includes DELETE, so a compromised service could delete/redeclare craig.dlx and silently suppress forensic delivery without ever reading the DLQ, falsifying ADR-022’s #1181 bound. The user-steered v2 reversal: exchange ownership moves to the operator/topology plane.

Decision Substance

Operator-owned exchanges (the suppression fix)

Remove both runtime exchange_declare calls from open_channel; delete craig_mq::declare_dlx outright (pre-1.0 — the sole production caller, bootstrap init_mq, updated in the same diff, no shims; the one test caller, services/craig-security/tests/api/dead_letter.rs, migrates to the test-plane helper). devstack/rabbitmq/definitions.json pre-declares craig.dlx (durable topic) beside craig.events. Recovery-semantics change, recorded in ADR-003: a WIPED broker is no longer silently self-healed — binds fail loudly under the #1127 supervisor backoff until topology is restored. Deliberate: the silent redeclare masked the data loss a wipe implies. Scratch-vhost tests (queue_contract.rs) provision both exchanges explicitly (test-plane helper).

Credential model (9 users, vhost /)

8 service principals (devstack passwords = username — a public ACL fixture, hashes committed via rabbitmqctl hash_password); craig-test = the broad AMQP test identity (no tags); craig keeps administrator as the operator/mgmt-plane identity only (RabbitMgmt + scratch-vhost creation need the admin tag). Per-service grants are derived op-by-op from the verified AMQP inventory and land as anchored regexes (RabbitMQ ACLs are substring searches — anchors mandatory; fixed names enumerated, no glob overbreadth): configure = own queues ONLY (no shared exchange appears in any service’s configure); write = craig.events (outbox) + craig.dlx (DLX-arg at own-queue declare) + own queues; read = craig.events (binds) + own queues. craig-security additionally holds the DLQ-plane grants: configure/read on its dlq/parking (quarantine: configure only — no consumer), write on amq.default (capture publishes) + its dlq, read on craig.dlx (the dlq.# bind). The executable form is definitions.json itself; the deployment guide re-renders it as JSON + shell rabbitmqctl forms with the production-password and cutover/rollback runbook.

Recorded bounds (ADR-003 amendment)

(1) security’s amq.default write is write-anywhere-via-default-exchange for the audit principal (per-queue default-exchange scoping is inexpressible; a dedicated capture exchange is future work); (2) injection residuals — any service can publish any routing key to craig.events (event forgery) and any dlq. key to craig.dlx (forensic-record forgery); topic permissions named as the future hardening; (3) events-plane eavesdropping (any service may bind any routing key); (4) the enforcement line: read-on-craig.dlx only in security’s set = "only craig-security reads dlq.`"; delete-on-shared-exchanges denied fleet-wide. Devstack = `rabbitmq:4.2-management-alpine; the 4.3.1+ passive-declare ACL is not yet enforced — grants are forward-compatible, never claimed as current enforcement. Honesty posture: devstack credentials are a FUNCTIONAL ACL FIXTURE, not a containment boundary.

Wiring

8 compose URL swaps; .env.example — seven craig:craig service URLs become per-service placeholders + composition’s missing block ADDED; xtask gets ONE shared URL-builder for both env writers, the in-memory pin updated, the persisted-writer test gains a credential assertion; TestConfig AMQP default → craig-test; queue_contract.rs grants the username PARSED from the configured URL (env overrides keep working).

Tests (broker_permissions.rs)

Mgmt-exact fleet pin: all 9 users' tags + permission triples EQUAL the expectation read from definitions.json at test time (file↔broker drift pin — proves absence, which boot cannot). Compose-URL static pin. Live-state-safe probes, fresh channel per probe, identity URLs derived by parsing the configured URL and force-replacing userinfo: tx-rollback publish probes (nothing delivered — security’s wildcard binding would otherwise pollute the live audit DB), production-shape own-queue declare, foreign-NAMED scratch declares, the anchor pin (evil.craig-rules.events.evil-{uuid}), amq.default denial, and the suppression pin the whole unit exists for: exchange_delete on both shared exchanges → 403 for services AND for the audit principal. RED is labeled an environment-dependent MIGRATION red (per-service users absent on the OLD broker), sequenced before the first broker rebuild.

Out of scope (routed)

  • The W1f retry redesign, error taxonomy, parking/quarantine, settlement-failure teardown, malformed-envelope capture — #1197 (C3, ADR-059, § C3 design above).

  • Queue-type/delivery-limit contract — #1198 (C4, § C4 design above).

  • Metrics, runbook, false-alert disclosure — #1199 (C5, § C5 design above).

  • Operator-visible threshold-alert routing — #1205; quarantine drain/replay tooling (the C5-declined optional scope) — #1206; the stale opentelemetry-observability plan page — #1207.

  • Broker permission separation — #1202; handler-panic containment — #1203 (epic addenda).

  • Inbox exactly-once redesign — #1178 (held for contract steer).

Edit this page · latest