ADR-067: Two-Layer Fault Injection for Contested-Environment Testing — Native Levers + Toxiproxy, with a Feature-Gated Seam Doctrine

On this page

Status

Accepted (2026-08-18) as the doctrine of record for epic &83 (contested-environment component testing); drafted FIRST per the draft-first path and implemented incrementally across units C3–C24 (this ADR precedes the code, and unit C1 lands it before any hook/armer unit). Plan: Contested-Environment Component Testing (epic &83, anchor #1493). This ADR is the C1 governance deliverable; it SUPERSEDES the named-only base-injector approach of the archived Test-Framework Hardening plan (§D2.0a/§D2.0b — see D7 — Disposition of the named-only injectors (supersedes Test-Framework Hardening §D2.0a/§D2.0b)).

Context

CRAIG’s battery has strong in-process concurrency coverage — the barrier-race harness (craig_test_lib::concurrent), the ADR-062 attempt machine, the ADR-059 DLQ tier, ~148 conc-tagged + ~238 fault-tagged tests — but adversity at the transport and infrastructure boundary is undeliverable today: no test can force a broker disconnect mid-consume, a TCP reset, a confirm timeout, a bursty/duplicate/out-of-order delivery flood, or DB-pool exhaustion. When a battery flakes, nothing separates "the component mishandles adversity" from "the test/fixture/ envelope is bad" — the 2026-08-17 docker-devtools forensics cost three cycles for want of that signal.

The Test-Framework Hardening plan named four base injectors (CipherErrorInjector, RabbitDownInjector, ObjectStoreErrorInjector, PublishInTxFailureInjector) as §D2.0a follow-ups, but they were never built — they are doc-comment placeholders in craig-test-lib/src/fault/mod.rs:19-23 and restart.rs:34-38, awaiting "production seam refactors". A maintainer stop-ship review of the first design (v2) found ten blocker-class defects in the naive realization of those placeholders — including a rollback-losing trigger audit, an async assertion in Drop, a publish-fault hook at a boundary that does not exist (CrashPoint::PlacementPostUpdate), an unviable feature graph, silent-green skips, and a production durability bypass. This ADR records the corrected doctrine.

Two facts drive the core shape:

  • Graceful protocol close and a TCP reset are different failures. A RabbitMQ management close_connection is an AMQP-level graceful close; a mid-stream TCP RST or a half-open freeze is a transport-level failure. lapin’s recovery path differs between them (clean channel teardown vs. a stalled socket the client must time out and re-dial). A single "rabbit is down" abstraction cannot exercise both, and real production loss takes both forms.

  • A test fault hook is a liability if it can reach production. Any in-process injector adds a seam to a real type; the binding question is not "can we hide it" (Kerckhoffs forbids that) but "can we prove, mechanically, that it is absent from every shipped artifact."

Decision

D1 — Two complementary fault layers, both required per surface

Adversity is injected at two layers, and an MQ surface’s contract is not covered until BOTH are exercised:

  • L1 — native levers (protocol-graceful / in-process): the RabbitMQ management API (RabbitMgmt::close_connection, rabbitmq_mgmt.rs:489) for a graceful AMQP close; a new pg_terminate helper (D5) for graceful backend termination; and the feature-gated in-process injectors (D4/D6) for commit-boundary and error-class faults.

  • L2 — Toxiproxy (transport-level): a devstack sidecar that shapes the TCP path — reset-peer, latency, timeout, bandwidth, slicer — for the failures L1 cannot express (D2).

The two-path rule (M6): every enumerated MQ consumer references BOTH a graceful-close leg and a tcp-reset leg; a surface with only one is a coverage failure at the ratchet gate (C23). This is the machine encoding of "graceful close ≠ TCP reset".

+

As-built (C10, #1505) — the two-path rule proven LIVE at the library layer. craig-mq/tests/transport_faults.rs exercises ALL FOUR subscribe variants (subscribe, subscribe_exclusive, subscribe_idempotent, subscribe_dlq) under BOTH a graceful AMQP close (L1, RabbitMgmt::close_connection) and an abrupt transport drop (L2, the tcp-reset path) — the first application connections routed through the sidecar. The L2 lever is Toxiproxy proxy disable (drops all of the proxy’s sockets so the broker sees the connection die with NO AMQP Close), NOT the ResetPeer toxic: ResetPeer fires only on the next byte in its direction, so against an idle AMQP consumer socket it no-ops (severing at most the client-facing half while the broker still believes the connection is live — a vacuous "reset"); disable is the deterministic, traffic-independent socket death the tcp-reset path requires. Each leg runs on a TEST-OWNED scratch vhost with its own supervised consumer (M5 — never a live devstack consumer; craig_test_lib::fault::mq_transport::ScratchVhost + the direct/proxied URL rewriters), severs the connection, and asserts — via a NEW-connection barrier (wait_reconnect polls until the queue’s consumer is attached on a connection whose name differs from the pre-sever one), so reconnect can never be satisfied vacuously — that the supervisor reconnects and resumes: at-least-once for plain/exclusive (a drop between handler-success and ack redelivers — the accepted AMQP semantic), exactly-once for the idempotent variant, proven NON-vacuously by REDELIVERING an already-processed envelope after reconnect and asserting the handler effect row was written exactly once (the inbox ON CONFLICT dedup absorbs the redelivery). Plus a confirm-timeout leg (a Timeout toxic armed AFTER a clean confirm surfaces PublishError::ConfirmTimeout, never Amqp — the channel-established-first barrier) and a slicer leg (delivery survives TCP fragmentation). The PER-SERVICE registry surfaces get their two-path legs at C13/C14 (this discharges the library-layer variants; the coverage schedule in the plan is the map).

Barrier forensics (#1528). A barrier that expires must say WHY, or the only move left is to calibrate it. #1509 took the floor 45 s → 90 s and sized it by ratio — 3× craig-mq’s 30 s RECONNECT_BACKOFF_CAP plus margin, against a leg measured at ~17 s in isolation. That sizing was reasoned, but it was UNDIAGNOSED: nothing in the leg’s output could say which of the three causes below had fired, so ratio-to-the-cap was the only available response, and a genuinely pinned supervisor would have been absorbed by the wider floor rather than surfaced. The forensics are what make the next expiry answerable instead of re-calibratable. wait_reconnect therefore panics with the last poll result plus a live broker snapshot (the vhost’s connections and states, the queue’s consumer count and per-consumer connection attachment), and arm_leg_diagnostics — called by every entry point of the sever harness, not one designated first step — installs the #1319 process-global test writer so craig-mq’s own supervisor narration (consume session lost … delay_ms=N and reconnect attempt failed at warn, subscriber reconnected at info) lands in the failing leg’s captured output. The last of those is why the shared default filter is warn,craig_mq=info: at warn alone the "it DID reconnect" line — the evidence for exactly one of the three causes — is the one that gets dropped. That separates the three causes an expiry can have: a rebuild attempt pinned with no re-dial, a rebuild loop failing fast under backoff, and a consumer that DID reattach on a connection the management API had not yet surfaced. The first of those turned out to be a real production defect with no bound at all — see ADR-003 § Amendment — #1528.

D2 — Toxiproxy is the L2 sidecar; pinned, isolated, non-root

  • Image: ghcr.io/shopify/toxiproxy 2.12.0, digest-pinned to the multi-arch index sha256:9378ed52a28bc50edc1350f936f518f31fa95f0d15917d6eb40b8e376d1a214e (the linux/amd64 leaf is sha256:a3e244375123dad8849091bcc59775e188624d3f602db01901f9af855682fef8). Pinned by digest in the C4 compose fault profile, per container-ops (latest stable, pinned per-project).

  • Control plane security posture: the Toxiproxy control API (:8474) AND the 64-port proxy listen range are published on the host loopback only (127.0.0.1); the container runs non-root (65534), drops all Linux capabilities, sets no-new-privileges, has a read-only root filesystem, and is CPU/memory capped. It is a test-plane control surface and MUST NOT be reachable off-host; C4’s fault-preflight verifies the loopback host binding at runtime via docker port (a proven non-loopback publish is a hard refusal regardless of --required), and C24 re-asserts it.

    Amendment (#1499, C4 as-built). The original wording said the sidecar sits on an isolated compose network. That is mechanically wrong for a proxy: to shape a devstack service’s TCP path it must reach that upstream by its stable compose service name (postgres-test:5432, rabbitmq:5672), which requires sharing the app’s default network — a dedicated network cannot route to it, and hair-pinning through the host breaks under cargo xtask dev’s ephemeral port remapping. Isolation is therefore delivered by host-boundary confinement (loopback-only publish of every port), privilege confinement (non-root, `cap_drop: ALL, no-new-privileges, read-only rootfs), resource limits, and the default-off fault profile (a raw docker compose up never starts it) — a stronger and honest posture than a network label that a proxy cannot honor.

  • Client: a hand-rolled reqwest client mirroring RabbitMgmt (rabbitmq_mgmt.rs:188) — bounded timeout, bounded transport-retry, run+test-scoped proxy names — NOT the toxiproxy_rust crate (see Alternatives considered).

  • Supply chain: Toxiproxy is a container image, outside `cargo-deny’s crate graph. It rides the pinned-image SBOM/license path; the digest, an SBOM reference, and a named digest-refresh owner are C1/C4 acceptance criteria (a digest bump is a reviewed change, not a floating tag).

  • Healthcheck: the sidecar carries a HEALTHCHECK on GET /version; ensure_ready (devstack.rs:129) gains fault-profile health so the battery never runs against a half-up fault layer.

    As-built (C11, #1506) — the DB + object-store L2 legs on the D2 sidecar. craig-db/tests/l2_db_faults.rs proves THREE DISTINCT DB failure modes with distinct typed-error oracles (never a hang): (A) pool-acquire exhaustion cliffs at the fixed 5 s bound as PoolTimedOut — the #1160 policy restated, a NON-tunable constant, capacity is the remedy (no Toxiproxy: a capacity fault, not transport); (B) connect refused — a DIRECT PgConnection through a DISABLED proxy fails as sqlx::Error::Io (through a pool the 5 s acquire loop retries the refused connect until it collapses to PoolTimedOut, indistinguishable from A — so the distinct Io assertion requires the no-retry direct connect, recorded); © in-query drop — a downstream ResetPeer fires while a statement’s response is in flight, yielding the connection-lost FAMILY (Io / Protocol / a connection-class Database SQLSTATE — timing-dependent and unpinned before C11, so the family is asserted, never a single variant) and the pool recovers on a fresh connection. New helper: craig_test_lib::fault::pg_transport::pg_url_via_proxy (the DB-DSN-through-proxy rewriter, the DB counterpart to mq_transport::amqp_url_via_proxy). craig-store/tests/l2_store_ambiguous_put.rs proves the ambiguous-put oracle: a downstream reset severs the PUT response AFTER the object lands, the put surfaces StoreError::ObjectStore, and the readback+digest oracle resolves it — verify_stored_blob → Verified (no silent orphan) and the attempt-machine store_blob_create → Converged (no double-write). To make that leg deterministic + fast (a persistent transport fault would otherwise ride object_store’s default 10-retry / ~55 s backoff to exhaustion), a behavior-preserving Store builder refactor adds the test-only Store::from_config_with_max_retries (the production from_config keeps the default retry policy). Follow-up: object_store’s default retry means request_timeout_secs bounds each ATTEMPT but not TOTAL request latency — filed for a production retry-bound decision.

D3 — Native Postgres lever: pg_terminate, not a mocked pool

L1’s DB-disconnect leg terminates the real backend: capture the target session’s pg_backend_pid(), exclude admin PIDs, cross a pg_advisory_lock in-phase barrier so the terminate lands mid-transaction deterministically, then pg_terminate_backend(pid) under the same-role privilege the app uses. Reconnect/recovery is asserted against real rows. No pool double stands in for a disconnect — a fake cannot prove sqlx’s reconnection behavior.

D4 — Feature-gated seam doctrine (the hooks cannot reach release artifacts)

In-process injectors add hook FIELDS to real host structs, gated so they are absent from every normal-dependency graph:

  • Host crates expose default-off features craig-mq/fault-injection, craig-crypto/fault-injection, craig-store/test-util; each gates the hook field in its host struct (a field must live where it is used).

  • craig-test-lib has its OWN default-off forwarding feature; its normal dependencies NEVER enable the host features. Integration tests self-dev-dependency on the host crate with the feature (resolver-v2 test-only activation); cross-crate consumers dev-dependency on craig-test-lib with its forwarding feature.

  • xtask depends on craig-test-lib WITHOUT the feature (lint-guarded).

  • Release-artifact gate (new blocking lint): cargo tree -e normal,build over the Dockerfile -p service list + xtask must show all three features ABSENT. This is the machine proof.

Honest claim: the hooks are "absent from release artifacts and every normal-dependency graph, machine-enforced" — NOT "impossible" (they compile under --all-features, by design). Kerckhoffs is satisfied: the seam is fully public and its exclusion is auditable from the tree.

D5 — PgFaultArmer does NOT implement FaultInjector; its cadence survives rollback

A DB-level fault armer cannot honor the FaultInjector contract (fault/mod.rs:35): its recorded_attempts() audit would live in a row that the very rollback it induces destroys (the inbox_tx.rs:439 precedent leaves no trace). Therefore:

  • PgFaultArmer is its OWN type returning a typed PgFaultReport { invocations, injected, passed }, NOT an impl FaultInjector.

  • Cadence that survives rollback (M1): one non-transactional SEQUENCE per armed fault — IF nextval('craig_fault_seq_{id}') = N THEN RAISE — because nextval() persists across the rollback the RAISE triggers. The audit is read Rust-side from last_value/is_called between phases, never from a rolled-back row.

  • No async assertion in Drop (M2): armers with an external (DB) oracle expose async fn finish(self) → PgFaultReport / assert_consumed(self), called in the test body; Drop only panics on a forgotten finished flag and never touches the DB. The sync ScenarioGuard (fault/mod.rs:77) remains for in-process injectors whose counters are process-local.

  • DSN hard guard: the armer refuses any DSN that is not an ephemeral test/scratch database — it issues DDL (CREATE SEQUENCE, triggers) and must never touch a shared DB.

  • Statement classes (decided now): event_outbox.published_at, event_inbox.processed_at, exchange_send_jobs.status, upload_attempts.status.

  • Cleanup: explicit disarm().await plus a scratch-DB disposal backstop (the DDL dies with the per-test database) — never Drop-dependent.

    As-built (C12, #1507) — the inbox burst/permutation floods + the ordering taxonomy. craig-mq/tests/burst_floods.rs drives duplicate/shuffled floods through the REAL ADR-062 inbox. Effects-exactly-once is a SUBSTRATE property (the ON CONFLICT (envelope_id) claim + same-tx effect commit), so a SYNTHETIC one-row-per-invocation handler is the correct probe — it isolates the substrate (a real handler’s own idempotency would conflate + weaken the test). Two flood paths: a duplicate STORM (republish-same-id k×, seed-shuffled) and a genuine broker nack/requeue (the FIRST flood-scale PgFaultArmer consumer — arming event_inbox.processed_at aborts the Nth completion stamp, rolling back claim+effect+stamp → the delivery requeues → commits once); each yields exactly one effect per distinct envelope, checked against an order-INDEPENDENT quiescence barrier (processed_count == N, never a later-marker barrier — that would bake in in-order single-queue delivery, invalid for a shuffle). Ordering-under-shuffle is a HANDLER property, not the substrate’s: the per-handler contract is declared in contested-surfaces.toml (ordering = commutative | revision-gated | buffered | strictly-ordered; the canonical taxonomy is defined on xtask VALID_ORDERING), set for the 5 verified mq-consumers (exchange/financial/ placement → revision-gated per ADR-060’s strict-greater assignment_revision guard; cases/ reporting → commutative). C12’s shuffled_* legs validate the DRIVER + ORACLE with SYNTHETIC contract handlers (converge to max-revision / full-set regardless of arrival order — never "last-delivered wins"); REAL per-service handler-ordering proof + the buffered/strictly-ordered shapes are C13/C14 (they need real schema + handlers). The reusable seeded driver (craig_test_lib::fault::burst: CRAIG_BURST_SEED resolution + replay-command print
    seeded_shuffle) is craig_mq-agnostic so C13/C14 reuse it. The ordering-declaration gap-check is report-only until the C23 ratchet.

D6 — Publish-fault boundary split; CrashPoint::PlacementPostUpdate retired (M3)

CrashPoint::PlacementPostUpdate (restart.rs:85) is unimplementable at the boundary it names: placement stages events IN the SQL transaction (events.rs), the handler’s Publisher extractor is unused (placements.rs:669), and only the outbox worker later publishes. The single crash point is replaced by two real boundaries:

  • StageFaultInjector — a hook inside craig_mq::stage_event (outbox.rs:86); its failure rolls back the domain UPDATE + the outbox row together (the atomicity leg). As-built (C7, #1502): stage_event is a FREE function with no struct to hold a field, so the seam is the process-global armer craig_mq::fault::arm_stage_fault(event_type, N) returning an RAII StageFaultGuard — the same process-local pattern as PgFaultArmer (nextest runs one process per test). The hook fires AFTER the INSERT (the row genuinely enters the tx, then rolls back).

  • OutboxPreSendInjector — the D4 feature-gated hook on Publisher before basic_publish (publisher.rs:141), armed during drain: a committed row, a pending outbox entry, and a later exactly-once publish (the semantics PlacementPostUpdate intended, at the real seam). As-built (C7, #1502): a craig_mq::fault::PreSendInjector field attached via Publisher::with_pre_send_injector (Arc-shared so the drain worker’s clone sees one counter); fail_first(event_type, n) fails the first n matching publishes, then passes.

  • The third cell (publish ok, post-publish stamp fails) is D5’s event_outbox.published_at trigger.

CrashPoint::PlacementPostUpdate is removed; OutboxPreSend + StageFault replace it in the crash-point table.

D7 — Disposition of the named-only injectors (supersedes Test-Framework Hardening §D2.0a/§D2.0b)

  • RabbitDownInjector — retired, not built. A single "rabbit down" abstraction violates D1’s two-path rule; its intent is served better by L1 close_connection (graceful) + L2 Toxiproxy reset/timeout (transport). Removed from the injector roster.

  • CipherErrorInjector → built at C8 under D4 (craig-crypto/fault-injection), operation-enum exposure only (Encrypt|Decrypt|Hmac; kcv excluded), affirming D8. As-built (C8, 1503): CipherErrorInjector::fail(op, count) is an Arc-shared field on FieldEncryptor (attached via with_fault_injector; clones share the counter — axum clones the encryptor per request), consulted at encrypt, decrypt, and the shared blind_index_hmac core (covering hmac
    hmac_domain). kcv is excluded BY CONSTRUCTION — there is no Kcv op and kcv does its own HKDF, never routing through blind_index_hmac. The new hook field carries     [zeroize(skip)] (it holds no secret; the derive walks every field) — the "zeroize-skip hook". A Decrypt fault returns the opaque CryptoError::Decrypt, a Hmac fault CryptoError::HmacInit.

    Surface coverage as-built (C8, #1503). The five named decrypt-path surfaces — persons_search, persons_federal_export, reports/summary, referrals (Decrypt), plus the SSN blind-index search + ssn_promotion write (Hmac/Encrypt) — ALL funnel through the SAME injectable boundary: craig_search::decrypt_row → FieldEncryptor::decrypt, and craig_search::encrypt_row’s blind-index derivation → `FieldEncryptor::hmac_domain. The craig-cases decrypt/encrypt wrappers are pub(crate) AND the release service binary carries NO fault hook (the release-artifact gate proves it, §D4), so per-endpoint HTTP fault injection is structurally impossible. The firing fault legs are therefore proven at that shared boundary in craig-search/tests/cipher_fault.rs (self-dev-dep activation, resolver-v2), and the redacted-500 mapping (SearchError::Crypto → ApiError::Internal → redacted 500) is pinned in craig-common (search_server_faults_map_500 + internal_redacts_detail).

  • ObjectStoreErrorInjector → built at C9 under D4 (craig-store/test-util), with a request_timeout config (default 30s; error class unchanged, StoreError::ObjectStore). As-built (C9, 1504): ObjectStoreErrorInjector::fail(op, count) is an Arc-shared field on Store (attached via with_error_injector; clones share the counter — axum clones the store per request), consulted at each op boundary (StoreOp::{Put, PutCreate, Get, Delete, Exists, List}). A fired fault returns a synthetic StoreError::ObjectStore(object_store::Error::Generic{…}), riding the existing redacted-500 mapping. request_timeout_secs (default 30) is threaded into the S3 client via ClientOptions::with_timeout (the Local FS backend has no client). The injector
    hook are gated     [cfg(any(test, feature = "test-util"))], so the pure tempdir tests run in the default battery (cfg(test)) with no self-dev-dep. The countdown body is the shared craig_fault_core::OpCountdown<StoreOp> primitive (a new leaf crate) — CipherErrorInjector (C8) was refactored to wrap the same primitive so the two do not duplicate the countdown (quality-budget B8); it is pulled only behind each host’s fault feature.

  • PublishInTxFailureInjector → split + renamed to StageFaultInjector + OutboxPreSendInjector (D6).

  • FaultyAttachmentStore/FaultyOutboxStore/FaultyInboxStore/FaultySendJobStore → superseded by PgFaultArmer (D5) at the statement boundary + the typed injectors above; the restart.rs crash-point wiring table is updated accordingly.

D8 — Cipher faults preserve the redacted-500 contract

A decryption/HMAC fault surfaces as the existing ADR-020 fail-closed redacted 500 — never a degraded 200 or a leakier error. The cipher injector PINS that contract; any non-500 degradation would be an API change and is out of scope for this program.

D9 — Required-mode fault stage, no silent skips (M9)

There is no fault_layer_available() boolean that lets a green battery skip the faults. Fault tests are #[ignore = "requires devstack+fault"]; cargo xtask fault-preflight (a validate step before nextest) HARD-FAILS when the fault layer is absent (Toxiproxy /version healthy, loopback binding verified, fault services up, lease table reachable). Executed-fault accounting (test-results/fault/*.json → fault-report.json) asserts every armed fault fired and every IN-class surface has >0 executed faults — a green battery with zero executed faults is a FAILURE.

+

As-built (C24, #1519). fault-preflight runs --required inside validate’s devstack phase; the no-devstack path (--skip-devstack`, battery = --lib --bins) never reaches it and is the one sanctioned fault-optional mode. The accounting is a PAIR around the battery — program_gate::reset purges every record and writes test-results/fault/.gate-stamp before nextest, program_gate::verify aggregates after — so "the battery executed faults" is a claim about THIS run, not about the disk; a record predating the stamp fails the gate rather than being counted, and the stamp is SINGLE-USE (a successful validate verification consumes it and its absence under that policy is itself a failure, so no later run can inherit the claim; a FAILED verification leaves all evidence — stamp included — intact for triage, and the standalone command never consumes it). Unreadable record files are COUNTED by the aggregation and refuse certification — a test killed mid-write leaves missing evidence, not a silent zero. The gate also pins the program’s two self-tests by NAME against the nextest inventory (the panic/process-death cleanup contract and the PgFaultArmer canonical-DSN refusal), so deleting one is a gate failure rather than a silently retired guarantee. That last check pins EXISTENCE, not behavior.

Two honest limits ride the accounting: the armed⇒fired invariant is only as sharp as the recorders (the burst legs hardcode fired: true, so today only the pool harness can report an unfired fault — #1547), and the per-class floor is partial (below).

The per-class floor is enforced HONESTLY: only mq-consumer and pool call the recorder today, so those two are asserted and the other nine IN classes are NAMED on every run as owed (#1546). Enforcing the floor wholesale would fail the gate for a bookkeeping gap rather than a coverage gap; a class joins RECORDING_CLASSES when it records, the same per-class ratchet C23 uses for the registry. Records land crate-local for integration tests (the recorder writes CWD-relative); the gate discovers every test-results/fault directory by WALKING the checkout (symlinks unfollowed; build-output dirs skipped) so a new workspace member’s records are found without anyone remembering to list its parent, and #1524 owns the real fix.

D10 — No durability bypass (M10)

Store::from_parts is #[cfg(any(test, feature = "test-util"))] and takes the typed StoreBackend enum, never a backend_is_local: bool; is_local_backend() keeps deriving from the typed enum that retention’s boot guard trusts (craig-retention/src/boot.rs:37). from_config is untouched. A test injector may fail an object-store call; it may not make a non-local backend claim to be local.

+

As-built (C9, 1504). from_parts(inner, max_upload_bytes, &StoreBackend) derives backend_is_local = *backend == StoreBackend::Local internally (it never accepts the bool), and is gated [cfg(any(test, feature = "test-util"))] so it compiles out of every release artifact. from_config is unchanged. The from_parts_derives_is_local_from_the_typed_backend test pins that an S3 backend can NEVER report local — the retention boot_probe durability refusal cannot be spoofed. The feature was renamed test-support → test-util in C7 (clippy redundant_feature_names rejects the -support suffix); the release-artifact gate’s FORBIDDEN list carries ("craig-store", "test-util").

D11 — Scope: IN vs OUT (recorded)

IN: S2S HTTP clients, external partners over transport, MQ surfaces, background workers, DB pool (harness + report-only), crypto/keyring, browser/BFF (e2e), the auth-plane, and the authenticated CLI. OUT (with reasons): the SDKs (already ratcheted — retry-same-id pins + shared test vectors); tool-time surfaces (craig-cli ops / craig-seed / xtask migrate,import,archive-fetch / mock-server — no production request path); and the composition engine (pure/deterministic, property-covered; its runtime rides the IN classes). The full taxonomy is the plan’s coverage map.

Alternatives considered

  • turmoil (deterministic network sim). Rejected: lapin and sqlx expose no transport seam turmoil can drive, and a real broker/DB cannot enter the simulation. It would test a mock, not the client we ship.

  • pumba (chaos via container kill/netem). Rejected: its blast radius is the whole host’s Docker/network namespace — unacceptable beside an 8-way nextest devstack and a shared daemon (the very contention that started this program).

  • Raw tc netem. Rejected: a leaked qdisc silently shapes unrelated traffic; no per-test scoping; a cleanup miss corrupts the next battery.

  • The toxiproxy_rust crate. Rejected: an unpinned third-party client adds crate-graph surface for what is a thin HTTP API; we already have the RabbitMgmt reqwest-client pattern to mirror, keeping the dependency footprint and the pinning story identical.

  • Keep a single RabbitDownInjector abstraction. Rejected: it cannot express both a graceful AMQP close and a TCP reset, which drive different lapin recovery paths (D1).

  • A fault_layer_available() boolean probe. Rejected: it converts a missing fault layer into a silent green (M9) — the opposite of the arbitration instrument this program exists to build.

Consequences

  • Deterministic contested-environment coverage across the whole codebase, plus the arbitration instrument: a green per-component contested contract beside a flaky battery points the finger at the test/fixture/envelope (or a capacity attribution), not the component — the objective backstop the no-environmental-blame rule lacked (see the arbitration-ladder runbook, operations/contested-environment.adoc).

  • A coverage ratchet (contested-surfaces.toml + census scanners, report-only at C3 → blocking per class at C23) so new surfaces land covered by construction — the "easy integration mechanism" requirement.

  • Costs, accepted: Toxiproxy lives outside cargo-deny (mitigated by the pinned-image SBOM path + a named refresh owner); the fault stage adds bounded battery wall-time (ceilings in the plan’s budget methodology — a breach is a blocking finding, not an envelope widening); and the feature-graph discipline is a standing obligation the release-artifact gate enforces.

  • Standing constraint (maintainer): tests never get easier — no added serialization beyond semantically-required ordering, no envelope widening, no assertion loosening. C1 creates the never-easier checklist (in testing-reference.adoc); every J-review in this program runs it.

Open questions

  • The Toxiproxy digest-refresh cadence and its named owner are finalized as a C1/C4 acceptance criterion (recorded here on completion).

  • The boundary with #1404 (production typed-degradation posture): this program ships only the pool-contention harness + a report-only characterization (C21); #1404 keeps the production degradation design, and C21’s enforcement promotes only when #1404’s design merges. RESOLVED 2026-08-23: ADR-068 (#1404) merged the design and promoted C21 to load-robust latency-shape enforcement — every winner acquires below the bound, every loser waits out at least ~the bound, never an exact-millisecond pin (crates/craig-test-lib/tests/pool_contention.rs + the registry entry).

Amendments

Amendment — #1515 (2026-08-20): the third lever class — OS-level process death

§D1’s two layers deliberately excluded container-kill tooling (the pumba rejection below records why: host-wide blast radius). C20 adds the lever WITHOUT the blast radius by scoping it to a DEDICATED rig: docker kill -s KILL against a replica of the self-contained docker-compose.cluster.yml project (craig-cluster-{run}, no fixed host-port publishes, own postgres/rabbitmq/keycloak, resource limits) — never a devstack container. The three lever classes are now:

  • L1 — native graceful (RabbitMgmt::close_connection, pg_terminate_backend): the peer is TOLD; the process survives and its supervisor recovers.

  • L2 — transport fault (Toxiproxy disable/latency): the peer is NOT told; the process survives and times out into recovery (#1528 bounds the establish).

  • L3 — OS process death (docker kill, cluster rig only): nothing survives on the victim — no SIGTERM, no drain, no AMQP close; RabbitMQ learns via dead sockets, Postgres reaps the victim’s backends (which is exactly the mechanism that frees session advisory leases — the generator/sweep crash-safety design), exclusive queues auto-delete, claimed outbox rows go dark until CLAIM_LEASE expiry.

The rig’s lifecycle is STAGE-OWNED (cargo xtask cluster-tests: stale craig-cluster-* sweep → build → up -d --wait → the env-gated legs → artifacts to test-results/cluster/ → signal-safe teardown) and OUT-OF-BAND ONLY (weekly discriminated schedule via CLUSTER_TESTS=true + the cluster-tests MR label + manual — the F1/L10 ratified fork; no battery budget). The two rig legs pin cluster recovery per-connection- name (the two-replica identity caveat: a bare count is satisfied vacuously by the survivor); the three C19 worker kill-leg oracles land on this rig as #1539.

Edit this page · latest