Plan: Test Framework Hardening — 8 Test Categories, Happy/Sad/Evil Coverage, Failure-Path Infrastructure

On this page
Contents

Status

Step Description Status

1

Plan + sibling platform-stab-2 plan + epics &21/&22 + step issues #273–#291 (existing) + 9 new step issues for Phase A.9–A.17 + nav.adoc Active entry. No code changes.

Done (pre-ADR-030) — this MR

2

Phase A.1 — concurrent_fire(n, builder) consolidation in craig-test-lib. In-place extension of platform-stab-2 Step 2’s minimal helper: adds concurrent_fire_synchronized (auto-barrier, default), concurrent_fire_with_barrier (caller-managed sync point), concurrent_fire_collect_ordered, concurrent_fire_until_first_success, concurrent_fire_first_n, ConcurrentRunReport. Per-task latency, ordered-by-start timestamps, panic-info surface.

Done (pre-ADR-030)

3

Phase A.2 — Fault-injection harness consolidation. Promotes platform-stab-2 Step 2’s 5 inline injectors into craig_test_lib::fault::* module + adds LatencyInjector, IntermittentFlapInjector, ConsumerBackpressureInjector + named scenarios.

Done (pre-ADR-030) — harness foundation; base injectors deferred per §Errata

4

Phase A.3 — Multi-replica devstack profile. cargo xtask dev start --replicas <SVC>=<N> via docker-compose --scale. MultiReplicaCluster test helper.

Done (pre-ADR-030) — verification test deferred to platform-stab-2 #274 per §Errata

5

Phase A.4 — State-machine matrix-testing helper + Payment first adoption. Hand-rolled (bounded alphabet); assert_only_valid_transitions_accepted enumerates the rejection set across all (from, to) pairs.

Done (pre-ADR-030)

6

Phase A.5 — JWT mutation library. ClaimsBuilder + mutation_matrix for parametric strictness rejection tests across all 8 services.

Done (pre-ADR-030) — per-service adoption is a Phase B follow-up; library + smoke landed

7

Phase A.6 — Magic-byte upload + filename-attack fixtures. PE/ELF/zip-bomb/script-disguised + filename-attack table (CRLF, NUL, RTLO, UNC, etc.).

Done (pre-ADR-030) — upload corpus + loader; filename-attack table deferred to platform-stab-2 §D8.1 per §Errata

8

Phase A.7 — .claude/docs/testing.md "Test Categories" taxonomy (8 categories + happy/sad/evil sub-axis) + file-naming convention + Failure-path testing helpers section + process rule. CONTRIBUTING.adoc + delivery-protocol.md cross-references.

Done (pre-ADR-030)

9

Phase A.8 — Typed DTOs across 8 test clients (Contract tests). Full per-client rewrite, 8 small MRs starting with CasesClient. Decision: per-service -contracts crates (e.g. craig-cases-contracts/) own the public DTOs; the service crate depends on its contracts crate; the test-lib dev-depends on each -contracts crate. Avoids the cycle that a service-crate re-export would create with the existing services/<svc>[dev-dependencies] craig-test-lib edge.

Done (pre-ADR-030) — full CasesClient migration — 28 DTOs in contracts crate, 62 method signatures typed, 17 test files rewritten; #308 closed

10

Phase A.9 — proptest scaffolding + first 7 properties. State transitions, pagination, filename sanitization, Content-Disposition encoding, canonical JSON signing, idempotency body fingerprint, reference-data validation never panics.

Done (pre-ADR-030) — 6 of 7 properties shipped; Content-Disposition deferred per §Errata; surfaced + fixed a real sanitize_filename non-idempotency bug

11

Phase A.10 — cargo-mutants smoke on 5 critical crates (craig-api, craig-mq, craig-auth, craig-store, transitions modules). Baseline report committed; surviving mutants catalogued. NOT a pre-push gate.

Done (pre-ADR-030) — config + empty baseline + docs; population deferred to post Step 12 xtask wrapper per §Errata

12

Phase A.11 — xtask subcommands: reliability / contracts / mutants / coverage-matrix / invariants / quality-budgets. Available tools, NOT enforced gates.

Done (pre-ADR-030) — 6 subcommands wired; coverage-matrix and quality-budgets started as stubs in this MR but Steps 13/14 promoted them to full implementations; CONTRIBUTING + testing.md + README cross-references updated

13

Phase A.12 — Code-quality budget warnings. Module-LOC, function-LOC, serde_json::Value-count, #[allow]-count, unwrap_or_default-count, dep-dupe count, untyped-test-client-method count. Lock-file baseline; warnings only.

Done (pre-ADR-030) — full B1–B7 counters wired; xtask/quality-budgets.toml + xtask/quality-budgets.lock baselined; CONTRIBUTING.adoc Quality budgets subsection added; 15 unit + integration tests; report-only, no enforcement gate per §D12

14

Phase A.13 — Coverage Matrix tooling: cargo xtask coverage-matrix inventories OpenAPI routes + maps to tests by 6 axes (happy/sad/evil/concurrency/replay/fault). REPORT-only Markdown table at target/coverage-matrix.md. Initial baseline committed.

Done (pre-ADR-030) — full subcommand wired; live-fetch via docker::get_host_port + cache mode; baseline committed to operations/coverage-matrix.adoc; 12 unit + integration tests; --diff flag deferred per §Errata

15

Phase A.14 — DB Constraint Evil Tests harness (hand-curated). assert_constraint_violation_returns + per-service tests/constraints/<constraint>.rs convention. Seed: 2 representative constraints per service (16 tests).

Done (pre-ADR-030) — helper module + status-only + type-URL assertion modes; 16/16 tests passing — 2 per service across all 8 services; first 8 shipped in MR !209, second 8 in the review-fix MR !212; RFC 9457 type-URL standardization tracked as #311

16

Phase A.15 — Central Evil Input Corpus consolidation. craig_test_lib::evil module wraps Steps 6+7 fixtures + new categories (overlong strings, invalid UUIDs, JSON depth bombs, path traversal, HTML/script, Unicode normalization, bad enum values, bad dates, multipart weirdness, malformed signatures). Single iterator all_evil_cases().

Done (pre-ADR-030) — 12-category corpus with 83 cases; all_evil_cases / evil_cases_for / parametric_evil_test! public API; self-test + global contract test + macro-demo + 8 per-service tests/api/evil_corpus.rs adoption files all passing; review-fix MR !212 shipped the per-service adoption fan-out (originally deferred — ; craig-web BFF upstream-evil-corpus test deferred to platform-stab-2 §D7 work)

17

Phase A.16 — Restart/Replay light helpers. simulate_crash_after, restart_service, assert_recovers_to. Adoption sites paired with platform-stab-2 §D2/D3/D4/D5 recovery sweeps.

Done (pre-ADR-030) — helpers + CrashHarness façade per §D16.1; bare-closure entry for self-tests + harness entry for adoption; 6 named CrashPoints + Custom variant; 10 self-tests covering both entry points; per-fix adoption tests paired with platform-stab-2 fixes per §D16.5

18

Phase A.17 — Invariant Sweeper. cargo xtask invariants runs SQL post-test-run checks (no orphans, no stuck pending tx, no unpublished outbox beyond grace, no inbox processed-after-failure, no missing audit/event rows, no idempotency processing past TTL).

Done (pre-ADR-030) — sweeper replaces Step-12 stub; 4 SQL invariants seeded; xtask::dsn helper + craig-test-lib::invariant_catalog parser; runtime SKIP for missing tables; live devstack run: 22 checks / 0 failures / 1 SKIP — surfaced craig-placement missing event_inbox migration as a real finding

19

Phase B — adoption bundle. 10 specific failure-path tests (concurrent same-key idempotency, multi-replica outbox, inbox handler-fail-then-retry, upload content-spoofing, JWT strictness × 8 services, /livez vs /readyz differentiation, state-machine matrix on all 16 machines, cross-service reconciliation orphan-injection, outbox publish-failure retry, exchange crash-recovery) + risk-coverage scorecard for top-20 critical workflows.

Done (pre-ADR-030) — 3 of 10 tests shipped — Test 8 BFF malformed upstream, Test 9 craig-store filename × evil corpus, Test 10 typed-DTO pilot; 7 platform-stab-2-dependent tests deferred to backlog; risk-coverage scorecard at docs/…​/test-coverage-scorecard.adoc + nav entry; cargo xtask coverage-matrix --emit-scorecard flag wired

20

Plan completion audit + archive. Spawn audit subagent per delivery-protocol.md; flip Status all-Complete; move plan from Active to archive Infrastructure & Reliability; update CLAUDE.md Phase Status (add Test Framework Hardening row); CHANGELOG wrap-up entry; close epic &22.

Done (pre-ADR-030) — audit subagent run; Status table reconciled with shipped reality across all 19 prior steps; Step 12/15/16 rows + 2 §Errata/§Potential bullets corrected; plan moved from Active to archive; CLAUDE.md Phase Status updated; CHANGELOG wrap-up entry; epic &22 closed

Epic: &22
Issues: #283–#291 (existing — 8 original Phase A foundation issues + 1 Phase B adoption bundle) plus 9 new issues to file in Step 1 of this MR for Phase A.9–A.17
Sibling: epic &21 Platform Stabilization Phase 2 — its P0/P1 fixes ship with inline-minimal versions of A.1+A.2 (concurrent_fire + 5 fault injectors) so they’re not blocked on Phase A consolidation
Branch prefix: feat/test-framework- / fix/test-framework- / chore/test-framework-
*Milestone
: 2026 Q3 — Platform Reliability

Context

The 2026-05-03 external review surfaced 3 P0 race conditions (idempotency, outbox, inbox) plus 4 P1 durability gaps. The first 5-agent audit had marked the relevant invariants "verified shipped" because they verified STRUCTURE (table exists, helper exposed, worker spawned) without verifying SEMANTICS (concurrency safety, retry behavior, atomic claim ordering).

But there’s a second-order finding the user surfaced: the existing test framework structurally cannot catch these races. Even if the audits had been semantically rigorous, the suite would not surface the gaps in CI:

  • All integration tests are sequential single-fire — no helper for concurrent same-key requests

  • Devstack is single-instance — no multi-replica mode; outbox double-publish never triggers

  • No fault injection — cipher / DB / RabbitMQ / object-store errors can’t be deterministically simulated

  • Nextest concurrency is per-test-file, not per-request

  • No property-based / state-space testing

  • No mutation testing — coverage-by-line says "covered" while assertions never bite

  • No invariant sweeper — DB-state violations after test runs go undetected

  • k6 load tests run against a single replica

  • Test clients use serde_json::Value everywhere, so contract drift surfaces only at runtime, not at compile time

This epic builds the missing infrastructure first (Phase A) so that platform-stab-2’s fixes can ship with proper coverage (Phase B), and so future race conditions can be caught in CI rather than by external review.

The user has further specified an 8-category test taxonomy with a happy/sad/evil sub-axis, and listed 10 specific tests that should ship first. Both are folded into this plan.

The methodology lesson — that storage-shape verification (table exists, helper exposed, worker spawned) is not equivalent to semantic verification (locking, claim ordering, retry behavior) under concurrency — is captured in the two-pass platform-invariant audit and reinforced in platform-stab-2’s Context section. This plan operationalizes that lesson by giving every architectural promise a testable form: idempotent means raced (Phase B test 1), durable means failed and replayed (test 5), typed means compile-checked (Phase A.8), secure means bad inputs and bad config are exercised (Phase A.5/A.6/A.15).

Test Categories taxonomy

This plan operationalizes 8 explicit test categories. Phase A.7 (Step 8) encodes them into .claude/docs/testing.md + a per-category file-naming convention; Phase A.13 (Step 14) builds tooling that surfaces coverage gaps.

Category What it asserts Helpers (Phase A piece)

Functional

Endpoint behaves as specified on the happy path

Existing nextest harness — status quo

Invariant

Facts that must always hold (no duplicate side effects, no over-capacity placements, no orphan rows, no double-processed events)

xtask invariants (A.17) + DB-constraint harness (A.14)

Concurrency

Behavior under intentional races (same-key POSTs, multi-worker drains, simultaneous state changes)

concurrent_fire (A.1) + multi-replica devstack (A.3)

Fault-injection

Behavior when a specific dependency fails at a specific boundary

Fault injectors (A.2) + restart helpers (A.16)

Recovery

Behavior across crash/restart between phases of multi-step workflows

Restart helpers (A.16) + invariant sweeper (A.17)

Contract

Schema drift breaks at compile time, not runtime

Typed DTOs (A.8) + coverage-matrix (A.13)

Property-based

Invariants on unbounded inputs (filename sanitization, JSON canonicalization, fingerprinting)

proptest scaffolding (A.9)

Mutation

The tests assert meaningfully — surviving mutants surface lines that are exercised but not asserted

cargo-mutants smoke (A.10)

Sub-axis: every Functional / Invariant / Fault-injection test should be classified as one or more of:

  • happy — the success path the spec describes

  • sad — predictable rejections (validation, 4xx, expected errors)

  • evil — adversarial inputs (oversized strings, malformed signatures, path traversal, magic-byte spoofing, JWT strictness, etc.)

Coverage-matrix tooling (A.13) inventories which axes have a test for each route. Risk-coverage > line-coverage.

Scope

In scope:

  • 17 Phase A foundation pieces: helpers, harnesses, fixtures, xtask subcommands, code-quality budgets, evil corpus, invariant sweeper, coverage-matrix tooling

  • 10 Phase B adoption tests

  • Risk-coverage scorecard for top-20 critical workflows

  • .claude/docs/testing.md extension (Test Categories + helper docs + process rule)

  • .claude/docs/delivery-protocol.md failure-path-test process rule

  • CONTRIBUTING.adoc pointer

  • 9 new GitLab issues filed in Step 1 for Phase A.9–A.17

Out of scope (deliberate or deferred):

  • CI-pipeline gating of any new xtask subcommand. Per user directive: tools available, no CI pressure today. Future plan covers gate graduation.

  • Pre-push enforcement of code-quality budgets. Lock-file baseline ships; reviewer discipline holds the line; opt-in --fail-on-regression available.

  • Auto-derived DB-constraint evil tests from migration SQL. Pushed back: hand-curated harness + per-service convention instead.

  • Untested-large-file targeted sad/evil test backfill (7 files) — overlaps with platform-stab-2 §D10.1 large-file decomposition; tracked as follow-up plan.

  • New domain features (any of the standing-backlog items)

  • AWS deployment thread (#163 / #172 / #199 — owned by external team)

Two-phase structure

Phase A — Foundation (build the helpers, harnesses, tooling)

Seventeen discrete pieces — each is a small, focused MR; most are parallelizable across the team:

  1. concurrent_fire consolidation (extends platform-stab-2 inline)

  2. Fault-injection harness consolidation (extends platform-stab-2 inline + adds 3 new injectors)

  3. Multi-replica devstack profile (cargo xtask dev start --replicas)

  4. State-machine matrix-testing pattern + Payment first adoption

  5. JWT mutation library — mint malformed tokens for strictness tests

  6. Magic-byte upload-spoofing + filename-attack fixtures

  7. testing.md "Test Categories" taxonomy + file-naming convention + Failure-path testing helpers section + process rule

  8. Typed DTO migration across 8 test clients (Contract tests)

  9. proptest scaffolding + first 7 properties

  10. cargo-mutants smoke on 5 critical crates

  11. xtask subcommands: reliability / contracts / mutants / coverage-matrix / invariants / quality-budgets (available, not enforced)

  12. Code-quality budget warnings (warnings only, lock-file baseline)

  13. Coverage Matrix tooling (xtask coverage-matrix REPORT-only)

  14. DB Constraint Evil Tests — hand-curated harness

  15. Central Evil Input Corpus consolidation

  16. Restart/Replay light helpers

  17. Invariant Sweeper — xtask invariants

Phase B — Adoption (10 specific tests + risk-coverage scorecard)

Ten user-listed failure-path tests, each tagged with which Phase A piece it depends on. Plus a risk-coverage scorecard at docs/modules/ROOT/pages/test-coverage-scorecard.adoc listing top-20 critical workflows × 6 coverage axes.

Many Phase B tests double as the failing-then-passing test for a sibling platform-stab-2 fix. Coordination expected.

Wave sequencing

  • W1: Step 1 (this plan + sibling plan + epics + step issues; solo, blocks all)

  • W2 — parallelizable: Phase A foundation work — Steps 2–18 are largely independent; can be picked up in any order. Long-running pieces (A.8 typed DTO migration, A.10 mutation testing, A.13 coverage matrix tooling) run in parallel with shorter pieces. Recommended order for the team to start with the highest leverage items: A.1 → A.2 → A.7 → A.13 → A.15 → others.

  • W3: Phase B adoption (Step 19). Each Phase B test depends on its corresponding Phase A helper landing first. Many Phase B tests are also the failing-then-passing test for a sibling platform-stab-2 (epic &21) fix — coordinate landings.

  • W4: Step 20 (audit + archive).

Design

D1. concurrent_fire helper consolidation (Phase A.1 — Step 2)

Sibling plan platform-stabilization-2 (epic &21) Step 2 introduces a minimal inline concurrent_fire(n, builder) at crates/craig-test-lib/src/concurrent.rs and 5 fault injectors at crates/craig-test-lib/src/fault_injection.rs. Those modules ship as the smallest viable harness to verify epic &21’s Steps 3–9; this plan’s Phase A.1 (Step 2) and A.2 (Step 3) consolidate and extend them rather than re-introducing parallel surfaces.

The minimal concurrent_fire from epic &21 Step 2 has the shape:

// crates/craig-test-lib/src/concurrent.rs (epic &21 Step 2 — current state)
pub async fn concurrent_fire<F, R, Fut>(n: usize, builder: F) -> Vec<R>
where
    F: Fn(usize) -> Fut + Send + Sync,
    Fut: std::future::Future<Output = R> + Send + 'static,
    R: Send + 'static,
{
    let mut handles = Vec::with_capacity(n);
    for i in 0..n {
        handles.push(tokio::spawn(builder(i)));
    }
    let mut results = Vec::with_capacity(n);
    for h in handles {
        results.push(h.await.expect("concurrent_fire task panicked"));
    }
    results
}

It covers exactly one shape: spawn-all-then-await-all, panic-propagating via .expect, results in spawn-task-completion order (JoinHandle await order, not spawn order — which the doc string implies but the impl does not guarantee). Several Phase B failure-path tests (#291) need shapes this helper cannot express:

  • Multi-replica outbox concurrent test (sibling Step 4 of &21 — and Phase B test for #275) needs a synchronized "go" so both worker drains start within the same tokio-tick window. Without a barrier, worker 1 typically wins the race and worker 2 finds an empty event_outbox table.

  • Idempotency atomic-claim test needs to assert that exactly one handler-side effect lands AND that the per-task latency distribution shows a winner-shaped first response and N-1 loser-shaped replays (loser latency ≥ winner latency by construction). The current helper drops latency.

  • State-machine transition reject tests often need "fire many; first K successes are enough" semantics — currently expressible only via ad-hoc loops outside the helper.

  • Panic-as-bug surfacing — the current .expect panics the test thread, which under nextest is recoverable but loses provenance (which task panicked? what input?).

D1.1. Decision: in-place extension, not migration

Two viable consolidation strategies were considered:

Option Trade-offs

(a) In-place extension — extend crates/craig-test-lib/src/concurrent.rs with new functions; existing concurrent_fire keeps its name and signature; new helpers live alongside.

Pro: zero churn for the &21 Phase 2 fix tests already authored against the minimal API. Pro: single module, single import path. Con: the module accretes a non-trivial surface (~6 functions + a helper struct).

(b) Migrate to richer module + re-export shim — create crates/craig-test-lib/src/concurrent/{fire.rs,barrier.rs,first.rs,report.rs} with the new richer API; keep the old concurrent_fire as a thin re-export at the crate root.

Pro: cleaner separation of variants. Con: doubles the import surface during the transition. Con: the re-export shim is dead weight as soon as the &21 callers migrate; no other consumer.

Decision: (a) in-place extension. The richer API is naturally expressible as 4 sibling functions + 1 struct in one file (~250 LOC target). There are no other consumers to keep stable; the &21 Phase 2 tests are the only callers and they live in this same workspace. Splitting into submodules buys nothing while doubling import paths.

The one rename this plan accepts: the existing minimal helper is renamed to concurrent_fire_collect (matching the new naming family — _collect, _collect_ordered, _with_barrier, _until_first_success, _first_n) and concurrent_fire becomes a deprecated alias re-exported for one minor version. Errata gets an entry; epic &21 Phase 2 callers swap names in the same MR that lands this consolidation.

D1.2. New helper inventory

// crates/craig-test-lib/src/concurrent.rs

use std::future::Future;
use std::pin::Pin;
use std::sync::Arc;
use std::time::{Duration, Instant};
use tokio::sync::Barrier;
use tokio::task::JoinHandle;

/// Outcome of a single task in a concurrent run.
#[derive(Debug)]
pub struct TaskOutcome<R> {
    /// 0-based index passed to the builder.
    pub index: usize,
    /// Wall-clock duration from spawn to handler return.
    pub latency: Duration,
    /// Wall-clock instant the task was spawned (monotonic, set under
    /// the same `Instant::now()` epoch as `latency`'s start).
    pub started_at: Instant,
    /// Value returned by the builder future, or panic info.
    pub result: TaskResult<R>,
}

#[derive(Debug)]
pub enum TaskResult<R> {
    Ok(R),
    Panicked { message: String },
}

/// Aggregate report over a `concurrent_fire_*` invocation.
pub struct ConcurrentRunReport<R> {
    /// One entry per task, ordered by `started_at` ascending.
    pub outcomes: Vec<TaskOutcome<R>>,
    /// Wall-clock duration from first-spawn to last-return.
    pub total: Duration,
}

impl<R> ConcurrentRunReport<R> {
    /// Number of tasks whose result is `TaskResult::Ok(_)`.
    pub fn ok_count(&self) -> usize { /* ... */ }
    /// Number of tasks whose result is `TaskResult::Panicked { .. }`.
    pub fn panic_count(&self) -> usize { /* ... */ }
    /// Returns `Ok(values)` if all tasks completed without panic, else
    /// `Err(panic_messages)`. Most tests want this.
    pub fn into_results(self) -> Result<Vec<R>, Vec<String>> { /* ... */ }
}

/// Spawn N concurrent tokio tasks, await all, return values in *spawn
/// order* (index 0..N-1). Panics in tasks panic the test thread —
/// matches the current minimal helper's behavior, kept for back-compat.
///
/// When-to-use: drop-in replacement for the &21 Step 2 minimal helper.
pub async fn concurrent_fire_collect<F, Fut, R>(n: usize, builder: F) -> Vec<R>
where
    F: Fn(usize) -> Fut + Send + Sync,
    Fut: Future<Output = R> + Send + 'static,
    R: Send + 'static;

/// Same as `concurrent_fire_collect` but returns a full `ConcurrentRunReport`
/// with per-task latency, ordered-by-start-instant, and panic capture.
///
/// When-to-use: any test that needs to assert *winner-vs-loser shape*
/// (idempotency replay tests), latency distribution, or that wants
/// panic-as-data instead of panic-as-test-failure.
pub async fn concurrent_fire_collect_ordered<F, Fut, R>(
    n: usize,
    builder: F,
) -> ConcurrentRunReport<R>
where
    F: Fn(usize) -> Fut + Send + Sync,
    Fut: Future<Output = R> + Send + 'static,
    R: Send + 'static;

/// Spawn N tasks, all wait on a `tokio::sync::Barrier`, then proceed
/// together. Closes the spawn-skew window: without a barrier, task 0
/// is typically 50–500µs ahead of task N-1; with a barrier they all
/// release within one tokio-scheduler tick.
///
/// When-to-use: outbox/inbox concurrent-claim tests where the test is
/// asserting "two workers race to claim the same row" *and* needs an
/// explicit pre-claim semantic synchronization point (e.g. all N
/// pre-fetch claims_state, then barrier-wait, then race to UPDATE).
/// The barrier is exposed to the caller so the builder can decide
/// where to synchronize.
///
/// Caveat: this variant cannot guarantee the caller actually awaited
/// the barrier. A builder that ignores its `Arc<Barrier>` argument
/// produces the same staggered-start behavior as plain `concurrent_fire`.
/// For tests that just want "all start together, no caller-controlled
/// sync point", use `concurrent_fire_synchronized` (below).
pub async fn concurrent_fire_with_barrier<F, Fut, R>(
    n: usize,
    builder: F,
) -> ConcurrentRunReport<R>
where
    F: Fn(usize, Arc<Barrier>) -> Fut + Send + Sync,
    Fut: Future<Output = R> + Send + 'static,
    R: Send + 'static;

/// Spawn N tasks; the helper itself awaits a `Barrier::new(n)` *immediately
/// before* invoking each `builder(i)` closure, so all N closures begin
/// execution at the same scheduler tick. The barrier is internal to the
/// helper; the caller's builder receives only `i: usize` and runs as if
/// the synchronization never existed.
///
/// When-to-use: tests that simply need "all start together" without a
/// caller-controlled sync point. Idempotency same-key concurrent POST
/// (Phase B test 1), JWT-mutation parametric tests, basic outbox-claim
/// races. Strictly stronger than `concurrent_fire` for catching staggered-
/// start false-negatives, with no API surface caller can leak.
///
/// This is the recommended default. Reach for `concurrent_fire_with_barrier`
/// only when the test needs a *post-prep, pre-race* sync point inside the
/// builder body.
pub async fn concurrent_fire_synchronized<F, Fut, R>(
    n: usize,
    builder: F,
) -> ConcurrentRunReport<R>
where
    F: Fn(usize) -> Fut + Send + Sync,
    Fut: Future<Output = R> + Send + 'static,
    R: Send + 'static;

/// Spawn N tasks, return as soon as the *first* task returns `Ok(_)`
/// (or all N return Err). Cancels the remaining tasks' futures on
/// success.
///
/// When-to-use: state-machine "transition is allowed from this state"
/// tests where any successful transition proves the invariant; or
/// JWT-mutation tests (Phase A.5) where any one of N malformed tokens
/// being accepted is a failure case.
pub async fn concurrent_fire_until_first_success<F, Fut, R, E>(
    n: usize,
    builder: F,
) -> Result<R, Vec<E>>
where
    F: Fn(usize) -> Fut + Send + Sync,
    Fut: Future<Output = Result<R, E>> + Send + 'static,
    R: Send + 'static,
    E: Send + 'static;

/// Spawn N tasks, return as soon as `k` tasks return `Ok(_)`. Aborts
/// the remaining tasks. Returns the first `k` successes in completion
/// order; if fewer than `k` succeed before all N finish, returns Err
/// with the collected errors.
///
/// When-to-use: "exactly K of N concurrent claims succeed" assertions.
/// E.g., concurrent same-key idempotency: K=1 of N=5; concurrent foster
/// home capacity: K=capacity of N=capacity+overflow.
pub async fn concurrent_fire_first_n<F, Fut, R, E>(
    n: usize,
    k: usize,
    builder: F,
) -> Result<Vec<R>, Vec<E>>
where
    F: Fn(usize) -> Fut + Send + Sync,
    Fut: Future<Output = Result<R, E>> + Send + 'static,
    R: Send + 'static,
    E: Send + 'static;

D1.3. Implementation notes

  • All variants spawn via tokio::spawn (not tokio::task::spawn_blocking) — concurrency targets I/O-bound async code paths.

  • concurrent_fire_collect_ordered instruments via Instant::now() inside the spawned future (post-tokio::spawn, before awaiting the builder) so that started_at reflects the moment the runtime actually picked up the task, not the moment tokio::spawn returned. The difference is microseconds in CI, but matters for asserting "barrier-synchronized" claims in tests that exercise the barrier variant alongside the ordered variant.

  • concurrent_fire_with_barrier constructs Arc::new(Barrier::new(n)) internally so callers do not have to thread the barrier construction. Each task receives the Arc<Barrier> and is expected to call barrier.wait().await at its own synchronization point — typically the first line of the builder closure body.

  • concurrent_fire_until_first_success and concurrent_fire_first_n use JoinSet::abort_all() for cancellation. The aborted tasks' futures get a Cancelled join error which is dropped (not surfaced in Vec<E>) — only futures that naturally completed with Err(e) contribute to the error vector.

  • Panic capture uses JoinError::is_panic() + Any::downcast_ref::<&str> / Any::downcast_ref::<String>. Falls back to "panic with non-string payload" for non-string panic payloads.

D1.4. Test descriptions

For each new helper, smoke tests live at crates/craig-test-lib/tests/concurrent.rs (consolidating the &21 Step 2 file of the same name). Tests:

  1. fire_collect_zero_tasksconcurrent_fire_collect(0, |_| async { 1 }) returns empty Vec. No spawns, no panics.

  2. fire_collect_single_task — N=1; builder receives index 0; result vector has length 1.

  3. fire_collect_panic_propagates — N=3; builder panics on index=1; the test wraps the call in tokio::spawn(…​).await.unwrap_err().is_panic() to assert the panic surfaces.

  4. fire_collect_all_panic — N=5; every builder panics; assertion as above.

  5. fire_collect_mixed_latency — N=10; builders sleep 100ms * (i+1); result vector still has length 10 and contains every index value (set-equality, not order-equality).

  6. fire_collect_ordered_winner_shape — N=5 against an AtomicUsize fetch_add(1); into_results() returns Ok(vec![0,1,2,3,4]); outcomes[0].started_at ⇐ outcomes[4].started_at; panic_count() == 0.

  7. fire_collect_ordered_panic_capture — N=3; index=1 panics with "boom"; panic_count() == 1; the corresponding outcome’s TaskResult::Panicked { message } contains "boom".

  8. fire_with_barrier_synchronizes_start — N=5; each builder records Instant::now() after barrier.wait().await. The max(started_at) - min(started_at) delta is asserted < 5ms (one scheduler tick). Compare against `fire_collect_ordered’s typical delta of ≥ 50µs to demonstrate the barrier reduces skew.

  9. fire_until_first_success_short_circuits — N=10; builders 0–8 return Err(i); builder 9 returns Ok(99) after a 50ms delay; index-3 builder also has a 100ms sleep + would have returned Ok(33); the helper returns Ok(99) (the first to flip to Ok wins) and the index-3 task is aborted before it can complete.

  10. fire_until_first_success_all_err — N=5; every builder returns Err(i); helper returns Err(vec![0,1,2,3,4]) (order-independent).

  11. fire_first_n_exact_count — N=10, K=3; 5 builders return Ok, 5 return Err; helper returns Ok with exactly 3 values; the remaining 7 tasks are aborted.

  12. fire_first_n_under_target — N=5, K=3; only 2 builders return Ok; helper returns Err with 3 collected errors.

Path A/B/C structure for the most concurrency-critical case:

  1. Path A — barrier reduces start-skew below tokio tick boundary: fire_with_barrier_synchronizes_start (above).

  2. Path B — without barrier, skew exceeds tick: control test using fire_collect_ordered with the same builder; assert max(started_at) - min(started_at) ≥ 20µs (almost always true on CI).

  3. Path C — barrier deadlock surfaced as test timeout: N=5 with one builder that returns without calling barrier.wait().await; the remaining 4 tasks block on the barrier forever; the test wraps the call in tokio::time::timeout(Duration::from_secs(2), …) and asserts Err. This documents the failure mode for callers.

D2. Fault-injection harness consolidation (Phase A.2 — Step 3)

D2.0. Dependency injection seams (precondition for everything in §D2)

The injector designs in §D2.1+ assume that production code accepts an abstraction (trait, generic, enum-dispatch wrapper, fault-aware concrete type) at the seam where the test wants to fail. Today, most production code accepts concrete PgPool / Publisher / FieldEncryptor / Store directly, or calls free functions that take &PgPool. Without addressing this, the injectors are beautiful API paper over uninjectable code: the test compiles but no production codepath ever reaches the wrapped type.

Three viable seam patterns. Each Phase A.2 candidate-injector callsite picks one based on the actual production surface; this is not a one-size-fits-all decision.

Pattern When to use Cost Limit

Trait-object injection at construction

Production callsite already takes a long-lived dep through a struct field (e.g. OutboxWorker { publisher: Publisher }). Make the field a Box<dyn PublisherLike> (or a generic <P: PublisherLike> if zero-overhead matters). Production passes the concrete Publisher; tests pass RabbitDownInjector::wrap(publisher).

Touches every constructor; one trait per dep family.

dyn only works if the trait’s surface is small (≤4 methods); otherwise generics-with-where blow up call-graph monomorphization.

Test-only enum dispatch

Production callsite is a free fn that takes a small typed dependency (e.g. &Publisher, &Store) — the dependency’s surface is the only path queries flow through. Wrap callers in an enum: enum Pub { Real(Publisher), Faulty(FaultyPublisher) } — dispatch in the impl block. Not for &PgPool: PgPool queries flow into sqlx::query() after the dependency boundary, so any enum wrapping PgPool cannot intercept; use the codepath-specific store trait pattern from sub-pattern (b) below for DB faults.

Module-internal change; no signature churn at API boundary.

Adds an enum match per call; risk of forgotten match arm on new variants. Not applicable to types whose API exposes a primitive (&PgPool, &[u8]) that callers further unwrap.

Real container failure

Failure mode is systemic (RabbitMQ unreachable; DB connection refused; object store returns 503). Bring up devstack with the dependency stopped (docker compose stop rabbitmq) and assert behavior end-to-end.

Slow (5-10s startup); requires cargo xtask dev integration.

Cannot inject selective failures (e.g. "fail this query but not that"); cannot scope failures to a specific test.

Concrete seam audit (per the 5 inline injectors from epic &21 Step 2 + the 3 new ones in §D2):

  1. FieldEncryptor (CipherErrorInjector + LatencyInjector + IntermittentFlapInjector): production already takes Option<FieldEncryptor> through AppState. Pattern: trait-object — define pub trait FieldEncrypt: Send + Sync { fn encrypt_str(&self, …​) → Result<…​>; fn decrypt_str(&self, …​) → Result<…​>; }. Implement on FieldEncryptor and on each injector wrapper. Change AppState::encryptor: Option<Box<dyn FieldEncrypt>>. Audit ~5 callsites in services/craig-cases/src/api/persons.rs (already enumerated in platform-stab-2 §D9.3).

  2. PgPool (DbErrorInjector + LatencyInjector + IntermittentFlapInjector): production stores PgPool directly in many App* structs and 100+ store functions take &PgPool and call sqlx::query(…​). A wrapper that exposes &PgPool is a wrapper in name only: once the test code obtains &PgPool it bypasses any interception. The faux-Db-newtype-with-inner() design rejected here. Three real options, picked per-codepath:

    Sub-pattern When Trade-off

    (a) Real container-level failure

    The test cares about systemic DB unavailability (connection refused, cluster failover, network partition). Bring up devstack, docker compose stop postgres, run the test, observe /readyz flip + handler 503 + outbox row stays unpublished.

    Cannot inject selective failures (one query fails, others succeed). Cannot scope failures to a specific test invocation. Slow (5-10s setup + teardown).

    (b) Repository / store trait at the codepath under test

    The test wants to inject a failure at this specific store function. Define pub trait FinalizeIdempotencyStore { async fn finalize_succeeded(…​) → Result<…​>; … }. Production impl wraps &PgPool and calls sqlx::query. Tests pass FaultyFinalizeIdempotencyStore that errors on demand. Worker / handler holds Arc<dyn FinalizeIdempotencyStore> (with async_trait per §D2.0c below) or generic <S: FinalizeIdempotencyStore> if codegen overhead matters.

    One trait + one prod impl + one test impl per faultable codepath. ~3-5 traits per service in scope (FinalizeIdempotencyStore, AttachmentStore, SendJobStore, OutboxStore, InboxStore). Adds an indirection at every call site that wants to participate in fault injection.

    (c) No injection at this layer

    The test’s failure mode is at a different layer (cipher, RMQ, object store) and DB calls are not the seam. Use the cipher/RMQ/object-store injectors instead and let DB calls run real.

    None — the right answer when DB is not the failure point.

    Decision rule: every Phase B test that wants a DB failure declares which sub-pattern it uses in its test file’s module-level doc comment. Reviewer enforces. If sub-pattern (b) is chosen, the trait is added in the same MR that lands the test (per the failing-test-first protocol §D2.0 last paragraph). Speculative trait extraction without a test target is rejected.

    Concrete trait inventory expected (each ships in a Phase A.2 sub-MR alongside its first failing-test-first user):

    • FinalizeIdempotencyStore — for platform-stab-2 §D1 finalize-failure → live-response Phase B test 3.

    • AttachmentStore — for platform-stab-2 §D4 commit-then-blob-fails Phase B test 6.

    • SendJobStore — for platform-stab-2 §D5 mid-send-failure Phase B test 7.

    • OutboxStore — for platform-stab-2 §D2 publish-then-update-fails (single-process simulation).

    • InboxStore — for platform-stab-2 §D3 retry-after-failure Phase B test 5.

      There is no Db newtype. The inner() → &PgPool accessor pattern is removed from this plan.

  3. Publisher (RabbitDownInjector + LatencyInjector + ConsumerBackpressureInjector): production stores Publisher directly in OutboxWorker, ExchangeSendWorker, etc. Pattern: trait-object via async_trait (or generic structs — see below). Native async fn in dyn-compatible traits is technically supported on Rust 1.75+, but the dyn-compatibility rules around Send/'static futures make a hand-rolled Arc<dyn PublishLike> brittle in the worker context (workers spawn the future onto a runtime, so the future must be Send + 'static). Two object-safe paths:

    // Path 1: async_trait macro (most ergonomic; small per-call boxing cost).
    #[async_trait::async_trait]
    pub trait PublishLike: Send + Sync {
        async fn publish(&self, env: &EventEnvelope) -> Result<(), PublishError>;
        async fn publish_in_tx(
            &self,
            tx: &mut sqlx::PgConnection,
            env: &EventEnvelope,
        ) -> Result<(), PublishError>;
        async fn publish_dlx(
            &self,
            env: &EventEnvelope,
            original_queue: &str,
            retry_count: i32,
            last_error: &str,
        ) -> Result<(), PublishError>;
    }
    // Worker holds Arc<dyn PublishLike + Send + Sync>; `async_trait` produces
    // the BoxFuture indirection automatically.
    // Path 2: generic worker struct (zero per-call cost; codegen monomorphizes).
    pub struct OutboxWorker<P: Publisher + Send + Sync + 'static> {
        pool: PgPool,
        publisher: Arc<P>,
    }
    // Production: OutboxWorker<RealPublisher>. Tests: OutboxWorker<RabbitDownInjector>.
    // Trade-off: each worker is a separate concrete type; harder to swap at runtime,
    // fine at compile-time.

    Decision rule: use Path 1 (async_trait) when the worker is constructed in main.rs and the choice of publisher impl varies per environment. Use Path 2 (generic) when the publisher impl is fixed at compile time and the worker is hot-path. Default proposal: Path 1 for the consolidation — single rebuild swaps the impl in tests, matches the existing Box<dyn ObjectStore> precedent in craig-store, and async_trait is already a workspace dep (verify via Cargo.toml [workspace.dependencies]; if absent, add async-trait = "0.1").

    Implement on Publisher (production) and on each injector wrapper. Change worker fields to Arc<dyn PublishLike> (with async_trait’s `Send + Sync + 'static bounds satisfied by the macro-generated impl).

  4. Store (object_store-backed) (ObjectStoreErrorInjector + LatencyInjector): craig-store already exposes a Store struct that wraps Arc<dyn object_store::ObjectStore>. Pattern: leverage the existing dyn boundary. Each injector wraps Arc<dyn ObjectStore> and delegates to inner; Store::wrap_for_test(injected_arc) constructor.

  5. PublishInTxFailureInjector (free fn craig_mq::stage_event): free fn signature is pub async fn stage_event(executor, envelope) → Result<…​>. Pattern: wrap in a small EventStager trait inside craig-test-lib and route test-mode calls through the trait; production keeps the free fn (no production change). Tests opt in via a thread-local override or per-test EventStager::install_for_test(…​).

Refactor cost summary:

  • Trait-object additions: ~20 LOC of trait + impl per dep family + ~5 callsite signature updates per dep family.

  • Worker struct fields: change publisher: Publisherpublisher: Arc<dyn PublishLike> in 7 services × 1-2 worker structs each = ~10 sites.

  • Db newtype rollout: ~15 sites across App* structs + ~50 store-fn callsites that today pass app.db.inner() already use the right shape — only the App* field type changes.

  • Total: ~3 LOC × ~80 callsites + ~150 LOC of new trait machinery = bounded; one focused MR per dep family. Fits as the first commit of Step 3 before any new injector lands.

Failing-test-first protocol for §D2: every new injector ships with (a) the seam refactor, (b) the injector wrapper, (c) one test that uses the injector against a real codepath. If (c) requires a codepath that doesn’t yet have the seam, file a follow-up issue and skip the injector until the seam lands. Do not merge an injector whose test target is uninjectable.

D2.0a. Initial state (epic &21 Step 2 inline injectors)

Epic &21 Step 2 introduces 5 minimal injectors:

  • CipherErrorInjector — wraps FieldEncryptor; inject_error_on_next_n(n)

  • DbErrorInjector — wraps PgPool; inject_error_after(n)brittle by design (per §D2.0 above): wrapping PgPool with a type that exposes &PgPool cannot intercept queries that flow downstream into sqlx::query(). Platform-stab-2 Step 2 ships this as a placeholder so its dependent fixes have something compiling, but the inline DbErrorInjector is structurally limited to scenarios where the test happens to use the wrapped pool’s surface methods directly. Phase A.2 (Step 3) of THIS plan replaces the inline DbErrorInjector with the codepath-specific store traits enumerated in §D2.0: FinalizeIdempotencyStore, AttachmentStore, OutboxStore, InboxStore, SendJobStore. Tests landed alongside platform-stab-2 fixes that today depend on DbErrorInjector will be migrated to the corresponding store trait as the trait lands — typically in the same MR, using the failing-test-first protocol (§D2.0 last paragraph).

  • RabbitDownInjector — wraps Publisher; inject_unavailable_for(duration). The Publisher API is a small surface (publish, publish_in_tx, publish_dlx) so the wrapper genuinely intercepts. This pattern stays valid.

  • ObjectStoreErrorInjector — wraps Store; inject_put_failure_for_keys(prefix). Store already wraps Arc<dyn ObjectStore> so the dyn boundary makes interception trivial. This pattern stays valid.

  • PublishInTxFailureInjector — wraps craig_mq::stage_event; fail_for_event_type(event_type). Free fn, intercepted via thread-local override. This pattern stays valid.

Each is a thin AtomicBool/AtomicUsize/Mutex<Instant> wrapper with a Drop impl that asserts no remaining injected-but-not-consumed faults. They cover the single-shot patterns: "fail once", "fail N", "fail for duration", "fail keys with prefix".

Failure-path tests in epic &21 Phase 2 (#274–#280) and the broader Phase B adoption (#291) need richer scenarios that cannot be expressed with single-shot counters:

  • Latency injection — "this DB query takes 4s instead of 4ms" — needed to assert await_winner ceiling-409 path (D1 Path C in the &21 plan) actually triggers under controlled conditions.

  • Partial-write simulation — "the blob put succeeds but the status-promotion UPDATE fails" — needed to assert AttachmentScanner reaps orphans correctly (&21 D4.5).

  • Intermittent flapping — "cipher fails on attempts 1, 3, 5; succeeds on 2, 4, 6" — needed to assert the inbox three-state retry’s backoff schedule actually retries to cap (&21 D3 Path C).

  • Timestamp-controlled scenarios — "at t=0 RMQ is up; at t=2s RMQ is down; at t=5s RMQ is up again" — needed for outbox-during-RMQ-blip tests.

  • Subscriber back-pressure — "consumer accepts the deliver but takes 60s to ack" — needed for any test that wants to assert outbox-depth alarms or DLX surface under slow-consumer conditions. No injector exists for this today.

D2.1. Decision: in-place extension via a module promotion

Same trade-off as D1: option (a) extend crates/craig-test-lib/src/fault_injection.rs in place, or (b) split into a fault/ subdirectory.

Unlike D1, the natural shape of the extension is multiple files — the existing 5 injectors (~50 LOC each) plus 3 new ones (~80 LOC each) plus a scenarios.rs registry plus a harness.rs trait module gives ~600 LOC total. That crosses the threshold where one file is unwieldy.

Decision: option (b) — promote fault_injection.rs to a fault/ module directory, keep pub use re-exports at the existing path:

// crates/craig-test-lib/src/fault_injection.rs (post-promotion)
//! Compatibility re-exports. All concrete types now live under
//! `crate::fault::*`. Will be removed in a future release once
//! callers migrate.
pub use crate::fault::cipher::CipherErrorInjector;
pub use crate::fault::rabbit::RabbitDownInjector;
pub use crate::fault::object_store::ObjectStoreErrorInjector;
pub use crate::fault::publish_in_tx::PublishInTxFailureInjector;
// `DbErrorInjector` is intentionally NOT re-exported. Per §D2.0,
// the `PgPool`-wrapping pattern is replaced by codepath-specific
// store traits (FinalizeIdempotencyStore, AttachmentStore, etc.).
// Tests using the inline DbErrorInjector are migrated to the
// matching trait as the seam lands; see §D2.0 sub-pattern (b).

The new layout:

crates/craig-test-lib/src/fault/
├── mod.rs                  # FaultInjector trait + Attempt + ScenarioGuard
├── cipher.rs               # CipherErrorInjector (migrated from inline)
├── rabbit.rs               # RabbitDownInjector
├── object_store.rs         # ObjectStoreErrorInjector
├── publish_in_tx.rs        # PublishInTxFailureInjector
├── stores/                 # NEW — codepath-specific store traits + faulty impls
│   ├── mod.rs              # trait definitions per §D2.0 sub-pattern (b)
│   ├── finalize_idempotency.rs   # FinalizeIdempotencyStore + Faulty impl
│   ├── attachment.rs       # AttachmentStore + Faulty impl
│   ├── outbox.rs           # OutboxStore + Faulty impl
│   ├── inbox.rs            # InboxStore + Faulty impl
│   └── send_job.rs         # SendJobStore + Faulty impl
├── latency.rs              # NEW — LatencyInjector wrapper
├── flap.rs                 # NEW — IntermittentFlapInjector
├── backpressure.rs         # NEW — ConsumerBackpressureInjector
└── scenarios.rs            # NEW — named pre-built Scenario constants

(The inline db.rs from platform-stab-2 Step 2 is retired in this consolidation; its callers migrate to a stores/ trait per §D2.0a above. The shim file keeps the *valid &21 Phase 2 cipher/rabbit/object-store/publish-in-tx injector imports unchanged.)

D2.2. The FaultInjector trait

// crates/craig-test-lib/src/fault/mod.rs

/// Common shape every fault injector implements.
pub trait FaultInjector: Send + Sync + 'static {
    /// The `Scenario` type drives this injector's behavior. Distinct
    /// per injector — `CipherScenario`, `DbScenario`, etc.
    type Scenario: Send + Sync + 'static;

    /// Configure the injector with a scenario. Replaces any prior
    /// scenario; returns a `ScenarioGuard` that, when dropped,
    /// asserts the scenario was *consumed* (consume == "the wrapped
    /// callsite was invoked under the scenario at least the expected
    /// number of times"). Use `scenario_guard.disarm()` to opt out
    /// of the assertion (e.g., for pre-condition setup that didn't
    /// actually need the injection to fire).
    #[must_use]
    fn inject(&self, scenario: Self::Scenario) -> ScenarioGuard<'_>;

    /// Audit log: every wrapped-callsite invocation under the active
    /// scenario, in invocation order. Tests use this to assert "the
    /// codepath was reached N times" without coupling to internal
    /// counters.
    fn recorded_attempts(&self) -> Vec<Attempt>;
}

/// One invocation of a fault-injector-wrapped callsite.
#[derive(Debug, Clone)]
pub struct Attempt {
    /// 1-based attempt number under the current scenario.
    pub n: u32,
    /// Wall-clock instant the attempt started.
    pub at: Instant,
    /// Outcome: did the injector return Err (fault injected) or pass
    /// the call through to the underlying real impl (success)?
    pub injected: bool,
    /// Optional per-injector context (e.g., for `ObjectStoreErrorInjector`,
    /// the object key the call was for).
    pub context: Option<String>,
}

/// RAII handle returned by `FaultInjector::inject`. On drop, asserts
/// the scenario's expected-consumption invariant. `disarm()` opts out.
pub struct ScenarioGuard<'a> {
    inner: Box<dyn FnOnce() + 'a>,  // the assertion closure
    armed: bool,
}

impl<'a> ScenarioGuard<'a> {
    pub fn disarm(mut self) { self.armed = false; }
}

impl Drop for ScenarioGuard<'_> {
    fn drop(&mut self) {
        if self.armed && !std::thread::panicking() {
            // Move the closure out via Option-trick.
            // Asserts "scenario was consumed as advertised".
        }
    }
}

The Drop assertion catches a class of false-positive tests that the &21 minimal injectors already attempt to catch: a test sets up inject_error_after(3) but the codepath only invokes the wrapped callsite twice, so the injection never fires, the test passes, and the "failure-path coverage" claim is structurally false. The ScenarioGuard makes that explicit — drop fires the assertion at test-teardown time, and the panic-checking guard means the assertion is suppressed if the test was already failing for an unrelated reason.

D2.3. LatencyInjector — wraps any of the 5 base injectors

// crates/craig-test-lib/src/fault/latency.rs

/// Wrap any `FaultInjector` to add a configurable latency before and/or
/// after each wrapped-callsite invocation. The wrapped injector's
/// scenario fires unchanged; the latency is purely additive.
pub struct LatencyInjector<I: FaultInjector> {
    inner: I,
    before: Option<Duration>,
    after: Option<Duration>,
}

impl<I: FaultInjector> LatencyInjector<I> {
    pub fn new(inner: I) -> Self;
    pub fn with_pre_delay(mut self, d: Duration) -> Self;
    pub fn with_post_delay(mut self, d: Duration) -> Self;
}

// Re-implements FaultInjector by delegating + sleeping.
impl<I: FaultInjector> FaultInjector for LatencyInjector<I> {
    type Scenario = I::Scenario;
    fn inject(&self, scenario: I::Scenario) -> ScenarioGuard<'_> { /* ... */ }
    fn recorded_attempts(&self) -> Vec<Attempt> { /* ... */ }
}

When-to-use: the &21 D1 Path C "loser observes processing for 5s, ceiling-409" test needs to make the winner’s handler genuinely take ≥5s. Wrap a FinalizeIdempotencyStore impl with LatencyInjector::new(real_store).with_pre_delay(Duration::from_secs(6)) on the finalize_succeeded call (the §D2.0 sub-pattern (b) faultable trait identified for §D1’s idempotency Path C). That makes the winner stall inside the trait method, the loser poll, and the ceiling fire deterministically.

Smoke test (crates/craig-test-lib/tests/fault_latency.rs): wrap a no-op FinalizeIdempotencyStore impl (or any CipherErrorInjector / ObjectStoreErrorInjector instance — they all implement the FaultInjector trait) with with_pre_delay(50ms); invoke the wrapped callsite; assert the recorded Attempt.at deltas are ≥ 50ms.

(Per §D2.0, there is no DbErrorInjector wrapping PgPool directly. DB fault injection happens at codepath-specific store traits — these traits each implement FaultInjector and compose with LatencyInjector via the same generic wrapper.)

D2.4. IntermittentFlapInjector — alternating Ok/Err on cadence

// crates/craig-test-lib/src/fault/flap.rs

/// Configurable cadence pattern of pass-through / inject-error.
pub struct IntermittentFlapInjector<I: FaultInjector> {
    inner: I,
    pattern: Vec<bool>,           // true = inject, false = pass-through
    counter: AtomicUsize,
}

impl<I: FaultInjector> IntermittentFlapInjector<I> {
    /// `pattern.len() >= 1`. The pattern repeats indefinitely:
    /// `[true, false]` = inject every odd attempt;
    /// `[false, false, true]` = inject every 3rd attempt;
    /// `[true, true, true, false]` = "fails 3-of-4".
    pub fn new(inner: I, pattern: Vec<bool>) -> Self;

    /// Convenience: alternating fail-then-succeed.
    pub fn alternating(inner: I) -> Self {
        Self::new(inner, vec![true, false])
    }
}

impl<I: FaultInjector> FaultInjector for IntermittentFlapInjector<I> {
    type Scenario = I::Scenario;
    /* ... */
}

When-to-use: inbox three-state retry test (epic &21 D3) needs to assert "handler fails attempt 1, succeeds attempt 2; processed_at IS NOT NULL, error_count == 1". The minimal CipherErrorInjector from &21 Step 2 takes inject_error_on_next_n(1), which works for that test. Phase B test #283 (the &22 adoption bundle’s "inbox flap then recover") needs pattern = [true, true, false] — fail twice, succeed on third — to exercise backoff schedule positions 1 and 2, then success.

Smoke test: wrap a CipherErrorInjector with [true, false, true]; invoke 6 times; assert attempts 1, 3, 5 errored and 2, 4, 6 succeeded (via recorded_attempts()).

D2.5. ConsumerBackpressureInjector — slow consumer simulation

The 6th injector. No equivalent exists in &21 Step 2; this is fully new in Phase A.2.

// crates/craig-test-lib/src/fault/backpressure.rs

/// Simulate a slow RabbitMQ consumer that delays its `delivery.ack()`.
/// Wraps a real `lapin::Consumer`-driven loop and inserts a configurable
/// pre-ack sleep; can also drop deliveries silently to simulate hung
/// consumers.
pub struct ConsumerBackpressureInjector {
    /// Per-delivery pre-ack delay. Default `Duration::ZERO`.
    pub ack_delay: Duration,
    /// If `Some(n)`, every nth delivery is *not acked* (simulates a
    /// permanently hung consumer slot).
    pub drop_every_nth: Option<u32>,
    counter: AtomicU32,
    recorded: Mutex<Vec<Attempt>>,
}

impl ConsumerBackpressureInjector {
    pub fn new() -> Self;
    pub fn with_ack_delay(mut self, d: Duration) -> Self;
    pub fn drop_every(mut self, n: u32) -> Self;

    /// Wrap a delivery handler closure to apply the configured back-pressure.
    /// Used by tests that drive a real or fake `lapin::Consumer` stream
    /// through this injector before asserting outbox-depth / DLX behavior.
    pub async fn handle_delivery<F, Fut>(
        &self,
        delivery: lapin::message::Delivery,
        handler: F,
    ) -> anyhow::Result<()>
    where
        F: FnOnce(lapin::message::Delivery) -> Fut,
        Fut: Future<Output = anyhow::Result<()>>;
}

impl FaultInjector for ConsumerBackpressureInjector {
    type Scenario = BackpressureScenario;
    /* ... */
}

pub enum BackpressureScenario {
    Steady { ack_delay: Duration },
    SpikeFor { duration: Duration, ack_delay: Duration },
    HangEveryNth { n: u32 },
}

When-to-use:

  • DLX surface under slow consumer — Phase B test for #291: stage 100 events, run a ConsumerBackpressureInjector with ack_delay = 5s, assert RabbitMQ’s per-consumer prefetch hits the cap, outbox-depth alarm fires.

  • Inbox claim-recovery under hung consumerHangEveryNth { n: 3 } drops every 3rd delivery; redelivery surfaces; the inbox row’s attempt_started_at is past the recovery window; the next worker re-claims.

  • /livez vs /readyz semantic-differentiation (Phase B test for #291): RMQ-side back-pressure should not flip /livez to false, but should flip /readyz to false. Without this injector, the test cannot induce the condition.

Smoke test (crates/craig-test-lib/tests/fault_backpressure.rs): drive a fake lapin::message::Delivery stream of 5 items through with_ack_delay(50ms); assert per-delivery handling time ≥ 50ms; assert recorded_attempts().len() == 5.

D2.6. craig_test_lib::fault::scenarios — named pre-built scenarios

// crates/craig-test-lib/src/fault/scenarios.rs

//! Named pre-built scenarios for the most common failure-path tests.
//! Tests should prefer these named constants over open-coding scenarios
//! — keeps test intent legible and centralizes "what does 'flaky DB'
//! mean for this codebase" in one place.

use super::*;

pub mod db {
    /// Pool exhaustion after `n` successful queries. Mirrors the
    /// production failure where pgbouncer hits its connection cap.
    pub fn pool_exhausted_after_n(n: u32) -> DbScenario { /* ... */ }

    /// Connection refused on every query. Mirrors the production
    /// failure where the DB is unreachable (e.g., during a Postgres
    /// failover).
    pub fn connection_refused() -> DbScenario { /* ... */ }

    /// Serialization-failure on every query. Mirrors the production
    /// failure where row-level locking + concurrent updates trigger
    /// `serializable_isolation_failure`.
    pub fn serialization_failure() -> DbScenario { /* ... */ }
}

pub mod rmq {
    /// RMQ unreachable for `dur`, then comes back. Models a
    /// short-lived broker outage / failover.
    pub fn flap_for_5s() -> RabbitScenario { flap_for(Duration::from_secs(5)) }
    pub fn flap_for(dur: Duration) -> RabbitScenario { /* ... */ }

    /// Permanently down. Models broker-side cluster failure with no
    /// recovery in test runtime.
    pub fn permanently_down() -> RabbitScenario { /* ... */ }
}

pub mod cipher {
    /// `FieldEncryptor` panics once. Models the corrupted-key case
    /// where the cipher impl panics on bad key material.
    pub fn panic_once() -> CipherScenario { /* ... */ }

    /// `FieldEncryptor` returns Err on next 3 attempts (matches the
    /// inbox max_retries default of 5 attempts so the test can
    /// exercise "3 fails, then succeed" — under-cap retry).
    pub fn err_3_then_succeed() -> CipherScenario { /* ... */ }
}

pub mod object_store {
    /// `Store::put` returns Err for any key matching the prefix.
    pub fn put_failure_for_prefix(prefix: &'static str) -> ObjectStoreScenario { /* ... */ }

    /// `Store::head` returns 404 (key not found) for any key — models
    /// the post-Phase-2-orphan condition where DB has a `pending` row
    /// but the blob never landed.
    pub fn head_404_all() -> ObjectStoreScenario { /* ... */ }
}

When-to-use: every Phase B failure-path test should reach for a named scenario first; falls back to open-coding only if the named registry doesn’t cover the case. New named scenarios are added here as post-merge follow-ups whenever a test introduces an open-coded scenario that’s likely to recur.

D2.7. Test descriptions

For each new injector, smoke tests live at crates/craig-test-lib/tests/fault_<name>.rs:

  1. fault_latencywith_pre_delay(50ms) adds ≥50ms; with_post_delay similarly; both compose; recorded_attempts() shows correct timing.

  2. fault_flap_alternating — pattern [true, false], 6 invocations, attempts 1/3/5 errored, 2/4/6 succeeded (via recorded_attempts()).

  3. fault_flap_repeating_pattern — pattern [true, true, true, false], 8 invocations, attempts 1/2/3, 5/6/7 errored, 4/8 succeeded.

  4. fault_backpressure_ack_delay — drive 5 fake deliveries, ack_delay=50ms; per-delivery ≥50ms.

  5. fault_backpressure_drop_every_nthdrop_every(3) over 9 deliveries; deliveries 3, 6, 9 are recorded as injected=true (no ack); rest are recorded as injected=false.

  6. fault_scenario_guard_drop_asserts — set up db.inject(pool_exhausted_after_n(2)), invoke wrapped callsite once, drop the guard; the test should panic with "scenario not consumed". Wrap in std::panic::catch_unwind to assert.

  7. fault_scenario_guard_disarm_does_not_assert — same setup, guard.disarm(), drop; no panic.

  8. fault_scenario_named_constants_compile — call every named scenario in scenarios::* to ensure the registry is wired up (compile-time check; runtime assertions optional per-scenario).

  9. fault_scenario_named_db_pool_exhaustedpool_exhausted_after_n(3) applied to a wrapped no-op pool; 4 invocations; first 3 succeed, 4th errors with sqlx::Error::PoolTimedOut (or chosen variant).

Path A/B/C structure for the most safety-critical case (the Drop assertion):

  1. Path A — guard drops with consumed scenario: scenario fires the expected number of times; drop is silent.

  2. Path B — guard drops with under-consumed scenario, no test panic in flight: drop panics with assertion message; test catches via catch_unwind and asserts the panic message.

  3. Path C — guard drops during an unrelated test panic: simulate a test panicking before the guard drops (std::panic::panic_any); the guard’s Drop checks std::thread::panicking() and skips its assertion so the original panic surfaces unmuddled. Assert the surfaced panic carries the original payload, not the guard’s.

D3. Multi-replica devstack profile (Phase A.3 — Step 4)

Today the devstack is single-instance for every CRAIG application service. docker-compose.yml declares each service exactly once with a fixed --ports 800X:800X host-port mapping. Multi-replica behavior — two OutboxWorker`s draining the same `event_outbox, two craig-cases instances racing the idempotency-claim middleware — is simulated single-process in epic &21’s tests by spawning multiple worker tasks against one PgPool. That works for the per-test verification of &21 Steps 4 and 5, but it does not exercise:

  • Two real OS processes contending on FOR UPDATE SKIP LOCKED against a shared Postgres connection (separate connection pools, separate prepared-statement caches, separate Rust task schedulers).

  • Two real HTTP listeners behind a load-balancer answering same-key POSTs — needed to assert idempotency atomic-claim works across replicas, not just across tasks.

  • Two real RabbitMQ consumers competing for the same queue — needed for the inbox three-state-retry test under realistic broker prefetch and ack semantics.

Phase B failure-path tests (#291) — specifically "Multi-replica outbox" and "Concurrent same-key idempotency" — need a real multi-replica mode. This step builds it.

D3.1. Open-question default: docker-compose --scale

The plan’s existing Open Questions section flags this as decision-pending between --scale and custom xtask logic, with default proposal --scale. This step adopts the default. Rationale:

Path Verdict

docker-compose --scale <service>=N

Chosen. Compose-native, zero new orchestration code, every CRAIG service is already 12-factor-stateless (state in shared Postgres / RabbitMQ / Garage). The only friction is host-port pinning — fixed by switching scaled services to ephemeral ports + a Docker-internal hostname (which already exists for cross-service traffic via the compose network DNS).

Custom xtask N-instance launcher with manual port allocation

Rejected. Re-implements docker-compose’s load-balancer-pattern hostname resolution. ~300 LOC of orchestration code with no capability gain over --scale.

Kubernetes / k3d local cluster

Rejected. Multiple-orders-of-magnitude more setup cost; CI runners would need k3d in their image; docs would diverge from production manifests anyway.

The trade-off: --scale requires the host-port mapping to not be fixed for the scaled service, since Docker can’t bind 8002:8002 twice. Compose handles this by binding <port> (no host pin) and assigning ephemeral host ports — and tests reach replicas by name via the internal compose-network DNS, not by host port.

D3.2. xtask interface

Extend xtask/src/cmd/dev.rs::DevCommand::Start with a --replicas arg. Builds on the existing Start { force: bool } shape:

// xtask/src/cmd/dev.rs

#[derive(Subcommand)]
pub enum DevCommand {
    /// Start/rebuild all services.
    Start {
        /// Force full rebuild, skip staleness check.
        #[arg(long)]
        force: bool,

        /// Scale specific services. Format: `SERVICE=N[,SERVICE=N…]`.
        /// Each entry must name a service from CRAIG_SERVICES;
        /// N must be ≥1. Repeats with `=1` are no-ops.
        ///
        /// Example: `--replicas craig-cases=2,craig-exchange=2`
        ///
        /// Services not listed default to scale 1 (current behavior).
        ///
        /// Limitations:
        /// * Only stateless CRAIG application services are scalable
        ///   (the 8 in CRAIG_SERVICES). Postgres/RabbitMQ/Keycloak/
        ///   Garage scale to 1 — sharing state requires external
        ///   replication this devstack doesn't model.
        /// * `craig-web` (BFF) is scalable but the 8080 host port
        ///   pins to the first replica. Use `craig-web=1` for now.
        /// * Scaled services lose their fixed host ports; reach them
        ///   by Docker-internal DNS (`craig-cases-1`, `craig-cases-2`)
        ///   from inside the compose network, or by the ephemeral
        ///   host ports surfaced via `docker compose port`.
        #[arg(long, value_parser = parse_replicas)]
        replicas: Vec<ReplicaSpec>,
    },
    // ... rest unchanged
}

#[derive(Debug, Clone)]
pub struct ReplicaSpec {
    pub service: String,
    pub count: u32,
}

fn parse_replicas(raw: &str) -> Result<Vec<ReplicaSpec>, String> {
    let mut out = Vec::new();
    for entry in raw.split(',') {
        let (svc, n) = entry.split_once('=')
            .ok_or_else(|| format!("expected SERVICE=N, got {entry}"))?;
        if !CRAIG_SERVICES.contains(&svc) {
            return Err(format!(
                "unknown service '{svc}'. Allowed: {}",
                CRAIG_SERVICES.join(", ")
            ));
        }
        let count: u32 = n.parse().map_err(|_| format!("non-integer count: {n}"))?;
        if count == 0 {
            return Err(format!("count must be >= 1, got 0 for {svc}"));
        }
        out.push(ReplicaSpec { service: svc.to_string(), count });
    }
    Ok(out)
}

D3.3. Threading replicas through devstack::bring_up

xtask/src/cmd/dev.rs::run passes the replicas list to devstack::ensure_ready, which threads it through to bring_up:

// xtask/src/devstack.rs

pub fn ensure_ready(force: bool, replicas: &[ReplicaSpec]) -> Result<()> { /* ... */ }

pub fn bring_up(no_cache: bool, replicas: &[ReplicaSpec]) -> Result<()> {
    // ... existing port-reservation, compose-up logic ...

    let mut up_args: Vec<String> = vec!["up".into(), "-d".into(), "--build".into()];
    for spec in replicas {
        up_args.push("--scale".into());
        up_args.push(format!("{}={}", spec.service, spec.count));
    }
    docker::compose(&up_args.iter().map(String::as_str).collect::<Vec<_>>())?;

    // health-wait (next section) ...
}

When replicas is empty, the args degenerate to the current ["up", "-d", "--build"] — zero behavior change for the default case.

D3.4. docker-compose.yml adjustments

Three categories of change. All are minimal and --scale=1-compatible (no behavior change when the flag is absent):

  1. Drop fixed container_name — none of the CRAIG services declare container_name today, so this is a no-op verification step. Confirm via Grep "container_name:" docker-compose.yml.

  2. Switch host-port pins from "8002:8002" to "8002" for scalable servicesonly when scaling. The default-scale-1 case keeps the current 8002:8002 mapping because tests that connect from the host (integration tests, k6, ZAP) need a stable port. The cleanest path: introduce a docker-compose override file docker-compose.scale.yml that re-declares the scalable services with floating-port mappings, loaded only when --replicas is passed.

  3. Confirm shared-state-only — Postgres, RabbitMQ, Garage, Keycloak remain scale=1; verify each scaled service’s environment block references shared infra (postgres/rabbitmq/keycloak/garage hostnames) and not per-instance state.

# docker-compose.scale.yml — loaded only when --replicas is non-empty.
# Removes host-port pins from scalable services so docker-compose can
# bind the container ports to ephemeral host ports across replicas.
services:
  craig-rules:
    ports: !override
      - "8001"
  craig-cases:
    ports: !override
      - "8002"
  craig-placement:
    ports: !override
      - "8003"
  craig-exchange:
    ports: !override
      - "8004"
  craig-financial:
    ports: !override
      - "8005"
  craig-reporting:
    ports: !override
      - "8006"
  craig-security:
    ports: !override
      - "8007"
  craig-intake:
    ports: !override
      - "8008"

The !override tag (compose v2.20+) replaces the parent’s ports list rather than appending. Confirmed against the compose version pinned in .gitlab-ci.yml.

D3.5. Compose-override vs in-place compose edit

Two strategies for managing the scale-mode port adjustments:

Path Verdict

Compose-override file (docker-compose.scale.yml)

Chosen. Loaded conditionally via docker compose -f docker-compose.yml -f docker-compose.scale.yml up … only when --replicas is non-empty. Default flow unchanged. CI / pre-push hooks / E2E unaffected. Override file lives next to the main compose, version-controlled, easy to audit.

In-place edit of docker-compose.yml to floating ports

Rejected. Breaks every test/script that connects to host localhost:8002. Would force a parallel migration of integration tests + k6 + ZAP configs to docker compose port discovery — a large, mostly-mechanical diff with no benefit when most of the time scaling is unused.

Programmatic compose-rendering inside xtask

Rejected. Re-implements compose-merge semantics. Overrides are the built-in mechanism; no benefit to recreating them in Rust.

D3.6. Health-wait under scaling

devstack::wait_for_service_health(svc) (in xtask/src/devstack.rs) calls docker::is_healthy("craig-cases"). Under --scale=2 there are two containers — craig-cases-1 and craig-cases-2 — and the helper needs to wait for both. Strategy: enumerate replica container names via docker compose ps --services --filter status=running, then poll each. Returns Ok only when every replica reports healthy.

// xtask/src/devstack.rs (additions)

fn replica_names(svc: &str) -> Result<Vec<String>> {
    // `docker compose ps -a <svc> --format {{.Name}}` returns
    // one line per replica. Trim, filter empties.
    let raw = docker::compose_output(&["ps", "-a", svc, "--format", "{{.Name}}"])?;
    Ok(raw.lines().filter(|l| !l.is_empty()).map(String::from).collect())
}

fn wait_for_service_health_all_replicas(svc: &str) -> Result<()> {
    let replicas = replica_names(svc)?;
    if replicas.is_empty() {
        bail!("no running replicas of {svc} to wait for");
    }
    for name in replicas {
        wait_for_named_health(&name)?;
    }
    Ok(())
}

docker::is_healthy already accepts a container name; the adjustment is per-replica iteration only.

D3.7. MultiReplicaCluster test helper

For tests that need multi-replica directly (not via the global devstack that pre-push runs), expose a builder:

// crates/craig-test-lib/src/multi_replica.rs (new)

/// Programmatic multi-replica devstack control for tests that drive a
/// dedicated cluster (separate from the pre-push devstack). Tests that
/// run against the *existing* devstack should use the existing
/// `TestHarness` and reach replicas by their compose-network names.
pub struct MultiReplicaCluster {
    workspace_root: std::path::PathBuf,
    replicas: Vec<ReplicaSpec>,
    compose_files: Vec<std::path::PathBuf>,
}

pub struct MultiReplicaClusterBuilder { /* … */ }

impl MultiReplicaCluster {
    pub fn builder() -> MultiReplicaClusterBuilder;

    /// Bring up the cluster. Idempotent — if already up with the same
    /// scale config, no-op.
    pub async fn start(&mut self) -> anyhow::Result<()>;

    /// Tear down the cluster. Always wipes volumes.
    pub async fn stop(self) -> anyhow::Result<()>;

    /// Discover replica host:port endpoints. Iterates `docker compose
    /// port --index N <svc> <port>` for each `index in 1..=count` —
    /// the `--index` flag is **mandatory** here; without it,
    /// `docker compose port <svc> <port>` returns the FIRST replica's
    /// host port only (a silent ambiguity that would make tests pass
    /// against one replica while claiming to test both).
    /// Returns a Vec of `http://localhost:<ephemeral>` URLs, one per
    /// replica of `svc`.
    pub async fn replica_urls(&self, svc: &str) -> anyhow::Result<Vec<String>>;

    /// Reach a specific replica by 1-based index (matching docker
    /// compose's `--index` numbering). Useful for asserting which
    /// replica handled which request (e.g., outbox claim coverage).
    pub async fn replica_url(&self, svc: &str, index: usize) -> anyhow::Result<String>;
}

impl MultiReplicaClusterBuilder {
    pub fn replica(mut self, svc: &str, count: u32) -> Self;
    pub fn extra_compose_file(mut self, path: impl Into<std::path::PathBuf>) -> Self;
    pub fn build(self) -> MultiReplicaCluster;
}

When-to-use: integration tests that require multi-replica (Phase B test for #275 and the verification test below). Tests that don’t need multi-replica continue to use the global TestHarness against the default scale=1 devstack.

Implementation note: start() shells to docker compose -p <unique-project> -f … --scale … up -d. Health waits use the same poll loop as the xtask path, calling out to docker compose -p <unique-project> ps … --format {{.Name}}. Port discovery uses docker compose -p <unique-project> port --index <N> <svc> <internal-port>, iterating N from 1 to the configured replica count. The helper does not re-implement compose orchestration in Rust — it shells out so the production-path cargo xtask dev start --replicas and the test-path MultiReplicaCluster exercise the same compose invocation.

Project isolation (critical): every MultiReplicaCluster MUST construct a unique --project-name (-p) per test instance — proposal: format!("craig-mrc-{}", uuid::Uuid::now_v7().simple()). Without this, the test cluster shares the docker-compose project name with the developer’s running cargo xtask dev devstack, and cluster.stop() (which always wipes volumes via down -v) would destroy the developer’s local DB / RMQ / Garage state. The start()/stop() implementations bail with a clear error if the project name collides with craig (the default xtask devstack project name) or any other running project visible via docker compose ls --format json.

D3.8. Verification test

A new integration test at crates/craig-test-lib/tests/multi_replica_idempotency.rs exercises the full path:

#[tokio::test]
async fn concurrent_same_key_post_to_two_replicas() {
    // Path A: spin up dedicated 2-replica cluster.
    let mut cluster = MultiReplicaCluster::builder()
        .replica("craig-cases", 2)
        .build();
    cluster.start().await.unwrap();

    let urls = cluster.replica_urls("craig-cases").await.unwrap();
    assert_eq!(urls.len(), 2);

    let harness = TestHarness::new().await.unwrap();
    let token = harness.tokens.get_token("admin", "password").await.unwrap();
    let body = serde_json::json!({ "case_number": "FULTON-20260503-DEAD" });
    let key = uuid::Uuid::now_v7().to_string();

    // Fire 5 same-key POSTs across both replicas — alternate by index.
    let report = concurrent_fire_collect_ordered(5, |i| {
        let url = urls[i % 2].clone();
        let token = token.clone();
        let body = body.clone();
        let key = key.clone();
        async move {
            shared_http_client()
                .post(format!("{url}/v1/cases"))
                .bearer_auth(&token)
                .header("Idempotency-Key", &key)
                .json(&body)
                .send()
                .await
                .unwrap()
                .status()
                .as_u16()
        }
    }).await;

    let statuses = report.into_results().unwrap();
    assert_eq!(statuses.len(), 5);
    // Exactly one 200 (winner), four 200s as replays, none as 409
    // (assuming the winner's handler completes inside the 5s ceiling).
    assert!(statuses.iter().all(|s| *s == 200));

    // Path B: query both replicas; assert there's exactly one row in
    // craig-cases' DB matching the case_number. (Single Postgres
    // backs both replicas; if the atomic-claim worked, there's
    // exactly one INSERT.)
    let admin_client = harness.admin_cases_client().await.unwrap();
    let listed = admin_client.list_cases(/* filter by number */).await.unwrap();
    assert_eq!(listed.total, 1);

    cluster.stop().await.unwrap();
}

Three paths under verification:

  1. Path A — concurrent fire across replicas: 5 POSTs split between 2 replicas, all return 200, all return identical body (winner
    replays; loser path runs against the other replica’s handler-side effect by reading from the shared Postgres).

  2. Path B — single handler-side effect lands: assert cases table has exactly one row matching the test’s case_number, regardless of which replica wrote it.

  3. Path C — replicas are genuinely contending: instrument with a per-request response header X-Served-By: craig-cases-{1|2} (added via tower middleware in the test build) and assert the 5 responses span both replica hostnames. Without this, the test could pass with a load-balancer that pinned all 5 to one replica — which would silently degrade the test’s coverage.

D4. State-machine matrix-testing pattern (Phase A.4 — Step 5)

CRAIG ships 16 hand-rolled state machines across 6 services. Each one is a validate_<entity>_transition(from, to) → Result<(), String> wrapper over an enum’s can_transition_to(target) matches! arm. Today the only coverage is hand-picked positive + a handful of negative cases per machine (see services/craig-financial/src/transitions.rs:141-272 for the densest example). That pattern guarantees nothing about the invalid half of the state space — a regression that adds an unintended (Cleared, Pending) arm to PaymentStatus::can_transition_to would not be caught by any existing test, because no test enumerates the rejection set.

This step lands a generic helper that, given the enumeration of valid transitions and a rejection function, asserts every (from, to) pair not in the input list is rejected. First adoption is the financial-payment state machine.

D4.1. Decision: hand-rolled vs proptest

Option Rejected because

proptest Strategy<(State, State)>

proptest’s strength is unbounded input generation with shrinking. CRAIG state alphabets are bounded (`PaymentStatus = 5 variants, the largest in this repo, IcpcStatus = 7); the full Cartesian square is at most 49 pairs. Generators add a dependency, an integration learning curve, and shrinking machinery for a problem already small enough to enumerate.

test-strategy proc-macro on each enum

Adds another macro to debug. Same finite-alphabet objection.

Hand-rolled assert_only_valid_transitions_accepted(&[…], reject_fn)

Selected. Bounded enums + a single helper crate-internal to craig-test-lib. Trade-off documented: if a future state machine grows past ~20 variants (400 pairs) reconsider.

This is the same hand-rolled-default already flagged in platform-stab-2 §Open Questions item 5 and in the parent epic plan; this section formalizes it.

D4.2. Helper module

New file crates/craig-test-lib/src/state_machine_matrix.rs. Re-export from crates/craig-test-lib/src/lib.rs as pub mod state_machine_matrix.

Public API (builder pattern, not trait — every CRAIG state machine is a closed enum, no need for generic StateMachine plumbing):

/// One row of the truth-table input.
#[derive(Debug, Clone, Copy)]
pub struct ValidTransition<S: Copy + Eq + std::fmt::Debug + 'static> {
    pub from: S,
    pub to: S,
}

/// Outcome bucket for a single (from, to) pair.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum Outcome {
    /// `reject_fn` returned `Ok(())` and the pair was in the valid list.
    AcceptedValid,
    /// `reject_fn` returned `Err(_)` and the pair was NOT in the valid list.
    RejectedInvalid,
    /// **Bug:** valid pair rejected.
    AcceptedExpectedReject,
    /// **Bug:** invalid pair accepted.
    RejectedExpectedAccept,
    /// `reject_fn` panicked. Always a bug — transition validators must
    /// never panic on bad input strings.
    Panicked(String),
}

/// Result of running a full matrix.
#[derive(Debug, Default)]
pub struct MatrixReport<S> {
    pub total: usize,
    pub accepted_valid: Vec<(S, S)>,
    pub rejected_invalid: Vec<(S, S)>,
    pub mismatches: Vec<(S, S, Outcome)>,
    pub panics: Vec<(S, S, String)>,
}

impl<S> MatrixReport<S>
where
    S: Copy + Eq + std::fmt::Debug,
{
    /// Convenience: `assert!()` no mismatches and no panics. Pretty-prints
    /// the offending pairs into the assertion message.
    pub fn assert_clean(&self);
}

/// Drive a transition predicate against the full Cartesian square of an
/// enum's variants and assert the rejection set is exactly the
/// complement of `valid`.
///
/// `all` enumerates every variant; `valid` lists every accepted ordered
/// pair; `reject_fn` is the function under test (returns `Ok(())` on
/// accept, `Err(_)` on reject; must never panic on well-formed input).
///
/// `reject_fn` runs inside `std::panic::catch_unwind` so a panic on any
/// pair is captured as `Outcome::Panicked` rather than aborting the
/// suite.
pub fn assert_only_valid_transitions_accepted<S, F>(
    all: &[S],
    valid: &[ValidTransition<S>],
    reject_fn: F,
) -> MatrixReport<S>
where
    S: Copy + Eq + std::fmt::Debug + std::panic::UnwindSafe + 'static,
    F: Fn(S, S) -> Result<(), String> + std::panic::UnwindSafe + Copy;

D4.3. First adoption: PaymentStatus

The financial-payment state machine lives at services/craig-financial/src/transitions.rs:11-32 (PaymentStatus, PaymentStatus::can_transition_to, validate_payment_transition). The user spec referenced a transitions/payment.rs submodule path; current layout is single-file. No submodule split here — keep the diff minimal; future decomposition is out of scope.

New test file: services/craig-financial/tests/transitions_matrix.rs.

use craig_financial::transitions::{PaymentStatus, validate_payment_transition};
use craig_test_lib::state_machine_matrix::{
    ValidTransition as VT, assert_only_valid_transitions_accepted,
};

const ALL: &[PaymentStatus] = &[
    PaymentStatus::Pending,
    PaymentStatus::Approved,
    PaymentStatus::Issued,
    PaymentStatus::Cleared,
    PaymentStatus::Voided,
];

const VALID: &[VT<PaymentStatus>] = &[
    VT { from: PaymentStatus::Pending,  to: PaymentStatus::Approved },
    VT { from: PaymentStatus::Approved, to: PaymentStatus::Issued },
    VT { from: PaymentStatus::Issued,   to: PaymentStatus::Cleared },
    // any -> voided
    VT { from: PaymentStatus::Pending,  to: PaymentStatus::Voided },
    VT { from: PaymentStatus::Approved, to: PaymentStatus::Voided },
    VT { from: PaymentStatus::Issued,   to: PaymentStatus::Voided },
    VT { from: PaymentStatus::Cleared,  to: PaymentStatus::Voided },
];

#[test]
fn payment_state_machine_matrix_is_exhaustive() {
    let report = assert_only_valid_transitions_accepted(ALL, VALID, |from, to| {
        validate_payment_transition(&from.to_string(), &to.to_string())
    });
    report.assert_clean();
    // 5 × 5 = 25 pairs. 7 valid (above). 18 must reject — including 5 self-loops.
    assert_eq!(report.total, 25);
    assert_eq!(report.accepted_valid.len(), 7);
    assert_eq!(report.rejected_invalid.len(), 18);
}

validate_payment_transition accepts string args (&str → Result<(), String>); the closure adapts the strongly-typed enum back to that shape via Display (already derived on the enum). Future migration that takes typed enum args at the validator boundary is a follow-up — out of scope for this step.

D4.4. Failing-test-first protocol

To prove the matrix actually catches a regression, the MR sequence is:

  1. Land state_machine_matrix.rs with the helper.

  2. Write payment_state_machine_matrix_is_exhaustive per D4.3 but omit (Issued, Cleared) from VALID. Expect: assertion fires with Outcome::AcceptedExpectedReject for the omitted pair, listing (Issued, Cleared) in MatrixReport::mismatches. Commit a screenshot of the failure into the MR description.

  3. Restore (Issued, Cleared) to VALID. Test passes. Land the MR.

  4. Subsequent state-machine adoption MRs (D4.5) repeat the same dance.

This protocol is the on-ramp for the Process Rule encoded in §D7.4.

D4.5. Cross-references — all 16 documented state machines

Future Phase B adoption issue (#291 per the parent epic; the bundle issue tracking the matrix rollout across all 16) covers each. Itemized so a contextless agent can locate them:

# Service File:Symbol Variants

1

craig-financial

services/craig-financial/src/transitions.rs:11 PaymentStatus (validate_payment_transition)

5

2

craig-financial

services/craig-financial/src/transitions.rs:39 AdjustmentStatus (validate_adjustment_transition)

3

3

craig-financial

services/craig-financial/src/transitions.rs:59 ClaimingStatus (validate_claiming_transition)

3

4

craig-cases

services/craig-cases/src/transitions.rs (validate_investigation_transition, line 97)

TBD via local read

5

craig-cases

services/craig-cases/src/transitions.rs:112 validate_case_transition

TBD

6

craig-cases

services/craig-cases/src/transitions.rs:127 validate_case_plan_transition

TBD

7

craig-cases

services/craig-cases/src/transitions.rs:142 validate_task_transition

TBD

8

craig-placement

services/craig-placement/src/transitions.rs:57 validate_placement_transition

TBD

9

craig-placement

services/craig-placement/src/transitions.rs:72 validate_license_transition

TBD

10

craig-exchange

services/craig-exchange/src/transitions.rs:92 validate_agreement_transition

TBD

11

craig-exchange

services/craig-exchange/src/transitions.rs:111 validate_transaction_transition

TBD

12

craig-exchange

services/craig-exchange/src/transitions.rs:126 validate_icpc_transition

TBD

13

craig-reporting

services/craig-reporting/src/transitions.rs:40 validate_afcars_transition

TBD

14

craig-reporting

services/craig-reporting/src/transitions.rs:61 validate_ncands_transition

TBD

15

craig-security

services/craig-security/src/transitions.rs:64 validate_review_transition

TBD

16

craig-security

services/craig-security/src/transitions.rs:88 validate_nist_transition

TBD

Each of items 4–16 is a follow-up MR under Phase B. Step 5 lands the helper + item 1 only; the remaining 15 are tracked but not blocking this step’s completion.

D4.6. Test-coverage description

The helper itself ships with two unit tests in crates/craig-test-lib/tests/state_machine_matrix.rs:

  1. Happy path: a 3-variant enum (enum E { A, B, C }), valid = &[(A, B), (B, C)], predicate consistent with valid. Assert MatrixReport::mismatches.is_empty(), total == 9, accepted_valid.len() == 2, rejected_invalid.len() == 7.

  2. Mismatch detection: same enum, predicate that erroneously also accepts (C, A). Assert mismatches contains (C, A, Outcome::AcceptedExpectedReject).

  3. Panic capture: predicate that panic!()`s on `(B, C). Assert panics contains the pair with the panic message; assert_clean returns failure rather than aborting.

Plus the financial-payment integration test from D4.3.

D5. JWT mutation library (Phase A.5 — Step 6)

ADR-021 commits craig-auth to a strictness policy: tokens must carry aud matching the service name, typ: Bearer, valid iss, signing alg in the JWKS-advertised allowlist, present-and-recent exp/iat, optional azp/scope checked when service config requires. The current test surface for that policy is exactly two cases per service (a passing token + "invalid.token.here" per crates/craig-auth/src/middleware.rs:178-205). Strict-validator regressions — accidentally widening Validation::set_audience to multiple values, dropping the alg-pin, accepting none-alg tokens — pass that surface trivially.

This step lands a single-axis mutation library: a builder that mints valid baseline tokens, plus a mutation_matrix iterator yielding one variant per axis with one field tampered. Each variant is asserted to fail with a specific ApiError variant.

D5.1. Helper module

New file crates/craig-test-lib/src/jwt_mutation.rs. Re-export from lib.rs as pub mod jwt_mutation.

D5.2. Test signing-key fixture

craig-auth/test-fixtures/test_rsa_private.der already exists (used by crates/craig-auth/src/middleware.rs:104). Reuse it. Expose a JwksFixture API on the auth crate so the test-lib doesn’t have to reach into private fixtures:

// crates/craig-auth/src/jwks.rs (additive)

#[cfg(any(test, feature = "test-fixtures"))]
pub mod test_fixtures {
    use super::*;
    use jsonwebtoken::jwk::{Jwk, JwkSet};
    use jsonwebtoken::{Algorithm, EncodingKey};

    /// The shared RSA private key that test middlewares + the
    /// `craig-test-lib::jwt_mutation` helpers sign tokens with.
    /// Identical bytes to `crates/craig-auth/test-fixtures/test_rsa_private.der`.
    pub fn test_encoding_key() -> EncodingKey {
        let der = include_bytes!("../test-fixtures/test_rsa_private.der");
        EncodingKey::from_rsa_der(der)
    }

    /// JWKS containing the public half of [`test_encoding_key`], with
    /// the given `kid`. Mount this on a [`JwksProvider`] via
    /// [`JwksProvider::inject_keys`] (test-only fn — make `pub(crate)`
    /// → `pub` under the same `test-fixtures` feature).
    pub fn test_jwk_set(kid: &str) -> JwkSet {
        let key = test_encoding_key();
        let mut jwk = Jwk::from_encoding_key(&key, Algorithm::RS256)
            .expect("build JWK from RSA test key");
        jwk.common.key_id = Some(kid.to_string());
        JwkSet { keys: vec![jwk] }
    }
}

Cargo.toml: add a test-fixtures feature on craig-auth that re-exports the same module to consumers (craig-test-lib, integration tests of every service that mounts auth_middleware). This is the minimum visibility change; the alternative (a separate craig-auth-test crate) is a heavier refactor for marginal isolation gain.

D5.3. ClaimsBuilder

/// Mutable JWT claim set under construction. All getters/setters are
/// chainable; finalize with [`mint`].
///
/// Defaults (set by `ClaimsBuilder::baseline`): valid sub, future exp
/// (1h), past iat, `iss = TEST_ISSUER`, `aud = ["craig-cases"]` (caller
/// overrides for other services), `typ = "Bearer"`, no scope.
#[derive(Debug, Clone)]
pub struct ClaimsBuilder {
    sub: String,
    iss: String,
    aud: Option<Vec<String>>,
    azp: Option<String>,
    typ: Option<String>,
    scope: Option<String>,
    exp: i64,
    nbf: Option<i64>,
    iat: i64,
    realm_roles: Vec<String>,
    extra: serde_json::Map<String, serde_json::Value>,
    alg: jsonwebtoken::Algorithm,
    kid: String,
}

impl ClaimsBuilder {
    /// Construct a baseline that, when minted, validates against a
    /// JwksProvider configured for `service_aud`. All single-axis
    /// mutators below derive from this.
    pub fn baseline(service_aud: &str) -> Self;

    /// Override the audience claim. Pass `&["craig-cases"]` to scope
    /// to a single service; pass `&[]` then `.without_aud()` to drop
    /// the claim entirely.
    pub fn with_aud(mut self, aud: &[&str]) -> Self;

    /// Drop the `aud` claim. Used to assert the validator rejects
    /// audience-less tokens (ADR-021).
    pub fn without_aud(mut self) -> Self;

    /// Override the `typ` header/claim. ADR-021 requires `"Bearer"`;
    /// passing `"Refresh"` or `"ID"` must be rejected.
    pub fn with_typ(mut self, typ: &str) -> Self;

    /// Set the scope claim. RFC 6749 space-separated string.
    pub fn with_scope(mut self, scope: &str) -> Self;

    /// Set the authorized-party claim. ADR-021 surface; for `craig-ui`
    /// vs `craig-api` distinction.
    pub fn with_azp(mut self, azp: &str) -> Self;

    /// Construct an expired token. `expired(Duration::from_secs(60))`
    /// sets `exp = now - 60s`.
    pub fn expired(mut self, ago: std::time::Duration) -> Self;

    /// Construct a not-yet-valid token via `nbf`.
    pub fn not_yet_valid(mut self, ahead: std::time::Duration) -> Self;

    /// Override the signing algorithm. Pass `Algorithm::HS256` to
    /// assert RS256-only validators reject. Note: `jsonwebtoken` 10.3
    /// does NOT expose `Algorithm::None`. The "alg=none" attack is
    /// covered by `with_unsigned()` below, which manually constructs
    /// the compact JWT with header `{"alg":"none"}` and an empty
    /// signature segment per RFC 7515 §6.1.
    pub fn with_alg(mut self, alg: jsonwebtoken::Algorithm) -> Self;

    /// Manually construct an unsigned JWT (`{"alg":"none"}` header +
    /// claims + empty signature segment). Cannot use `jsonwebtoken::encode`
    /// because the crate intentionally drops `Algorithm::None` support.
    /// Mint path:
    ///   header_b64  = base64url("{\"alg\":\"none\",\"typ\":\"JWT\"}")
    ///   payload_b64 = base64url(serde_json::to_vec(&self.claims))
    ///   token       = format!("{header_b64}.{payload_b64}.")  // empty signature
    /// Validators that fail to enforce an algorithm allowlist will
    /// accept this token; the test asserts they reject.
    pub fn with_unsigned(mut self) -> Self;

    /// Sign with a kid that does NOT exist in the JWKS — exercises
    /// kid-mismatch rejection.
    pub fn with_unknown_kid(mut self) -> Self;

    /// Inject an additional top-level claim (negative-test for unknown
    /// claim acceptance, or positive-test for service-specific claims).
    pub fn with_extra_claim(mut self, key: &str, value: serde_json::Value) -> Self;

    /// Mint the token. Fails only on RSA signing errors (i.e. never
    /// in supported configs).
    pub fn mint(&self) -> String;
}

D5.4. Mutation matrix

/// Identifies the single axis a mutation perturbs. Used in test
/// failure messages so a 1-of-N parametric test pinpoints which
/// strictness rule the validator broke.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum MutationKind {
    AudMissing,
    AudWrongRealm,
    AudWrongService,
    TypMissing,
    TypNotBearer,
    AzpWrongClient,
    ScopeMissing,        // only emitted when baseline declares a required scope
    ScopeInsufficient,
    Expired1s,
    Expired1h,
    NotYetValid1m,
    AlgNone,             // alg=none attack
    AlgConfusionHs256,   // HS256 against an RS256-pinned validator
    UnknownKid,
    SubMissing,
    IssWrongRealm,
}

pub struct MutatedToken {
    pub kind: MutationKind,
    pub jwt: String,
    /// The expected `craig_common::error::ApiError` variant. Matched
    /// stringly by the harness because the variants are large and
    /// `PartialEq` is not derived.
    pub expected_api_error: &'static str,
}

/// Yield one [`MutatedToken`] per single-axis mutation against the
/// given baseline. The base must already mint a valid token against
/// the service under test.
pub fn mutation_matrix(base: ClaimsBuilder) -> impl Iterator<Item = MutatedToken>;

mutation_matrix returns ~16 items today. Each is constructed by cloning the baseline, applying one mutator, and minting. The expected ApiError is hard-coded per kind and aligned to ADR-021 (e.g. AudMissing/AudWrongService"Unauthorized: invalid audience"; AlgNone"Unauthorized: alg not allowed"). When the validator is later widened to accept a previously-rejected variant, that test must be deleted and an ADR updated — making the strictness regression surface in code review.

D5.5. Cross-cuts with craig-auth

The mutator covers axes ADR-021 already names. To stay in sync:

  • New strictness fields in ADR-021 → new MutationKind variant. Reviewer must check.

  • The expected_api_error string is the public error message; if craig_common::error::ApiError is restructured (e.g. variant renamed), this file is one of the call-out review locations. Loose coupling is acceptable here because the test failure message is what the developer reads, not a programmatic match.

  • JwksProvider::with_audience is the validator hook. If a future ADR-021 amendment adds required azp (currently optional per ADR), JwksFixture gains an expect_azp builder and the matrix grows accordingly.

D5.6. Test description — parametric rejection across 8 services

New file crates/craig-test-lib/tests/jwt_mutation_smoke.rs:

use craig_test_lib::jwt_mutation::{ClaimsBuilder, mutation_matrix};

#[test]
fn baseline_token_validates() {
    // Sanity check: the base case mints a token the validator accepts.
    let token = ClaimsBuilder::baseline("craig-cases").mint();
    let provider = test_provider_for_audience("craig-cases").await;
    assert!(provider.validate_token(&token).await.is_ok());
}

#[test]
fn every_single_axis_mutation_is_rejected_by_craig_cases_validator() {
    let base = ClaimsBuilder::baseline("craig-cases");
    let provider = test_provider_for_audience("craig-cases");
    let mut failures = Vec::new();
    for m in mutation_matrix(base) {
        match provider.validate_token(&m.jwt).await {
            Ok(_) => failures.push(format!("{:?} unexpectedly accepted", m.kind)),
            Err(e) if e.to_string().contains(m.expected_api_error) => {} // pass
            Err(e) => failures.push(
                format!("{:?} rejected with wrong error: got {e}, want {}",
                    m.kind, m.expected_api_error)),
        }
    }
    assert!(failures.is_empty(), "JWT mutation regressions: {failures:#?}");
}

A per-service variant of the second test lives under each service’s tests/api/auth_strictness.rs (8 files), each iterating mutation_matrix(ClaimsBuilder::baseline("<service-aud>")) against a router built with that service’s auth_middleware. The iteration replaces every present hand-rolled "invalid token" check.

The eight per-service test files are owned by Phase B follow-ups; this step lands the helper + the smoke test only.

D6. Magic-byte upload-spoofing fixtures (Phase A.6 — Step 7)

Pairs with platform-stab-2 §D8.1, where validate_upload grows a content-sniffing arm via the infer crate. That work implements magic-byte rejection. This step ships the test corpus — a catalogued set of byte sequences that exercise every spoofing class plus a parallel set of valid happy-path companions. Without the corpus, every uploader test re-invents fixtures inline and the cases drift between sites.

D6.1. Fixture directory

New tree: crates/craig-test-lib/fixtures/upload_spoofing/. Every file is checked into git (each <2KB). LICENSE-SPDX header appended as a sibling README.md (binaries themselves can’t carry SPDX in-band). The directory is loaded via include_bytes! at compile time so tests don’t depend on cwd.

File Size Description

pe_disguised_as_pdf.bin

64B

MZ\x90\x00\x03\x00\x00\x00… Windows PE header followed by NUL padding. Declared MIME at upload time: application/pdf. Expected: rejected with MimeMagicMismatch { declared: "application/pdf", detected: "application/x-msdownload" }.

elf_disguised_as_jpeg.bin

64B

\x7fELF\x02\x01\x01\x00… Linux ELF header + arch padding. Declared: image/jpeg. Expected: rejected; detected application/x-executable.

zip_bomb_disguised_as_csv.bin

1.2KB

Pre-baked tiny ZIP archive whose decompressed size is ~10MiB (10 nested deflate streams of zeros). Declared: text/csv. Expected: rejected on magic-byte check before the decompressor ever sees it (detected: "application/zip"); §D8.1 in platform-stab-2 covers this.

script_disguised_as_png.bin

32B

#!/bin/sh\necho pwned\n shell script. Declared: image/png. Expected: rejected; detected text/x-shellscript (or text/plain on infer fallback — both must reject because neither is in DEFAULT_ALLOWED_MIME_TYPES).

truncated_pdf_first_8_bytes.bin

8B

Exactly %PDF-1.4 then EOF. Declared: application/pdf. Expected: rejected as truncated; the magic-byte check passes but the structural-validity arm of platform-stab-2 §D8.1 fails.

xml_with_external_entity.bin

384B

XXE bomb (<!DOCTYPE foo [<!ENTITY xxe SYSTEM "file:///etc/passwd">]> etc.). Declared: application/xml. Expected: rejected because application/xml is not in DEFAULT_ALLOWED_MIME_TYPES today; this is a pin against accidentally adding XML to the allowlist without an XXE-disabling parser config.

valid_pdf.bin

256B

Real %PDF-1.4 header + minimal %%EOF trailer.

valid_jpeg.bin

134B

Minimal valid JPEG (\xff\xd8\xff\xe0\x00\x10JFIF…).

valid_png.bin

67B

Minimal valid PNG (\x89PNG\r\n\x1a\n + IHDR + IEND).

valid_csv.bin

64B

name,value\na,1\nb,2\n plain ASCII.

The valid-companion fixtures exist so happy-path tests don’t have to re-invent fixtures for every adopter; they are the negative-control half of every spoof test pair.

D6.2. Fixture loader

New file crates/craig-test-lib/src/upload_fixtures.rs. Re-export from lib.rs as pub mod upload_fixtures.

/// Stable identifier for a fixture file. Variants map 1:1 to D6.1 entries.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum FixtureName {
    PeDisguisedAsPdf,
    ElfDisguisedAsJpeg,
    ZipBombDisguisedAsCsv,
    ScriptDisguisedAsPng,
    TruncatedPdfFirst8Bytes,
    XmlWithExternalEntity,
    ValidPdf,
    ValidJpeg,
    ValidPng,
    ValidCsv,
}

/// Static fixture bytes. Compiled into the test binary via `include_bytes!`.
pub fn fixture(name: FixtureName) -> &'static [u8];

/// One spoof case: the bytes, the lying declared MIME, and the
/// `StoreError` variant the validator must surface.
pub struct SpoofFixture {
    pub name: FixtureName,
    pub declared_mime: &'static str,
    pub body: &'static [u8],
    /// Stringly matched against `StoreError::to_string()` because
    /// `StoreError` does not implement `PartialEq`.
    pub expected_rejection: &'static str,
}

/// Iterate every spoofing case (excludes the valid-companion fixtures).
/// 6 cases today; grows as new spoof classes are added.
pub fn enumerate_spoofs() -> impl Iterator<Item = SpoofFixture>;

/// Iterate every valid-companion fixture for happy-path tests.
pub fn enumerate_valid() -> impl Iterator<Item = SpoofFixture>; // body + true MIME

D6.3. Filename-attack fixtures

A second axis. Today crates/craig-store/src/validation.rs::sanitize_filename strips path separators, NUL, and trims leading/trailing whitespace + dots. Coverage at crates/craig-store/src/validation.rs:124-163 is good for those cases; missing: CRLF, RTLO, UNC, fullwidth-Unicode confusables, surrogate-pair edge cases, RFC-5987 round-trip through Content-Disposition (which platform-stab-2 §D8.1 also lands).

New crates/craig-test-lib/src/filename_attack.rs:

/// One filename-attack case. `expected_sanitized` is the exact bytes
/// `sanitize_filename` must return; `expected_disposition` is the exact
/// header value `RFC 5987` Content-Disposition encoding must produce
/// (None when the case is rejected outright).
pub struct FilenameCase {
    pub name: &'static str,
    pub raw: &'static str,
    pub expected_sanitized: Result<&'static str, &'static str>,
    pub expected_disposition: Option<&'static str>,
}

pub const CASES: &[FilenameCase] = &[
    FilenameCase {
        name: "double_quote",
        raw: r#"my"file.pdf"#,
        // Quote is preserved by sanitize_filename today. The download
        // path Content-Disposition encoder must escape it.
        expected_sanitized: Ok(r#"my"file.pdf"#),
        expected_disposition: Some(
            r#"attachment; filename="my\"file.pdf"; filename*=UTF-8''my%22file.pdf"#),
    },
    FilenameCase {
        name: "crlf_injection",
        raw: "ok.pdf\r\nSet-Cookie: x=y",
        // CRLF is not stripped today; the encoder is responsible for
        // % -encoding it. If sanitize_filename later starts rejecting
        // CRLF, flip expected_sanitized to Err and drop disposition.
        expected_sanitized: Ok("ok.pdfSet-Cookie: x=y"),
        expected_disposition: Some(
            r#"attachment; filename="ok.pdfSet-Cookie: x=y"; filename*=UTF-8''ok.pdf%0D%0ASet-Cookie%3A%20x%3Dy"#),
    },
    FilenameCase {
        name: "nul_byte",
        raw: "ok\0.pdf",
        expected_sanitized: Ok("ok.pdf"),
        expected_disposition: Some(r#"attachment; filename="ok.pdf"; filename*=UTF-8''ok.pdf"#),
    },
    FilenameCase {
        name: "backslash",
        raw: r"a\b\c.pdf",
        expected_sanitized: Ok("abc.pdf"),
        expected_disposition: Some(r#"attachment; filename="abc.pdf"; filename*=UTF-8''abc.pdf"#),
    },
    FilenameCase {
        name: "forward_slash",
        raw: "a/b/c.pdf",
        expected_sanitized: Ok("abc.pdf"),
        expected_disposition: Some(r#"attachment; filename="abc.pdf"; filename*=UTF-8''abc.pdf"#),
    },
    FilenameCase {
        name: "leading_trailing_whitespace",
        raw: "   spaced.pdf   ",
        expected_sanitized: Ok("spaced.pdf"),
        expected_disposition: Some(r#"attachment; filename="spaced.pdf"; filename*=UTF-8''spaced.pdf"#),
    },
    FilenameCase {
        name: "length_256",
        raw: "<256 'a' chars>.pdf",
        // 255-byte char-boundary truncation — see validation.rs
        // sanitize_truncates_at_char_boundary_for_multibyte_input.
        expected_sanitized: Ok("<255 'a' chars>"),
        expected_disposition: Some(r#"attachment; filename="<255 'a' chars>"; filename*=UTF-8''<...>"#),
    },
    FilenameCase {
        name: "unicode_resume",
        raw: "résumé.pdf",
        expected_sanitized: Ok("résumé.pdf"),
        expected_disposition: Some(r#"attachment; filename="r__sum__.pdf"; filename*=UTF-8''r%C3%A9sum%C3%A9.pdf"#),
    },
    FilenameCase {
        name: "rtlo_override",
        raw: "fdp.\u{202E}exe", // U+202E RIGHT-TO-LEFT OVERRIDE — renders "exe.pdf" in many UIs
        // RTLO must be stripped: it has no semantic in a filename and
        // the rendering surface gets to lie to the user.
        expected_sanitized: Ok("fdp.exe"),
        expected_disposition: Some(r#"attachment; filename="fdp.exe"; filename*=UTF-8''fdp.exe"#),
    },
    FilenameCase {
        name: "unc_path",
        raw: r"\\evil.example.com\share\f.pdf",
        expected_sanitized: Ok("evil.example.comsharef.pdf"),
        expected_disposition: Some(
            r#"attachment; filename="evil.example.comsharef.pdf"; filename*=UTF-8''evil.example.com%5Cshare%5Cf.pdf"#),
    },
    FilenameCase {
        name: "all_dots",
        raw: "...",
        expected_sanitized: Err("InvalidFilename"),
        expected_disposition: None,
    },
    FilenameCase {
        name: "fullwidth_dot",
        raw: "f\u{FF0E}exe", // U+FF0E FULLWIDTH FULL STOP — visually `.` but not the path-separator char
        expected_sanitized: Ok("f\u{FF0E}exe"),
        expected_disposition: Some(r#"attachment; filename="f.exe"; filename*=UTF-8''f%EF%BC%8Eexe"#),
    },
];

The expected outputs are frozen at the table — every change to sanitize_filename or the disposition encoder must also touch CASES (and therefore the MR diff), making sanitization-rule changes auditable.

D6.4. Test-coverage description

crates/craig-test-lib/tests/upload_fixtures_smoke.rs:

  1. every_spoof_is_rejected_by_validate_upload: parametric loop over enumerate_spoofs, each calls validate_upload(case.declared_mime, case.body.len(), &v) (where v includes the §D8.1 magic-byte check) and asserts the resulting StoreError to_string() contains case.expected_rejection.

  2. every_valid_companion_passes_validate_upload: parametric loop over enumerate_valid, each must return Ok(()).

  3. filename_attack_round_trip: for each FilenameCase, sanitize_filename(case.raw) is matched against case.expected_sanitized; if Ok, the result is fed through the (platform-stab-2 §D8.1) disposition encoder and matched against case.expected_disposition.

  4. Failing-test-first protocol (mirrors §D4.4): land the helper, write the test with truncated_pdf_first_8_bytes.bin deliberately omitted from enumerate_spoofs, observe the spoof rejection coverage drop, restore.

These tests run without devstack — pure in-process validation — and gate at the workspace cargo nextest run --workspace --lib level. Phase B uploader-handler integration tests (4 sites: contact_attachments, court_orders, ICPC, report_attachments) consume these fixtures via craig_test_lib::upload_fixtures::enumerate_spoofs and POST each through their respective endpoint, asserting 415 / 422 per platform-stab-2 §D8.1.

D7. testing.md "Failure-path testing" + Test Categories taxonomy + Process rule (Phase A.7 — Step 8)

This is the meta-deliverable. It is the only step in this plan that is documentation-only — no Rust, no migration. It encodes the test-categorization taxonomy that today is implicit in the parent epic plan, formalizes per-category file-naming conventions so a contextless agent can place a new test correctly, links every helper landed in §D4–§D6 (and the Phase 2 / Step 2 helpers from platform-stab-2) into a usage cheat-sheet, and writes down the Process Rule that closes the loop on the two-pass platform-invariant audit.

D7.1. New "Test Categories" section in .claude/docs/testing.md

Inserted after "Critical Rules" / "Mandatory Rules" and before "Migration Naming Convention". Eight categories. The descriptions below are the exact text to land verbatim:

Category Description

Functional

Single-fire request → single response → assert status + body. The status quo. Catches "does this endpoint exist and return the right shape." Example: services/craig-cases/tests/api/cases.rs::create_case_returns_201. Cheap, fast, and the dominant test class today.

Invariant

Asserts a property that must hold across many states or across many calls. Distinct from functional in that the property is a universal claim, not a single observation. Example: "every successful POST /v1/cases/persons produces exactly one person.created outbox event with the same UUID as the response body’s id." A failing invariant test usually points at a missing transactional bracket, not a typo.

Concurrency

Spawns N parallel calls against the same endpoint or store function and asserts a no-double / no-drop / consistent-state property. Uses craig_test_lib::concurrent::concurrent_fire. Example: 5 simultaneous POSTs with the same Idempotency-Key produce one side-effect (platform-stab-2 §D1).

Fault-injection

Wraps a dependency (DB, RMQ, cipher, object store, stage_event) in a wrapper that errors on demand, then asserts the error path is reached and behaves correctly. Uses craig_test_lib::fault_injection::*. Example: RabbitDownInjector::inject_unavailable_for(5s) while the outbox worker drains, asserting publish-failure rows accumulate attempts > 0 instead of marking published_at.

Recovery

Like fault-injection but the test continues past the fault and asserts the system recovers — orphans get reaped, in-flight jobs get reclaimed, retries succeed. Example: "kill craig-cases between Phase 1 commit and Phase 2 put in attachment upload; restart; assert scanner reaps orphan within 5 min" (platform-stab-2 §D4).

Contract

End-to-end typed-DTO round-trip through a service’s HTTP boundary. The test client uses the same DTO type the service handler imports. A breaking schema change to either side fails compilation rather than passing through a serde_json::Value and only failing in production. See §D8 for the migration of the 8 typed clients.

Property-based

Generators (proptest) produce inputs and the test asserts a property holds for all of them. Useful for matchers, encoders, sanitizers, RFC parsers. Example: `proptest!(

(s: String) { assert!(sanitize_filename(&s).map(

n

!n.contains('\0')).unwrap_or(true)) })`. Available but not required; introduce on the rare functions whose input space is too large to enumerate.

Mutation

The taxonomy is descriptive of how tests are written across the codebase, not prescriptive of a category quota per service. Most tests are Functional today; the goal of epic &22 is to make Concurrency / Fault-injection / Recovery the second-most-common classes, with Invariant tests added as platform invariants get encoded.

D7.2. New "File-naming convention for test categories" section

Category Path convention

Functional

tests/api/<endpoint>.rs (status quo) or tests/<feature>.rs for cross-endpoint. Indexed via the tests/api.rs module file per current pattern.

Invariant

tests/invariants/<invariant>.rs. New top-level dir; module index file tests/invariants.rs declares submodules.

Concurrency

tests/concurrency/<scenario>.rs. Each file names the property: tests/concurrency/idempotency_no_double_handler.rs, tests/concurrency/outbox_no_double_publish.rs.

Fault-injection

tests/fault/<failure_mode>.rs. Filename names the upstream fault: rmq_publish_unavailable.rs, db_pool_timeout.rs.

Recovery

tests/recovery/<scenario>.rs. Filename names what is recovered: attachment_orphan_reaped.rs, exchange_send_in_flight_reclaimed.rs.

Contract

typed-client tests live in crates/craig-test-lib/tests/typed/<svc>_client.rs (per §D8.6).

Property-based

tests/properties/<property>.rs. Filename names the property under test: sanitize_filename_no_path_separators.rs.

Mutation

No per-test file. Configured at the workspace level via .cargo/mutants.toml; runs via cargo xtask mutants.

Existing tests are not mass-relocated. The convention applies to net-new test files per Phase B and per future MRs.

D7.3. New "Failure-path testing helpers" section

A usage cheat-sheet pointing at every helper landed by this plan + by platform-stab-2 Step 2:

Helper Imported as Use-case

concurrent_fire

craig_test_lib::concurrent::concurrent_fire

Concurrency tests — spawn N parallel tasks, await all.

CipherErrorInjector

craig_test_lib::fault_injection::CipherErrorInjector

Fault-injection on FieldEncryptor.

Codepath-specific store traits + faulty impls (e.g. FaultyFinalizeIdempotencyStore, FaultyAttachmentStore, FaultyOutboxStore, FaultyInboxStore, FaultySendJobStore)

craig_test_lib::fault::stores::*

DB fault injection — at the codepath under test (§D2.0 sub-pattern b). For systemic DB failures use container-level (sub-pattern a) instead. There is no DbErrorInjector wrapping PgPool directly — that pattern was rejected because &PgPool flowing into sqlx::query() bypasses any wrapper.

RabbitDownInjector

craig_test_lib::fault_injection::RabbitDownInjector

Fault-injection on Publisher.

ObjectStoreErrorInjector

craig_test_lib::fault_injection::ObjectStoreErrorInjector

Fault-injection on Store::put.

PublishInTxFailureInjector

craig_test_lib::fault_injection::PublishInTxFailureInjector

Fault-injection on craig_mq::stage_event.

assert_only_valid_transitions_accepted

craig_test_lib::state_machine_matrix::*

State-machine matrix testing (§D4).

ClaimsBuilder + mutation_matrix

craig_test_lib::jwt_mutation::*

JWT strictness rejection tests (§D5).

enumerate_spoofs + fixture

craig_test_lib::upload_fixtures::*

Upload-spoofing rejection tests (§D6).

filename_attack::CASES

craig_test_lib::filename_attack::CASES

Filename-sanitization round-trip tests (§D6.3).

multi-replica devstack

cargo xtask dev start --replicas N

Phase A.3 (sibling step in this plan, parent epic Phase A.3).

proptest

dev-dependency on proptest = "1"

Property-based tests (introduce per-test-file).

cargo mutants --smoke

cargo xtask mutants

Mutation testing (periodic, not per-push).

Each row carries a one-line example invocation in the actual testing.md so it is grep-able.

D7.4. New "Process rule" section

Verbatim text to land:

Process rule (epic &21 / &22 onward): every concurrency-relevant fix, fault-tolerance fix, or platform-invariant fix MUST ship with at least one Concurrency, Fault-injection, Invariant, or Recovery test — in any combination — in the same MR as the production-code change. The choice of category(ies) is the author’s; the existence of some failure-path test is non-negotiable. Failing-test-first (commit failing test, then commit fix in the same MR) is preferred — it makes the regression demonstrable from the diff alone — but not mandated; reviewer-requested addition is acceptable.

Reviewer responsibility: when reviewing an MR labeled concurrency, platform-invariant, durability, or any sub-label of those, the reviewer must explicitly confirm in CR that a test of one of the four categories exists. The MR is blocked until either (a) the test is added or (b) the author argues — and the reviewer accepts — that the change is structurally incapable of regressing the invariant (e.g. a comment-only edit). Discipline-only enforcement; no pre-commit hook (default proposal in the parent epic Open Questions item 4).

Linkage: this rule is the code-and-test embodiment of the two-pass platform-invariant audit's storage-shape-vs-semantics lesson. An audit that verifies "table exists" without the matching Concurrency / Recovery test does not constitute platform-invariant verification.

D7.5. Cross-references to update

  • .claude/docs/delivery-protocol.md — add a one-paragraph pointer in the "Plan completion audit" section: "When the audit is verifying a platform invariant, the auditor must follow the two-pass platform-invariant audit — name the failure mode in plain English, cite (or write) the failing test that catches it, then verify the fix. Storage-shape verification alone is insufficient."

  • CONTRIBUTING.adoc — add a new section "Failure-path testing" with a one-paragraph pointer to the testing.md sections D7.1–D7.4 above. Keep CONTRIBUTING.adoc the public-facing entry; testing.md is the agent-facing detail.

  • .claude/CLAUDE.md "Required Reading" table — testing.md row already exists; no change. (No new MUST-READ; this is documentation expansion within an existing required doc.)

D7.6. Quality budgets — available, not enforced

A separate sub-section of testing.md, after Process rule. Eight numeric budgets, each implemented as a script in xtask/src/budget.rs (Phase B follow-up — not this plan’s step):

Budget Threshold (proposed) Command

Max route module length

500 LOC (post platform-stab-2 §D10.1 decomposition)

cargo xtask budget --metric route-loc

Max function length

80 LOC

cargo xtask budget --metric fn-loc

Max serde_json::Value count outside boundary modules

declining baseline (start: current count)

cargo xtask budget --metric value-count

Max #[allow(…​)] count

declining baseline

cargo xtask budget --metric allow-count

Max unwrap_or_default in non-test prod code

declining baseline (post platform-stab-2 §D7)

cargo xtask budget --metric unwrap-or-default

New dependency duplicates

0 (post platform-stab-2 §D10.2 cleanup)

cargo tree -d

New untyped test clients

0 among trusted-write clients post-§D8; IntakeClient stays &Value by design (untrusted-ingress simulator — see §Errata, #307)

cargo xtask budget --metric untyped-clients

Mutation-test surviving-mutant ratio on critical modules

0 in craig-mq, craig-api/idempotency.rs, craig-store/validation.rs

cargo xtask mutants --smoke

Threshold + command shape are reserved here; no enforcement in this plan. The pre-push CI gates listed by the user (cargo xtask reliability, contracts, mutants --smoke) are deliberately deferred — the plan documents the call-shape so the future enforcement MR has a stable target name, but the gate itself is not part of any step in this plan or epic &22 Phase A.

D8. Typed DTOs across 8 test clients (Phase A.8 — Step 9)

Every typed test client today (crates/craig-test-lib/src/clients/*.rs) uses serde_json::Value for both request bodies and response bodies. 211 occurrences across 8 files. The cost: a hand-rolled JSON shape mismatch in a test does not fail at compile time; it fails at the assertion site with a null field, days after the actual schema regression in the service-side handler. A typed client would have caught every "this field renamed" / "field type changed" / "required field made optional" regression in a single recompile.

D8.1. Decision: full per-client rewrite, not gradual

Option Outcome

Gradual within each client (mix Value and DTOs)

Rejected. A half-typed client carries the worst of both: review still has to read the JSON shape for the un-typed methods, and the type system has to be re-asserted at every typed/untyped boundary. The migration would stall.

Full per-client rewrite, 8 small MRs

Selected. Each MR is independently reviewable, the partial state across the codebase is "this client typed, that one not yet" (clean) instead of "this method typed, the next not" (unclean), and the order can be rebalanced as bandwidth allows. Default per the parent epic Open Questions item 3.

Single big-bang rewrite

Rejected. Diff size unreviewable.

D8.2. DTO ownership: per-service *-contracts crates, not central re-export

A previous draft had test-client typed DTOs use-ing the service crate directly: craig-test-lib would [dev-dependencies] craig-cases = { workspace = true } and use craig_cases::api::dto::*. This creates a Cargo dependency cycle: services already dev-depend on craig-test-lib (e.g. services/craig-cases/Cargo.toml:34). Adding a back-edge from craig-test-libcraig-cases produces:

craig-cases → [dev-dep] → craig-test-lib → [dep] → craig-cases   # cycle

Cargo’s resolver routes around this for the test target (dev-deps don’t propagate via the normal graph), but the cycle is brittle: any non-dev import path breaks cargo build workspace-wide. Production-safety lives in not creating the cycle in the first place.

Option Trade-off

Service crate exposes api::dto as pub; test-lib dev-depends on the service crate

Rejected. Creates the cycle described above. Worked while the prior draft assumed services don’t dev-depend on test-lib; that assumption is false today.

New craig-contracts/ workspace crate aggregating all DTOs

One canonical home, easy use craig_contracts::cases::CreateCaseRequest. But: every service-side handler change to its DTO drives a ripple through the contracts crate first, then the service, then the test client — three-MR sequence per DTO change. Bottleneck.

Per-service -contracts crates — a new tiny craig-cases-contracts/ for each service that owns *only the public DTO types. The service crate depends on its contracts crate (re-exports the same types from api::dto::* to avoid breaking existing service-internal callers). The test-lib dev-depends on each service’s -contracts crate.

Selected. No cycle (contracts crates are leaf nodes; nothing depends back on test-lib). DTO change is a 1-MR diff that the contracts crate compiles, the service compiles, the test client compiles — three crates, one MR, mechanical change. Adds 8 new crates to the workspace; each is ~50-200 LOC of pure data types. Mitigates the cycle without paying the central-bottleneck cost of a single craig-contracts crate.

Move typed clients into each service’s own test tree (services/craig-cases/tests/clients/cases.rs)

Eliminates the test-lib boundary entirely — each service owns both its DTOs and its typed test client. But: breaks the "test-lib has the cross-service test infrastructure" pattern that’s already established for the 8 untyped clients. Cross-service tests (e.g. integration tests in craig-test-lib/tests/) lose the typed clients. Acceptable for some narrow cases but doesn’t scale to cross-service tests.

Per-service -contracts crate shape*:

# crates/craig-cases-contracts/Cargo.toml (new)
[package]
name = "craig-cases-contracts"
version = "0.1.0"
edition = "2024"

[dependencies]
serde = { workspace = true, features = ["derive"] }
serde_json = { workspace = true }     # only if any DTO field needs Value
chrono = { workspace = true }
uuid = { workspace = true, features = ["serde"] }
utoipa = { workspace = true, features = ["chrono", "uuid"] }
craig-reference = { workspace = true } # if DTOs use shared enums
# NO dependency on craig-cases or craig-test-lib
// crates/craig-cases-contracts/src/lib.rs (new)
pub mod cases;
pub mod persons;
pub mod referrals;
// ... one module per api submodule with public DTOs
// services/craig-cases/Cargo.toml — add a non-dev dep
[dependencies]
craig-cases-contracts = { workspace = true }

// services/craig-cases/src/api/dto.rs (existing)
pub use craig_cases_contracts::cases::*;
pub use craig_cases_contracts::persons::*;
// internal re-export so existing handler code at `crate::api::dto::*` keeps working
// crates/craig-test-lib/Cargo.toml — add per-service contracts dev-deps only
[dev-dependencies]
craig-cases-contracts = { workspace = true }
craig-placement-contracts = { workspace = true }
craig-exchange-contracts = { workspace = true }
craig-financial-contracts = { workspace = true }
craig-reporting-contracts = { workspace = true }
craig-rules-contracts = { workspace = true }
craig-security-contracts = { workspace = true }
craig-intake-contracts = { workspace = true }

// crates/craig-test-lib/src/clients/cases.rs (rewritten in Step 9)
use craig_cases_contracts::cases::{CreateCaseRequest, CaseResponse};
use craig_cases_contracts::persons::{CreatePersonRequest, PersonResponse, ListPersonsResponse};
// no `use craig_cases::*;` — no cycle.

Migration ordering: each per-service -contracts crate is filed as the *first commit of its respective Phase A.8 sub-MR. Step 9 of THIS plan files the CasesClient pilot, which means: Step 9 lands craig-cases-contracts/ first, then migrates the existing DTOs into it (one move-per-file), then re-exports from the service, then rewrites the test client to use the contracts crate. 7 follow-up MRs each repeat the pattern for one service.

Cycle-detection guard: add a cargo deny bans.deny rule name = "craig-cases", deny-from = ["craig-test-lib"] (and similarly for the other 7 services). Any future PR that introduces the back-edge fails CI. Lightweight; pure prevention.

D8.3. Inventory — current state of the 8 clients

# Client (file) Method count Body type today

1

crates/craig-test-lib/src/clients/cases.rs

~60 methods

&serde_json::Value arg, ApiResponse<serde_json::Value> return

2

crates/craig-test-lib/src/clients/placement.rs

~32

same

3

crates/craig-test-lib/src/clients/exchange.rs

~22

same

4

crates/craig-test-lib/src/clients/financial.rs

~17

same

5

crates/craig-test-lib/src/clients/reporting.rs

~15

same

6

crates/craig-test-lib/src/clients/rules.rs

~7

same

7

crates/craig-test-lib/src/clients/security.rs

~46

same

8

crates/craig-test-lib/src/clients/intake.rs

~12

same (mix of authenticated + unauthenticated; both use Value)

Total: ~211 methods, 1:1 with the grep count. Migration order is not alphabetical:

Order Client Why this position

1

CasesClient

Highest churn (60 methods); piloting here surfaces every wrinkle (path-param escaping, multipart upload, query param threading) before the smaller clients.

2

PlacementClient

Second-largest, similar concurrency-test load coming in Phase B.

3

ExchangeClient

Heavy in epic &21 Step 7 send-worker — typed responses pay off there.

4

FinancialClient

First full state-machine matrix consumer (§D4); typed PaymentStatus round-trip is leverage.

5

ReportingClient

Lower endpoint count, lower test churn — schedule before security to keep momentum.

6

RulesClient

Tiny client; lands fast, provides confidence-builder mid-migration.

7

SecurityClient

46 methods; large diff but isolated from the epic &21 P0/P1 timeline.

8

IntakeClient

The mixed authenticated/unauthenticated split means this one needs a builder method per auth posture; saves the trickiest for last when the pattern is firm.

Each migration is a single MR with branch prefix feat/test-framework-typed-clients-<svc>.

D8.4. Worked example: CasesClient::list_persons

Today (crates/craig-test-lib/src/clients/cases.rs:41):

pub async fn search_persons(&self, params: &[(&str, &str)]) -> Result<ApiResponse<Value>> {
    self.inner
        .get_with_query("/v1/cases/persons", &params.to_vec())
        .await
}

Tomorrow:

use craig_cases::{ListPersonsResponse, ListPersonsQuery};

pub async fn search_persons(&self, query: &ListPersonsQuery) -> Result<ApiResponse<ListPersonsResponse>> {
    self.inner
        .get_with_query("/v1/cases/persons", &query.to_query_pairs())
        .await
}

Where ListPersonsQuery is a new typed struct in services/craig-cases/src/api/persons.rs (deserialize from URL query via serde_urlencoded already in the deps tree) re-exported from the lib root. ListPersonsQuery::to_query_pairs(&self) → Vec<(&'static str, String)> lives next to the type.

ApiResponse<T> (in crates/craig-test-lib/src/client.rs) is already generic; this migration is a body-type swap, not a client-shape change.

For create_person, the diff:

// today
pub async fn create_person(&self, body: &Value) -> Result<ApiResponse<Value>> {
    self.inner.post("/v1/cases/persons", body).await
}

// tomorrow
use craig_cases::{CreatePersonRequest, PersonResponse};

pub async fn create_person(&self, body: &CreatePersonRequest) -> Result<ApiResponse<PersonResponse>> {
    self.inner.post("/v1/cases/persons", body).await
}

ServiceClient::post<T: Serialize, R: DeserializeOwned> already supports generic body and response — the wrapper at crates/craig-test-lib/src/client.rs is generic at the inner layer, only the per-client typed wrappers were Value-specialized.

Test impact: every existing test calling cases_client.create_person(&json!({…})) now writes cases_client.create_person(&CreatePersonRequest { … }). The json! body is replaced with a struct literal; serde inference does the rest. The diff is mechanical except for two patterns:

  1. Pre-existing JSON malformation as test input. A few tests deliberately POST malformed JSON to assert 422. These tests bypass the typed client and call ServiceClient::post_raw (existing escape hatch — preserved).

  2. Open-shape responses. A handful of endpoints return { …​heterogeneous fields…​ } based on a discriminator; these get a #[serde(tag = "kind")] enum DTO on the service side and the test client benefits identically.

D8.5. Quality-budget tie-in (warnings-only)

Per §D7.6, a cargo xtask budget --metric value-count walks crates/craig-test-lib/src/clients//.rs and counts serde_json::Value. Budget threshold: declining baseline. After all 8 clients migrate, the budget is 0 in clients/. Value remains valid in client.rs (the boundary layer that handles raw response bytes / problem+json) — those occurrences are excluded by path. The check is warnings-only* in this plan; promotion to a hard gate is filed as a future MR after the 8 MRs land.

D8.6. Test-coverage description — per-client smoke

Each per-client migration MR ships one Contract test under crates/craig-test-lib/tests/typed/<svc>_client.rs:

// tests/typed/cases_client.rs (lands with MR 1)

use craig_cases::{CreatePersonRequest, PersonResponse};
use craig_test_lib::{TestHarness, devstack_available};

#[tokio::test]
async fn create_person_typed_round_trip() {
    if !devstack_available().await { return; }
    let harness = TestHarness::new().await;
    let client = harness.caseworker_cases_client();

    let req = CreatePersonRequest {
        first_name: "Round".into(),
        last_name:  "Trip".into(),
        // … every required field. Compile fails if a field is missing
        // or renamed — caught in CI rather than in a flaky `null` deref.
    };
    let resp = client.create_person(&req).await.unwrap();
    assert_eq!(resp.status, reqwest::StatusCode::CREATED);

    let person: &PersonResponse = resp.body.as_ref().unwrap();
    assert_eq!(person.first_name, "Round");
    // No more `body["first_name"].as_str().unwrap()` — direct field access.
}

The test pins one round-trip per client. A subsequent service-side rename to first_given_name fails at craig-cases::CreatePersonRequest’s definition; the test client fails to compile; the test client’s caller (`harness.create_person(&req)) fails to compile; the regression is caught at cargo build rather than at integration runtime.

This is the only Contract test landed by the migration; deeper Contract coverage is per-endpoint and accumulates as Phase B tests adopt the new clients.

D9. proptest scaffolding + first 7 properties (Phase A.9 — Step 10)

The eight Phase-A foundation pieces enumerated in the existing draft are imperative helpers (one helper at a time, hand-coded scenarios). Property-based testing is the declarative counterpart: you describe the input space and the invariant the code must satisfy, and the runner generates+shrinks counter-examples. Adding it now closes the gap between the small-case integration suite and the unbounded input space the production services actually see.

The user steered the existing Step 5 open question ("proptest crate vs hand-rolled state-machine matrix") toward hand-rolled for the bounded state-machine alphabet. That conclusion stands for state machines. Property testing — by contrast — is exactly where proptest’s strength lies: filename sanitization, JSON canonicalization, idempotency-fingerprinting, RFC-encoded headers all live in unbounded input spaces where a hand-curated case list is structurally inadequate.

D9.1 Workspace dependency

# Cargo.toml [workspace.dependencies]
# Property-based testing — used in `tests/properties/<prop>.rs` per testing.md.
proptest = "1.7"   # latest stable as of 2026-05; verify on commit via `cargo search proptest`

Per-crate adoption opts in via [dev-dependencies] only — proptest never reaches a release artefact:

# crates/craig-store/Cargo.toml
[dev-dependencies]
proptest = { workspace = true }

D9.2 Module convention

testing.md (extended in Step 8 of this plan with the "Failure-path testing" section) gains a sibling subsection: "Property tests". The convention:

Layout Rule

File path

<crate-or-service>/tests/properties/<property>.rs

Module index

<crate>/tests/properties.rs declares submodules with #[path = "properties/<file>.rs"] mod <name>;. Mirrors the existing tests/api.rs + tests/api/*.rs pattern from testing.md.

Per-property test name

prop_<invariant> — e.g. prop_sanitize_filename_idempotent, prop_pagination_no_duplicates.

Cases override

Default proptest!{ ProptestConfig::with_cases(256), .. }. Override per property in code, not via env var, so CI and local runs match.

Shrinking budget

Use proptest defaults. If a property repeatedly times out shrinking on CI, raise max_shrink_iters rather than disabling shrinking.

cargo xtask reliability

The Step 12 subcommand globs tests/properties/*.rs via the nextest expression test(::properties::) to opt all properties into the reliability suite.

The Step 8 testing.md edit also documents the fail-first discipline: introduce a deliberate one-character bug in the unit-under-test, run the prop, watch shrinking land on a minimal counter-example, then revert. Land each property with a one-line PR comment quoting the shrunk input that the un-bugged code now passes. This is the property-test analogue of the failing-then-passing rule for concurrency tests.

D9.3 Property 1 — state-machine round-trip (PaymentStatus)

File: services/craig-financial/tests/properties/payment_status.rs

The state-machine matrix helper from Step 5 (Phase A.4) covers reject-set verification by enumeration. The property version covers a different invariant: that an enum’s can_transition_to agrees with the validator function used at the API boundary, across the full cross-product.

use proptest::prelude::*;
// PaymentStatus actually lives at services/craig-financial/src/transitions.rs:11,
// NOT in craig-reference. Five variants only — no Draft, Disbursed, Reversed, Cancelled.
use craig_financial::transitions::{PaymentStatus, validate_payment_transition};

fn any_status() -> impl Strategy<Value = PaymentStatus> {
    prop_oneof![
        Just(PaymentStatus::Pending),
        Just(PaymentStatus::Approved),
        Just(PaymentStatus::Issued),
        Just(PaymentStatus::Cleared),
        Just(PaymentStatus::Voided),
    ]
}

proptest! {
    #![proptest_config(ProptestConfig::with_cases(512))]
    #[test]
    fn prop_payment_can_transition_matches_validator(
        from in any_status(),
        to in any_status(),
    ) {
        let enum_says = from.can_transition_to(to);
        let validator_result = validate_payment_transition(from.as_str(), to.as_str());
        prop_assert_eq!(enum_says, validator_result.is_ok(),
            "enum/validator disagree for {from:?} -> {to:?}");
    }
}

Fail-first sketch: temporarily flip one can_transition_to arm (e.g. allow Cleared → Pending); proptest shrinks to exactly that pair.

Expected runtime: 25 ordered pairs × 512 cases = 12,800 calls; <50ms total.

D9.4 Property 2 — pagination

File: crates/craig-api/tests/properties/pagination.rs

Pagination invariants are heavily relied on across all 8 services and have escaped review in the past (see Phase 10 Web UI work that surfaced the per_page=25 vs per_page=10 interaction). Property protects three invariants simultaneously:

  1. Each page contains at most per_page items.

  2. Items are stably ordered (the sort key is monotonic across pages).

  3. No item appears on two pages.

use proptest::prelude::*;
use craig_api::pagination::{PageRequest, paginate};

fn any_dataset() -> impl Strategy<Value = Vec<u64>> {
    proptest::collection::vec(0u64..1_000_000, 0..500)
        .prop_map(|mut v| { v.sort(); v.dedup(); v })
}

proptest! {
    #[test]
    fn prop_pagination_no_dupes_no_drops(
        data in any_dataset(),
        per_page in 1usize..50,
    ) {
        let total = data.len();
        let pages: Vec<_> = (1..=total.div_ceil(per_page).max(1))
            .map(|p| paginate(&data, PageRequest { page: p, per_page }))
            .collect();

        // each page len <= per_page
        for p in &pages { prop_assert!(p.items.len() <= per_page); }
        // ordering monotonic across pages
        let flat: Vec<_> = pages.iter().flat_map(|p| p.items.iter().copied()).collect();
        for w in flat.windows(2) { prop_assert!(w[0] <= w[1]); }
        // no duplicates across pages
        let mut seen = std::collections::HashSet::new();
        for x in &flat { prop_assert!(seen.insert(*x)); }
        // total reachable equals input
        prop_assert_eq!(flat.len(), total);
    }
}

Expected runtime: 256 cases × ~10 page builds = 2.5k calls; <100ms.

D9.5 Property 3 — filename sanitization

File: crates/craig-store/tests/properties/sanitize_filename.rs

craig_store::validation::sanitize_filename (signature pub fn sanitize_filename(&str) → Result<String, StoreError> at crates/craig-store/src/validation.rs:58) is on the upload hot path and has been touched repeatedly (most recently by platform-stab-2 §D8.1’s RFC 5987 work). Property locks four invariants on the Ok arm (the Err arm fires for inputs that sanitize to empty, which is its own correctness requirement):

  1. Output never contains /, \, NUL, or other ASCII controls.

  2. Output is non-empty (the only return path; Err is the empty case).

  3. Idempotent: sanitize_filename(sanitize_filename(s).unwrap())? == sanitize_filename(s)?.

  4. Output byte length ⇐ 255 (the function’s documented truncation cap).

use proptest::prelude::*;
use craig_store::validation::sanitize_filename;

proptest! {
    #![proptest_config(ProptestConfig::with_cases(1024))]
    #[test]
    fn prop_sanitize_filename_invariants(s in ".{0,256}") {
        let Ok(out) = sanitize_filename(&s) else {
            // Empty-after-sanitization → Err is correct behavior; skip this case.
            return Ok(());
        };
        for ch in out.chars() {
            prop_assert!(!matches!(ch, '/' | '\\' | '\0'),
                "forbidden char {ch:?} survived sanitization of {s:?}");
            prop_assert!(!ch.is_control(),
                "control char {ch:?} survived sanitization");
        }
        prop_assert!(!out.is_empty(), "Ok arm returned empty string");
        let twice = sanitize_filename(&out)
            .expect("Ok output must itself be sanitizable to Ok");
        prop_assert_eq!(twice, out, "non-idempotent: f(f(x)) != f(x)");
        prop_assert!(out.len() <= 255, "output exceeds 255-byte cap: {}", out.len());
    }
}

Fail-first sketch: replace the c != '/' filter arm in the chain at validation.rs:60 with c != ';' so / re-appears in output; proptest shrinks to a single-character input "/".

Expected runtime: 1024 cases × short string ops; <50ms.

D9.6 Property 4 — Content-Disposition encoding round-trip

File: crates/craig-store/tests/properties/content_disposition.rs

Pairs with platform-stabilization-2 §D8.1’s RFC 6266 encoder. Property: the encoded header round-trips through a parser back to the original filename, modulo the documented ASCII-fallback substitution.

use proptest::prelude::*;
use craig_store::validation::content_disposition_attachment;

/// Reference parser using the `headers` crate (or hand-rolled per RFC 6266 §5).
fn parse_filename_star(header_value: &str) -> Option<String> { /* ... */ }

proptest! {
    #![proptest_config(ProptestConfig::with_cases(512))]
    #[test]
    fn prop_content_disposition_round_trip(name in "[\\PC]{1,200}") {
        // \PC = any non-control char, including non-ASCII for the filename* field.
        let header = content_disposition_attachment(&name);
        prop_assert!(header.starts_with("attachment;"),
            "header missing attachment disposition: {header}");
        prop_assert!(header.contains("filename*=UTF-8''"),
            "header missing RFC 5987 extended form: {header}");

        let recovered = parse_filename_star(&header)
            .expect("parser must recover filename*");
        prop_assert_eq!(recovered, name);
    }
}

Expected runtime: 512 cases × encode+parse; <100ms.

D9.7 Property 5 — canonical JSON signing

File: crates/craig-signing/tests/properties/canonical_json.rs

craig-signing exposes two primitives at crates/craig-signing/src/lib.rs: canonicalize_json(&serde_json::Value) → String (line 40 — recursive sort-keys + minimal whitespace) and hash_payload(&str) → String (line 73 — SHA-256 hex of the canonical string). Together they form the signing input for inter-service envelopes. Insignificant whitespace and key-order shouldn’t change the canonical output (and therefore must not change the hash). Property:

  1. Canonicalize v, hash; re-serialize v with serde_json::to_string_pretty (adds whitespace + reshuffles HashMap key order), parse back, canonicalize, hash — hashes must match.

  2. Replace any leaf value with a different leaf — canonical strings must differ (and thus hashes must differ).

use proptest::prelude::*;
use craig_signing::{canonicalize_json, hash_payload};
use serde_json::Value;

fn any_json_leaf() -> impl Strategy<Value = Value> {
    prop_oneof![
        Just(Value::Null),
        any::<bool>().prop_map(Value::Bool),
        (-1_000_000i64..1_000_000).prop_map(|n| Value::from(n)),
        "[\\PC]{0,32}".prop_map(Value::from),
    ]
}
fn any_json() -> impl Strategy<Value = Value> {
    let leaf = any_json_leaf();
    leaf.prop_recursive(4, 16, 8, |inner| {
        prop_oneof![
            proptest::collection::vec(inner.clone(), 0..6).prop_map(Value::Array),
            proptest::collection::hash_map("[a-z]{1,8}", inner, 0..6)
                .prop_map(|m| serde_json::to_value(m).unwrap()),
        ]
    })
}

proptest! {
    #![proptest_config(ProptestConfig::with_cases(256))]
    #[test]
    fn prop_canonical_hash_invariant_to_whitespace_and_key_order(v in any_json()) {
        let canonical_a = canonicalize_json(&v);
        let hash_a = hash_payload(&canonical_a);

        // Round-trip through pretty-printer reshuffles HashMap iteration order
        // and inserts whitespace; canonicalize must recover identical bytes.
        let pretty = serde_json::to_string_pretty(&v).unwrap();
        let v_round: Value = serde_json::from_str(&pretty).unwrap();
        let canonical_b = canonicalize_json(&v_round);
        let hash_b = hash_payload(&canonical_b);

        prop_assert_eq!(canonical_a, canonical_b,
            "canonical string changed under whitespace/key-order normalization");
        prop_assert_eq!(hash_a, hash_b,
            "hash changed under whitespace/key-order normalization");
    }
}

Fail-first sketch: replace canonicalize_json recursion arm for Value::Object with serde_json::to_string(&map).unwrap() (HashMap iteration order is non-deterministic in Rust); pretty-printed input shrinks to a 2-key object that hashes differently on different runs.

Expected runtime: 256 cases × recursive json (depth 4); <500ms (canonicalization dominates).

D9.8 Property 6 — idempotency body fingerprint

File: crates/craig-api/tests/properties/idempotency_fingerprint.rs

Pairs with platform-stab-2 §D1’s idempotency claim. The body-fingerprint function gates the loser-side fingerprint comparison; if it ever returns the same fingerprint for two different bodies, two requests with mismatched bodies are quietly accepted as duplicates.

Visibility note: body_fingerprint is currently a private free function at crates/craig-api/src/idempotency.rs:105. Two paths to make it testable:

Option Trade-off

Promote to pub(crate) and expose via #[cfg(test)] pub use from a test-only re-export module

Keeps the function private to non-test consumers; one-line addition.

Promote to pub fn outright

Makes the fingerprint algorithm a public API contract, which it arguably already is (since the value is persisted in idempotency_responses.body_fingerprint and platform-stab-2 §D1 references it across paths). Default proposal.

Adopt option 2: change fn body_fingerprint(…​) at idempotency.rs:105 to pub fn body_fingerprint(…​), document it as part of the crate’s public API. Step 10 of THIS plan includes the visibility flip as a one-line change in crates/craig-api/src/idempotency.rs.

Two complementary property assertions:

  1. Equality preservation: fingerprint(b1) == fingerprint(b2) iff b1 == b2.

  2. Small-flip distinctness: differ-by-one-byte inputs produce different fingerprints (catches a checksum that ignores high-order bits or has a small fingerprint domain).

use proptest::prelude::*;
use craig_api::idempotency::body_fingerprint; // requires the pub-visibility flip noted above

proptest! {
    #![proptest_config(ProptestConfig::with_cases(512))]
    #[test]
    fn prop_fingerprint_equal_iff_bytes_equal(
        a in proptest::collection::vec(any::<u8>(), 0..2048),
        b in proptest::collection::vec(any::<u8>(), 0..2048),
    ) {
        let fa = body_fingerprint(&a);
        let fb = body_fingerprint(&b);
        prop_assert_eq!(fa == fb, a == b);
    }

    #[test]
    fn prop_fingerprint_one_byte_flip_differs(
        bytes in proptest::collection::vec(any::<u8>(), 1..1024),
        idx in any::<prop::sample::Index>(),
    ) {
        let i = idx.index(bytes.len());
        let mut flipped = bytes.clone();
        flipped[i] = flipped[i].wrapping_add(1);
        prop_assert_ne!(body_fingerprint(&bytes), body_fingerprint(&flipped));
    }
}

Expected runtime: 1024 cases × two SHA-256-ish hashes; <300ms.

D9.9 Property 7 — reference-data validation never panics

File: crates/craig-reference/tests/properties/parse_never_panics.rs

The 48-test reference suite covers the happy paths for each enum. Property complements: parsing arbitrary garbage strings against any reference enum returns Ok() or Err() — never panics, never out-of-bounds, never overflows.

use proptest::prelude::*;
use std::str::FromStr;
use craig_reference::{
    PaymentStatus, PlacementStatus, ContactType, ReportStatus,
    AdminUnit, FipsCode, EventType,
};

macro_rules! prop_no_panic {
    ($t:ty, $name:ident) => {
        proptest! {
            #![proptest_config(ProptestConfig::with_cases(2048))]
            #[test]
            fn $name(s in ".{0,64}") {
                // Catch panics; success or failure is fine, panic is not.
                let _ = std::panic::catch_unwind(|| <$t>::from_str(&s));
            }
        }
    };
}

prop_no_panic!(PaymentStatus,    prop_payment_status_no_panic);
prop_no_panic!(PlacementStatus,  prop_placement_status_no_panic);
prop_no_panic!(ContactType,      prop_contact_type_no_panic);
prop_no_panic!(ReportStatus,     prop_report_status_no_panic);
prop_no_panic!(EventType,        prop_event_type_no_panic);
prop_no_panic!(FipsCode,         prop_fips_code_no_panic);

Fail-first sketch: introduce s.split_at(s.len() + 1) into one parser; proptest shrinks to a single character.

Expected runtime: 6 enums × 2048 cases × cheap parse; <500ms total.

D9.10 Aggregate budget

Property Cases Wall (ms) Crate

D9.3 PaymentStatus round-trip

512

<50

craig-financial

D9.4 Pagination

256

<100

craig-api

D9.5 sanitize_filename

1024

<50

craig-store

D9.6 Content-Disposition

512

<100

craig-store

D9.7 Canonical JSON signing

256

<500

craig-signing

D9.8 Body fingerprint (×2)

1024

<300

craig-api

D9.9 Reference parse no-panic (×6)

12,288

<500

craig-reference

Total

~16k

<2s

The full property suite fits in cargo nextest run --workspace -E 'test(::properties::)' under 2 seconds. Reasonable to run on every developer push under cargo xtask reliability (Step 12) without making the existing pre-push gate slower.

D9.11 Test description (per property)

Each property carries:

  1. Strategy — the impl Strategy<Value = T> quoted verbatim above.

  2. Invariant — the prop_assert! body.

  3. Fail-first sketch — the one-character flip in production code that demonstrates the property catches the bug, with the expected shrunk counter-example documented in a // FAIL-FIRST: comment retained in the test file.

  4. Cases-override — explicit when not the default 256.

  5. Runtime budget — measured on the contributor’s machine; recorded in a // PROPTEST_BUDGET_MS: comment so reviewers spot regressions.

D10. cargo-mutants smoke on 5 critical crates (Phase A.10 — Step 11)

Mutation testing answers a question coverage cannot: do the existing tests actually catch wrong code, or do they just execute it? cargo-mutants mutates production code (flips comparisons, deletes statements, replaces Ok)) with Ok(( no-ops, etc.) and re-runs the test suite per mutant. A mutant that survives (test suite still passes) is a hard signal that the test suite has a real semantic gap on that line — exactly the storage-shape-vs-semantics gap the 2026-05-03 review surfaced for the platform invariants.

Per the user-stated constraints, this is a smoke deployment: the tool is available via cargo xtask mutants --smoke, the baseline report is committed, and surviving mutants are catalogued — but mutation pass-rate is not a CI gate. Tightening the budget happens in a future plan.

D10.1 Tool installation + version pin

# .cargo/mutants.toml — workspace-level config (committed)
# cargo-mutants 25.x is the actively-maintained line as of 2026-05.
# Pin via `cargo install cargo-mutants --version "^25.0" --locked`.
timeout = 300                    # per-mutant cap (seconds); kills hangs
test_tool = "nextest"            # use the same runner as the rest of CRAIG
test_package = ["--locked"]
exclude_globs = [
    "**/tests/**",
    "**/examples/**",
    "**/benches/**",
    "**/build.rs",
    "**/main.rs",                # boot/wiring code; mutants here surface as test-harness gaps, not invariants
]

CONTRIBUTING.adoc gains a one-line "Optional dev tools" section pointing at cargo install cargo-mutants --version "^25.0" --locked. No-tool installation does not break any existing workflow — cargo xtask mutants is opt-in.

D10.2 The five critical crates (rationale)

Crate Why it’s critical Mutant surface

crates/craig-api

Hosts idempotency middleware (platform-stab-2 §D1), problem-details mapper, auth wrappers. Test failures here have correctness blast-radius across all 8 services.

idempotency.rs, problem.rs, auth_layer.rs, pagination.rs

crates/craig-mq

Outbox + inbox durability (§D2 + §D3 of platform-stab-2). Concurrency-correctness pivot point.

outbox.rs, inbox.rs, publisher.rs

crates/craig-auth

JWT validation — typ/aud/azp/scope/exp/nbf strictness. Direct attacker-facing surface.

validator.rs, claims.rs

crates/craig-store

Upload validation, encryption boundaries, object-store interactions.

validation.rs, store.rs, scanner.rs (after platform-stab-2 §D4 lands)

services/craig-cases/src/transitions/

5 state machines, the platform’s largest cross-product of permitted transitions.

transitions/*.rs

The 5 are deliberately the highest-leverage crates — not the most-touched. Other crates remain runnable via cargo xtask mutants --full but are not in the smoke baseline.

D10.3 Smoke-profile bounds

  • Per crate: cap mutant count at ~50 via cargo mutants --in-place --no-shuffle --jobs 1 --shard 0/N with N chosen so each shard runs <5min on the contributor machine.

  • Per-mutant timeout: 300s (above).

  • Total wall-time budget: <30 minutes for the full smoke. Documented in CONTRIBUTING as "expect a coffee".

  • Reports written to target/mutants-<crate>/ (cargo-mutants default), then a single summary copied to target/mutants-summary.json by the xtask wrapper (D11.3).

D10.4 Surviving-mutant cataloguing

Each smoke run produces three lists per crate:

Bucket Definition Action

Caught

Mutant produced a test failure. Healthy.

None.

Missed (survived)

Test suite passed unchanged. Real test gap.

Catalogue in xtask/mutants-baseline.toml with crate, file, line, mutant_kind and a one-line // reason: <why-it’s-acceptable-or-tracked>.

Unviable / timeout

Mutant didn’t compile, or hit the 300s timeout.

Skip from the baseline; cargo-mutants reports them but they’re not test-quality signal.

The baseline file is committed as the current floor. A future plan can tighten the budget by requiring:

  1. New CRs may not increase the missed count (delta=0).

  2. Each // reason: tracked-issue-#NNN line must reference a real backlog issue.

The first iteration intentionally does not enforce either — per the user’s "available, not gated" constraint.

D10.5 Test description

Treat the smoke run itself as the verification:

  1. cargo install cargo-mutants --version "^25.0" --locked

  2. cargo xtask mutants --smoke (Step 12)

  3. Inspect target/mutants-summary.json; assert at least one caught mutant per crate (sanity floor — if a crate has zero caught mutants the smoke is misconfigured).

  4. Commit xtask/mutants-baseline.toml with the missed list.

  5. Fail-first sketch: on a clean working tree, manually delete one prop_assert! from a property in §D9; rerun smoke; observe the corresponding mutant survive where it previously didn’t. Document this as the regression-detection demonstration.

D11. xtask subcommands: reliability + contracts + mutants + coverage-matrix + invariants (Phase A.11 — Step 12)

Five thin orchestration subcommands surface the new tooling. Each follows the existing xtask/src/cmd/*.rs pattern verbatim — clap Args struct, pub fn run(args) → Result<()>, sectioned === …​ === output, OK / FAIL / SKIP markers per item, no panics.

The shared design point — non-negotiable per user constraint — is that none of these subcommands run as part of the pre-push hook today. .githooks/pre-push is unchanged. Each subcommand prints, on entry, a one-liner identifying itself as opt-in.

D11.1 Subcommand registration

Extend xtask/src/cmd/mod.rs:

pub mod reliability;
pub mod contracts;
pub mod mutants;
pub mod coverage_matrix;
pub mod invariants;
pub mod quality_budgets;        // §D12

#[derive(Subcommand)]
pub enum Command {
    Dev(dev::DevArgs),
    Test(test::TestArgs),
    E2e(e2e::E2eArgs),
    Validate(validate::ValidateArgs),
    CheckDocs(check_docs::CheckDocsArgs),
    Perf(perf::PerfArgs),
    Security(security::SecurityArgs),
    Sbom,
    ApiDocs(api_docs::ApiDocsArgs),
    Reconcile(reconcile::ReconcileArgs),

    /// Run reliability suite (concurrency / recovery / fault / invariants / properties)
    Reliability(reliability::ReliabilityArgs),
    /// Run typed-client contract tests
    Contracts(contracts::ContractsArgs),
    /// Run cargo-mutants on the 5 critical crates (smoke) or workspace (--full)
    Mutants(mutants::MutantsArgs),
    /// Generate the route × test-axis coverage matrix
    CoverageMatrix(coverage_matrix::CoverageMatrixArgs),
    /// Run platform-invariant tests (atomic-claim, FUSL, processed-state, ...)
    Invariants(invariants::InvariantsArgs),
    /// Report or enforce code-quality budgets (§D12)
    QualityBudgets(quality_budgets::QualityBudgetsArgs),
}

xtask/src/main.rs gets six new match arms wired identically to the existing pattern.

D11.2 reliability subcommand

File: xtask/src/cmd/reliability.rs

Glob nextest filter expression covers all four reliability axes from the testing.md taxonomy added in Step 8 of this plan, plus the property suite from §D9:

use anyhow::{Context, Result, bail};
use clap::Args;
use std::process::Command;

const ENTRY_BANNER: &str =
    "Reliability suite (opt-in; not part of pre-push gate). \
     See plans/test-framework-hardening.adoc §D11.";

const NEXTEST_FILTER: &str = "test(::concurrency::) \
                            | test(::recovery::) \
                            | test(::fault::) \
                            | test(::invariants::) \
                            | test(::properties::)";

#[derive(Args)]
pub struct ReliabilityArgs {
    /// Optional nextest filter to AND with the default reliability filter.
    #[arg(long)]
    pub filter: Option<String>,
}

pub fn run(args: ReliabilityArgs) -> Result<()> {
    println!("=== {ENTRY_BANNER} ===");
    let mut filter = NEXTEST_FILTER.to_string();
    if let Some(extra) = args.filter {
        filter = format!("({filter}) & ({extra})");
    }
    let status = Command::new("cargo")
        .args(["nextest", "run", "--workspace", "--locked", "-E", &filter])
        .status()
        .context("spawn cargo nextest")?;
    if !status.success() { bail!("reliability suite failed with {status}"); }
    println!("=== Reliability OK ===");
    Ok(())
}

D11.3 mutants subcommand

File: xtask/src/cmd/mutants.rs

use anyhow::{Context, Result, bail};
use clap::Args;
use std::process::Command;

const SMOKE_CRATES: &[&str] = &[
    "crates/craig-api",
    "crates/craig-mq",
    "crates/craig-auth",
    "crates/craig-store",
    "services/craig-cases",
];

#[derive(Args)]
pub struct MutantsArgs {
    /// Run full workspace mutation testing (no per-crate filter)
    #[arg(long, conflicts_with = "smoke")]
    pub full: bool,
    /// Run the smoke profile against the 5 critical crates (default)
    #[arg(long)]
    pub smoke: bool,
}

pub fn run(args: MutantsArgs) -> Result<()> {
    println!("=== cargo-mutants (opt-in; not part of pre-push gate). \
              Surviving mutants catalogued in xtask/mutants-baseline.toml. ===");

    if !is_installed() {
        bail!("cargo-mutants is not installed. Install with:\n\
               cargo install cargo-mutants --version \"^25.0\" --locked");
    }

    let crates: Vec<&str> = if args.full { vec![] } else { SMOKE_CRATES.into() };

    if crates.is_empty() {
        run_one(None)?;
    } else {
        for c in &crates { run_one(Some(c))?; }
    }
    summarize()?;
    println!("=== Mutants OK (see target/mutants-summary.json) ===");
    Ok(())
}

fn is_installed() -> bool {
    Command::new("cargo").args(["mutants", "--version"]).output()
        .map(|o| o.status.success()).unwrap_or(false)
}
fn run_one(scope: Option<&str>) -> Result<()> { /* ... cargo mutants --in-place --jobs 1 [--in-dir scope] ... */ Ok(()) }
fn summarize() -> Result<()> { /* aggregate per-crate target/mutants-*/outcomes.json into target/mutants-summary.json */ Ok(()) }

D11.4 contracts subcommand

File: xtask/src/cmd/contracts.rs

The typed-client tests (Phase A.8 in the existing draft) live under tests/typed/<service>/ per the testing.md convention. The subcommand pins them as a callable axis.

use anyhow::{Context, Result, bail};
use clap::Args;
use std::process::Command;

#[derive(Args)]
pub struct ContractsArgs {
    /// Restrict to a specific service (cases, exchange, ...)
    #[arg(long)]
    pub service: Option<String>,
}

pub fn run(args: ContractsArgs) -> Result<()> {
    println!("=== Typed-client contract tests (opt-in; not part of pre-push gate). ===");
    let filter = match args.service.as_deref() {
        Some(s) => format!("test(::typed::{s}::)"),
        None    => "test(::typed::)".to_string(),
    };
    let status = Command::new("cargo")
        .args(["nextest", "run", "--workspace", "--locked", "-E", &filter])
        .status().context("spawn cargo nextest")?;
    if !status.success() { bail!("contracts suite failed"); }
    Ok(())
}

D11.5 coverage-matrix subcommand

File: xtask/src/cmd/coverage_matrix.rs — full implementation belongs to §D13 (Step 14). The subcommand here is a one-line shell-out.

use anyhow::Result;
use clap::Args;

#[derive(Args)]
pub struct CoverageMatrixArgs { /* see §D13 */ }

pub fn run(args: CoverageMatrixArgs) -> Result<()> {
    println!("=== Coverage matrix (opt-in; report-only, no enforcement gate). ===");
    crate::cmd::coverage_matrix::generate(args)   // module-private fn from §D13
}

D11.6 invariants subcommand

File: xtask/src/cmd/invariants.rs — wraps a nextest filter targeting the platform-invariant tests added across this plan and platform-stab-2.

pub fn run(_: InvariantsArgs) -> Result<()> {
    println!("=== Invariant tests (opt-in; not part of pre-push gate). ===");
    let status = Command::new("cargo")
        .args(["nextest", "run", "--workspace", "--locked",
               "-E", "test(::invariants::)"])
        .status()?;
    if !status.success() { bail!("invariant suite failed"); }
    Ok(())
}

D11.7 Documentation surface

  • CONTRIBUTING.adoc — new "Optional reliability tooling" section enumerating the five subcommands as available tools (not required for MR landing).

  • .claude/docs/testing.md — Step 8’s edits already document the file-naming convention; this step appends a "Local subcommands" subsection cross-referencing the xtask entry-points.

  • README.adoc — one-paragraph note in the Development section.

D11.8 Test description

Per subcommand, a tests/ integration test exercises the subcommand once against a known-good fixture and asserts:

  1. Exit 0 on a passing project.

  2. Exit non-zero on an artificially-failed project (delete one prop_assert!, run reliability, expect failure).

  3. Output prints the entry banner verbatim — guards against the "available, not gated" message ever drifting.

D12. Code-quality budget warnings (Phase A.12 — Step 13)

User-stated budgets ship as a report, never as a pre-push gate. The mechanism is opt-in: contributors run cargo xtask quality-budgets --report and see a table; reviewers read the same table on each MR by manually invoking it. The --fail-on-regression flag exists for future CI activation but is not wired into pre-push or CI today.

D12.1 Budget catalogue

ID Budget Initial threshold Scope

B1

Max LOC per route module

500

services//src/api/.rs, services/craig-web/src/routes/*.rs

B2

Max LOC per function

100

All *.rs outside tests/

B3

serde_json::Value count

0 in src/ outside **/api/dto.rs; 5 in tests/

All *.rs

B4

#[allow(…​)] count

20 across workspace

All *.rs

B5

unwrap_or_default() count in production paths

50

src/ only; tests excluded

B6

Duplicate dependency versions

0 (zero new dupes vs baseline)

cargo tree -d parsed output

B7

Untyped methods on core test clients

0 new (paired with Phase A.8)

crates/craig-test-lib/src/clients/*.rs

Each budget has a per-path exception list (xtask/quality-budgets.toml); the budget compares the count outside the exception list against the threshold. Exception entries require a reason = "…​" per the workspace lint convention.

D12.2 Config file shape

# xtask/quality-budgets.toml
# Code-quality budgets — report-only as of test-framework-hardening Step 13.
# `--fail-on-regression` reads thresholds; `--report` emits a table.

[B1.route_module_loc]
threshold = 500
glob      = ["services/*/src/api/*.rs", "services/craig-web/src/routes/*.rs"]
exceptions = [
    # Examples — empty initially; new exceptions require a reason field
    # { path = "services/craig-cases/src/api/reports.rs", reason = "tracked: split-pending #282" },
]

[B2.function_loc]
threshold = 100
glob      = ["**/*.rs"]
exclude   = ["**/tests/**"]

[B3.serde_json_value]
threshold_src   = 0
threshold_tests = 5
src_glob        = ["**/src/**/*.rs"]
src_exclude     = ["**/api/dto.rs", "**/api/dto/**/*.rs"]
tests_glob      = ["**/tests/**/*.rs"]

[B4.allow_attributes]
threshold = 20

[B5.unwrap_or_default]
threshold = 50
glob      = ["**/src/**/*.rs"]

[B6.duplicate_deps]
threshold = 0
# Baseline captured in xtask/quality-budgets.lock; B6 reports delta vs lock.

[B7.untyped_test_client_methods]
threshold = 0
# Counts methods returning `serde_json::Value` from craig-test-lib clients.
glob      = ["crates/craig-test-lib/src/clients/*.rs"]

D12.3 Lockfile / baseline

# xtask/quality-budgets.lock — generated by `cargo xtask quality-budgets --report --write-lock`
# Tracks the COUNT (not threshold) at the time of the lock. Regression check
# under `--fail-on-regression` reads BOTH the threshold and the lock and fails
# if `count > max(threshold, lock)`.
[counts]
B1_route_module_loc      = 0      # LOC over threshold; populated on first run
B2_function_loc          = 0
B3_serde_json_value_src  = 0
B3_serde_json_value_test = 0
B4_allow_attributes      = 0
B5_unwrap_or_default     = 0
B6_duplicate_deps        = 0
B7_untyped_test_methods  = 0
generated_at = "2026-05-03T00:00:00Z"
generated_from_sha = "<git-rev>"

The lock is committed. New CRs that increase a count must justify in CR description (reviewer discipline; the lockfile diff is the obvious signal).

D12.4 Subcommand shape

File: xtask/src/cmd/quality_budgets.rs

use anyhow::Result;
use clap::Args;

#[derive(Args)]
pub struct QualityBudgetsArgs {
    /// Print the budget table (default mode)
    #[arg(long, conflicts_with = "fail_on_regression")]
    pub report: bool,
    /// Exit non-zero if any budget exceeded (NOT run in pre-push today)
    #[arg(long)]
    pub fail_on_regression: bool,
    /// Update xtask/quality-budgets.lock with current counts
    #[arg(long, requires = "report")]
    pub write_lock: bool,
}

#[derive(Debug)]
struct BudgetReport {
    id: &'static str,
    name: &'static str,
    threshold: usize,
    locked: usize,
    actual: usize,
}

pub fn run(args: QualityBudgetsArgs) -> Result<()> {
    println!("=== Quality budgets (report-only; not a pre-push gate). ===");
    let cfg = read_config()?;
    let lock = read_lock()?;
    let reports = vec![
        run_b1_route_module_loc(&cfg, &lock)?,
        run_b2_function_loc(&cfg, &lock)?,
        run_b3_serde_json_value(&cfg, &lock)?,
        run_b4_allow_attributes(&cfg, &lock)?,
        run_b5_unwrap_or_default(&cfg, &lock)?,
        run_b6_duplicate_deps(&cfg, &lock)?,
        run_b7_untyped_test_methods(&cfg, &lock)?,
    ];
    print_table(&reports);
    if args.write_lock { write_lock(&reports)?; }
    if args.fail_on_regression {
        let regressed: Vec<_> = reports.iter()
            .filter(|r| r.actual > r.threshold.max(r.locked))
            .collect();
        if !regressed.is_empty() {
            anyhow::bail!("{} budget(s) regressed", regressed.len());
        }
    }
    Ok(())
}

fn print_table(rows: &[BudgetReport]) { /* tabled — same crate already in workspace */ }
fn read_config()  -> Result<Config> { /* parse xtask/quality-budgets.toml */ Ok(Default::default()) }
fn read_lock()    -> Result<Lock>   { /* parse xtask/quality-budgets.lock */ Ok(Default::default()) }
fn write_lock(_reports: &[BudgetReport]) -> Result<()> { Ok(()) }
fn run_b1_route_module_loc(_: &Config, _: &Lock) -> Result<BudgetReport> { todo!() }
// ... B2..B7

The implementations are mostly directory walks (walkdir = "2" already in workspace via several deps) plus regex counts (regex = "1"). B6 shells out to cargo tree -d. B7 uses syn to parse client files and inspect return types — already a workspace dep via cargo-deny derivations.

D12.5 Sample output (target shape)

=== Quality budgets (report-only; not a pre-push gate). ===
+-----+----------------------------------+--------+--------+--------+--------+
| ID  | Name                             | Thresh | Locked | Actual | Status |
+-----+----------------------------------+--------+--------+--------+--------+
| B1  | Route module LOC                 |    500 |    893 |    893 | LOCKED |
| B2  | Function LOC                     |    100 |    142 |    142 | LOCKED |
| B3a | serde_json::Value in src         |      0 |     12 |     12 | LOCKED |
| B3b | serde_json::Value in tests       |      5 |    187 |    187 | LOCKED |
| B4  | #[allow(...)] count              |     20 |     34 |     34 | LOCKED |
| B5  | unwrap_or_default in production  |     50 |    101 |    101 | LOCKED |
| B6  | Duplicate dependency versions    |      0 |      2 |      2 | LOCKED |
| B7  | Untyped test-client methods      |      0 |     56 |     56 | LOCKED |
+-----+----------------------------------+--------+--------+--------+--------+
Status: LOCKED = at-or-below current lockfile baseline.
        OVER   = above lockfile baseline; explain in CR description.

D12.6 Test description

  1. Unit: each run_b<N>_* function has a fixture-directory test asserting it counts deterministically.

  2. Snapshot: integration test runs the subcommand against a fixed fixture, snapshots the rendered table; visible diff on regression.

  3. Lockfile-init: first --write-lock produces a lockfile that round-trips parse → render.

D13. Coverage Matrix tooling — xtask coverage-matrix (Phase A.13 — Step 14)

Coverage tools historically conflate two different signals: line coverage (which lines executed during tests) and risk coverage (which behaviors of the system were tested). Line coverage is what cargo-tarpaulin and cargo-llvm-cov produce. Risk coverage — the matrix of "for each route, which adversarial axes have we tested?" — has no off-the-shelf tool. This subcommand builds it.

Per the user’s "report-only" directive: the output is a Markdown table at target/coverage-matrix.md, plus a single percentage at the end. No threshold, no gate.

D13.1 Inputs

Source Used for Pull mechanism

OpenAPI specs (each service exposes /api-doc/openapi.json)

Route inventory: (method, path) per service

xtask/src/cmd/coverage_matrix.rs reuses the xtask api-docs HTTP client (already present at xtask/src/cmd/api_docs.rs:46).

Test source files

Test inventory by axis

walkdir over crates//tests/, services//tests/. Match path prefixes per the testing.md taxonomy.

Test names within those files

Axis tagging via _happy / _sad / _evil suffixes

Lightweight regex on [test] / [tokio::test] / proptest! blocks. The user explicitly directed not to introduce a #[test_category(…​)] proc-macro.

D13.2 Six axes

Axis What it asserts How counted

happy

Documented success path

Test name ends _happy OR is the only test for the route

sad

Documented failure path (4xx)

Test name ends _sad

evil

Adversarial input (attack-shaped)

Test name ends _evil OR lives under tests/security/

concurrency

Concurrent same-key behavior

Test file matches tests/concurrency/

replay

Crash-and-restart / redelivery semantics

Test file matches tests/recovery/

fault

Dependency error handling

Test file matches tests/fault/

For each (service, method, path, axis) cell, the cell is if at least one test references the route — reference meaning either the URL appears as a string literal in the test source, or the typed-client method name corresponding to the route is invoked. The typed-client mapping comes from a single xtask/route-to-client-method.toml table maintained alongside the typed clients (Phase A.8 deliverable; this plan adds an empty starter file with the manually-mapped seed entries).

D13.3 Output format

Markdown table at target/coverage-matrix.md:

# CRAIG Coverage Matrix
Generated: 2026-05-03T12:00:00Z
Route: 247 endpoints across 8 services
Risk axes: happy / sad / evil / concurrency / replay / fault

| Service | Method | Path | happy | sad | evil | conc | replay | fault |
|---------|--------|------|-------|-----|------|------|--------|-------|
| craig-cases | GET | /v1/cases       | ✓ |   |   |   |   |   |
| craig-cases | POST | /v1/cases      | ✓ | ✓ |   | ✓ |   |   |
| craig-cases | GET | /v1/cases/{id}  | ✓ |   |   |   |   |   |
| ...
**Risk-coverage score: 412 / 1482 cells (27.8%)**

The percentage is total_cells_filled / total_cells. total_cells = endpoints × 6. Score is reported but not gated.

D13.4 Subcommand shape

File: xtask/src/cmd/coverage_matrix.rs

use anyhow::{Context, Result};
use clap::Args;
use std::collections::BTreeMap;
use std::path::PathBuf;

#[derive(Args)]
pub struct CoverageMatrixArgs {
    /// Output path (default: target/coverage-matrix.md)
    #[arg(long, default_value = "target/coverage-matrix.md")]
    pub output: PathBuf,
    /// Base URL for OpenAPI scraping (default: http://localhost)
    #[arg(long, default_value = "http://localhost")]
    pub base_url: String,
    /// Skip OpenAPI fetch; use cached `target/openapi-cache.json`
    #[arg(long)]
    pub use_cache: bool,
}

#[derive(Debug, Clone)]
struct Route { service: String, method: String, path: String }

#[derive(Debug)]
struct Cell { axes: [bool; 6] }   // happy, sad, evil, conc, replay, fault

pub fn generate(args: CoverageMatrixArgs) -> Result<()> {
    println!("=== Coverage matrix (report-only). ===");
    let routes = if args.use_cache { read_cache()? }
                 else                { fetch_openapi(&args.base_url)? };
    let tests  = scan_tests()?;
    let matrix = build_matrix(&routes, &tests);
    write_markdown(&args.output, &routes, &matrix)?;
    let (filled, total) = score(&matrix);
    println!("=== Risk-coverage: {filled}/{total} ({:.1}%) — see {}",
             100.0 * filled as f64 / total as f64,
             args.output.display());
    Ok(())
}

fn fetch_openapi(_base: &str) -> Result<Vec<Route>> { /* see api_docs.rs:46-72 */ Ok(vec![]) }
fn scan_tests() -> Result<Vec<TestRef>> { /* walkdir + regex */ Ok(vec![]) }
fn build_matrix(_: &[Route], _: &[TestRef]) -> BTreeMap<(String,String,String), Cell> { Default::default() }
fn write_markdown(_: &std::path::Path, _: &[Route], _: &BTreeMap<(String,String,String), Cell>) -> Result<()> { Ok(()) }
fn score(m: &BTreeMap<(String,String,String), Cell>) -> (usize, usize) {
    let total = m.len() * 6;
    let filled: usize = m.values().map(|c| c.axes.iter().filter(|x| **x).count()).sum();
    (filled, total)
}

#[derive(Debug)] struct TestRef { /* axis, route_match */ }
fn read_cache() -> Result<Vec<Route>> { Ok(vec![]) }

D13.5 Decision: regex tagging, not proc-macro

The user explicitly directed not to introduce a #[test_category(…​)] proc-macro. Rationale:

Approach Cost Benefit

Proc-macro #[test_category(happy)]

New crate (craig-test-tags); pulls in syn/quote/proc-macro2; one more compilation step on every test rebuild; macro versioning vs runtime crate versioning matrix.

Compile-time guarantee that an axis tag is one of the six known.

Regex over name suffix (_happy, _sad, _evil) + path prefix (tests/concurrency/)

Trivial; lives entirely inside xtask.

No compile-time guarantee — typo _evel produces no axis tag. Mitigated by a CI-time lint inside the subcommand: any test name in tests/api/ that doesn’t end in one of the recognized suffixes is reported as "untagged" in a footer block.

The footer-block mitigation is enough at the current scale (~1259 Rust tests). Revisit if untagged-test count climbs over a threshold.

D13.6 Initial baseline commit

The Step 14 MR commits both the subcommand and a first-run target/coverage-matrix.md snapshot under docs/modules/ROOT/pages/operations/coverage-matrix.adoc (the markdown table is wrapped in [source,markdown]). Subsequent CRs do not update this file; it’s a one-shot baseline. A future plan revisits "live coverage matrix in CI" once the foundation has bedded in.

D13.7 --diff main flag for scorecard delta

Reviewer-discipline lift: the scorecard at docs/modules/ROOT/pages/test-coverage-scorecard.adoc lives in the repo and changes per MR. Without tooling, reviewers must eyeball cell flips. Add --diff <REF> (default main) to the subcommand:

#[derive(Args)]
pub struct CoverageMatrixArgs {
    /// Compare against a base ref (default: main). Prints filled/empty cell deltas.
    #[arg(long)]
    pub diff: Option<String>,
    // ...other args from D13.4
}

Behavior:

  1. Run the matrix against the current tree → current_matrix.

  2. git stash (preserve uncommitted) → git checkout <REF> → re-run subcommand → base_matrixgit checkout -git stash pop.

  3. Print:

    === Coverage-matrix delta vs main ===
    + POST /v1/cases :: evil   (filled by tests/api/cases_evil.rs::create_case_oversize_name_evil)
    + GET  /v1/cases :: replay (filled by tests/recovery/cases_list_replay.rs)
    - DELETE /v1/cases/:id :: happy   (regression: previously filled, no longer matched)
    === Risk-coverage delta: +2 cells, -1 cell, net +1 ===
  4. Exit 0 always — report-only, even on regressions, per the no-CI-pressure directive.

This is the lightest possible "pre-commit hint": developers run cargo xtask coverage-matrix --diff main before pushing; reviewers can run it during review. No pre-commit hook, no CI gate.

D13.8 Test description

  1. Fixture: a tiny mock OpenAPI spec (3 routes) + 3 mock test files, one tagged each axis. Subcommand should produce a 3-row matrix with the expected cells filled.

  2. Untagged warning: extra test file with no recognized suffix; subcommand prints it under "Untagged tests" in the footer.

  3. Score arithmetic: 3 routes × 6 axes = 18 cells; 5 cells filled → score string 5/18 (27.8%).

D14. DB Constraint Evil Tests — hand-curated harness (Phase A.14 — Step 15)

The user’s framing is correct: auto-deriving negative tests from migrations is brittle. CHECK constraints are arbitrary boolean expressions; UNIQUE constraints are tuples; FK constraints have ON DELETE semantics that vary; NOT NULL is the only constraint where the negative space is trivial. Even when the SQL parser succeeds, the interesting assertion is at the API boundary ("does the handler return 409 Conflict with the right RFC 9457 type URL?") rather than at the DB layer ("does the INSERT raise SQLSTATE 23514?"). The DB-layer signal is necessary but not sufficient.

The deliverable is therefore a hand-curated harness: a small helper module + a per-service convention for placing constraint tests + a seed of 2 representative constraints per service (16 tests total, 8 services). The remainder is tracked as a backlog issue and grown alongside future migrations.

D14.1 Helper module

File: crates/craig-test-lib/src/constraints.rs (new; exported from lib.rs)

//! Hand-curated DB constraint test harness.
//!
//! For each constraint we want to verify, write one test in
//! `services/<svc>/tests/constraints/<constraint>.rs` that uses
//! `assert_constraint_violation_returns(...)` to drive an API request
//! known to violate the constraint and assert the resulting 4xx +
//! RFC 9457 problem-type URL.

use crate::client::TypedClient;
use reqwest::StatusCode;

#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum ExpectedRejection {
    BadRequest400,
    UnprocessableEntity422,
    Conflict409,
    NotFound404,
}

impl From<ExpectedRejection> for StatusCode {
    fn from(e: ExpectedRejection) -> Self {
        match e {
            ExpectedRejection::BadRequest400         => StatusCode::BAD_REQUEST,
            ExpectedRejection::UnprocessableEntity422=> StatusCode::UNPROCESSABLE_ENTITY,
            ExpectedRejection::Conflict409           => StatusCode::CONFLICT,
            ExpectedRejection::NotFound404           => StatusCode::NOT_FOUND,
        }
    }
}

/// Drive `request` against `client`; assert the response carries
/// `expected` status and a problem+json body whose `type` field equals
/// `problem_type_url`.
///
/// `request` is a closure so the harness can rerun it (state-machine
/// constraint tests sometimes need a setup step before triggering).
pub async fn assert_constraint_violation_returns<R, Fut>(
    client: &TypedClient,
    request: R,
    expected: ExpectedRejection,
    problem_type_url: &str,
) where
    R: FnOnce(TypedClient) -> Fut,
    Fut: std::future::Future<Output = reqwest::Response>,
{
    let resp = request(client.clone()).await;
    let status = resp.status();
    let body: serde_json::Value = resp.json().await.expect("problem+json");
    assert_eq!(status, StatusCode::from(expected),
        "expected {expected:?} got {status}; body={body}");
    assert_eq!(body.get("type").and_then(|v| v.as_str()), Some(problem_type_url),
        "wrong problem type; full body: {body}");
}

D14.2 Per-service file convention

Service Test directory Module index

services/craig-cases

tests/constraints/

tests/constraints.rs declaring #[path = "constraints/<file>.rs"] mod <file>;

services/craig-placement

tests/constraints/

same

services/craig-exchange

tests/constraints/

same

services/craig-financial

tests/constraints/

same

services/craig-reporting

tests/constraints/

same

services/craig-rules

tests/constraints/

same

services/craig-security

tests/constraints/

same

services/craig-intake

tests/constraints/

same

The pattern matches the existing tests/api.rs + tests/api/*.rs taxonomy from testing.md.

D14.3 Constraint inventory (CRAIG-wide)

A first pass via Grep "(CHECK|UNIQUE|REFERENCES|NOT NULL)" services//migrations/.sql yields the constraints worth exercising. Indicative, not exhaustive — full enumeration is a backlog issue tracked under epic &22:

Service Table.constraint Kind Negative request

craig-mq (shared)

event_outbox.published_at IS NULL OR published_at >= created_at

CHECK

not API-driven; covered by §D2 outbox tests

craig-mq (shared)

idempotency_responses.status IN ('processing','succeeded','failed')

CHECK (after platform-stab-2 §D1)

POST with hand-crafted state isn’t reachable through the handler — this is a DB-layer integrity guard; cover via a focused sqlx::query integration test, not via the API harness

craig-cases

report_attachments.object_status IN ('pending','present','failed')

CHECK (after platform-stab-2 §D4)

same as above

craig-financial

payments.amount > 0

CHECK

POST /v1/payments with amount=-1 → 400 problem+json type https://docs.craig/problems/invalid-amount

craig-financial

payments.case_id REFERENCES cases.cases(id)

FK (cross-service; reconcile-walked, not DB-enforced)

POST /v1/payments with non-existent case_id → 404 problem+json type …​/case-not-found

craig-placement

placements.foster_home_id REFERENCES foster_homes(id)

FK

POST /v1/placements with deleted home_id → 409 problem+json type …​/home-not-active

craig-placement

placements.status_check (CHECK status IN (…​))

CHECK

POST /v1/placements body.status="in_orbit" → 422 problem+json type …​/invalid-placement-status

craig-exchange

icpc.transactions.partner_id REFERENCES partners(id) ON DELETE RESTRICT

FK ON DELETE RESTRICT

DELETE /v1/partners/{id} with active ICPC tx → 409 problem+json type …​/partner-in-use

craig-security

partners.code UNIQUE

UNIQUE

second POST /v1/partners with same code → 409 problem+json type …​/duplicate-partner-code

craig-rules

rule_sets.name UNIQUE

UNIQUE

second POST /v1/rules/sets same name → 409

craig-reporting

report_runs.status IN (…​)

CHECK

not API-reachable (handler validates first)

craig-intake

intake_submissions.api_key_id REFERENCES partner_api_keys(id)

FK

submission with rotated/deleted key → 401 problem+json type …​/api-key-revoked

Step 1 of this plan files the full enumeration as a tracking issue; the seed below picks 2 representative API-reachable constraints per service.

D14.4 Seed — 16 tests, 2 per service

File Constraint ExpectedRejection Problem-type URL

services/craig-cases/tests/constraints/case_number_format.rs

regex CHECK on cases.case_number

BadRequest400

https://docs.craig/problems/invalid-case-number

services/craig-cases/tests/constraints/contact_type_enum.rs

CHECK on contacts.contact_type

UnprocessableEntity422

https://docs.craig/problems/invalid-contact-type

services/craig-placement/tests/constraints/placement_status_enum.rs

CHECK on placements.status

UnprocessableEntity422

https://docs.craig/problems/invalid-placement-status

services/craig-placement/tests/constraints/home_in_use_on_delete.rs

RESTRICT FK

Conflict409

https://docs.craig/problems/home-in-use

services/craig-exchange/tests/constraints/partner_in_use_on_delete.rs

RESTRICT FK

Conflict409

https://docs.craig/problems/partner-in-use

services/craig-exchange/tests/constraints/icpc_status_enum.rs

CHECK

UnprocessableEntity422

https://docs.craig/problems/invalid-icpc-status

services/craig-financial/tests/constraints/payment_amount_positive.rs

CHECK > 0

BadRequest400

https://docs.craig/problems/invalid-amount

services/craig-financial/tests/constraints/rate_table_unique_name.rs

UNIQUE

Conflict409

https://docs.craig/problems/duplicate-rate-table

services/craig-reporting/tests/constraints/report_run_unique_name.rs

UNIQUE

Conflict409

https://docs.craig/problems/duplicate-report-run

services/craig-reporting/tests/constraints/report_kind_enum.rs

CHECK

UnprocessableEntity422

https://docs.craig/problems/invalid-report-kind

services/craig-rules/tests/constraints/rule_set_unique_name.rs

UNIQUE

Conflict409

https://docs.craig/problems/duplicate-rule-set-name

services/craig-rules/tests/constraints/jurisdiction_enum.rs

CHECK

UnprocessableEntity422

https://docs.craig/problems/invalid-jurisdiction

services/craig-security/tests/constraints/partner_unique_code.rs

UNIQUE

Conflict409

https://docs.craig/problems/duplicate-partner-code

services/craig-security/tests/constraints/admin_unit_fips_format.rs

CHECK regex

BadRequest400

https://docs.craig/problems/invalid-fips-code

services/craig-intake/tests/constraints/api_key_revoked.rs

FK with active=false

Conflict409 (mapped to 401 by handler)

https://docs.craig/problems/api-key-revoked

services/craig-intake/tests/constraints/captcha_required.rs

settings-validation (paired §D8.2)

BadRequest400

https://docs.craig/problems/captcha-required

D14.5 Sample test body

// services/craig-financial/tests/constraints/payment_amount_positive.rs
use craig_test_lib::{
    constraints::{assert_constraint_violation_returns, ExpectedRejection},
    TestHarness,
};
use craig_test_lib::clients::caseworker_financial_client;

#[tokio::test]
async fn payment_amount_must_be_positive_evil() {
    let h = TestHarness::new().await;
    if !h.devstack_available().await { return; }
    let client = caseworker_financial_client(&h);

    assert_constraint_violation_returns(
        &client,
        |c| async move {
            c.create_payment_raw(serde_json::json!({
                "case_id":      h.fixtures.case_id,
                "placement_id": h.fixtures.placement_id,
                "amount":       -1,
                "kind":         "ive_maintenance",
            })).await
        },
        ExpectedRejection::BadRequest400,
        "https://docs.craig/problems/invalid-amount",
    ).await;
}

D14.6 Test description

  1. Initial seed verification: all 16 tests pass against a fresh devstack on first run.

  2. Regression marker: the test names include _evil so they’re picked up by §D13’s coverage matrix as the evil cell for the relevant routes.

  3. Pushback assurance: no migration parsing, no auto-generation. The harness is 50 lines of Rust + 16 hand-written tests; debugging a failure points directly at the API boundary, not at a generator.

D15. Central Evil Input Corpus consolidation (Phase A.15 — Step 16)

Steps 6 (D5 — JWT mutation library) and 7 (D6 — magic-byte + filename-attack fixtures) ship as standalone fixture modules first, because the platform-stab-2 sibling fixes need them inline. D15 consolidates those two seed corpora into a single craig_test_lib::evil module and extends the corpus across nine new attack categories so every parametric API test, fuzz harness, contract-test, and security regression draws from one canonical iterator. Two motivations, in order:

  1. Single source of truth for "what does a hostile input look like". Today the magic-byte fixtures, the JWT mutations, and the ad-hoc 422 tests scattered across 8 services each carry their own corpus. When a reviewer adds "we should also reject CRLF in filenames", that addition lands in one fixture file and is silently absent from the other 7. A consolidated corpus means one EvilCase added once is exercised against every public-ingestion endpoint that opts into the unified contract test (§D15.6).

  2. Discovery via iteration, not enumeration. The current pattern asks the test author to list which inputs are evil. The corpus inverts the question: for every endpoint, the test loops all_evil_cases() and asserts each is rejected with a clean 4xx. New evil categories propagate to all endpoints automatically.

D15.1 Consolidation: from two seeds to one module

The Step-6 JWT mutation library lives at crates/craig-test-lib/src/jwt_mutations.rs (filed by the prior agent). The Step-7 upload fixtures live at crates/craig-test-lib/src/upload_fixtures.rs. Both are kept callable as-is (their callers in platform-stab-2 §D5 / §D8.1 don’t move). D15 re-homes them under crates/craig-test-lib/src/evil/:

crates/craig-test-lib/src/evil/
├── mod.rs           — re-exports + EvilCase / EvilCategory / EvilPayload / ExpectedRejection
├── jwt.rs           — wraps Step-6 jwt_mutations; emits EvilCase::Jwt(JwtMutation)
├── upload.rs        — wraps Step-7 upload_fixtures; emits EvilCase::Upload(UploadFixture)
├── string.rs        — overlong / invalid-UTF-8 / NUL / CRLF / RTLO / ZWSP    (NEW)
├── uuid.rs          — wrong-version / wrong-length / all-zeros / non-hex     (NEW)
├── json.rs          — depth bomb / int overflow / extra fields / type swap   (NEW)
├── path.rs          — `../`, double-encoded, NUL-truncated, UNC, ADS         (NEW)
├── html.rs          — script tags, javascript: URLs, polyglot SVG/HTML       (NEW)
├── unicode.rs       — NFC vs NFD, homoglyph, BiDi override, BOM              (NEW)
├── enum_value.rs    — unknown / case-mismatched / boundary-adjacent          (NEW)
├── date.rs          — leap-second, year-9999, negative, fractional overflow  (NEW)
├── multipart.rs     — missing boundary, conflicting Content-Length, …        (NEW)
└── signature.rs     — wrong-length, wrong-key, replay, alg-confusion         (NEW)

jwt.rs and upload.rs are thin wrappers — they import the existing seed modules and emit EvilCase enum variants. The seed modules continue to expose their direct iterator API for callers that want raw JwtMutation/UploadFixture types (e.g. the platform-stab-2 §D5 JWT-strictness test does not need the unified EvilCase envelope; it iterates jwt_mutations::all() directly).

D15.2 Public types

// crates/craig-test-lib/src/evil/mod.rs
#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)]
pub enum EvilCategory {
    Jwt, Upload, String, Uuid, Json, Path, Html, Unicode,
    EnumValue, Date, Multipart, Signature,
}

#[derive(Clone, Debug)]
pub struct EvilCase {
    /// Stable, human-meaningful identifier. Used as the test-block name when
    /// fed into `parametric_evil_test!`. Examples:
    ///   "string.overlong_1mb"
    ///   "json.depth_bomb_4096"
    ///   "path.dot_dot_double_url_encoded"
    pub name: &'static str,
    pub category: EvilCategory,
    pub payload: EvilPayload,
    pub expected: ExpectedRejection,
}

#[derive(Clone, Debug)]
pub enum EvilPayload {
    /// String body — fed as the field value in a request DTO.
    Str(String),
    /// Raw bytes — fed as a multipart body or as a typed Bytes argument.
    Bytes(Vec<u8>),
    /// JSON value — fed as a request body, typically with a wrapper DTO.
    Json(serde_json::Value),
    /// Pre-built JWT (header.payload.signature) — fed via Authorization header.
    BearerToken(String),
    /// Multipart fragment — already encoded; injected as the request body.
    MultipartRaw { content_type: String, body: Vec<u8> },
}

#[derive(Clone, Copy, Debug)]
pub enum ExpectedRejection {
    /// Any 4xx status acceptable — used for "reject with a clean error".
    AnyClientError,
    /// Specific 4xx status — used when the contract pins it (e.g. JWT → 401).
    Status(u16),
    /// 4xx with a specific RFC 9457 type pointer.
    ProblemType(&'static str),
}

pub fn all_evil_cases() -> impl Iterator<Item = EvilCase> { /* chain all categories */ }

pub fn evil_cases_for(category: EvilCategory) -> impl Iterator<Item = EvilCase> {
    all_evil_cases().filter(move |c| c.category == category)
}

The choice of &'static str for name is deliberate: every case must be a compile-time literal so test names are stable across runs (snapshot-testing parity, JUnit history matching).

D15.3 Corpus content per new category

Category Cases (default count) Coverage rationale

string.rs

1 KB / 1 MB / 16 MB overlong; invalid UTF-8 (lone surrogate, truncated continuation byte); NUL injection mid-string; CRLF injection in filename / display-name fields; leading + trailing whitespace; RTLO U+202E; ZWSP U+200B; NULL-truncated string ("abc\0evil")

Hits length bounds, encoder strictness, log-injection, RTLO header spoofing.

uuid.rs

Wrong version (v4 token where v7 expected); 31 hex chars; 33 hex chars; all-zeros (00000000-…); dash position scrambled; non-hex (gggggggg-…); valid UUID with embedded NUL

Hits UUID parser strictness + idempotency-key strictness.

json.rs

Depth bomb at 1024, 4096, 16384 nesting; ref cycle attempt ({"a":{"$ref":"#"}}); i64::MAX + 1 integer; schema-stuffing 100 unknown extra fields; missing one required field; type-swap ({"id": 42} where string expected)

Hits serde_json depth limits + DTO deny_unknown_fields posture + numeric coercion.

path.rs

../etc/passwd; ..%2fetc%2fpasswd; ..%252fetc%252fpasswd (double); NUL-truncated legit.pdf\0/../../etc/passwd; UNC \\server\share; alt-stream legit.pdf:secret; backslash-as-separator on POSIX

Hits path-sanitization in upload + download handlers.

html.rs

<script>alert(1)</script>; <img onerror=alert(1)>; javascript:alert(1) URL; base64-payload data URL; double-decode trick <script>; polyglot HTML/SVG file

Hits any field that’s later rendered without escape.

unicode.rs

Same string in NFC vs NFD (deduplication test); homoglyph — Cyrillic а (U+0430) where Latin a expected; BiDi LRO/RLO override; UTF-8 BOM at start of body; UTF-16 BOM in JSON body

Hits normalization-mismatch person-match collisions + display-name strictness.

enum_value.rs

Unknown variant name; case-mismatched (PENDING where pending expected); boundary-adjacent (pendinG, pendng, pending_); serializer-quirk ("PENDING " with trailing space)

Hits string-typed enum boundaries before the §D10.7 type-system fix lands; protects post-fix.

date.rs

Leap second 2016-12-31T23:59:60Z; year-9999 (9999-12-31T23:59:59Z); negative timestamp (-62135596800); fractional second precision overflow (.123456789012); UTC vs local timezone confusion

Hits chrono/jiff parser quirks + audit-log timestamp invariants.

multipart.rs

Missing boundary= parameter; malformed Content-Type: multipart/form-data;; conflicting Content-Length vs body bytes; oversized field count (1024 fields); empty filename (filename=""); content-type spoofing (declared application/json, body is multipart)

Hits multer / axum-extra multipart parsing.

signature.rs

Wrong-length JWS (truncated); signed with wrong key; replay (timestamp from 6 hours ago); alg-confusion (HS256 token forwarded to RS256 verifier); none alg attempt

Hits craig-signing canonical-JWS path + JWT validator strictness.

Default total corpus size: ~120 cases. Cheap (no I/O, no async); a single endpoint loop costs <2s.

D15.4 Helper macro: parametric_evil_test!

Boilerplate-free adoption from a service test file:

// services/craig-cases/tests/api/evil_corpus.rs
use craig_test_lib::evil::{parametric_evil_test, EvilCategory};

parametric_evil_test! {
    name: cases_post_rejects_evil_inputs,
    endpoint: |client, case| async move {
        client.post_case_with_evil(case).await
    },
    skip_categories: [EvilCategory::Multipart, EvilCategory::Signature],
}

The macro expands to one #[tokio::test] per EvilCase, naming each cases_post_rejects_evil_inputs<case.name>. Names are stable, individually addressable in cargo nextest run -E 'test(/path\.dot_dot/)'. skip_categories is the escape hatch for endpoints where a category is structurally inapplicable (POST /v1/cases doesn’t accept multipart, so skip it rather than expecting a useless reject).

D15.5 Reusability test: the global ingestion-rejection contract

A single integration test in crates/craig-test-lib/tests/evil_corpus_global_contract.rs opts the most-public endpoint of every API service into the contract:

#[tokio::test]
async fn every_evil_case_is_rejected_at_public_intake() {
    if !devstack_available().await { return; }
    let h = TestHarness::new().await;
    let intake = h.public_intake_client();

    for case in craig_test_lib::evil::all_evil_cases() {
        let res = intake.submit_with_evil(&case).await;
        match (res.status().as_u16(), case.expected) {
            (s, ExpectedRejection::AnyClientError) if (400..500).contains(&s) => {}
            (s, ExpectedRejection::Status(want)) if s == want => {}
            // ProblemType: parse RFC 9457 body, compare type field
            (s, ExpectedRejection::ProblemType(t)) if (400..500).contains(&s) => {
                assert_problem_type(res, t).await;
            }
            (s, exp) => panic!(
                "evil case {:?} (cat={:?}) returned status {} (expected {:?}) — \
                 server accepted hostile input or 5xx-ed instead of clean reject",
                case.name, case.category, s, exp,
            ),
        }
    }
}

The contract: for every EvilCase, the public ingestion endpoint returns a 4xx with the expected shape — no 500s, no 200s, no silent acceptance. A 500 is a test failure even if the input was hostile, because hostile input is a normal-business-hours occurrence and the service must classify it cleanly. This is the single test that establishes the boundary contract globally; per-endpoint parametric_evil_test! invocations harden each service-specific surface.

D15.6 Adoption: where the corpus is wired in

Site What changes

services/craig-cases/tests/api/evil_corpus.rs

New file: parametric_evil_test! against every POST /v1/cases* and PATCH /v1/cases* endpoint.

services/craig-exchange/tests/api/evil_corpus.rs

New file: same shape against POST /v1/exchange/transactions and ICPC submit endpoints.

services/craig-{financial,placement,reporting,rules,security}/tests/api/evil_corpus.rs

New file each: same pattern. Aggregate ~50 new test blocks per service.

services/craig-intake/tests/api/evil_corpus.rs

Existing file extended — replaces the ad-hoc seed corpus with the unified module. Public-intake is the most hostile surface.

crates/craig-test-lib/tests/evil_corpus_global_contract.rs

New file: §D15.5 global contract.

services/craig-web/tests/upstream_evil_corpus.rs

New file: feeds the corpus into the BFF’s typed clients to assert the BFF doesn’t propagate hostile values upstream. (Defends platform-stab-2 §D7.)

D15.7 Test coverage

Self-test for the corpus: crates/craig-test-lib/tests/evil_corpus_self.rs asserts (a) every category contributes at least one case; (b) every case has a non-empty name; (c) names are unique across all categories; (d) the iterator is deterministic across runs (sort + diff against a snapshot file).

Cross-reference: the two-pass platform-invariant audit's lesson reified as data — the corpus is the structural-vs-semantic boundary check.

D16. Restart/Replay light helpers (Phase A.16 — Step 17)

A heavyweight restart-replay harness was considered (full process spawn, signal-based crash, supervisor reattach) and rejected: it conflates testing with init systems, introduces flakiness from PID racing, and the only behavior we need is "drive the recovery code path of a service that crashed at a chosen semantic point". A small set of helpers, paired with the existing test-pattern docs, gives 100% of the value with <200 LOC and zero process-management state.

D16.1 Module shape

crates/craig-test-lib/src/restart.rs (new):

/// A semantic point at which the service simulates a crash. The injector at
/// this point hard-errors with a panic-equivalent (returns Err that the
/// surrounding code unwraps in test mode), forcing the in-flight tx to abort
/// and the calling fn to return.
#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)]
pub enum CrashPoint {
    /// Inside an attachment upload, after Phase 1 commit and BEFORE Phase 2 put.
    AttachmentMidUpload,
    /// Inside an attachment upload, after Phase 2 put and BEFORE status promote.
    AttachmentPostPut,
    /// Inside outbox drain, after publish() returns Ok and BEFORE post-publish UPDATE.
    OutboxPostPublish,
    /// Inside inbox handler, between handler success and processed_at UPDATE.
    InboxPostHandle,
    /// Inside exchange send, after adapter.send() returns Ok and BEFORE finalize tx.
    ExchangePostSend,
    /// Inside placement update, after update_placement succeeds and BEFORE event publish.
    PlacementPostUpdate,
    /// Generic — caller-supplied predicate (for ad-hoc test scenarios).
    Custom(&'static str),
}

/// Result of a simulated crash: the service was driven up to the crash point
/// and then forcibly errored. The DB / blob / queue state is whatever the
/// simulated point left behind.
pub struct SimulatedCrash {
    pub at: CrashPoint,
    pub error: anyhow::Error,
    /// Useful diagnostics — e.g. the row id that was being processed.
    pub artifact: Option<serde_json::Value>,
}

/// Drive a builder closure that constructs and exercises the service-under-test
/// up to `point`. Uses the Phase A.2 fault-injection harness to deliver the
/// crash at exactly the named site.
pub async fn simulate_crash_after<F, Fut>(
    point: CrashPoint,
    builder: F,
) -> SimulatedCrash
where
    F: FnOnce(CrashHarness) -> Fut,
    Fut: std::future::Future<Output = anyhow::Result<()>>;

/// Re-run a service's `main_inner(args, env)` (see §Open Questions) to drive
/// recovery sweeps after a simulated crash. Mocks the binary entrypoint without
/// spawning a process. Returns a handle that the test can await/cancel.
pub fn restart_service<F, Fut>(svc_main: F) -> SimulatedRestart
where
    F: FnOnce() -> Fut + Send + 'static,
    Fut: std::future::Future<Output = anyhow::Result<()>> + Send + 'static;

/// Poll until the system reaches the expected recovery state, or timeout.
/// `state` is a closure that queries DB / blob / queue and returns Ok(true)
/// when the invariant holds.
pub async fn assert_recovers_to<F, Fut>(
    state: F,
    timeout: Duration,
) -> anyhow::Result<()>
where
    F: Fn() -> Fut,
    Fut: std::future::Future<Output = anyhow::Result<bool>>;

The CrashHarness passed to the builder closure is a thin façade exposing the same fault injectors from Phase A.2 plus a crash_now(point) method that the test author calls at the chosen semantic point in service code (or, more commonly, the crash injector is armed before exercising the production handler, and the production handler hits the injector mid-execution because the injector is wired into the dependency the handler depends on — DB pool, publisher, store).

D16.2 How the crash points are wired

Each CrashPoint variant maps to a specific Phase A.2 injector configuration. The mapping table is the contract:

CrashPoint Underlying injector Activation

AttachmentMidUpload

ObjectStoreErrorInjector::inject_put_failure_for_keys(prefix) armed before Phase 1 commit

Phase 1 succeeds, Phase 2 put errors

AttachmentPostPut

FaultyAttachmentStore::fail_status_promote_after(N) (one of the §D2.0 sub-pattern (b) store traits — pub trait AttachmentStore { async fn mark_object_present(…​); …​ }; faulty impl errors on the Nth call to mark_object_present)

Status-promote UPDATE errors

OutboxPostPublish

FaultyOutboxStore::fail_mark_published_after(N) (§D2.0 store trait OutboxStore with mark_published(id) faultable)

Post-publish UPDATE errors

InboxPostHandle

FaultyInboxStore::fail_finalize_processed_after(N) (§D2.0 store trait InboxStore with finalize_processed(envelope_id) faultable)

processed_at UPDATE errors

ExchangePostSend

FaultySendJobStore::fail_finalize_sent_after(N) (§D2.0 store trait SendJobStore with finalize_sent(job_id, response) faultable)

Finalize tx errors after partner accepted

PlacementPostUpdate

PublishInTxFailureInjector::fail_for_event_type("placement.ended")

Tx aborts; whole update rolls back

Custom(name)

Caller passes their own armed_injector_fn (any FaultInjector impl)

Caller-defined site

(Per §D2.0, there is no DbErrorInjector::inject_error_after(N) operating on a generic PgPool wrapper. Every DB fault is scoped to a named store trait whose method the production codepath actually depends on; faulty impls of those traits are what the harness wires.)

The contract for each CrashPoint is also encoded as a doc-test in restart.rs — a compile-time check that every variant has a wiring match in the harness setup.

D16.3 Adoption pattern

// services/craig-cases/tests/api/contact_attachments_recovery.rs
#[tokio::test]
async fn upload_recovers_after_post_put_crash() {
    if !devstack_available().await { return; }

    let crash = simulate_crash_after(CrashPoint::AttachmentPostPut, |h| async move {
        let client = h.cases_client();
        client.upload_contact_attachment(case_id(), b"hello world").await?;
        Ok(())
    }).await;

    assert!(crash.error.to_string().contains("simulated"));

    // Now drive recovery — the AttachmentScanner re-runs and promotes the row.
    restart_service(|| async {
        craig_cases::main_inner(test_args(), test_env()).await
    });

    assert_recovers_to(|| async {
        let row = query_attachment_row(crash.artifact.as_ref().unwrap()).await?;
        Ok(row.object_status == "present")
    }, Duration::from_secs(30)).await.unwrap();
}

The pattern is: arm the crash → exercise the handler → restart the service (or just spawn the recovery worker; for many cases a full restart isn’t needed, only the scanner) → assert the post-recovery state.

D16.4 Adoption sites — paired with platform-stab-2 fixes

Each platform-stab-2 fix that ships a recovery sweep gets a paired test under the new module:

Platform-stab-2 fix Crash point exercised Test file

§D2 Outbox FUSL

OutboxPostPublish (publish ok, UPDATE fails → row reclaimable)

crates/craig-mq/tests/outbox_post_publish_crash.rs

§D3 Inbox three-state

InboxPostHandle (handler ok, processed_at UPDATE fails → reprocesses safely)

crates/craig-mq/tests/inbox_post_handle_crash.rs

§D4 Atomic blob+DB+event upload

AttachmentMidUpload + AttachmentPostPut (scanner reaps both flavors)

services/craig-cases/tests/api/attachments_recovery.rs

§D5 Exchange send worker

ExchangePostSend (partner accepted, finalize tx aborted → recovery sweep finishes)

services/craig-exchange/tests/api/exchange_send_recovery.rs

Each test asserts: (a) the crash genuinely aborted the in-flight operation, (b) the DB / blob / queue state matches the documented post-crash invariant, (c) restart drives the recovery sweep to completion, (d) the post-recovery state matches the success-path invariant for the same operation.

D16.5 Documentation

The pattern is documented in .claude/docs/testing.md under "Restart/replay testing" (Phase A.7 already adds the parent section). Convention: every platform-invariant fix that has a recovery sweep ships a paired restart.rs test as the failing-test-first artifact. Reviewer responsibility — a fix without a paired test fails the reviewer-discipline gate from coding-conventions.md Plan Authoring rules.

D16.6 Test description (parametric over CrashPoint)

A meta-test in crates/craig-test-lib/tests/restart_helpers_self.rs: for each CrashPoint variant (excluding Custom), simulate a crash + restart against an in-memory minimal harness, assert the helper observes the crash error and the recovery completes within a 5s timeout. Catches drift between the variant set and the wiring table from §D16.2.

D17. Invariant Sweeper — xtask invariants (Phase A.17 — Step 18)

After cargo xtask test or cargo xtask e2e runs, the dev DB is in some terminal state. Today nothing checks whether that terminal state is consistent. Steps 3-8 of platform-stab-2 introduce a new class of multi-row invariants — "no orphan attachment rows", "no stuck in_flight send jobs", "no `processed_at IS NOT NULL AND failed_at IS NOT NULL`" — that no individual test asserts because the invariants are cross-test. The invariant sweeper turns these into post-test assertions.

D17.0 DB connection scope

CRAIG runs 7 stateful services, each with its own Postgres DB (craig_cases, craig_placement, craig_exchange, craig_financial, craig_reporting, craig_security, craig_rules). The invariant sweeper iterates per-service-DB rather than against a unified read replica or per-service xtask invocation. Rationale:

  • Catalog files at crates/craig-test-lib/sql/invariants/<svc>/*.sql are physically partitioned per service (§D17.3) — one xtask process loads all 8 buckets and visits each.

  • Per-service xtask invocation would multiply CI / pre-push runtime by 8× and complicate aggregation of the pass/fail summary.

  • A unified read replica does not exist in CRAIG’s deployment topology (each service owns its DB; no cross-DB read access by design).

DSN discovery: reuse the existing devstack-bringup environment variables (CRAIG_CASESDATABASE_URL, CRAIG_PLACEMENTDATABASE_URL, …) that cargo xtask dev already populates in .env. The xtask reads them via std::env::var(); if any required DSN is missing the subcommand bails with a clear message ("Run cargo xtask dev start first" or "Set CRAIG_<SVC>__DATABASE_URL"). Use a thin helper enumerate_services() that returns a Vec<ServiceDb { name: &str, dsn: String }> for the 7 stateful services; the helper lives in xtask/src/dsn.rs so it can be reused by other subcommands later.

Connection pool sizing: open a small pool per service (max_connections=2) inside connect_to(&dsn) since the workload is read-only and short-lived; close after each service is done.

D17.1 Output format and contract

cargo xtask invariants is a report-only subcommand by default. It prints a per-invariant pass/fail summary, exits 0 on all-pass, and exits 1 on any-fail. It is wired into cargo xtask test --post-check as opt-in (NOT a pre-push gate per the user directive that all xtask reliability/contracts/mutants/coverage-matrix/invariants/quality-budgets commands ship report-only).

A failed invariant surfaces as:

[invariants] craig-cases :: orphan_pending_attachments
  query: SELECT id, object_status, created_at FROM contact_attachments
         WHERE object_status='pending' AND created_at < now() - INTERVAL '24 hours'
  expected: 0 rows
  got: 3 rows
  sample: [
    { id: 0193…, object_status: "pending", created_at: "2026-05-01T…" },
    …
  ]
  remediation: see plans/platform-stabilization-2.adoc §D4.5 — scanner should
               have reaped this row; investigate scanner logs / DB connectivity.

The remediation text is part of each invariant’s metadata (see §D17.4) — never a free-form panic; always a pointer to the doc that specifies the fix.

D17.2 Implementation

xtask/src/cmd/invariants.rs (new):

pub async fn run(args: InvariantsArgs) -> anyhow::Result<()> {
    let services = enumerate_services(&args)?;        // default: all 8 stateful
    let mut all_passed = true;
    for svc in services {
        let pool = connect_to(&svc.dsn).await?;
        let invariants = load_invariants_for(&svc)?;  // §D17.3
        for inv in invariants {
            let rows = sqlx::query(&inv.sql)
                .fetch_all(&pool)
                .await
                .with_context(|| format!("running invariant {} on {}", inv.name, svc.name))?;
            if rows.is_empty() {
                println!("[invariants] {} :: {} OK", svc.name, inv.name);
            } else {
                all_passed = false;
                report_failure(&svc, &inv, &rows);
            }
        }
    }
    if !all_passed { std::process::exit(1); }
    Ok(())
}

D17.3 Invariant catalog file layout

Each service ships SQL files under crates/craig-test-lib/sql/invariants/<svc>/ (chosen over per-service service trees because the catalog is a test artifact, not a service artifact, and the test-lib already owns cross-service test infra):

crates/craig-test-lib/sql/invariants/
├── craig-cases/
│   ├── orphan_pending_attachments.sql
│   ├── duplicate_active_contact_persons.sql
│   └── audit_row_per_state_mutation.sql
├── craig-exchange/
│   ├── stuck_pending_transactions.sql
│   ├── stuck_in_flight_send_jobs.sql
│   └── orphan_pending_attachments.sql           — ICPC mirror
├── craig-mq/                                     — global cross-service
│   ├── unpublished_outbox_beyond_grace.sql
│   └── inbox_processed_with_failed.sql
├── craig-financial/…
├── craig-placement/…
├── craig-reporting/…
├── craig-rules/…
└── craig-security/…

Each .sql file is a single SELECT that must return zero rows. The first comment line of the file is metadata in TOML-fragment form:

-- name = "orphan_pending_attachments"
-- service = "craig-cases"
-- grace = "24h"
-- remediation = "plans/platform-stabilization-2.adoc#D4.5"
-- severity = "high"
SELECT id, object_status, created_at, file_name
FROM contact_attachments
WHERE object_status = 'pending'
  AND created_at < now() - INTERVAL '24 hours';

xtask reads the comment header at load time, strips it from the SQL it actually executes, and embeds the metadata in the report.

D17.4 The full invariant catalog (initial set)

# Service Name Query (one-line summary)

1

craig-cases / craig-exchange

orphan_pending_attachments

…WHERE object_status='pending' AND created_at < now() - INTERVAL '24 hours'

2

every soft-delete table

duplicate_active_<table>

Per-table: no two rows with the same business key and active=true

3

craig-exchange

stuck_pending_transactions

…WHERE status='pending' AND created_at < now() - INTERVAL '6 hours'

4

craig-exchange

stuck_in_flight_send_jobs

…WHERE status='in_flight' AND next_attempt_at < now() - INTERVAL '60 seconds'

5

craig-mq (per service DB)

unpublished_outbox_beyond_grace

…WHERE published_at IS NULL AND created_at < now() - INTERVAL '5 minutes'

6

craig-mq (per service DB)

inbox_processed_with_failed

…WHERE processed_at IS NOT NULL AND failed_at IS NOT NULL

7

craig-api (per service DB)

idempotency_processing_past_ttl

…WHERE status='processing' AND claim_expires_at < now() - INTERVAL '1 minute'

8

every state-mutating endpoint per service

audit_row_per_state_mutation

Per-entity: every row in <entity> has at least one matching row in audit (per-service catalog of (entity, audit-pattern) pairs).

D17.5 Grace windows as constants

To prevent drift between the SQL grace literal and the semantic constant the codepath uses:

// crates/craig-common/src/constants.rs
use std::time::Duration;

pub const ORPHAN_PENDING_ATTACHMENT_GRACE: Duration = Duration::from_secs(24 * 60 * 60);
pub const OUTBOX_PENDING_GRACE: Duration = Duration::from_secs(5 * 60);
pub const STUCK_EXCHANGE_TX_GRACE: Duration = Duration::from_secs(6 * 60 * 60);
pub const STUCK_SEND_JOB_GRACE: Duration = Duration::from_secs(60);
pub const IDEMPOTENCY_CLAIM_TTL: Duration = Duration::from_secs(30);
pub const IDEMPOTENCY_PROCESSING_GRACE_PAST_TTL: Duration = Duration::from_secs(60);

A unit test in xtask/src/cmd/invariants.rs parses each .sql file, extracts the INTERVAL '<n> <unit>' literal, and asserts it matches the corresponding constant. Drift fails the test.

D17.6 Per-service audit-row catalog (invariant #8)

Built progressively. Each service contributes a crates/craig-test-lib/sql/invariants/<svc>/audit_catalog.sql listing (entity_table, audit_table, mutation_event_types) rows that the invariant joins against. Initial seed: cases, placements, exchange transactions, financial payments, security partners. The remaining ~10 entities ship in W3 hardening passes.

D17.7 Wiring into cargo xtask test --post-check

// xtask/src/cmd/test.rs (extension)
if args.post_check {
    super::invariants::run(InvariantsArgs::default()).await?;
}

--post-check is opt-in. CI does not gate on it. Pre-push does not gate on it. Local developers run it on demand. Quarterly cadence: a scheduled CI job runs cargo xtask invariants --against staging-db for trend tracking.

D17.8 Test description

xtask/tests/invariants.rs: parses every .sql file in the catalog, validates header metadata, asserts every name is unique, asserts every grace literal matches the corresponding craig_common::constants::*_GRACE value, runs each query against an empty test DB and asserts zero rows. Catches catalog drift at unit-test time without devstack.

DB. Phase B — adoption bundle (Step 19)

Phase B of this plan does not duplicate the platform-stab-2 sibling plan’s per-step concurrent/fault/recovery tests. Steps 3 (idempotency), 4 (outbox), 5 (inbox), 6 (attachments), 7 (exchange send), 8 (placement) of platform-stab-2 each ship their own concurrent/fault/recovery test inline, using the platform-stab-2 §D2 minimal helpers. Phase B of this plan adopts the consolidated helpers from Phase A (which extend those minimal helpers) for the additional tests on the user’s "Specific Tests I’d Add First" list that platform-stab-2 does not already cover.

The user’s two-list overlap analysis: the original 10 Phase B tests in the current draft and the 10 "Specific Tests I’d Add First" share 6 tests. The deduplicated list of unique adoption tests is 10 tests + a risk-coverage scorecard. Each one is a paired failing-test-first artifact for either a platform-stab-2 fix (cross-referenced) or a hardening item that has no platform-stab-2 counterpart.

DB.1 The 10 adoption tests

# Service Test Helpers used Sibling fix

1

craig-api

Concurrent same-key Idempotency-Key + same body → exactly one handler side effect

Phase A.1 concurrent_fire; happy / sad / evil / concurrency cells

platform-stab-2 §D1 (Step 3)

2

craig-api

Same key + different body → 422 + does NOT execute the second handler

Phase A.1 concurrent_fire; sad / concurrency cells

platform-stab-2 §D1 (Step 3)

3

craig-api

Idempotency storage failure during finalize → returns the live response body, not empty success

Phase A.2 FaultyFinalizeIdempotencyStore (§D2.0 sub-pattern (b) trait FinalizeIdempotencyStore; faulty impl errors on finalize_succeeded after handler returns) + Phase A.16 simulate_crash_after; sad / fault cells

platform-stab-2 §D1 (Step 3)

4

craig-mq

Two outbox workers cannot publish same row twice

Phase A.1 + Phase A.3 multi-replica devstack OR single-process simulation

platform-stab-2 §D2 (Step 4)

5

craig-mq

Inbox handler failure doesn’t permanently suppress redelivery (under-cap retry)

Phase A.2 fault injector

platform-stab-2 §D3 (Step 5)

6

craig-cases / craig-placement

Object-store write succeeds, DB insert fails → no untracked blobs

Phase A.2 fault injector + Phase A.17 invariant sweep + Phase A.16 restart

platform-stab-2 §D4 (Step 6)

7

craig-exchange

Crash/failure between transaction creation and adapter completion → leaves recoverable state

Phase A.16 restart_service + assert_recovers_to

platform-stab-2 §D5 (Step 7)

8

craig-web

Malformed upstream JSON → renders explicit service error, NOT default empty data

Phase A.2 fault injector for upstream

platform-stab-2 §D7 (Step 9)

9

craig-store

Filename + Content-Disposition fuzzing: quotes, CRLF, Unicode, long names, path tricks

Phase A.6 + Phase A.15 evil_cases_for(EvilCategory::Path) and (EvilCategory::String)

platform-stab-2 §D8.1 (Step 10) — FIXTURES SIDE

10

craig-test-lib + craig-cases

Typed-client pilot for craig-cases — proves Phase A.8 migration runs end-to-end

Phase A.8 typed DTOs

n/a (this plan only)

Each test file lives under the service’s tests/ directory, named <test>_evil.rs / _recovery.rs / _concurrent.rs per its dominant axis. Naming convention matches the regex tagging discussion in §Open Questions — suffixes _happy, _sad, _evil, _concurrent, _fault, _replay are claimed as test-category markers consumed by the coverage-matrix xtask.

DB.2 Risk-coverage scorecard

docs/modules/ROOT/pages/test-coverage-scorecard.adoc is a living artifact tracking, for each of the top-20 critical workflows, which of six test-category cells is covered:

= Test Coverage Scorecard
:description: Risk-based coverage for the top 20 critical workflows.

[cols="3,1,1,1,1,1,1"]
|===
| Workflow | Happy | Sad | Evil | Concurrency | Replay | Fault

| Idempotent POST handler (cross-cutting)
| ✅ | ✅ | ✅ | ✅ | ✅ | ✅
| Outbox publish (cross-cutting)
| ✅ | ✅ | ⏳ | ✅ | ✅ | ✅
| Inbox handler (cross-cutting)
| ✅ | ✅ | ⏳ | ⏳ | ✅ | ✅
| Attachment upload (4 sites)
| ✅ | ✅ | ⏳ | ⏳ | ✅ | ✅
| Exchange send (1 path)
| ✅ | ✅ | ⏳ | ⏳ | ✅ | ✅
| Placement transition guard
| ✅ | ✅ | ⏳ | ✅ | ⏳ | ⏳
| BFF page render under upstream failure
| ✅ | ✅ | ⏳ | ⏳ | ⏳ | ✅
| Person match
| ✅ | ⏳ | ⏳ | ⏳ | ⏳ | ⏳
| Rules evaluation
| ✅ | ⏳ | ⏳ | ⏳ | ⏳ | ⏳
| Public intake submit
| ✅ | ✅ | ✅ | ⏳ | ⏳ | ⏳
| (10 more — see source) | | | | | | |
|===

Legend: ✅ covered, ⏳ planned, ❌ gap.

The 6 cells are the canonical test-category taxonomy enforced via reviewer discipline (Phase A.7 — Step 8). Each Phase B test from §DB.1 ticks one or more cells; the scorecard is updated in the same MR that lands the test (delivery checklist gate).

The scorecard is generated and hand-maintained: a script-section at the top is auto-populated by cargo xtask coverage-matrix --report --emit-scorecard (Phase A.11/A.12 deliverable) by reading test-name-suffix tags; the workflow rows are hand-curated from the operational risk register. Drift between the auto-section and the curated section flags a stale scorecard.

DB.3 Step 19 deliverable list

  1. 10 adoption test files (one per row of §DB.1).

  2. docs/modules/ROOT/pages/test-coverage-scorecard.adoc (new — top-20 workflows × 6-cell grid).

  3. cargo xtask coverage-matrix --emit-scorecard flag (extension to A.11 deliverable).

  4. nav.adoc entry under Architecture & Design pointing at scorecard.

  5. CHANGELOG entry per landed test + a single entry for the scorecard introduction.

Steps

Step 1: Plan adoc + nav.adoc Active + CHANGELOG entry + 9 new GitLab issues

Files:

  • docs/modules/ROOT/pages/plans/test-framework-hardening.adoc (new — content of this entire plan)

  • docs/modules/ROOT/pages/plans/platform-stabilization-2.adoc (sibling — already written from snappy-napping-puzzle.md)

  • docs/modules/ROOT/nav.adoc (Active section: add both plans)

  • CHANGELOG.adoc (single combined entry under == Unreleased covering both plan adoptions)

Implementation:

  1. Replace the existing thin test-framework-hardening.adoc (137 lines) with this plan’s full content (~5100 lines). The thin draft on the current branch (chore/platform-stab-2-and-test-framework-plans) was an earlier scratch that the plan-mode review supersedes. Use a single git operation: git rm then re-add the new content, OR overwrite-in-place. Either way, the diff is replace file contents.

  2. Verify platform-stabilization-2.adoc is the sibling-plan content already written from snappy-napping-puzzle.md (the prior plan-mode review). No edits needed in this MR.

  3. Update nav.adoc Active section to include both plans (already done in this branch’s working tree; verify still in place).

  4. Append the combined CHANGELOG entry under == Unreleased covering both plan adoptions (already done in this branch’s working tree; verify still in place).

  5. File 9 new GitLab issues under epic &22 for Phase A.9–A.17 not covered by the existing #283–#291. Each issue title matches the Status-table description; body cross-references the corresponding §D section. Filed via glab issue create --epic 22 --title "…" --description "…" --label "P1-high,test-framework" (one per issue):

    • A.9 — proptest scaffolding + first 7 properties

    • A.10 — cargo-mutants smoke on 5 critical crates

    • A.11 — xtask subcommands (reliability / contracts / mutants / coverage-matrix / invariants / quality-budgets, REPORT-only)

    • A.12 — Code-quality budget warnings + lock-file baseline

    • A.13 — Coverage Matrix tooling (xtask coverage-matrix, REPORT-only)

    • A.14 — DB Constraint Evil Tests (hand-curated harness)

    • A.15 — Central Evil Input Corpus consolidation

    • A.16 — Restart/Replay light helpers

    • A.17 — Invariant Sweeper (xtask invariants)

No code, no migration, no tests. Pure docs + issue-tracker bookkeeping.

Verification:

  1. glab epic view 22 --with-issues — confirms 9 new issues are linked under &22 alongside the existing #283–#291 (total 18 issues + 1 Phase B bundle = 19)

  2. Antora docs render the new .adoc without errors (verifiable via local Antora build or cargo xtask api-docs)

  3. nav.adoc Active section visibly shows both plans (grep -A2 "** Active" docs/modules/ROOT/nav.adoc shows both xrefs)

  4. CHANGELOG == Unreleased section lists the combined entry as the topmost entry

  5. Pre-push gate passes — no Rust code touched

Cross-reference: top-matter Status table.

Step 2: Phase A.1 — concurrent_fire helper consolidation

Files:

  • crates/craig-test-lib/src/concurrent.rs (extend in place — adds concurrent_fire_collect, concurrent_fire_collect_ordered, concurrent_fire_synchronized (auto-barrier), concurrent_fire_with_barrier (caller-managed sync point), concurrent_fire_until_first_success, concurrent_fire_first_n, ConcurrentRunReport, TaskOutcome, TaskResult)

  • crates/craig-test-lib/src/lib.rs (re-exports + deprecated concurrent_fire alias)

  • crates/craig-test-lib/Cargo.toml (no new deps; tokio::sync::Barrier is in the existing tokio workspace dep)

  • crates/craig-test-lib/tests/concurrent.rs (extend with the 12 smoke tests from §D1.4)

  • .claude/docs/testing.md (update the Failure-path testing helpers section added by epic &21 Step 2 with the 4 new helpers + when-to-use table)

  • (No service-side or migration changes; pure test infra.)

Implementation:

  1. Land §D1.2’s full surface in crates/craig-test-lib/src/concurrent.rs, alphabetized, with rustdoc on every public item including the when-to-use guidance from §D1.2.

  2. Move existing minimal concurrent_fire body into concurrent_fire_collect. Add pub use self::concurrent_fire_collect as concurrent_fire; with #[deprecated(note = "renamed to concurrent_fire_collect; will be removed in next minor version")].

  3. Implement ConcurrentRunReport, TaskOutcome, TaskResult; into_results() collects into Vec on Ok and Vec<String> on any panic; ok_count() / panic_count() are simple iteration.

  4. concurrent_fire_collect_ordered: use tokio::task::JoinSet; record Instant::now() at task entry; sort outcomes by started_at before returning.

  5. concurrent_fire_with_barrier: build Arc::new(Barrier::new(n)); pass to each spawned future; started_at is recorded after barrier.wait().await. The caller’s builder receives the barrier and is expected to call .wait().await at its semantic sync point.

  6. concurrent_fire_synchronized: build Arc::new(Barrier::new(n)) internally; spawn each task as a wrapper that calls barrier.wait().await before invoking the caller’s builder(i) closure. Caller never sees the barrier; cannot leak the synchronization. Recommended default for "all start together" tests.

  7. concurrent_fire_until_first_success + concurrent_fire_first_n: use JoinSet::join_next; abort on threshold via JoinSet::abort_all; collect errors from naturally-completed Err results only.

  8. Panic capture per §D1.3: JoinError::is_panic()JoinError::into_panic()Any::downcast_ref::<&str> then Any::downcast_ref::<String> then fallback message.

  9. Add the 12 smoke tests from §D1.4 to tests/concurrent.rs. Existing &21 Step 2 tests remain green (calling deprecated alias).

  10. Update .claude/docs/testing.md Failure-path section: list 5 helpers + when-to-use cell per row + one-line code sketch per helper.

  11. Errata entry in this plan: "Existing concurrent_fire renamed to concurrent_fire_collect; back-compat alias retained for one minor version. Reason: consolidate naming family for 4 new variants."

Verification:

  1. cargo nextest run -p craig-test-lib --test concurrent — 12 smoke tests pass (the 1 from &21 Step 2 against the alias, plus 11 new).

  2. cargo clippy -p craig-test-lib --tests --locked — -D warnings — clean, including #[deprecated] warning suppression in the alias callsite.

  3. cargo build --workspace --locked — confirms no caller of concurrent_fire (alias) breaks; confirms one deprecation warning fires per existing call site so the migration scope is visible.

  4. Manual: count call sites via Grep "concurrent_fire\b" --type rust — expected to match the &21 Step 2 idempotency / outbox / inbox tests; each call site is candidate for a follow-up rename in epic &21’s MR series.

Cross-reference: §D1.

Step 3: Phase A.2 — Fault-injection harness consolidation

Files:

  • crates/craig-test-lib/src/fault/ (new module directory) with mod.rs, cipher.rs, db.rs, rabbit.rs, object_store.rs, publish_in_tx.rs (migrated from epic &21 Step 2 fault_injection.rs), latency.rs, flap.rs, backpressure.rs, scenarios.rs (new)

  • crates/craig-test-lib/src/fault_injection.rs (replaced with re-export shim per §D2.1)

  • crates/craig-test-lib/src/lib.rs (add pub mod fault; alongside existing pub mod fault_injection;)

  • crates/craig-test-lib/Cargo.toml (no new deps; uses existing tokio, lapin, sqlx workspace deps)

  • crates/craig-test-lib/tests/fault_latency.rs, crates/craig-test-lib/tests/fault_flap.rs, crates/craig-test-lib/tests/fault_backpressure.rs, crates/craig-test-lib/tests/fault_scenarios.rs (new smoke tests)

  • crates/craig-test-lib/tests/fault_injection.rs (existing &21 Step 2 smoke tests; remain green via the shim)

  • .claude/docs/testing.md (extend Failure-path section with the 6 injectors + named-scenarios registry + FaultInjector trait recipe; epic &21 Step 2 added the initial 5; this step extends)

Implementation:

  1. Migration step — move every existing injector body from fault_injection.rs into its sibling under fault/. Source diff is pure relocation; no behavior change; all &21 Step 2 tests should still pass against the shim. Run cargo test -p craig-test-lib --test fault_injection between this step and the next as the fast-feedback gate.

  2. Define FaultInjector trait + Attempt struct + ScenarioGuard per §D2.2 in fault/mod.rs. Implement on each migrated injector by promoting their existing internal state to fit the trait shape (each already has counter / Mutex internals; the trait just formalizes the access pattern).

  3. Implement LatencyInjector per §D2.3. Compose-style: it holds an inner I: FaultInjector and forwards inject / recorded_attempts unchanged, but inserts tokio::time::sleep before/after each wrapped call.

  4. Implement IntermittentFlapInjector per §D2.4. The pattern is a Vec<bool>; the AtomicUsize counter is fetch_add(1) modulo the pattern length; each wrapped call indexes the pattern.

  5. Implement ConsumerBackpressureInjector per §D2.5. The handle_delivery helper wraps a lapin::message::Delivery handler with tokio::time::sleep (ack_delay) + optional drop. recorded_attempts() records each delivery with injected = ack_delay > 0 || dropped.

  6. Build out scenarios.rs per §D2.6 — db::*, rmq::*, cipher::*, object_store::*, publish_in_tx::*, backpressure::* modules with named constructors. Each named constructor returns the right Scenario enum variant. Test with cargo doc -p craig-test-lib --no-deps to verify rustdoc rendering.

  7. Add the smoke tests from §D2.7 — 9 named cases across 4 new test files (fault_latency, fault_flap, fault_backpressure, fault_scenarios). The existing fault_injection.rs test file remains untouched.

  8. Drop-assertion path: implement Path A/B/C tests from §D2.7’s closing block. Use std::panic::catch_unwind to verify behavior in Path B; use std::panic::panic_any for Path C; ensure std::thread::panicking() check survives.

  9. Update .claude/docs/testing.md. Sections to add: "FaultInjector trait", "Latency / Flap / Backpressure wrappers", "Named scenarios in `craig_test_lib::fault::scenarios`", and a "Picking the right injector" decision table.

Verification:

  1. cargo nextest run -p craig-test-lib --test fault_injection — &21 Step 2 tests pass via shim.

  2. cargo nextest run -p craig-test-lib --test fault_latency — 1 case.

  3. cargo nextest run -p craig-test-lib --test fault_flap — 2 cases.

  4. cargo nextest run -p craig-test-lib --test fault_backpressure — 2 cases.

  5. cargo nextest run -p craig-test-lib --test fault_scenarios — 4 cases including drop-assertion Path A/B/C.

  6. cargo clippy -p craig-test-lib --tests --locked — -D warnings — clean.

  7. cargo doc -p craig-test-lib --no-deps — generates without warnings; manual eyeball of fault::scenarios module page confirms named scenarios are documented with when-to-use.

  8. Manual: Grep "use craig_test_lib::fault_injection" — every &21 Step 2 caller still imports cleanly (proves the shim is tight).

Cross-reference: §D2.

Step 4: Phase A.3 — Multi-replica devstack profile

Files:

  • xtask/src/cmd/dev.rs (extend DevCommand::Start with --replicas per §D3.2; add ReplicaSpec + parse_replicas)

  • xtask/src/devstack.rs (thread replicas: &[ReplicaSpec] through ensure_ready + bring_up per §D3.3; add replica_names
    wait_for_service_health_all_replicas per §D3.6)

  • xtask/src/docker.rs (add compose_with_overrides helper that takes an &[&str] of -f <file> args; compose and friends delegate)

  • docker-compose.scale.yml (new, root of repo, per §D3.4 — 8 scalable services with !override floating-port lists)

  • crates/craig-test-lib/src/multi_replica.rs (new — MultiReplicaCluster + builder per §D3.7)

  • crates/craig-test-lib/src/lib.rs (add pub mod multi_replica;
    re-export MultiReplicaCluster)

  • crates/craig-test-lib/Cargo.toml (no new deps — uses tokio
    existing process-shell pattern)

  • crates/craig-test-lib/tests/multi_replica_idempotency.rs (new — the §D3.8 verification test)

  • .claude/docs/local-dev.md (new section: "Multi-replica devstack")

  • .claude/docs/testing.md (update with MultiReplicaCluster recipe + when-to-use)

  • docs/modules/ROOT/pages/dev/devstack.adoc (or equivalent developer-guide page) — describe --replicas flag

Implementation:

  1. Add ReplicaSpec + parse_replicas to xtask/src/cmd/dev.rs per §D3.2. parse_replicas is unit-testable in isolation; add 4 #[cfg(test)] cases (happy path, missing =, unknown service, zero count).

  2. Thread replicas: &[ReplicaSpec] through cmd::dev::run, devstack::ensure_ready, devstack::bring_up, devstack::reload, devstack::reseed. Default at every other entry point (Stop, Restart, Clean, etc.) is &[].

  3. In devstack::bring_up: when replicas is non-empty, splice -f docker-compose.scale.yml into the compose invocation (docker compose -f docker-compose.yml -f docker-compose.scale.yml up -d --build --scale craig-cases=2 …). When empty, the existing docker compose up -d --build path runs unchanged.

  4. Implement replica_names + wait_for_service_health_all_replicas per §D3.6. Update wait_for_health (the all-services loop) to call the new replica-aware helper. Default-scale-1 case still returns one replica name and terminates immediately on health.

  5. Author docker-compose.scale.yml per §D3.4. Verify the !override tag behavior against the compose version pinned in .gitlab-ci.yml — local test: docker compose -f docker-compose.yml -f docker-compose.scale.yml config | grep -A2 'craig-cases:' | grep ports should show only the floating port, not the merged list.

  6. Implement MultiReplicaCluster + builder per §D3.7. Sub-implementation detail: start() chooses an isolated compose project name (e.g., craig-test-mrc-<uuid>) so the dedicated cluster does not collide with the developer’s persistent default-name devstack. replica_urls iterates docker compose -p <project> port --index N craig-cases 8002 for N in 1..=count, parsing each output line as a single host:port. The --index N flag is mandatory: without it, docker compose port craig-cases 8002 returns only the FIRST replica’s host port (a silent ambiguity that would make a 2-replica test pass against a 1-replica subset).

  7. Author the §D3.8 verification test. Covers Path A (concurrent fire across replicas), Path B (one handler-side effect lands), Path C (both replicas served traffic via the X-Served-By header trick). Note: the test is gated by if !devstack_available().await { return; } style guard from craig-test-lib, but it spins its own dedicated cluster — the global devstack is not affected and the test can run in parallel with other integration tests.

  8. Update .claude/docs/local-dev.md with a new section: how to invoke --replicas, the docker-compose.scale.yml rationale, the ephemeral-port discovery recipe (docker compose port), and the current limitations (postgres/keycloak/garage stay scale=1; craig-web’s host port pins to first replica).

  9. Update .claude/docs/testing.md with MultiReplicaCluster usage: "Use this when …" table + 1 minimal code snippet.

Verification:

  1. cargo build -p xtask --locked — clean build with new args.

  2. cargo nextest run -p xtask --libparse_replicas unit tests pass.

  3. cargo xtask dev start --replicas craig-cases=2,craig-exchange=2 on a clean dev box — both services come up healthy with 2 replicas each; cargo xtask dev status shows replica names; ephemeral port range shows two distinct host ports per scaled service.

  4. cargo xtask dev start without --replicas — default scale=1 behavior, host ports 8002:8002 etc. unchanged. Pre-push hook unaffected.

  5. cargo nextest run -p craig-test-lib --test multi_replica_idempotency — Path A/B/C all green. Test runtime ≤ 90s (cluster boot dominates). Also passes when run after cargo xtask dev restart (cluster boots in isolation regardless of global devstack state).

  6. Manual: docker compose -f docker-compose.yml -f docker-compose.scale.yml config — render output shows scalable services with floating ports and shared-state services unchanged.

  7. Manual: kill one replica of craig-cases mid-test (docker compose stop craig-cases-2); rerun the verification test; assert it still passes (the surviving replica handles all 5 POSTs; atomic-claim still wins). This proves the test asserts correctness, not just load distribution.

Cross-reference: §D3.

Step 5: Phase A.4 — State-machine matrix-testing pattern + Payment first adoption

Files:

  • crates/craig-test-lib/src/state_machine_matrix.rs (new)

  • crates/craig-test-lib/src/lib.rs (pub mod state_machine_matrix;)

  • crates/craig-test-lib/tests/state_machine_matrix.rs (new — helper unit tests per §D4.6 items 1–3)

  • services/craig-financial/tests/transitions_matrix.rs (new — first adoption per §D4.3)

  • .claude/docs/testing.md (testing.md gets the matrix helper added to the §D7.3 cheat-sheet — folded into Step 8 unless that step is in flight)

    1. Implement ValidTransition, Outcome, MatrixReport, assert_only_valid_transitions_accepted per §D4.2. std::panic::catch_unwind wraps each predicate call so a panic in one pair does not abort the suite.

    2. Implement MatrixReport::assert_clean with a multi-line message listing every mismatch + every panic.

    3. Add the three helper unit tests per §D4.6.

    4. Land the financial-payment matrix test per §D4.3, omitting (Issued, Cleared) from VALID first: confirm assertion fires; commit the failing-test screenshot/log into the MR description; restore the omission; confirm green; merge.

    5. Cross-link in §D4.5 table → file an issue per remaining state machine (15 issues) under the parent epic. Each is a one-file MR following the same pattern. Phase B adoption work, not blocking this step.

    6. Run: cargo nextest run -p craig-test-lib --test state_machine_matrix — green; cargo nextest run -p craig-financial --test transitions_matrix — green.

Cross-reference: §D4.

Step 6: Phase A.5 — JWT mutation library

Files:

  • crates/craig-test-lib/src/jwt_mutation.rs (new — per §D5.3 + §D5.4)

  • crates/craig-test-lib/src/lib.rs (pub mod jwt_mutation;)

  • crates/craig-test-lib/Cargo.toml (add jsonwebtoken dev-dep if absent at workspace-pin level; today craig-auth carries it — re-export rather than duplicate)

  • crates/craig-auth/src/jwks.rs (additive: pub mod test_fixtures gated on feature = "test-fixtures" per §D5.2)

  • crates/craig-auth/Cargo.toml (add test-fixtures feature; craig-test-lib declares the feature in its dev-dep edge)

  • crates/craig-test-lib/tests/jwt_mutation_smoke.rs (new — per §D5.6)

    1. Implement ClaimsBuilder per §D5.3. Defaults: now-relative iat/exp, iss = TEST_ISSUER, kid = "test-kid" (matches the existing crates/craig-auth/src/middleware.rs:108 fixture).

    2. Implement mint(&self) → String using jsonwebtoken::encode with EncodingKey::from_rsa_der from §D5.2. The Algorithm::None "alg=none" attack vector lives in a separate helper with_unsigned() that manually constructs the compact JWT (base64url header + base64url payload + empty signature segment), since jsonwebtoken 10.3 does NOT expose Algorithm::None in its enum.

    3. Implement mutation_matrix per §D5.4 — single-axis variants only; combinatoric explosion is out of scope.

    4. Add the smoke test from §D5.6 (baseline_token_validates + every_single_axis_mutation_is_rejected_by_craig_cases_validator).

    5. Open the 8 per-service follow-up issues (one per service that mounts auth_middleware) under epic &22 Phase B. Each lands a tests/api/auth_strictness.rs consuming mutation_matrix(ClaimsBuilder::baseline("<svc>")). Not blocking this step’s completion.

    6. Run: cargo nextest run -p craig-test-lib --test jwt_mutation_smoke — green; cargo nextest run -p craig-auth — green (existing tests survive the test-fixtures-feature reshuffle).

Cross-reference: §D5.

Step 7: Phase A.6 — Magic-byte upload-spoofing fixtures

Files:

  • crates/craig-test-lib/fixtures/upload_spoofing/ (new dir, 10 binary files per §D6.1)

  • crates/craig-test-lib/fixtures/upload_spoofing/README.md (new — SPDX header + table mapping each .bin to its description per §D6.1)

  • crates/craig-test-lib/src/upload_fixtures.rs (new — per §D6.2)

  • crates/craig-test-lib/src/filename_attack.rs (new — per §D6.3)

  • crates/craig-test-lib/src/lib.rs (two new pub mod lines)

  • crates/craig-test-lib/tests/upload_fixtures_smoke.rs (new — per §D6.4)

  • crates/craig-test-lib/tests/filename_attack_smoke.rs (new — round-trip table-driven test per §D6.4 item 3)

    1. Mint each spoof fixture per §D6.1 byte-by-byte. Use python3 -c 'import sys; sys.stdout.buffer.write(bytes.fromhex("…​"))' > pe_disguised_as_pdf.bin (or the equivalent xxd-roundtrip). Verify each .bin size matches §D6.1; verify each magic-byte prefix with infer::get(…​) in a one-shot script before commit.

    2. Mint the 4 valid-companion fixtures (valid_pdf.bin, valid_jpeg.bin, valid_png.bin, valid_csv.bin).

    3. Implement FixtureName, fixture(), SpoofFixture, enumerate_spoofs, enumerate_valid per §D6.2. Use include_bytes! for the static byte slices.

    4. Implement filename_attack::CASES per §D6.3. The 12-row table is hand-frozen; subsequent additions go in the same file alphabetically by name.

    5. Add upload_fixtures_smoke.rs per §D6.4 items 1–2 — both run as workspace unit tests (no devstack needed). Note: item 1 currently depends on platform-stab-2 §D8.1 having landed; if §D8.1 has not landed, the test asserts on the (existing) MIME-allowlist arm only and gets a // TODO(epic-21-D8.1) comment near the magic-byte assertion. Tracked.

    6. Add filename_attack_smoke.rs per §D6.4 item 3. The expected_disposition half is gated on platform-stab-2 §D8.1’s RFC 5987 encoder shipping; same // TODO pattern applies if not yet landed.

    7. Failing-test-first protocol: deliberately omit truncated_pdf_first_8_bytes.bin from enumerate_spoofs’s array, observe the smoke test’s coverage drop in the assertion (the test asserts `enumerate_spoofs().count() == 6); restore; merge.

Cross-reference: §D6 (and platform-stab-2 §D8.1 for the validator side).

Step 8: Phase A.7 — testing.md "Failure-path testing" + Test Categories taxonomy + Process rule

Files:

  • .claude/docs/testing.md (4 new sections per §D7.1, §D7.2, §D7.3, §D7.4 — and §D7.6 if the budget shape lands in this MR; the budget script is deferred per §D7.6)

  • .claude/docs/delivery-protocol.md (one-paragraph audit-rule pointer per §D7.5)

  • CONTRIBUTING.adoc (new "Failure-path testing" section per §D7.5)

    1. Insert §D7.1 "Test Categories" section into .claude/docs/testing.md after "Mandatory Rules". Eight categories, exact verbatim text. Cross-link each to its first example file.

    2. Insert §D7.2 "File-naming convention for test categories" section immediately after §D7.1.

    3. Insert §D7.3 "Failure-path testing helpers" section after §D7.2. Include one-line example invocation per row (grep-able).

    4. Insert §D7.4 "Process rule" section immediately after §D7.3, marked as a [NOTE] block in AsciiDoc-style emphasis (testing.md is markdown, so equivalent: a > Process rule blockquote).

    5. Insert §D7.6 "Quality budgets — available, not enforced" section after §D7.4. Mark every command as future (italics in the doc) so a contextless agent doesn’t try to invoke them.

    6. Update .claude/docs/delivery-protocol.md "Plan completion audit" section per §D7.5 — single paragraph.

    7. Update CONTRIBUTING.adoc per §D7.5 — single section, one paragraph, link to testing.md anchor.

    8. No code, no tests, no migration. Pre-push gate runs as usual; only Antora docs need to render cleanly. Run: cargo xtask api-docs --check (or local Antora build).

Cross-reference: §D7.

Step 9: Phase A.8 — Typed DTOs across 8 test clients (8 MRs, this step lands MR 1)

This step is the long-running migration umbrella. Step 9 in this plan lands MR 1 (CasesClient pilot). Each subsequent MR is filed and tracked under epic &22 but does not block the parent plan’s "Step 9" status from flipping to Complete — Complete here means "the migration pattern is proven on CasesClient and the remaining 7 issues are filed."

The contracts-crate pattern is mandatory. Per §D8.2, the prior draft had craig-test-lib dev-depending on craig-cases directly, which would create a cycle with the existing services/craig-cases/Cargo.toml:34 dev-dep on craig-test-lib. This step instantiates the per-service contracts-crate pattern; do not dev-dep the service crate from the test-lib.

Files (MR 1 — CasesClient):

  • crates/craig-cases-contracts/Cargo.toml (new — leaf crate, no deps on services/* or craig-test-lib)

  • crates/craig-cases-contracts/src/lib.rs (new — module per api submodule: pub mod cases; pub mod persons; pub mod referrals; …)

  • crates/craig-cases-contracts/src/cases.rs (new — DTOs moved from services/craig-cases/src/api/cases.rs)

  • crates/craig-cases-contracts/src/persons.rs (new — same)

  • crates/craig-cases-contracts/src/referrals.rs (new — same)

  • Cargo.toml workspace [workspace.dependencies] — add craig-cases-contracts = { path = "crates/craig-cases-contracts" }

  • services/craig-cases/Cargo.toml — add craig-cases-contracts = { workspace = true } to [dependencies]

  • services/craig-cases/src/api/cases.rs — delete the moved DTO definitions, replace with pub use craig_cases_contracts::cases::*; so handler code at crate::api::cases::CreateCaseRequest keeps working unchanged

  • services/craig-cases/src/api/persons.rs (same shape)

  • services/craig-cases/src/api/referrals.rs (same shape)

  • crates/craig-test-lib/Cargo.toml — add craig-cases-contracts = { workspace = true } to [dev-dependencies] (NOT craig-cases — that would reintroduce the cycle the design rejects)

  • crates/craig-test-lib/src/clients/cases.rs (~60-method rewrite per §D8.4)

  • crates/craig-test-lib/tests/typed/cases_client.rs (new — Contract round-trip per §D8.6)

  • crates/craig-test-lib/tests/typed.rs (module index file declaring mod cases_client;)

  • deny.toml — add a bans.deny entry preventing any future PR from adding craig-cases as a craig-test-lib dependency (cycle guard)

    1. Contracts crate first: create crates/craig-cases-contracts/ as a leaf crate. Contents: pure data types (request/response DTOs). Add only the deps the DTOs need (serde, chrono, uuid, utoipa, craig-reference where shared enums apply). Do not depend on services/craig-cases or craig-test-lib.

    2. Move DTOs: move every Create*Request, Update*Request, Response, paged-list response shape from services/craig-cases/src/api/{cases,persons,referrals,…​}.rs into the matching crates/craig-cases-contracts/src/<module>.rs. Each DTO retains its #[derive(Deserialize, Serialize, Debug, Clone, PartialEq, ToSchema)] derives. The service-internal api::::* modules then pub use craig_cases_contracts::*::*; so existing handler code compiles unchanged.

    3. Service-side audits:

      1. Every DTO touched by a test-client method needs #[derive(Serialize)] (today most are deserialize-only); add.

      2. Many response types are constructed inline from store rows and never named at the API boundary; lift those into named structs (PersonResponse, etc.) and move to the contracts crate.

      3. Rule of thumb: every test-client method’s response type must be a #[derive(Deserialize, Serialize, Debug, Clone, PartialEq)] named struct in the contracts crate; anonymous JSON shapes get a name during this step.

    4. Test-client-side: rewrite crates/craig-test-lib/src/clients/cases.rs per §D8.4. use craig_cases_contracts::cases::*; use craig_cases_contracts::persons::*; … — note the _contracts suffix on the crate. Methods that return non-JSON (e.g. list_persons returns paged metadata + data: Vec<Person>) get a ListPersonsResponse { data: Vec<PersonResponse>, total: u64, … } DTO in the contracts crate first.

    5. Test-callers: every tests/api/*.rs file under services/craig-cases/tests/ currently using the JSON-y forms breaks at compile. Rewrite each call-site: json!({"first_name": "Foo"})CreatePersonRequest { first_name: "Foo".into(), … }. Estimated touch: ~30 test files (count via Grep "cases_client\." services/craig-cases/tests/).

    6. Smoke: add tests/typed/cases_client.rs per §D8.6. The first round-trip test is the canary for the migration — failing here means the service-side DTOs and the test-client DTOs disagree.

    7. Quality budget: §D7.6 budget tracker checks crates/craig-test-lib/src/clients/cases.rs reports zero serde_json::Value hits. Warning, not failure (per §D8.5).

    8. File 7 follow-up issues under epic &22 Phase A.8 for the remaining 7 clients per the §D8.3 order (Placement → Exchange → Financial → Reporting → Rules → Security → Intake). Each issue links its own §D8.3 row, its own MR branch (feat/test-framework-typed-clients-<svc>), and the same MR template as MR 1.

    9. Run: cargo nextest run -p craig-cases — every call-site recompiles + passes; cargo nextest run -p craig-test-lib --test typed — green; cargo xtask budget --metric value-count (when shipped) — clients/cases.rs reports 0.

Cross-reference: §D8.

Step 10: proptest scaffolding + first 7 properties

Files:

  1. Cargo.toml (workspace) — add proptest = "1.7" to [workspace.dependencies]

  2. crates/craig-store/Cargo.toml — add proptest = { workspace = true } under [dev-dependencies]

  3. crates/craig-api/Cargo.toml — same

  4. crates/craig-signing/Cargo.toml — same

  5. crates/craig-reference/Cargo.toml — same

  6. services/craig-financial/Cargo.toml — same

  7. services/craig-financial/tests/properties.rs (new) + tests/properties/payment_status.rs (new)

  8. crates/craig-api/tests/properties.rs (new) + tests/properties/{pagination,idempotency_fingerprint}.rs (new)

  9. crates/craig-store/tests/properties.rs (new) + tests/properties/{sanitize_filename,content_disposition}.rs (new)

  10. crates/craig-signing/tests/properties.rs (new) + tests/properties/canonical_json.rs (new)

  11. crates/craig-reference/tests/properties.rs (new) + tests/properties/parse_never_panics.rs (new)

  12. .claude/docs/testing.md — append "Property tests" subsection under the Step 8 "Failure-path testing" section

Cross-reference: §D9.

  1. Workspace Cargo.toml: add proptest dep verbatim (verify version with cargo search proptest --limit 1).

  2. Per crate listed: add [dev-dependencies] proptest = { workspace = true }.

  3. Author the seven property files per §D9.3–§D9.9. Each carries the verbatim Strategy + invariant block + the FAIL-FIRST and PROPTEST_BUDGET_MS comments.

  4. Add the tests/properties.rs module-index file per crate (mirrors tests/api.rs pattern).

  5. Append the "Property tests" subsection to testing.md per §D9.2.

  6. Run cargo nextest run --workspace -E 'test(::properties::)'; assert all pass and total wall-time <2s.

  7. Fail-first verification (per property): introduce one-character bug in the unit-under-test; run prop; record shrunk counter-example; revert.

Step 11: cargo-mutants smoke on 5 critical crates

Files:

  1. .cargo/mutants.toml (new) — workspace mutants config per §D10.1

  2. xtask/mutants-baseline.toml (new) — committed missed-mutant catalogue, populated on first run

  3. CONTRIBUTING.adoc — new "Optional dev tools" subsection with cargo-install line

  4. .claude/docs/testing.md — new "Mutation testing" subsection cross-referencing the xtask subcommand

Cross-reference: §D10. (Subcommand wiring lives in Step 12 / §D11.3.)

  1. Author .cargo/mutants.toml per §D10.1. Verify version compatibility with cargo install cargo-mutants --version "^25.0" --locked.

  2. Document install line in CONTRIBUTING under "Optional dev tools".

  3. Run cargo xtask mutants --smoke (after Step 12 lands the subcommand) on a clean working tree.

  4. Inspect each crate’s target/mutants-<crate>/outcomes.json; for each "missed" mutant, append an entry to xtask/mutants-baseline.toml:

    [[missed]]
    crate       = "crates/craig-mq"
    file        = "src/inbox.rs"
    line        = 142
    mutant_kind = "replace == with !="
    reason      = "tracked: inbox-retry coverage gap, see #28X"
  5. Commit baseline.

  6. Append testing.md "Mutation testing" subsection: tool purpose, when to run, what surviving mutants mean, how to interpret the baseline.

  7. Fail-first verification: delete one prop_assert! from the §D9 properties; rerun smoke; observe the corresponding mutant survive that previously didn’t. Document this demonstration in the MR description, then revert.

Step 12: xtask subcommands: reliability + contracts + mutants + coverage-matrix + invariants

Files:

  1. xtask/src/cmd/mod.rs — register six new subcommands per §D11.1

  2. xtask/src/main.rs — match arms for the six new commands

  3. xtask/src/cmd/reliability.rs (new) — §D11.2

  4. xtask/src/cmd/contracts.rs (new) — §D11.4

  5. xtask/src/cmd/mutants.rs (new) — §D11.3

  6. xtask/src/cmd/coverage_matrix.rs (new, stub) — §D11.5; full impl in Step 14

  7. xtask/src/cmd/invariants.rs (new) — §D11.6

  8. xtask/src/cmd/quality_budgets.rs (new, stub) — §D11.1 registration only; full impl in Step 13

  9. CONTRIBUTING.adoc — "Optional reliability tooling" subsection §D11.7

  10. README.adoc — one-paragraph note in Development section

  11. .claude/docs/testing.md — "Local subcommands" subsection §D11.7

  12. xtask/tests/subcommand_smoke.rs (new) — per-subcommand smoke per §D11.8

Cross-reference: §D11.

  1. Edit cmd/mod.rs per §D11.1 — registers six new modules and six new Command enum variants.

  2. Author cmd/reliability.rs verbatim from §D11.2.

  3. Author cmd/contracts.rs verbatim from §D11.4.

  4. Author cmd/mutants.rs verbatim from §D11.3 (depends on Step 11 baseline file).

  5. Author cmd/coverage_matrix.rs per §D11.5 — stub that delegates to the §D13 implementation (which lands in Step 14).

  6. Author cmd/invariants.rs verbatim from §D11.6.

  7. Author cmd/quality_budgets.rs per §D11.1 registration; full impl lands in Step 13.

  8. Edit main.rs to add six match arms.

  9. Append "Optional reliability tooling" subsection to CONTRIBUTING.adoc enumerating each subcommand + the entry banner each prints.

  10. Add one-paragraph Development-section note to README.adoc.

  11. Append "Local subcommands" subsection to testing.md cross-referencing each subcommand and noting they’re opt-in (not pre-push gates).

  12. Author xtask/tests/subcommand_smoke.rs per §D11.8 — three integration tests per subcommand (passing, deliberately-failed, banner-text).

  13. Verification: run each subcommand once on a clean tree; observe the exact entry-banner text from §D11.

Step 13: Code-quality budget warnings

Files:

  1. xtask/src/cmd/quality_budgets.rs — full implementation per §D12.4

  2. xtask/quality-budgets.toml (new) — config per §D12.2

  3. xtask/quality-budgets.lock (new) — initial baseline per §D12.3

  4. xtask/Cargo.toml — add walkdir = "2", regex = "1", syn = "2" to [dependencies] if not already present

  5. CONTRIBUTING.adoc — new "Quality budgets" subsection

  6. xtask/tests/quality_budgets_test.rs (new) — fixtures + snapshot per §D12.6

Cross-reference: §D12.

  1. Replace the Step 12 stub in cmd/quality_budgets.rs with the full implementation per §D12.4. Each run_b<N>_* is a directory walk + regex count.

  2. Author xtask/quality-budgets.toml verbatim from §D12.2 (empty exception lists).

  3. Run cargo xtask quality-budgets --report --write-lock once; commit the resulting xtask/quality-budgets.lock. This captures the current counts as the baseline.

  4. Document in CONTRIBUTING: how to run, what regression means, where to add an exception (with a reason field).

  5. Author fixture-based unit tests per §D12.6 covering each budget.

  6. Append snapshot test of the rendered table.

  7. Verification: run cargo xtask quality-budgets --report on a clean tree; observe table matches lockfile values; intentionally add one extra unwrap_or_default() in a test; rerun with --fail-on-regression; observe non-zero exit; revert.

Step 14: Coverage Matrix tooling

Files:

  1. xtask/src/cmd/coverage_matrix.rs — full implementation per §D13.4 (replacing Step 12 stub)

  2. xtask/route-to-client-method.toml (new) — initial mapping per §D13.2; mostly empty, populated alongside Phase A.8 typed-client work

  3. target/coverage-matrix.md (committed once as baseline) — first-run output snapshot

  4. docs/modules/ROOT/pages/operations/coverage-matrix.adoc (new) — wraps the markdown table in [source,markdown]; linked from nav.adoc Operations section

  5. xtask/tests/coverage_matrix_test.rs (new) — fixture per §D13.7

Cross-reference: §D13.

  1. Replace the Step 12 stub in cmd/coverage_matrix.rs with the full implementation per §D13.4.

  2. Author the empty-but-shaped route-to-client-method.toml; document the column meanings + the convention for new entries.

  3. Run cargo xtask coverage-matrix on a healthy devstack; commit the resulting target/coverage-matrix.md as the baseline (copied into docs/modules/ROOT/pages/operations/coverage-matrix.adoc).

  4. Add nav entry under Operations.

  5. Author the three fixture tests per §D13.7.

  6. Decision documented in the MR description: name-suffix regex tagging, not proc-macro (per §D13.5).

  7. Verification: run subcommand against a 3-route mock OpenAPI fixture; observe expected matrix rows and score.

Step 15: DB Constraint Evil Tests — hand-curated harness

Files:

  1. crates/craig-test-lib/src/constraints.rs (new) — helper module per §D14.1

  2. crates/craig-test-lib/src/lib.rspub mod constraints;

  3. services/craig-cases/tests/constraints.rs + tests/constraints/{case_number_format,contact_type_enum}.rs

  4. services/craig-placement/tests/constraints.rs + tests/constraints/{placement_status_enum,home_in_use_on_delete}.rs

  5. services/craig-exchange/tests/constraints.rs + tests/constraints/{partner_in_use_on_delete,icpc_status_enum}.rs

  6. services/craig-financial/tests/constraints.rs + tests/constraints/{payment_amount_positive,rate_table_unique_name}.rs

  7. services/craig-reporting/tests/constraints.rs + tests/constraints/{report_run_unique_name,report_kind_enum}.rs

  8. services/craig-rules/tests/constraints.rs + tests/constraints/{rule_set_unique_name,jurisdiction_enum}.rs

  9. services/craig-security/tests/constraints.rs + tests/constraints/{partner_unique_code,admin_unit_fips_format}.rs

  10. services/craig-intake/tests/constraints.rs + tests/constraints/{api_key_revoked,captcha_required}.rs

  11. .claude/docs/testing.md — append "Constraint tests" subsection

  12. Backlog GitLab issue (filed in Step 1 of this plan) — full constraint enumeration grown from §D14.3 over time

Cross-reference: §D14.

  1. Author the helper module verbatim from §D14.1; export from lib.rs.

  2. For each of the 8 services: create tests/constraints.rs module-index file declaring the two #[path = …​] mod lines.

  3. Author each of the 16 test files following the §D14.5 sample shape. Each test:

    1. Skips if devstack absent (if !h.devstack_available().await { return; })

    2. Boots TestHarness

    3. Constructs the typed client

    4. Calls assert_constraint_violation_returns(…​) with the constraint-specific request body, expected rejection, and problem-type URL from §D14.4.

  4. Append testing.md "Constraint tests" subsection: discoverability convention, expected RFC 9457 problem types, link to backlog issue for full enumeration.

  5. File backlog issue under epic &22 with the §D14.3 enumeration as the seed; note that future migrations land with their constraint test in the same MR.

  6. Verification: cargo nextest run --workspace -E 'test(::constraints::)'; all 16 pass; runtime <30s under devstack.

  7. Pushback documentation: the MR description quotes §D14’s first paragraph verbatim — auto-generation rejected, hand-curation chosen.

Step 16: Phase A.15 — Central Evil Input Corpus

Files:

  • crates/craig-test-lib/src/evil/{mod,jwt,upload,string,uuid,json,path,html,unicode,enum_value,date,multipart,signature}.rs (new — 12 modules + mod)

  • crates/craig-test-lib/src/lib.rs (add pub mod evil)

  • crates/craig-test-lib/Cargo.toml (add deps: infer already added by platform-stab-2 §D8.1; proptest = "1" for the BiDi/RTLO/Unicode generators; unicode-normalization = "0.1" for NFC/NFD pairs)

  • crates/craig-test-lib/tests/evil_corpus_self.rs (new)

  • crates/craig-test-lib/tests/evil_corpus_global_contract.rs (new)

  • services/craig-{cases,exchange,financial,placement,reporting,rules,security,intake}/tests/api/evil_corpus.rs (new — 8 files, one per service)

  • services/craig-web/tests/upstream_evil_corpus.rs (new)

  • .claude/docs/testing.md ("Evil Input Corpus" section)

Implementation:

  1. Create the 12-module tree under crates/craig-test-lib/src/evil/ per §D15.1. jwt.rs and upload.rs import the existing Phase A.5/A.6 seed modules and emit EvilCase enum variants — no changes to the seeds.

  2. Implement the 9 NEW category modules per §D15.3 corpus content. For each: pub fn cases() → impl Iterator<Item = EvilCase>. Compile-time constants where possible; only multipart.rs and signature.rs need runtime construction (they wrap mints with timestamps).

  3. Implement mod.rs::all_evil_cases() as a chained iterator. evil_cases_for(category) is a thin filter.

  4. Implement parametric_evil_test! macro per §D15.4. Macro lives in crates/craig-test-lib/src/evil/mod.rs with #[macro_export].

  5. Implement self-test (tests/evil_corpus_self.rs) per §D15.7. Asserts non-empty per category, unique names, deterministic iteration.

  6. Implement global contract test (tests/evil_corpus_global_contract.rs) per §D15.5. Skips if devstack unavailable.

  7. Wire parametric_evil_test! into 8 service test trees per §D15.6. Each service file is ~30 LOC.

Verification:

  1. cargo nextest run -p craig-test-lib --test evil_corpus_self — passes

  2. cargo nextest run -p craig-test-lib --test evil_corpus_global_contract — passes (devstack required)

  3. Per-service: cargo nextest run -p craig-cases --test evil_corpus — every case rejected with 4xx

  4. .claude/docs/testing.md shows new "Evil Input Corpus" section

  5. cargo clippy -p craig-test-lib --tests --locked — -D warnings clean

Step 17: Phase A.16 — Restart/Replay light helpers

Files:

  • crates/craig-test-lib/src/restart.rs (new)

  • crates/craig-test-lib/src/lib.rs (add pub mod restart)

  • crates/craig-test-lib/tests/restart_helpers_self.rs (new — meta-test per §D16.6)

  • services/craig-{cases,exchange,placement}/src/main.rs (extract pub async fn main_inner(args, env) → anyhow::Result<()>; existing main() becomes tokio::main wrapper around it — see §Open Questions)

  • crates/craig-mq/tests/{outbox_post_publish_crash,inbox_post_handle_crash}.rs (new)

  • services/craig-cases/tests/api/attachments_recovery.rs (new)

  • services/craig-exchange/tests/api/exchange_send_recovery.rs (new)

  • .claude/docs/testing.md ("Restart/replay testing" section)

Implementation:

  1. Implement CrashPoint enum + CrashHarness + simulate_crash_after + restart_service + assert_recovers_to per §D16.1.

  2. Implement crash-point wiring table per §D16.2. Each variant maps to a specific Phase A.2 injector configuration. Doc-test enforces wiring exists for every variant.

  3. Refactor 8 service main.rs files to extract main_inner(args, env). The refactor is small (~10 LOC per service) — wrap existing main body, exposing the dependency-injection seam to tests. main() becomes a 5-line tokio::main wrapper.

  4. Implement 4 paired tests (one per platform-stab-2 fix with a recovery sweep) per §D16.4.

  5. Implement self-test per §D16.6.

  6. Document the pattern in testing.md.

Verification:

  1. cargo nextest run -p craig-test-lib --test restart_helpers_self — passes

  2. cargo nextest run -p craig-mq --test outbox_post_publish_crash — passes

  3. cargo nextest run -p craig-mq --test inbox_post_handle_crash — passes

  4. cargo nextest run -p craig-cases --test attachments_recovery — passes

  5. cargo nextest run -p craig-exchange --test exchange_send_recovery — passes

  6. Each service still builds; main_inner extraction is binary-compatible

  7. .claude/docs/testing.md shows new "Restart/replay testing" section

Step 18: Phase A.17 — Invariant Sweeper xtask

Files:

  • xtask/src/cmd/invariants.rs (new)

  • xtask/src/cmd/mod.rs (export + dispatch)

  • xtask/src/cmd/test.rs (add --post-check flag)

  • xtask/Cargo.toml (no new deps — sqlx + clap already present)

  • crates/craig-test-lib/sql/invariants/<svc>/*.sql (new — initial 8 invariants seeded across 8 services per §D17.4)

  • crates/craig-common/src/constants.rs (extend with grace constants from §D17.5)

  • xtask/tests/invariants.rs (new — catalog parser self-test per §D17.8)

  • .claude/docs/testing.md ("Invariant Sweeper" section)

Implementation:

  1. Implement xtask/src/cmd/invariants.rs per §D17.2. SQL loaded at runtime; metadata header parsed via simple string splits (no TOML dep — header is a tiny dialect).

  2. Seed initial 8-invariant catalog per §D17.4. Each is a single .sql file with comment header. The "audit_row_per_state_mutation" invariant ships seeded for cases / placements / exchange / financial / security only; remaining 3 services ship a TODO.sql placeholder that returns 0 rows.

  3. Add grace constants to crates/craig-common/src/constants.rs per §D17.5.

  4. Wire cargo xtask test --post-check to invoke the sweeper after the test battery returns.

  5. Implement xtask self-test per §D17.8: parse every .sql, validate header metadata, assert grace literals match constants, run each against empty DB.

  6. Document in testing.md, including the report-only contract.

Verification:

  1. cargo nextest run -p xtask --test invariants — catalog self-test passes

  2. cargo xtask invariants against a clean dev DB — exits 0, prints all-pass

  3. cargo xtask invariants against a DB seeded with one orphan row — exits 1, prints failure with sample row + remediation pointer

  4. cargo xtask test --post-check runs the test battery + sweeper end-to-end

  5. .claude/docs/testing.md shows "Invariant Sweeper" section

Step 19: Phase B — adoption bundle (10 tests)

Files:

  • services/craig-api/tests/idempotency_concurrent_same_body.rs (new — test 1)

  • services/craig-api/tests/idempotency_different_body_422.rs (new — test 2)

  • services/craig-api/tests/idempotency_finalize_failure_returns_live.rs (new — test 3)

  • crates/craig-mq/tests/outbox_two_workers_no_double_publish.rs (new — test 4)

  • crates/craig-mq/tests/inbox_failure_doesnt_suppress_redelivery.rs (new — test 5)

  • services/craig-cases/tests/api/upload_db_failure_no_blob.rs (new — test 6)

  • services/craig-exchange/tests/api/send_crash_recoverable_state.rs (new — test 7)

  • services/craig-web/tests/upstream_malformed_renders_error.rs (new — test 8)

  • services/craig-store/tests/filename_disposition_evil.rs (new — test 9)

  • services/craig-cases/tests/api/typed_client_pilot.rs (new — test 10)

  • docs/modules/ROOT/pages/test-coverage-scorecard.adoc (new — top-20 workflows × 6 cells)

  • docs/modules/ROOT/nav.adoc (Architecture & Design entry pointing at scorecard)

  • xtask/src/cmd/coverage_matrix.rs (extend with --emit-scorecard flag)

  • CHANGELOG.adoc (per-test + scorecard entries)

Implementation:

  1. Each test follows the helper pairings from §DB.1.

  2. Each test file’s first #[tokio::test] is the failing-test-first artifact; the test name mirrors the GitLab issue title for one-grep traceability.

  3. The scorecard is hand-curated for the workflow rows; the cell values are auto-emitted from cargo xtask coverage-matrix --emit-scorecard reading test-name suffix tags.

  4. Cross-reference each test to its sibling platform-stab-2 step in the test file’s module-level doc comment: //! Paired with platform-stab-2.adoc §D1 (Step 3) — failing-test-first artifact.

Recommended sub-commit ordering (single MR, but per-commit reviewer-friendly):

  1. Commit 1: Test 1 (concurrent same-body idempotency — Phase A.1 canary). Validates the consolidated concurrent_fire helper end-to-end against the platform-stab-2 §D1 fix.

  2. Commit 2: Test 2 (different-body 422). Tightens the §D1 contract.

  3. Commit 3: Test 3 (idempotency finalize-failure returns live response). Pairs with §D1 error mapping.

  4. Commit 4: Test 4 (two outbox workers, no double publish). Pairs with platform-stab-2 §D2.

  5. Commit 5: Test 5 (inbox handler-failure-then-retry). Pairs with platform-stab-2 §D3.

  6. Commit 6: Test 6 (object-store + DB-failure → no blob orphan). Pairs with platform-stab-2 §D4.

  7. Commit 7: Test 7 (exchange crash → recoverable state). Pairs with platform-stab-2 §D5; uses Phase A.16 restart helpers.

  8. Commit 8: Test 8 (BFF malformed-upstream renders error). Pairs with platform-stab-2 §D7.

  9. Commit 9: Test 9 (filename + Content-Disposition evil corpus). Pairs with platform-stab-2 §D8.1; uses Phase A.15 evil corpus.

  10. Commit 10: Test 10 (typed-client pilot). Pairs with Phase A.8.

  11. Commit 11: Top-20 risk-coverage scorecard authoring + nav.adoc xref + xtask --emit-scorecard flag.

  12. Commit 12: CHANGELOG entries (one per test + one for scorecard) + final scorecard run capturing the 10 new tests' cell flips.

Verification:

  1. cargo nextest run --workspace includes all 10 new test files; each passes against the post-platform-stab-2 binaries

  2. cargo xtask coverage-matrix --emit-scorecard produces a non-empty auto section

  3. cargo xtask coverage-matrix --diff main shows +N filled cells (one per Phase B test) and zero regressions

  4. Scorecard renders in Antora without errors

  5. nav.adoc Architecture & Design section shows the scorecard link

  6. Manual: review scorecard against ops risk register; confirm top-20 workflows are accurate

  7. CHANGELOG shows one entry per test + one for the scorecard

Step 20: Plan completion audit + archive

Files: docs/modules/ROOT/pages/plans/test-framework-hardening.adoc (Status table → all Complete), docs/modules/ROOT/nav.adoc (move from Active to archive), docs/modules/ROOT/pages/plans/archive.adoc (new row under Infrastructure & Reliability), .claude/CLAUDE.md (Phase Status — add "Test Framework Hardening" row), CHANGELOG.adoc (wrap-up entry).

  1. Spawn plan-completion-audit subagent per delivery-protocol.md

  2. Verify all 19 prior steps complete via MR list

  3. Flip Status table all-Complete

  4. Move plan from nav.adoc Active to archive.adoc Infrastructure & Reliability section

  5. Update CLAUDE.md Phase Status (add "Test Framework Hardening" row with helper count + test count + scorecard link)

  6. CHANGELOG wrap-up entry

  7. Close epic &22

Files Touched

File Change

docs/modules/ROOT/pages/plans/test-framework-hardening.adoc

Plan file (this content). Step 1 expansion of the prior thin draft.

docs/modules/ROOT/nav.adoc

Active section: confirm test-framework-hardening xref present (already added in prior MR).

docs/modules/ROOT/pages/plans/archive.adoc

Step 20 — new row under Infrastructure & Reliability with all step MR numbers.

docs/modules/ROOT/pages/test-coverage-scorecard.adoc

Step 19 — new file. Top-20 workflows × 6-cell grid.

CHANGELOG.adoc

Entry per step (Steps 1–20).

crates/craig-test-lib/src/concurrent.rs

Step 2 — concurrent_fire helper (consolidates platform-stab-2 inline helper).

crates/craig-test-lib/src/fault_injection.rs

Step 3 — extends platform-stab-2 inline injectors with RabbitDownInjector, ObjectStoreErrorInjector, PublishInTxFailureInjector, CipherErrorInjector.

crates/craig-test-lib/src/state_machine_matrix.rs

Step 5 — pattern helper for state-machine matrix testing; first adoption: Payment.

crates/craig-test-lib/src/jwt_mutations.rs

Step 6 — JWT mutation library (kept callable as-is by platform-stab-2 §D5).

crates/craig-test-lib/src/upload_fixtures.rs

Step 7 — magic-byte + filename-attack fixtures (kept callable as-is by platform-stab-2 §D8.1).

crates/craig-test-lib/src/evil/{mod,jwt,upload,string,uuid,json,path,html,unicode,enum_value,date,multipart,signature}.rs

Step 16 — central evil input corpus (12 modules + mod). Wraps Step 6/7 seeds.

crates/craig-test-lib/src/restart.rs

Step 17 — restart/replay light helpers.

crates/craig-test-lib/src/constraints.rs

Step 14 — DB constraint evil-test harness (curated; sibling to D17 sweeper).

crates/craig-test-lib/src/lib.rs

Re-exports for every new module (Steps 2/3/5/6/7/16/17).

crates/craig-test-lib/Cargo.toml

Deps: proptest = "1" (Step 11 / 16 generators), unicode-normalization = "0.1" (Step 16), infer (already in workspace via platform-stab-2 §D8.1).

Workspace Cargo.toml

Add proptest, unicode-normalization to [workspace.dependencies].

crates/craig-test-lib/sql/invariants/<svc>/*.sql

Step 18 — per-service invariant catalog (initial 8 invariants).

crates/craig-common/src/constants.rs

Step 18 — grace constants used both by codepaths and by the invariant sweeper SQL literals.

xtask/src/cmd/reliability.rs

Step 11 (deliverable) — surfaces concurrency/fault/recovery test results in a per-axis report. Report-only.

xtask/src/cmd/contracts.rs

Step 11 — contract-tests harness (consumer-driven contracts). Report-only.

xtask/src/cmd/mutants.rs

Step 12 — cargo-mutants smoke wrapper. Report-only.

xtask/src/cmd/coverage_matrix.rs

Step 12 — coverage-matrix report keyed by test-name-suffix tags + --emit-scorecard flag (Step 19 extension).

xtask/src/cmd/invariants.rs

Step 18 — invariant sweeper. Report-only.

xtask/src/cmd/quality_budgets.rs

Step 13 — quality-budgets aggregator (LOC, cyclomatic, fn count). Report-only.

xtask/src/cmd/mod.rs, xtask/src/cmd/test.rs

Wiring for the 6 new subcommands; --post-check flag on test.

services/craig-{cases,exchange,financial,placement,reporting,rules,security,intake}/tests/api/evil_corpus.rs

Step 16 — adoption: parametric_evil_test! against each service’s POST endpoints.

services/craig-cases/tests/api/{attachments_recovery,upload_db_failure_no_blob,typed_client_pilot}.rs

Steps 17 + 19.

services/craig-exchange/tests/api/{exchange_send_recovery,send_crash_recoverable_state}.rs

Steps 17 + 19.

services/craig-mq/tests/{outbox_post_publish_crash,inbox_post_handle_crash,outbox_two_workers_no_double_publish,inbox_failure_doesnt_suppress_redelivery}.rs

Steps 17 + 19.

services/craig-api/tests/{idempotency_concurrent_same_body,idempotency_different_body_422,idempotency_finalize_failure_returns_live}.rs

Step 19.

services/craig-web/tests/{upstream_evil_corpus,upstream_malformed_renders_error}.rs

Steps 16 + 19.

services/craig-store/tests/filename_disposition_evil.rs

Step 19.

services/craig-{cases,exchange,placement,financial,reporting,rules,security,intake}/src/main.rs

Step 17 — extract main_inner(args, env) (small refactor enabling restart helpers).

.claude/docs/testing.md

Steps 8 + 16 + 17 + 18 — Test Categories taxonomy + helper docs + Evil Input Corpus + Restart/replay testing + Invariant Sweeper sections.

.claude/docs/delivery-protocol.md

Step 8 — failure-path-test process rule.

CONTRIBUTING.adoc

Step 8 — pointer to testing.md test-categories. Step 11 — "Optional dev tools" cargo-install line for cargo-mutants. Step 12 — "Optional reliability tooling" subsection enumerating the 6 new xtask subcommands. Step 13 — "Quality budgets" subsection.

.claude/CLAUDE.md

Step 20 — Phase Status row added.

docs/modules/ROOT/nav.adoc

Step 19 — Architecture & Design entry pointing at test-coverage-scorecard. Step 20 — move plan to archive.

Step 1 of THIS plan files 9 GitLab issues for the new Phase A pieces (A.9 proptest; A.10 cargo-mutants smoke; A.11 xtask subcommands reliability/contracts/mutants/coverage-matrix/invariants/quality-budgets; A.12 code-quality budget warnings; A.13 coverage matrix tooling; A.14 DB constraint evil-test harness; A.15 central evil corpus; A.16 restart/replay light helpers; A.17 invariant sweeper). These join the existing 8 Phase A issues + 1 Phase B bundle (#283–#291) for a total of 18 step issues + 1 audit issue under epic &22.

Verification

  1. cargo nextest run --workspace — full unit + integration battery passes including the 10 Phase B adoption tests + 8 service evil-corpus suites + restart-helpers self-test + invariant sweeper self-test.

  2. cargo xtask reliability — surfaces concurrency / fault / recovery test results per axis; report-only.

  3. cargo xtask contracts — consumer-driven contract tests pass; report-only.

  4. cargo xtask mutants --smoke — surviving-mutant report; report-only (defaults flagged in §Open Questions).

  5. cargo xtask coverage-matrix --report — per-category coverage report produced from test-name-suffix tags.

  6. cargo xtask invariants — exits 0 on a clean dev DB after a successful e2e run; exits 1 with a remediation pointer when an invariant is violated.

  7. cargo xtask quality-budgets --report — LOC / cyclomatic / fn-count per crate; report-only.

  8. Manual: kill craig-cases mid-blob-write (paired with platform-stab-2 §D4 manual verification step); restart; assert scanner reaps within 5 min.

  9. Manual: stage 200 concurrent same-key POSTs against POST /v1/cases (paired with platform-stab-2 §D1 manual step); assert exactly one handler invocation via the side-effect counter.

  10. cargo xtask coverage-matrix --emit-scorecard runs and produces a non-empty auto-section in docs/modules/ROOT/pages/test-coverage-scorecard.adoc; the page renders in Antora without errors.

  11. cargo xtask api-docs — Antora docs build without error including new pages (plan adoc, scorecard, archive row).

  12. cargo xtask validate --skip-docker — fmt + clippy + cargo-deny clean.

Documentation Updates

  • .claude/docs/testing.md — Test Categories taxonomy (happy / sad / evil / concurrency / replay / fault) + per-helper usage docs + reviewer-discipline process rule (Step 8 deliverable). Plus Evil Input Corpus section (Step 16), Restart/replay testing section (Step 17), Invariant Sweeper section (Step 18).

  • .claude/docs/delivery-protocol.md — failure-path-test process rule under Delivery Checklist: "every concurrency-relevant fix ships with a failing-test-first artifact paired with the failing-test-first directory in test-coverage-scorecard.adoc".

  • CONTRIBUTING.adoc — pointer to .claude/docs/testing.md Test Categories.

  • CHANGELOG.adoc — entry per step (20 entries; the audit step is a wrap-up summary).

  • docs/modules/ROOT/pages/test-coverage-scorecard.adoc — top-20 workflows × 6-cell grid (Phase B Step 19 deliverable).

  • docs/modules/ROOT/pages/plans/archive.adoc — Infrastructure & Reliability row added on Step 20 with all step MR numbers.

  • docs/modules/ROOT/nav.adoc — Architecture & Design entry for scorecard (Step 19); plan moved from Active to archive (Step 20).

  • .claude/CLAUDE.md — Phase Status table: add "Test Framework Hardening" row (Step 20).

Open questions

  1. Coverage matrix tagging convention. Two viable options:

    1. Regex on test name — suffix tags _happy, _sad, _evil, _concurrent, _replay, _fault. Coverage-matrix xtask scans cargo test — --list output and bins by suffix. No proc-macro dep, no compile-time machinery. Loose discipline — typo’d suffix → silently uncategorized.

    2. #[test_category(…​)] proc-macro — strict, IDE-discoverable, compile-time-checked. Adds craig-test-macros proc-macro crate dep.

      Default proposal: regex tagging. Reason: keeps deps lean; the looseness is acceptable because the scorecard is a living artifact maintained by reviewer discipline, not a CI gate. Revisit if the suffix-typo rate exceeds 1% in W1.

  2. Quality-budgets enforcement. Initially report-only (warnings in xtask output, not pre-push gate). The user directive is explicit: D11/D12 + invariants + quality-budgets ship report-only. Default proposal: defer enforcement to a future plan after 90 days of report-only operation. Concretely: a follow-up issue filed in W3 of this plan to revisit at the 90-day mark; promotion to enforced gate requires a second plan with explicit Status-table entry, not a silent flip.

  3. proptest case count default. proptest’s own default is 256 cases per property. Default proposal: keep 256 globally, raise to 1024 specifically for canonical-JSON signing properties (where the bug surface is wide and determinism is high). Configurable per `proptest! { #![proptest_config(ProptestConfig::with_cases(1024))] } on individual properties; do not override the global default in proptest.toml.

  4. Mutation testing budget. cargo-mutants produces a "surviving mutants" count per crate. Where do we draw the line? Default proposal: 0 surviving mutants in craig-auth and craig-mq core paths (the security + concurrency boundaries); ≤5 elsewhere. The budget is a report-only warning in W3, not an enforced gate. Revisit at the 90-day post-report-only checkpoint together with §Open Question 2.

  5. Restart helpers vs full mock-binary. How does a test drive a service’s main() without spawning a real process? Default proposal: extract pub async fn main_inner(args: Args, env: Env) → anyhow::Result<()> from each service’s main.rs (~10 LOC refactor per service), so tests restart_service(|| async { craig_cases::main_inner(test_args(), test_env()).await }) directly with mocked dep injection. The existing tokio::main becomes a 5-line wrapper around main_inner. Binary-compatible; no behavioral change in production. Filed as part of Step 17.

  6. Phase B test 4 — multi-replica vs single-process simulation. Two outbox workers can be exercised by either (a) Phase A.3 multi-replica devstack (cargo xtask dev start --replicas craig-cases=2) or (b) single-process simulation (two OutboxWorker instances against a shared PgPool in one test process — the platform-stab-2 §D2 model). Default proposal: prefer single-process simulation as the primary test for fast feedback; multi-replica devstack ships as the secondary end-to-end smoke that runs only under cargo xtask e2e. Two tests, one fast and locally runnable, one slow and CI-only.

  7. proptest version pin — 1.7 was current as of 2026-05; verify against the registry on commit and bump in this MR if 1.8 has shipped.

  8. cargo-mutants 25.x release line — confirm ^25.0 resolves to a stable release; fall back to a hard-pin (=25.X.Y) if patch-version churn proves disruptive in the smoke run.

  9. Coverage-matrix axis tagging at scale — name-suffix regex is fine at 1259 tests; revisit when untagged-test count exceeds (e.g.) 200 untagged-in-tests/api/.

  10. Quality-budgets B6 (duplicate dependencies) — current zen-engine + jsonwebtoken dupes (per platform-stab-2 §D10.2) are the current state; the lockfile captures them as the floor. Confirm with reviewer that "current dupe count = floor" is the correct semantics for the seed lock.

  11. Constraint test backlog seeding — §D14.3 enumeration is indicative. The Step 1 backlog issue must be filed before Step 15 lands so seed contributors can append rather than fork the canonical list.

  12. xtask subcommand namingreliability vs reliability-suite vs rel. Default proposal: reliability (matches existing one-word subcommand convention: dev, test, e2e, validate, perf, security, reconcile).

Potential improvements

Documented per pre-commit Q6 — concrete improvements identified during implementation that are NOT in scope for this plan but worth tracking:

  • cargo xtask reliability — concurrency-and-fault selectivity flags. Today the suite runs every category in one shell-out (concurrency | recovery | fault | invariants | properties). Once Phase B adoption (Step 19) lands enough category-specific tests, splitting into --concurrency-only / --fault-only / --recovery-only flags would let contributors target the slowest category for iterative debugging without rerunning the rest. File a follow-up after Step 19 ships.

  • cargo xtask mutants --smoke — per-crate parallelism. run_one invokes cargo mutants --jobs 1 per crate sequentially. The 5 critical crates could fan out across cores if mutation testing sees regular use; deferred until the Step 11 follow-up clean-tree population run shows the actual wall-time delta.

  • cargo xtask contracts --service <name> — CSV form. Today --service cases filters to one service; a CSV (--service cases,exchange) would let CI dashboards group services per pipeline shard. Trivial nextest-filter join; deferred until contract test counts cross ~50 per service (Phase B).

  • cargo xtask quality-budgets — exception lists not yet enforced. §D12.1 specifies that each budget has a per-path exception list (xtask/quality-budgets.toml already declares exceptions = [] arrays). The catalogue parser reads them but the counters do not yet honour exception entries — every match within the configured glob counts. Adding a path-glob exception walker is straightforward but unnecessary today (zero exceptions are populated). Implement when the first reviewer-approved exception lands so the schema and the enforcement arrive together.

  • B6 distinct-package count vs. version-pair count. B6 today is "distinct package names with multiple versions" (67). An alternative metric is "total duplicate (name, version) pairs minus distinct names" — i.e. the number of redundant installs. Both are defensible; the current choice is more stable across cargo updates because added/removed minor versions of one package only swing the count by 1. Worth re-evaluating if a future MR consolidates a duplicated package and the count drops by an unintuitive amount.

  • cargo xtask coverage-matrix --diff <REF>. §D13.7’s deferred flag. Re-running the matrix against main and printing cell flips (+ POST /v1/cases :: evil, - DELETE /v1/cases/{id} :: happy) gives reviewers a per-MR delta view that no other tool produces. Implementation needs a clean git stash + git checkout + re-run dance — workable, but the working-tree contract pushed it out of Step 14. File a follow-up once the typed-method bridge populates non-zero cells.

  • Per-service evil-corpus adoption files (§D15.6) — RETIRED. Step 16 originally deferred the 8 service-side tests/api/evil_corpus.rs files; the post-Step-17 review-fix MR !212 implemented all 8 with appropriate skip_categories lists where the handler doesn’t validate today. Each skipped category is a documented finding tracked under #311 (RFC 9457 standardization initiative). The BFF services/craig-web/tests/upstream_evil_corpus.rs is the one piece still deferred — it depends on platform-stab-2 §D7’s BFF-upstream rendering work and will land as part of that initiative. Net: this §Potential improvements bullet is closed; the BFF upstream test remains as a distinct platform-stab-2 follow-up.

  • Constraint test seed expansion to 2 tests/service (16 total) — RETIRED. §D14.4 listed 16 tests; Step 15 originally shipped 8. The post-Step-17 review-fix MR !212 implemented the second 8 (one per service), bringing the total to 16/16 passing — the original §D14.4 promise is now fully delivered.

  • RFC 9457 problem-type URL standardization. Today only 2 problem-type URLs are emitted across the entire codebase (idempotency-conflict, payload-too-large — both in craig-api/idempotency.rs). The §D14.4 design asks each constraint test to assert a per-violation type URL; that requires defining ~20 type URLs and wiring each handler’s rejection path to emit them. This is its own multi-MR initiative and was scoped out of Step 15. The constraints helper supports both modes (status-only today, type-URL when handlers ship them) so adoption is incremental once the URLs are defined.

  • cargo xtask coverage-matrix typed-method bridge. The committed xtask/route-to-client-method.toml declares the entry shape but has zero entries; the scan_typed_calls() consumer is unimplemented. Today the matrix only matches literal URL strings in test source, which the typed-client migration removed from most tests — hence the honest 0/1308 baseline. Implementing the typed-method bridge requires (a) populating the TOML alongside Phase A.8 typed-DTO migrations and (b) adding a syn-based call-site scanner to the subcommand. Land both together when the typed migration crosses ~50% adoption.

Errata

Implementation deltas vs. the original plan, captured for reviewer clarity:

  • Step 2 — built from scratch, not extended. §D1 framed Phase A.1 as "in-place extension of platform-stab-2 Step 2’s minimal helper" because the sibling plan was expected to land its inline concurrent_fire first. Per the user’s directive to do test-framework-hardening before platform-stab-2 (so platform-stab-2’s P0 fixes ship with proper coverage from the start), there was no minimal prefix to extend; the consolidated surface was authored from scratch in crates/craig-test-lib/src/concurrent.rs. The deprecated concurrent_fire alias is retained per §D1.1 to keep platform-stab-2’s authored-but-unmerged tests forward-compatible. No semantic difference — same module, same surface, same test set.

  • Step 19 — 3 of 10 adoption tests shipped; 7 platform-stab-2-dependent tests deferred. §DB.1 lists 10 specific tests. Tests 1-7 each require a platform-stab-2 fix or store-trait seam to exist before the test has anything to exercise: Tests 1-3 need the persistent idempotency middleware (§D1); Test 4 needs the multi-replica outbox infrastructure (§D2); Test 5 needs the three-state inbox (§D3); Test 6 needs the atomic blob+DB+event upload semantics (§D4); Test 7 needs the exchange send-job state machine (§D5). All 7 are filed as separate GitLab issues. Tests 8-10 ship today: Test 8 (BFF malformed upstream JSON → service error not silent empty data) lands as 3 unit tests in services/craig-web/src/api_client.rs::tests exercising malformed JSON + empty body + upstream 5xx via wiremock; Test 9 (filename + Content-Disposition fuzzing) lands as 3 integration tests in crates/craig-store/tests/filename_evil_corpus.rs looping the central evil corpus’s String/Path/Unicode categories against sanitize_filename and asserting post-sanitize invariants (no /, no \, no \0, no .. segment, idempotent); Test 10 (typed-client pilot) lands as 7 round-trip + boundary-pinning tests in crates/craig-cases-contracts/tests/typed_dto_pilot.rs exercising every public DTO across persons/cases/contacts/referrals plus the deny_unknown_fields=off and unknown-string-accepted posture pins. The risk-coverage scorecard at docs/modules/ROOT/pages/test-coverage-scorecard.adoc + the new cargo xtask coverage-matrix --emit-scorecard flag close out §DB.2/§DB.3.

  • Step 18 — initial invariant set is 4 (3 cross-service + 1 exchange-specific); §D17.4’s full table is platform-stab-2-dependent and ships incrementally. §D17.4 lists 8 invariants in the initial catalog. Of those, 4 reference platform-stab-2 schema that doesn’t exist today: orphan_pending_attachments needs contact_attachments.object_status (platform-stab-2 §D4); stuck_in_flight_send_jobs needs a send_jobs table that doesn’t exist; inbox_processed_with_failed needs an event_inbox.failed_at column that doesn’t exist (today’s table has only processed_at); idempotency_processing_past_ttl needs idempotency_responses.{status, claim_expires_at} that don’t exist (today’s table has just expires_at for cleanup). Step 18 ships the framework + the 4 invariants that map to today’s actual schema: outbox_unpublished_beyond_grace, inbox_unprocessed_beyond_grace, idempotency_expired_not_cleaned (cross-service in craig-mq/), and stuck_pending_transactions (in craig-exchange/). The runtime SKIP-on-42P01 handling means as platform-stab-2 lands the missing schema, new SQL files dropped into the catalog light up automatically without runner changes. Audit-row invariant (§D17.6) is unimplemented; flagged in §D17.6 as "ships in W3 hardening passes" — deferred. The live sweep against the running devstack already paid for itself: it surfaced craig-placement missing the event_inbox migration that the other 6 stateful services received — filed as a separate platform finding (the other services' 20260430173943_event_inbox.sql migration is absent under services/craig-placement/migrations/). The sweeper’s runtime SKIP correctly classified this as "feature-rollout gap" rather than a 1-failure ceiling violation.

  • Step 17 — both CrashHarness and a bare-closure entry shipped; named CrashPoint variants pair with the 3 fault injectors that exist today, with the 4 store-trait injectors lighting up as platform-stab-2’s §D2.0 seam refactor lands. §D16.1 specified one entry point passing a CrashHarness to the builder. Step 17 ships two entry points: simulate_crash_after_with_harness(point, builder) matches §D16.1 (builder receives CrashHarness, requires live devstack) and is the canonical adoption shape for §D16.4’s platform-stab-2-paired tests; simulate_crash_after(point, builder) is a bare-closure entry that the helper’s own self-tests use to exercise the orchestration logic without devstack. CrashHarness wraps TestHarness for client construction, captures diagnostic artifacts via record_artifact, and exposes crash_now(msg) for manual aborts plus arm(injector, scenario) for fault-injection. The 3 wrapper injectors that ship today (Latency, IntermittentFlap, ConsumerBackpressure) are armable through CrashHarness::arm immediately; the 4 store-trait injectors §D16.2 references (FaultyAttachmentStore, FaultyOutboxStore, FaultyInboxStore, FaultySendJobStore) light up as platform-stab-2’s §D2.0a/§D2.0b store-seam refactor introduces them. Earlier draft of this MR shipped only the bare-closure entry on the (lazy) reasoning that the named-variant injectors don’t exist yet; corrected after user pushback because the façade itself — TestHarness wrapping, artifact capture, manual crash_now — is useful regardless of which injectors back the named variants. 10 self-tests cover both entry points (8 bare + 2 harness-integration that skip without devstack).

  • Step 16 — corpus + global contract + macro shipped; per-service adoption files deferred to backlog. §D15.6 lists 8 service-side adoption files (services/craig-<svc>/tests/api/evil_corpus.rs) plus the BFF upstream test (services/craig-web/tests/upstream_evil_corpus.rs). Each adoption file is mechanical fan-out using the parametric_evil_test! macro, but the per-service "endpoint:" closure needs raw-JSON-accepting client paths that don’t uniformly exist (cases is fully typed-DTO post Step-9 redo; financial / reporting / placement / security have a Value-accepting method per major resource). Authoring 8 files of ~30 LOC each + verifying each against a live devstack would have doubled the MR’s footprint without adding new contract surface — the global contract test (evil_corpus_global_contract.rs) already exercises every JSON/string/UUID/path/html/unicode/enum/date case end-to-end against the public-intake endpoint, which §D15.5 calls "the single test that establishes the boundary contract globally." Per-service fan-out is tracked in §Potential improvements. Three tactical adjustments inside the corpus itself: (1) string.overlong_16mb reduced to string.overlong_4mb to keep self-test wall-time under 1s while preserving the "above-cap-but-below-DoS" alarm shape; (2) json.depth_bomb_* cases emit pre-serialized bytes rather than constructed serde_json::Value trees because Value’s Drop recurses and overflowed the test-thread stack at depth 4096; (3) the parametric_evil_test! macro skips applicable-category checks in-test rather than gating expansion at compile-time, because category applicability depends on which Axum extractor the endpoint uses (Json vs Multipart vs raw Bytes) and is more honestly expressed as runtime configuration.

  • Step 15 — seed reduced from 2 tests/service (16) to 1 test/service (8); RFC 9457 type-URL portion of the helper deferred; signature simplified. §D14.4 listed 16 specific tests with §D14.1’s assert_constraint_violation_returns(client, request_closure, ExpectedRejection, problem_type_url) — closure-based, requiring a generic TypedClient trait that doesn’t exist in the codebase, and assuming RFC 9457 problem-type URLs that also don’t exist (only 2 such URLs are emitted across the entire codebase today, both in craig-api/idempotency.rs). The signature was simplified to two response-shape helpers: assert_constraint_violation_status(actual_status, expected, context) for status-only assertion and assert_constraint_violation_returns(actual_status, body_type, expected, problem_type_url, context) for the full-fidelity assertion. The seed shipped is 1 test per service (8 tests total) against real handler rejection paths verified against a live devstack. Each test surfaces a real boundary (FK lookup failure, UNIQUE-conflict mapping to 409, typed-Json-extractor 422 on bad enum, etc.). The §D14.4 list assumed several validation behaviors that don’t exist in the codebase today — e.g., cases.admin_unit is not validated against the FIPS registry on create; financial.payments.amount > 0 is not enforced as a CHECK constraint at the DB layer (financial has zero user-driven DB constraints); cases.case_number UNIQUE collision is auto-retried server-side and not surfaceable through the API. The second-test-per-service expansion + the type-URL standardization initiative are tracked in §Potential improvements; both are real platform-improvement candidates that this plan should not have absorbed.

  • Step 14 — --diff <REF> flag deferred; route-to-client-method bridge ships as a stub config file. §D13.7 specified a --diff main flag that runs the matrix twice (current + base) and prints filled/empty cell deltas. Implementing it requires a clean git stash + git checkout + re-run dance from inside the subcommand, which conflicts with the user’s working-tree convention (running tools while holding uncommitted changes). The flag is not mission-critical for the baseline — the per-MR review signal it provides can be obtained by hand-diffing two runs. Deferred to a follow-up MR; tracked under §Potential improvements. Separately, §D13.2 specified a route-to-client-method.toml mapping that lets the matrix count typed-method invocations (not just literal URL strings). The mapping’s file ships in this MR with the agreed-upon entry shape and zero entries; the consumer (a scan_typed_calls() step alongside scan_tests()) lands when the Phase A.8 typed-DTO migration is far enough along to populate the mapping with real entries. Today the matrix only matches literal "/v1/…​".to_string() references in test source, which is why the baseline reports 0/1308 cells — current tests use typed-client method calls, not URL literals. The 0% baseline is honest signal, not a tooling bug.

  • Step 13 — syn span locations swapped for a streaming brace-balance scan in B2; B7 regex broadened to bare Value. §D12.4 sketched B2 (function-LOC over threshold) using syn to walk ItemFn/ImplItemFn and reading Span::start()/end(). Those positions require proc-macro2’s `span-locations feature, which the workspace doesn’t enable. Implemented B2 with a streaming text scan that tracks brace depth, skips strings/chars/line+block comments/raw strings, and identifies fn-signature braces by walking backward from { looking for the fn keyword. Three unit tests cover the basic case, brace-in-string false-positive guard, and nested-closure handling. B7 (untyped test-client methods) was originally a serde_json::Value regex on return types, but the repository convention is use serde_json::Value; at the top of each clients/*.rs file followed by bare Value in signatures. B7 now also matches bare Value when an import is in scope — which is what bumped the actual count from 0 (false negative) to 194 (true count). B6 (duplicate dep versions) also hit a counting bug — the originally specified cargo tree -d --prefix none (without --depth 0) returned the full reverse-dep tree under each duplicate, inflating counts; switched to --depth 0 which returns only the duplicate roots, then dedup’d by package name in a BTreeSet.

  • Step 12 — coverage-matrix and quality-budgets ship as stubs; mutants-baseline.toml population still deferred to a follow-up clean-tree run. §D11 frames Step 12 as wiring 6 stable entry points; the two report-generators (coverage-matrix and quality-budgets) belong to later steps (§D13 / §D12 respectively) and are intentionally registered as stubs in this MR so callers (CI dashboards, contributors, scripts) can pin a stable command name today. Each stub prints a STUB: line naming the step that finishes it. The four runner subcommands (reliability, invariants, contracts, mutants) are fully functional. The mutants-baseline.toml population that Step 11’s errata deferred to "after Step 12 lands" is still pending — it requires a clean working tree and a single cargo xtask mutants --smoke run; that’s a separate housekeeping commit, not a code change, and is filed against the Step 11 follow-up backlog rather than rolled into this MR.

  • Step 11 — config + empty baseline shipped; population deferred until Step 12’s xtask subcommand exists. §D10.5 Step 5 calls for running cargo xtask mutants --smoke and committing the populated xtask/mutants-baseline.toml. The xtask subcommand itself is Step 12 work (per §D10’s own cross-reference: "Subcommand wiring lives in Step 12 / §D11.3"). Step 11 ships the infrastructure: .cargo/mutants.toml workspace config (timeout, nextest runner, exclude_globs), xtask/mutants-baseline.toml as an empty template with the entry-shape spec, CONTRIBUTING.adoc "Optional dev tools" install line, .claude/docs/testing.md Mutation testing sub-section. The baseline gets populated immediately after Step 12 lands (a single cargo xtask mutants --smoke run on a clean working tree, committed as the floor). Documenting this seam keeps the plan honest about the dependency.

  • Step 10 — Content-Disposition property deferred; sanitize_filename + pagination reframed against existing fns; reference-enum list corrected; surfaced + fixed a real sanitize_filename non-idempotency bug. §D9.6 (Content-Disposition encoding round-trip) targets craig_store::validation::content_disposition_attachment which is platform-stab-2 §D8.1 work and not yet shipped — deferred. §D9.4 wrote the pagination property against a paginate(&data, PageRequest) helper that doesn’t exist in the codebase; reframed against PageRequest::offset/limit arithmetic (the actual SQL-layer commitment), same invariants. §D9.9 enumerated PaymentStatus, PlacementStatus, ReportStatus, AdminUnit, FipsCode, EventType — none of those types exist in craig-reference; replaced with 11 actual EnumString-derived enums (Gender, Race, Ethnicity, ReporterType, Priority, AbuseType, Disposition, ContactType, CourtOrderType, PlacementType, PermanencyGoal). The §D9.5 prop_sanitize_filename_invariants property surfaced a real non-idempotency bug on first run: leading-.-then-\t sequences had \t survive the first sanitize_filename call (because trim() ran before trim_matches('.'), while the dot was still leading). Fixed in the same MR by replacing the two-pass trim with a single combined predicate (c.is_whitespace() || c == '.') and adding a post-truncation re-trim pass for the case where 255-byte truncation creates a trailing whitespace. Property bit; production code corrected. body_fingerprint in craig-api/src/idempotency.rs was promoted from fn to pub fn per §D9.8 so out-of-crate property tests can lock the equality and one-byte-flip invariants — the value was already part of the cross-service contract via the persisted idempotency_responses.body_fingerprint column.

  • §D8 batch — IntakeClient (#307) excluded from the typed-DTO test-client migration by design (2026-06-20). §D8.3’s inventory listed 8 clients to type, row 8 being IntakeClient, on the assumption it just needed "a builder method per auth posture." Implementing the sibling migrations (#303 financial / #304 reporting / #305 rules / #306 security — all landed) revealed a category error in that framing: IntakeClient is the test suite’s untrusted-ingress simulator, not a trusted-write client. Its submit_public_report is the anonymous public-intake endpoint and the target of the evil-corpus contract test — the EvilCategory::Json case (crates/craig-test-lib/src/evil/intake_body.rs) sends arbitrary non-object JSON that a typed &SubmitReportRequest structurally cannot carry, and the partner submit methods are likewise probed with malformed bodies + JWS byte-fidelity (jws_error_uniformity, sanitization, cross_partner_bola). Typing the client would delete the capability those security tests exist to exercise. The correct architecture is already in place: the server owns the typed contract — submit_report(Json(_): Json<SubmitReportRequest>) deserializes-and-validates untrusted bytes (or 400s) — and the test client, like a real external caller, correctly speaks raw Value. A typed-happy-path + raw-_raw(&Value)-escape-hatch split was considered and rejected as a fig leaf. Scope correction: the §D8.3 "8 clients" inventory and the §D8.5 "0 untyped test clients post-migration" budget stand corrected to trusted-write clients onlyIntakeClient remains intentionally &Value. Dead intake client methods (register_signer_key/get_signer_key_public/list_organizations, targeting /keys routes retired to craig-security per ADR-017 Step 6) and the craig-web↔intake SubmitReportRequest shape-duplication are filed as #636 / #637. (#307 closed not-applicable-by-design.)

  • Step 9 redo — full DTO lift; CasesClient method migration scoped to follow-up #308. The original Step 9 MR (!200) shipped only a persons-module skeleton — 1 of 12 DTO modules, 0 method swaps, 0 test-callsite rewrites. Per the user’s pushback ("are you being lazy and deferring things that don’t need to be deferred?"), Step 9 was redone: this MR lifts all 28 request-side DTOs spanning 9 modules (case_plans, cases, contacts, court_orders, investigations, persons, referrals, report_persons, reports) into crates/craig-cases-contracts/, and services/craig-cases/src/api/.rs retains the API surface via pub use craig_cases_contracts::::*;. Response types tied to store::models::* (ReferralDetail, ReportDetail) and craig_matching::JsonbEntry (EntrySuggestions, PersonSuggestionsResponse) remain in craig-cases — lifting them requires moving the underlying store models, which is a separate refactor outside the typed-DTO scope. The CasesClient method-signature migration (60 methods) + ~30 test-callsite rewrites under services/craig-cases/tests/api/*.rs are tracked as #308 (not blocking Step 9 completion). The 7 sibling-client migrations are #301–#307. This MR’s "Step 9" completion now means: contracts crate has the full DTO surface, the cycle-free dev-dep edge from craig-test-lib works, and downstream method/test migration is mechanical lift from typed DTOs that already exist.

  • Step 9 — architectural pilot only; per-method migration deferred to 8 follow-up issues. (Original errata for !200; superseded by the redo above.) §D8.1 frames Step 9 as "MR1 of an 8-MR migration umbrella" and §D8.4 walks through ~60 method signatures that need typed-DTO swaps in crates/craig-test-lib/src/clients/cases.rs, plus ~30 test files at services/craig-cases/tests/api/.rs whose json!({…​}) call-sites would all break at compile. Step 9 ships the architectural skeleton: a leaf crates/craig-cases-contracts/ crate with the persons-module DTOs lifted (PersonSearchQuery, CreatePersonRequest, UpdatePersonRequest); craig-cases retains the API surface via pub use craig_cases_contracts::persons::; so handler code is unchanged; craig-test-lib gains [dev-dependencies] craig-cases-contracts and a typed_contracts smoke test that round-trips a CreatePersonRequest through serde without pulling craig-cases. The cycle guard the plan §D8.2 reserved for deny.toml is documented in deny.toml comments rather than enforced today: craig-cases is currently bin-only, so the structural guard is "no lib target — nothing CAN depend on it." The actual entries with wrappers = […​] allowlists land when the per-module migrations introduce real consumers. The remaining 11 craig-cases DTO modules + the actual CasesClient method-signature migration + the 7 sibling-client migrations are 8 follow-up issues filed under epic &22.

  • Step 8 — delivery-protocol.md cross-reference dropped (Tier 1 doc). §D7.5 prescribed adding a one-paragraph audit-rule pointer to .claude/docs/delivery-protocol.md. That file is a Tier 1 quickstart-template doc enforced by cargo xtask check-docs; downstream edits fail validation. The storage-shape-vs-semantics rule lives in the Process rule sub-section of .claude/docs/testing.md instead, which delivery-protocol.md’s existing "Plan Completion Audit" section already cross-references via its general testing.md pointer. Same outcome, no Tier 1 drift.

  • Step 7 cleanup — partial filename-attack table shipped. The original Step 7 errata deferred the entire filename_attack table to platform-stab-2 §D8.1 because some cases (CRLF, RTLO, fullwidth) depend on a sanitizer extension that hasn’t shipped. The user pushed back that this was lazy — a subset of cases works against the current sanitize_filename. Shipped craig_test_lib::filename_attack::CASES with 8 cases the current sanitizer handles correctly (forward_slash, backslash, nul_byte, leading_trailing_whitespace, leading_trailing_dots, all_dots, empty_string, mixed_separators) plus a round-trip smoke test + count-pin. The CRLF / RTLO / fullwidth / length_256 / unicode_resume / double_quote / unc_path rows + the expected_disposition half of every row still ship with platform-stab-2 §D8.1 — the harness exists today so that MR is purely additive (append rows, bump the count-pin, add disposition assertions).

  • Step 7 — upload corpus + loader shipped; filename-attack table + magic-byte rejection assertion deferred to §D8.1. (Original errata; partially superseded by the cleanup above for the filename-attack table portion. Magic-byte rejection assertion remains §D8.1-dependent.)

  • Step 6 cleanup — nbf enforcement added; mutation restored. The original Step 6 errata dropped NotYetValidOneMinute because Validation::validate_nbf defaults to false and the validator didn’t opt in. The user pushed back that this was lazy ("ADR-021 is about strictness; that’s in-spirit"). Restored: craig_auth::jwks::JwksProvider::validate_token now sets validation.validate_nbf = true (defense-in-depth — tokens without nbf are unaffected since jsonwebtoken treats absent nbf as immediately valid; tokens with future nbf past the default 60s leeway now reject). The mutation is back in the matrix, renamed NotYetValid and using a +5-minute offset to clear the leeway window. The AlgNone substring narrowing remains (upstream library reality — jsonwebtoken’s header parser refuses alg: none at deserialization time, surfacing as a serde "unknown variant \`none\`" error rather than a clean alg-allowlist rejection).

  • Step 6 — nbf mutation dropped, alg=none expected-error substring narrowed. (Original errata; superseded by the cleanup above.)

  • Step 5 — minimal lib.rs added to craig-financial for integration-test access. §D4.3 specifies services/craig-financial/tests/transitions_matrix.rs as an integration test, but craig-financial was bin-only (no lib.rs), so tests/.rs could not import craig_financial::transitions::. Shipped a minimal services/craig-financial/src/lib.rs exposing pub mod transitions; — the bin target keeps its mod transitions; declaration in main.rs (separate compilation unit). This is the minimum change that unblocks the §D4.5 sibling adoptions across the other 5 services with state machines (each will land an analogous lib.rs in the MR that adopts the matrix). Errata documents the pattern.

  • Step 4 — --replicas infrastructure shipped; full §D3.8 verification test deferred. §D3.8’s verification test asserts "exactly one row in cases" given concurrent same-key POSTs across replicas — that’s the strong-claim assertion of the platform-stab-2 #274 atomic-claim fix. Without #274 the current Postgres-backed idempotency middleware may not handle the race correctly under contention, so the test as specified would be testing unfixed behavior. Step 4 ships the infrastructure (--replicas xtask flag, docker-compose.scale.yml override, per-replica health-wait, MultiReplicaCluster test helper with unique-project-name + start/stop/replica_urls) plus 3 lifecycle / project-name-isolation tests gated by CRAIG_MULTI_REPLICA_SMOKE=1 (opt-in, since spinning up a dedicated cluster takes ~30-60s and would slow pre-push). The full §D3.8 verification test ships in platform-stab-2 #274 alongside the atomic-claim fix it actually validates. The X-Served-By tower middleware (Path C in §D3.8) is a service-side change outside test-framework scope; ships when needed by Phase B tests.

  • Step 3 — harness foundation only; base injectors deferred per failing-test-first protocol. §D2.0 mandates that every new injector ships with (a) the production seam refactor, (b) the injector wrapper, and (c) one test that uses the injector against a real codepath. Step 3 ships the harness foundation (FaultInjector trait + Attempt + ScenarioGuard with Drop assertion), the 3 new generic wrappers (LatencyInjector, IntermittentFlapInjector, ConsumerBackpressureInjector), the scenarios::backpressure::* named-constructor sub-module, and 10 smoke tests. The 4 base injectors named in §D2.0a (CipherErrorInjector, RabbitDownInjector, ObjectStoreErrorInjector, PublishInTxFailureInjector) require production seam refactors and have no current test consumer; they are deferred — each ships in the MR that lands its first failing-test consumer (platform-stab-2 P0/P1 fixes for the seams the fixes need; Phase B Step 19 for the rest). The db::*, rmq::*, cipher::*, object_store::* sub-modules of scenarios.rs are populated as their base injectors land. ConsumerBackpressureInjector::handle_delivery is generic over an opaque message type rather than concrete lapin::message::Delivery so smoke tests can pass any value while production passes the real type.

After this plan lands

  • 8 test categories operational across the workspace: happy / sad / evil / concurrency / replay / fault / state-machine / property

  • The happy/sad/evil convention is enforced via reviewer discipline + the risk-coverage scorecard’s living top-20 grid — every code-changing MR’s diff is checked against the scorecard’s pending cells before merge

  • Race conditions surface in CI / pre-push instead of by external review — the platform-stab-2 sibling fixes are the last cohort to require external-review surfacing

  • Mutation testing surfaces test-without-assertion gaps quarterly via cargo xtask mutants reports; a dashboard graph tracks surviving mutants per crate over time

  • The invariant sweeper catches post-test-run state violations that no individual test asserts — orphan rows, stuck pending tx, missing audit rows — turning silent inconsistencies into actionable failures

  • Sibling epic &21 (Platform Stabilization Phase 2) ships with proper coverage; the platform-stab-2 plan’s "minimum inline test infrastructure" (Step 2) is folded into the consolidated Phase A modules without breaking callers

  • The CRAIG test framework is structurally and semantically positioned to catch the next class of correctness gap before it reaches an external reviewer

This is foundational work. Treat it as a dependency for everything safety-critical that follows.

Edit this page · latest