Plan: Test Framework Hardening — 8 Test Categories, Happy/Sad/Evil Coverage, Failure-Path Infrastructure
On this page
- Status
- Context
- Test Categories taxonomy
- Scope
- Two-phase structure
- Wave sequencing
- Design
- D1. concurrent_fire helper consolidation (Phase A.1 — Step 2)
- D2. Fault-injection harness consolidation (Phase A.2 — Step 3)
- D3. Multi-replica devstack profile (Phase A.3 — Step 4)
- D4. State-machine matrix-testing pattern (Phase A.4 — Step 5)
- D5. JWT mutation library (Phase A.5 — Step 6)
- D6. Magic-byte upload-spoofing fixtures (Phase A.6 — Step 7)
- D7. testing.md "Failure-path testing" + Test Categories taxonomy + Process rule (Phase A.7 — Step 8)
- D8. Typed DTOs across 8 test clients (Phase A.8 — Step 9)
- D9. proptest scaffolding + first 7 properties (Phase A.9 — Step 10)
- D10. cargo-mutants smoke on 5 critical crates (Phase A.10 — Step 11)
- D11. xtask subcommands: reliability + contracts + mutants + coverage-matrix + invariants (Phase A.11 — Step 12)
- D12. Code-quality budget warnings (Phase A.12 — Step 13)
- D13. Coverage Matrix tooling —
xtask coverage-matrix(Phase A.13 — Step 14) - D14. DB Constraint Evil Tests — hand-curated harness (Phase A.14 — Step 15)
- D15. Central Evil Input Corpus consolidation (Phase A.15 — Step 16)
- D16. Restart/Replay light helpers (Phase A.16 — Step 17)
- D17. Invariant Sweeper —
xtask invariants(Phase A.17 — Step 18) - DB. Phase B — adoption bundle (Step 19)
- Steps
- Step 1: Plan adoc + nav.adoc Active + CHANGELOG entry + 9 new GitLab issues
- Step 2: Phase A.1 — concurrent_fire helper consolidation
- Step 3: Phase A.2 — Fault-injection harness consolidation
- Step 4: Phase A.3 — Multi-replica devstack profile
- Step 5: Phase A.4 — State-machine matrix-testing pattern + Payment first adoption
- Step 6: Phase A.5 — JWT mutation library
- Step 7: Phase A.6 — Magic-byte upload-spoofing fixtures
- Step 8: Phase A.7 — testing.md "Failure-path testing" + Test Categories taxonomy + Process rule
- Step 9: Phase A.8 — Typed DTOs across 8 test clients (8 MRs, this step lands MR 1)
- Step 10: proptest scaffolding + first 7 properties
- Step 11: cargo-mutants smoke on 5 critical crates
- Step 12: xtask subcommands: reliability + contracts + mutants + coverage-matrix + invariants
- Step 13: Code-quality budget warnings
- Step 14: Coverage Matrix tooling
- Step 15: DB Constraint Evil Tests — hand-curated harness
- Step 16: Phase A.15 — Central Evil Input Corpus
- Step 17: Phase A.16 — Restart/Replay light helpers
- Step 18: Phase A.17 — Invariant Sweeper xtask
- Step 19: Phase B — adoption bundle (10 tests)
- Step 20: Plan completion audit + archive
- Files Touched
- Verification
- Documentation Updates
- Open questions
- Potential improvements
- Errata
- After this plan lands
Status
| Step | Description | Status |
|---|---|---|
1 |
Plan + sibling platform-stab-2 plan + epics &21/&22 + step issues #273–#291 (existing) + 9 new step issues for Phase A.9–A.17 + nav.adoc Active entry. No code changes. |
Done (pre-ADR-030) — this MR |
2 |
Phase A.1 — |
Done (pre-ADR-030) |
3 |
Phase A.2 — Fault-injection harness consolidation. Promotes platform-stab-2 Step 2’s 5 inline injectors into |
Done (pre-ADR-030) — harness foundation; base injectors deferred per §Errata |
4 |
Phase A.3 — Multi-replica devstack profile. |
Done (pre-ADR-030) — verification test deferred to platform-stab-2 #274 per §Errata |
5 |
Phase A.4 — State-machine matrix-testing helper + Payment first adoption. Hand-rolled (bounded alphabet); |
Done (pre-ADR-030) |
6 |
Phase A.5 — JWT mutation library. |
Done (pre-ADR-030) — per-service adoption is a Phase B follow-up; library + smoke landed |
7 |
Phase A.6 — Magic-byte upload + filename-attack fixtures. PE/ELF/zip-bomb/script-disguised + filename-attack table (CRLF, NUL, RTLO, UNC, etc.). |
Done (pre-ADR-030) — upload corpus + loader; filename-attack table deferred to platform-stab-2 §D8.1 per §Errata |
8 |
Phase A.7 — |
Done (pre-ADR-030) |
9 |
Phase A.8 — Typed DTOs across 8 test clients (Contract tests). Full per-client rewrite, 8 small MRs starting with CasesClient. Decision: per-service |
Done (pre-ADR-030) — full CasesClient migration — 28 DTOs in contracts crate, 62 method signatures typed, 17 test files rewritten; #308 closed |
10 |
Phase A.9 — proptest scaffolding + first 7 properties. State transitions, pagination, filename sanitization, Content-Disposition encoding, canonical JSON signing, idempotency body fingerprint, reference-data validation never panics. |
Done (pre-ADR-030) — 6 of 7 properties shipped; Content-Disposition deferred per §Errata; surfaced + fixed a real |
11 |
Phase A.10 — cargo-mutants smoke on 5 critical crates (craig-api, craig-mq, craig-auth, craig-store, transitions modules). Baseline report committed; surviving mutants catalogued. NOT a pre-push gate. |
Done (pre-ADR-030) — config + empty baseline + docs; population deferred to post Step 12 xtask wrapper per §Errata |
12 |
Phase A.11 — xtask subcommands: |
Done (pre-ADR-030) — 6 subcommands wired; |
13 |
Phase A.12 — Code-quality budget warnings. Module-LOC, function-LOC, |
Done (pre-ADR-030) — full B1–B7 counters wired; xtask/quality-budgets.toml + xtask/quality-budgets.lock baselined; CONTRIBUTING.adoc Quality budgets subsection added; 15 unit + integration tests; report-only, no enforcement gate per §D12 |
14 |
Phase A.13 — Coverage Matrix tooling: |
Done (pre-ADR-030) — full subcommand wired; live-fetch via docker::get_host_port + cache mode; baseline committed to operations/coverage-matrix.adoc; 12 unit + integration tests; --diff flag deferred per §Errata |
15 |
Phase A.14 — DB Constraint Evil Tests harness (hand-curated). |
Done (pre-ADR-030) — helper module + status-only + type-URL assertion modes; 16/16 tests passing — 2 per service across all 8 services; first 8 shipped in MR !209, second 8 in the review-fix MR !212; RFC 9457 type-URL standardization tracked as #311 |
16 |
Phase A.15 — Central Evil Input Corpus consolidation. |
Done (pre-ADR-030) — 12-category corpus with 83 cases; all_evil_cases / evil_cases_for / parametric_evil_test! public API; self-test + global contract test + macro-demo + 8 per-service |
17 |
Phase A.16 — Restart/Replay light helpers. |
Done (pre-ADR-030) — helpers + |
18 |
Phase A.17 — Invariant Sweeper. |
Done (pre-ADR-030) — sweeper replaces Step-12 stub; 4 SQL invariants seeded; |
19 |
Phase B — adoption bundle. 10 specific failure-path tests (concurrent same-key idempotency, multi-replica outbox, inbox handler-fail-then-retry, upload content-spoofing, JWT strictness × 8 services, /livez vs /readyz differentiation, state-machine matrix on all 16 machines, cross-service reconciliation orphan-injection, outbox publish-failure retry, exchange crash-recovery) + risk-coverage scorecard for top-20 critical workflows. |
Done (pre-ADR-030) — 3 of 10 tests shipped — Test 8 BFF malformed upstream, Test 9 craig-store filename × evil corpus, Test 10 typed-DTO pilot; 7 platform-stab-2-dependent tests deferred to backlog; risk-coverage scorecard at docs/…/test-coverage-scorecard.adoc + nav entry; cargo xtask coverage-matrix --emit-scorecard flag wired |
20 |
Plan completion audit + archive. Spawn audit subagent per delivery-protocol.md; flip Status all-Complete; move plan from Active to archive Infrastructure & Reliability; update CLAUDE.md Phase Status (add Test Framework Hardening row); CHANGELOG wrap-up entry; close epic &22. |
Done (pre-ADR-030) — audit subagent run; Status table reconciled with shipped reality across all 19 prior steps; Step 12/15/16 rows + 2 §Errata/§Potential bullets corrected; plan moved from Active to archive; CLAUDE.md Phase Status updated; CHANGELOG wrap-up entry; epic &22 closed |
Epic: &22
Issues: #283–#291 (existing — 8 original Phase A foundation issues + 1 Phase B adoption bundle) plus 9 new issues to file in Step 1 of this MR for Phase A.9–A.17
Sibling: epic &21 Platform Stabilization Phase 2 — its P0/P1 fixes ship with inline-minimal versions of A.1+A.2 (concurrent_fire + 5 fault injectors) so they’re not blocked on Phase A consolidation
Branch prefix: feat/test-framework- / fix/test-framework- / chore/test-framework-
*Milestone: 2026 Q3 — Platform Reliability
Context
The 2026-05-03 external review surfaced 3 P0 race conditions (idempotency, outbox, inbox) plus 4 P1 durability gaps. The first 5-agent audit had marked the relevant invariants "verified shipped" because they verified STRUCTURE (table exists, helper exposed, worker spawned) without verifying SEMANTICS (concurrency safety, retry behavior, atomic claim ordering).
But there’s a second-order finding the user surfaced: the existing test framework structurally cannot catch these races. Even if the audits had been semantically rigorous, the suite would not surface the gaps in CI:
-
All integration tests are sequential single-fire — no helper for concurrent same-key requests
-
Devstack is single-instance — no multi-replica mode; outbox double-publish never triggers
-
No fault injection — cipher / DB / RabbitMQ / object-store errors can’t be deterministically simulated
-
Nextest concurrency is per-test-file, not per-request
-
No property-based / state-space testing
-
No mutation testing — coverage-by-line says "covered" while assertions never bite
-
No invariant sweeper — DB-state violations after test runs go undetected
-
k6 load tests run against a single replica
-
Test clients use
serde_json::Valueeverywhere, so contract drift surfaces only at runtime, not at compile time
This epic builds the missing infrastructure first (Phase A) so that platform-stab-2’s fixes can ship with proper coverage (Phase B), and so future race conditions can be caught in CI rather than by external review.
The user has further specified an 8-category test taxonomy with a happy/sad/evil sub-axis, and listed 10 specific tests that should ship first. Both are folded into this plan.
The methodology lesson — that storage-shape verification (table exists, helper exposed, worker spawned) is not equivalent to semantic verification (locking, claim ordering, retry behavior) under concurrency — is captured in the two-pass platform-invariant audit and reinforced in platform-stab-2’s Context section. This plan operationalizes that lesson by giving every architectural promise a testable form: idempotent means raced (Phase B test 1), durable means failed and replayed (test 5), typed means compile-checked (Phase A.8), secure means bad inputs and bad config are exercised (Phase A.5/A.6/A.15).
Test Categories taxonomy
This plan operationalizes 8 explicit test categories. Phase A.7 (Step 8) encodes them into .claude/docs/testing.md + a per-category file-naming convention; Phase A.13 (Step 14) builds tooling that surfaces coverage gaps.
| Category | What it asserts | Helpers (Phase A piece) |
|---|---|---|
Functional |
Endpoint behaves as specified on the happy path |
Existing nextest harness — status quo |
Invariant |
Facts that must always hold (no duplicate side effects, no over-capacity placements, no orphan rows, no double-processed events) |
|
Concurrency |
Behavior under intentional races (same-key POSTs, multi-worker drains, simultaneous state changes) |
|
Fault-injection |
Behavior when a specific dependency fails at a specific boundary |
Fault injectors (A.2) + restart helpers (A.16) |
Recovery |
Behavior across crash/restart between phases of multi-step workflows |
Restart helpers (A.16) + invariant sweeper (A.17) |
Contract |
Schema drift breaks at compile time, not runtime |
Typed DTOs (A.8) + coverage-matrix (A.13) |
Property-based |
Invariants on unbounded inputs (filename sanitization, JSON canonicalization, fingerprinting) |
proptest scaffolding (A.9) |
Mutation |
The tests assert meaningfully — surviving mutants surface lines that are exercised but not asserted |
cargo-mutants smoke (A.10) |
Sub-axis: every Functional / Invariant / Fault-injection test should be classified as one or more of:
-
happy — the success path the spec describes
-
sad — predictable rejections (validation, 4xx, expected errors)
-
evil — adversarial inputs (oversized strings, malformed signatures, path traversal, magic-byte spoofing, JWT strictness, etc.)
Coverage-matrix tooling (A.13) inventories which axes have a test for each route. Risk-coverage > line-coverage.
Scope
In scope:
-
17 Phase A foundation pieces: helpers, harnesses, fixtures, xtask subcommands, code-quality budgets, evil corpus, invariant sweeper, coverage-matrix tooling
-
10 Phase B adoption tests
-
Risk-coverage scorecard for top-20 critical workflows
-
.claude/docs/testing.mdextension (Test Categories + helper docs + process rule) -
.claude/docs/delivery-protocol.mdfailure-path-test process rule -
CONTRIBUTING.adocpointer -
9 new GitLab issues filed in Step 1 for Phase A.9–A.17
Out of scope (deliberate or deferred):
-
CI-pipeline gating of any new xtask subcommand. Per user directive: tools available, no CI pressure today. Future plan covers gate graduation.
-
Pre-push enforcement of code-quality budgets. Lock-file baseline ships; reviewer discipline holds the line; opt-in
--fail-on-regressionavailable. -
Auto-derived DB-constraint evil tests from migration SQL. Pushed back: hand-curated harness + per-service convention instead.
-
Untested-large-file targeted sad/evil test backfill (7 files) — overlaps with platform-stab-2 §D10.1 large-file decomposition; tracked as follow-up plan.
-
New domain features (any of the standing-backlog items)
-
AWS deployment thread (#163 / #172 / #199 — owned by external team)
Two-phase structure
Phase A — Foundation (build the helpers, harnesses, tooling)
Seventeen discrete pieces — each is a small, focused MR; most are parallelizable across the team:
-
concurrent_fireconsolidation (extends platform-stab-2 inline) -
Fault-injection harness consolidation (extends platform-stab-2 inline + adds 3 new injectors)
-
Multi-replica devstack profile (
cargo xtask dev start --replicas) -
State-machine matrix-testing pattern + Payment first adoption
-
JWT mutation library — mint malformed tokens for strictness tests
-
Magic-byte upload-spoofing + filename-attack fixtures
-
testing.md "Test Categories" taxonomy + file-naming convention + Failure-path testing helpers section + process rule
-
Typed DTO migration across 8 test clients (Contract tests)
-
proptest scaffolding + first 7 properties
-
cargo-mutants smoke on 5 critical crates
-
xtask subcommands: reliability / contracts / mutants / coverage-matrix / invariants / quality-budgets (available, not enforced)
-
Code-quality budget warnings (warnings only, lock-file baseline)
-
Coverage Matrix tooling (
xtask coverage-matrixREPORT-only) -
DB Constraint Evil Tests — hand-curated harness
-
Central Evil Input Corpus consolidation
-
Restart/Replay light helpers
-
Invariant Sweeper —
xtask invariants
Phase B — Adoption (10 specific tests + risk-coverage scorecard)
Ten user-listed failure-path tests, each tagged with which Phase A piece it depends on. Plus a risk-coverage scorecard at docs/modules/ROOT/pages/test-coverage-scorecard.adoc listing top-20 critical workflows × 6 coverage axes.
Many Phase B tests double as the failing-then-passing test for a sibling platform-stab-2 fix. Coordination expected.
Wave sequencing
-
W1: Step 1 (this plan + sibling plan + epics + step issues; solo, blocks all)
-
W2 — parallelizable: Phase A foundation work — Steps 2–18 are largely independent; can be picked up in any order. Long-running pieces (A.8 typed DTO migration, A.10 mutation testing, A.13 coverage matrix tooling) run in parallel with shorter pieces. Recommended order for the team to start with the highest leverage items: A.1 → A.2 → A.7 → A.13 → A.15 → others.
-
W3: Phase B adoption (Step 19). Each Phase B test depends on its corresponding Phase A helper landing first. Many Phase B tests are also the failing-then-passing test for a sibling platform-stab-2 (epic &21) fix — coordinate landings.
-
W4: Step 20 (audit + archive).
Design
D1. concurrent_fire helper consolidation (Phase A.1 — Step 2)
Sibling plan platform-stabilization-2 (epic &21) Step 2 introduces a
minimal inline concurrent_fire(n, builder) at
crates/craig-test-lib/src/concurrent.rs and 5 fault injectors at
crates/craig-test-lib/src/fault_injection.rs. Those modules ship as
the smallest viable harness to verify epic &21’s Steps 3–9; this plan’s
Phase A.1 (Step 2) and A.2 (Step 3) consolidate and extend them rather
than re-introducing parallel surfaces.
The minimal concurrent_fire from epic &21 Step 2 has the shape:
// crates/craig-test-lib/src/concurrent.rs (epic &21 Step 2 — current state)
pub async fn concurrent_fire<F, R, Fut>(n: usize, builder: F) -> Vec<R>
where
F: Fn(usize) -> Fut + Send + Sync,
Fut: std::future::Future<Output = R> + Send + 'static,
R: Send + 'static,
{
let mut handles = Vec::with_capacity(n);
for i in 0..n {
handles.push(tokio::spawn(builder(i)));
}
let mut results = Vec::with_capacity(n);
for h in handles {
results.push(h.await.expect("concurrent_fire task panicked"));
}
results
}
It covers exactly one shape: spawn-all-then-await-all, panic-propagating
via .expect, results in spawn-task-completion order (JoinHandle await
order, not spawn order — which the doc string implies but the impl
does not guarantee). Several Phase B failure-path tests (#291) need
shapes this helper cannot express:
-
Multi-replica outbox concurrent test (sibling Step 4 of &21 — and Phase B test for #275) needs a synchronized "go" so both worker drains start within the same tokio-tick window. Without a barrier, worker 1 typically wins the race and worker 2 finds an empty
event_outboxtable. -
Idempotency atomic-claim test needs to assert that exactly one handler-side effect lands AND that the per-task latency distribution shows a winner-shaped first response and N-1 loser-shaped replays (loser latency ≥ winner latency by construction). The current helper drops latency.
-
State-machine transition reject tests often need "fire many; first K successes are enough" semantics — currently expressible only via ad-hoc loops outside the helper.
-
Panic-as-bug surfacing — the current
.expectpanics the test thread, which under nextest is recoverable but loses provenance (which task panicked? what input?).
D1.1. Decision: in-place extension, not migration
Two viable consolidation strategies were considered:
| Option | Trade-offs |
|---|---|
(a) In-place extension — extend |
Pro: zero churn for the &21 Phase 2 fix tests already authored against the minimal API. Pro: single module, single import path. Con: the module accretes a non-trivial surface (~6 functions + a helper struct). |
(b) Migrate to richer module + re-export shim — create
|
Pro: cleaner separation of variants. Con: doubles the import surface during the transition. Con: the re-export shim is dead weight as soon as the &21 callers migrate; no other consumer. |
Decision: (a) in-place extension. The richer API is naturally expressible as 4 sibling functions + 1 struct in one file (~250 LOC target). There are no other consumers to keep stable; the &21 Phase 2 tests are the only callers and they live in this same workspace. Splitting into submodules buys nothing while doubling import paths.
The one rename this plan accepts: the existing minimal helper is
renamed to concurrent_fire_collect (matching the new naming family —
_collect, _collect_ordered, _with_barrier, _until_first_success,
_first_n) and concurrent_fire becomes a deprecated alias re-exported
for one minor version. Errata gets an entry; epic &21 Phase 2 callers
swap names in the same MR that lands this consolidation.
D1.2. New helper inventory
// crates/craig-test-lib/src/concurrent.rs
use std::future::Future;
use std::pin::Pin;
use std::sync::Arc;
use std::time::{Duration, Instant};
use tokio::sync::Barrier;
use tokio::task::JoinHandle;
/// Outcome of a single task in a concurrent run.
#[derive(Debug)]
pub struct TaskOutcome<R> {
/// 0-based index passed to the builder.
pub index: usize,
/// Wall-clock duration from spawn to handler return.
pub latency: Duration,
/// Wall-clock instant the task was spawned (monotonic, set under
/// the same `Instant::now()` epoch as `latency`'s start).
pub started_at: Instant,
/// Value returned by the builder future, or panic info.
pub result: TaskResult<R>,
}
#[derive(Debug)]
pub enum TaskResult<R> {
Ok(R),
Panicked { message: String },
}
/// Aggregate report over a `concurrent_fire_*` invocation.
pub struct ConcurrentRunReport<R> {
/// One entry per task, ordered by `started_at` ascending.
pub outcomes: Vec<TaskOutcome<R>>,
/// Wall-clock duration from first-spawn to last-return.
pub total: Duration,
}
impl<R> ConcurrentRunReport<R> {
/// Number of tasks whose result is `TaskResult::Ok(_)`.
pub fn ok_count(&self) -> usize { /* ... */ }
/// Number of tasks whose result is `TaskResult::Panicked { .. }`.
pub fn panic_count(&self) -> usize { /* ... */ }
/// Returns `Ok(values)` if all tasks completed without panic, else
/// `Err(panic_messages)`. Most tests want this.
pub fn into_results(self) -> Result<Vec<R>, Vec<String>> { /* ... */ }
}
/// Spawn N concurrent tokio tasks, await all, return values in *spawn
/// order* (index 0..N-1). Panics in tasks panic the test thread —
/// matches the current minimal helper's behavior, kept for back-compat.
///
/// When-to-use: drop-in replacement for the &21 Step 2 minimal helper.
pub async fn concurrent_fire_collect<F, Fut, R>(n: usize, builder: F) -> Vec<R>
where
F: Fn(usize) -> Fut + Send + Sync,
Fut: Future<Output = R> + Send + 'static,
R: Send + 'static;
/// Same as `concurrent_fire_collect` but returns a full `ConcurrentRunReport`
/// with per-task latency, ordered-by-start-instant, and panic capture.
///
/// When-to-use: any test that needs to assert *winner-vs-loser shape*
/// (idempotency replay tests), latency distribution, or that wants
/// panic-as-data instead of panic-as-test-failure.
pub async fn concurrent_fire_collect_ordered<F, Fut, R>(
n: usize,
builder: F,
) -> ConcurrentRunReport<R>
where
F: Fn(usize) -> Fut + Send + Sync,
Fut: Future<Output = R> + Send + 'static,
R: Send + 'static;
/// Spawn N tasks, all wait on a `tokio::sync::Barrier`, then proceed
/// together. Closes the spawn-skew window: without a barrier, task 0
/// is typically 50–500µs ahead of task N-1; with a barrier they all
/// release within one tokio-scheduler tick.
///
/// When-to-use: outbox/inbox concurrent-claim tests where the test is
/// asserting "two workers race to claim the same row" *and* needs an
/// explicit pre-claim semantic synchronization point (e.g. all N
/// pre-fetch claims_state, then barrier-wait, then race to UPDATE).
/// The barrier is exposed to the caller so the builder can decide
/// where to synchronize.
///
/// Caveat: this variant cannot guarantee the caller actually awaited
/// the barrier. A builder that ignores its `Arc<Barrier>` argument
/// produces the same staggered-start behavior as plain `concurrent_fire`.
/// For tests that just want "all start together, no caller-controlled
/// sync point", use `concurrent_fire_synchronized` (below).
pub async fn concurrent_fire_with_barrier<F, Fut, R>(
n: usize,
builder: F,
) -> ConcurrentRunReport<R>
where
F: Fn(usize, Arc<Barrier>) -> Fut + Send + Sync,
Fut: Future<Output = R> + Send + 'static,
R: Send + 'static;
/// Spawn N tasks; the helper itself awaits a `Barrier::new(n)` *immediately
/// before* invoking each `builder(i)` closure, so all N closures begin
/// execution at the same scheduler tick. The barrier is internal to the
/// helper; the caller's builder receives only `i: usize` and runs as if
/// the synchronization never existed.
///
/// When-to-use: tests that simply need "all start together" without a
/// caller-controlled sync point. Idempotency same-key concurrent POST
/// (Phase B test 1), JWT-mutation parametric tests, basic outbox-claim
/// races. Strictly stronger than `concurrent_fire` for catching staggered-
/// start false-negatives, with no API surface caller can leak.
///
/// This is the recommended default. Reach for `concurrent_fire_with_barrier`
/// only when the test needs a *post-prep, pre-race* sync point inside the
/// builder body.
pub async fn concurrent_fire_synchronized<F, Fut, R>(
n: usize,
builder: F,
) -> ConcurrentRunReport<R>
where
F: Fn(usize) -> Fut + Send + Sync,
Fut: Future<Output = R> + Send + 'static,
R: Send + 'static;
/// Spawn N tasks, return as soon as the *first* task returns `Ok(_)`
/// (or all N return Err). Cancels the remaining tasks' futures on
/// success.
///
/// When-to-use: state-machine "transition is allowed from this state"
/// tests where any successful transition proves the invariant; or
/// JWT-mutation tests (Phase A.5) where any one of N malformed tokens
/// being accepted is a failure case.
pub async fn concurrent_fire_until_first_success<F, Fut, R, E>(
n: usize,
builder: F,
) -> Result<R, Vec<E>>
where
F: Fn(usize) -> Fut + Send + Sync,
Fut: Future<Output = Result<R, E>> + Send + 'static,
R: Send + 'static,
E: Send + 'static;
/// Spawn N tasks, return as soon as `k` tasks return `Ok(_)`. Aborts
/// the remaining tasks. Returns the first `k` successes in completion
/// order; if fewer than `k` succeed before all N finish, returns Err
/// with the collected errors.
///
/// When-to-use: "exactly K of N concurrent claims succeed" assertions.
/// E.g., concurrent same-key idempotency: K=1 of N=5; concurrent foster
/// home capacity: K=capacity of N=capacity+overflow.
pub async fn concurrent_fire_first_n<F, Fut, R, E>(
n: usize,
k: usize,
builder: F,
) -> Result<Vec<R>, Vec<E>>
where
F: Fn(usize) -> Fut + Send + Sync,
Fut: Future<Output = Result<R, E>> + Send + 'static,
R: Send + 'static,
E: Send + 'static;
D1.3. Implementation notes
-
All variants spawn via
tokio::spawn(nottokio::task::spawn_blocking) — concurrency targets I/O-bound async code paths. -
concurrent_fire_collect_orderedinstruments viaInstant::now()inside the spawned future (post-tokio::spawn, before awaiting the builder) so thatstarted_atreflects the moment the runtime actually picked up the task, not the momenttokio::spawnreturned. The difference is microseconds in CI, but matters for asserting "barrier-synchronized" claims in tests that exercise the barrier variant alongside the ordered variant. -
concurrent_fire_with_barrierconstructsArc::new(Barrier::new(n))internally so callers do not have to thread the barrier construction. Each task receives theArc<Barrier>and is expected to callbarrier.wait().awaitat its own synchronization point — typically the first line of the builder closure body. -
concurrent_fire_until_first_successandconcurrent_fire_first_nuseJoinSet::abort_all()for cancellation. The aborted tasks' futures get aCancelledjoin error which is dropped (not surfaced inVec<E>) — only futures that naturally completed withErr(e)contribute to the error vector. -
Panic capture uses
JoinError::is_panic()+Any::downcast_ref::<&str>/Any::downcast_ref::<String>. Falls back to"panic with non-string payload"for non-string panic payloads.
D1.4. Test descriptions
For each new helper, smoke tests live at
crates/craig-test-lib/tests/concurrent.rs (consolidating the &21
Step 2 file of the same name). Tests:
-
fire_collect_zero_tasks—concurrent_fire_collect(0, |_| async { 1 })returns emptyVec. No spawns, no panics. -
fire_collect_single_task— N=1; builder receives index 0; result vector has length 1. -
fire_collect_panic_propagates— N=3; builder panics on index=1; the test wraps the call intokio::spawn(…).await.unwrap_err().is_panic()to assert the panic surfaces. -
fire_collect_all_panic— N=5; every builder panics; assertion as above. -
fire_collect_mixed_latency— N=10; builders sleep100ms * (i+1); result vector still has length 10 and contains every index value (set-equality, not order-equality). -
fire_collect_ordered_winner_shape— N=5 against anAtomicUsizefetch_add(1);into_results()returnsOk(vec![0,1,2,3,4]);outcomes[0].started_at ⇐ outcomes[4].started_at;panic_count() == 0. -
fire_collect_ordered_panic_capture— N=3; index=1 panics with"boom";panic_count() == 1; the corresponding outcome’sTaskResult::Panicked { message }contains"boom". -
fire_with_barrier_synchronizes_start— N=5; each builder recordsInstant::now()afterbarrier.wait().await. The max(started_at) - min(started_at) delta is asserted < 5ms (one scheduler tick). Compare against `fire_collect_ordered’s typical delta of ≥ 50µs to demonstrate the barrier reduces skew. -
fire_until_first_success_short_circuits— N=10; builders 0–8 returnErr(i); builder 9 returnsOk(99)after a 50ms delay; index-3 builder also has a 100ms sleep + would have returnedOk(33); the helper returnsOk(99)(the first to flip to Ok wins) and the index-3 task is aborted before it can complete. -
fire_until_first_success_all_err— N=5; every builder returnsErr(i); helper returnsErr(vec![0,1,2,3,4])(order-independent). -
fire_first_n_exact_count— N=10, K=3; 5 builders returnOk, 5 returnErr; helper returnsOkwith exactly 3 values; the remaining 7 tasks are aborted. -
fire_first_n_under_target— N=5, K=3; only 2 builders return Ok; helper returnsErrwith 3 collected errors.
Path A/B/C structure for the most concurrency-critical case:
-
Path A — barrier reduces start-skew below tokio tick boundary:
fire_with_barrier_synchronizes_start(above). -
Path B — without barrier, skew exceeds tick: control test using
fire_collect_orderedwith the same builder; assert max(started_at) - min(started_at) ≥ 20µs (almost always true on CI). -
Path C — barrier deadlock surfaced as test timeout: N=5 with one builder that returns without calling
barrier.wait().await; the remaining 4 tasks block on the barrier forever; the test wraps the call intokio::time::timeout(Duration::from_secs(2), …)and asserts Err. This documents the failure mode for callers.
D2. Fault-injection harness consolidation (Phase A.2 — Step 3)
D2.0. Dependency injection seams (precondition for everything in §D2)
The injector designs in §D2.1+ assume that production code accepts an abstraction (trait, generic, enum-dispatch wrapper, fault-aware concrete type) at the seam where the test wants to fail. Today, most production code accepts concrete PgPool / Publisher / FieldEncryptor / Store directly, or calls free functions that take &PgPool. Without addressing this, the injectors are beautiful API paper over uninjectable code: the test compiles but no production codepath ever reaches the wrapped type.
Three viable seam patterns. Each Phase A.2 candidate-injector callsite picks one based on the actual production surface; this is not a one-size-fits-all decision.
| Pattern | When to use | Cost | Limit |
|---|---|---|---|
Trait-object injection at construction |
Production callsite already takes a long-lived dep through a struct field (e.g. |
Touches every constructor; one trait per dep family. |
|
Test-only enum dispatch |
Production callsite is a free fn that takes a small typed dependency (e.g. |
Module-internal change; no signature churn at API boundary. |
Adds an enum match per call; risk of forgotten match arm on new variants. Not applicable to types whose API exposes a primitive ( |
Real container failure |
Failure mode is systemic (RabbitMQ unreachable; DB connection refused; object store returns 503). Bring up devstack with the dependency stopped ( |
Slow (5-10s startup); requires |
Cannot inject selective failures (e.g. "fail this query but not that"); cannot scope failures to a specific test. |
Concrete seam audit (per the 5 inline injectors from epic &21 Step 2 + the 3 new ones in §D2):
-
FieldEncryptor(CipherErrorInjector + LatencyInjector + IntermittentFlapInjector): production already takesOption<FieldEncryptor>throughAppState. Pattern: trait-object — definepub trait FieldEncrypt: Send + Sync { fn encrypt_str(&self, …) → Result<…>; fn decrypt_str(&self, …) → Result<…>; }. Implement onFieldEncryptorand on each injector wrapper. ChangeAppState::encryptor: Option<Box<dyn FieldEncrypt>>. Audit ~5 callsites inservices/craig-cases/src/api/persons.rs(already enumerated in platform-stab-2 §D9.3). -
PgPool(DbErrorInjector + LatencyInjector + IntermittentFlapInjector): production storesPgPooldirectly in manyApp*structs and 100+ store functions take&PgPooland callsqlx::query(…). A wrapper that exposes&PgPoolis a wrapper in name only: once the test code obtains&PgPoolit bypasses any interception. The faux-Db-newtype-with-inner()design rejected here. Three real options, picked per-codepath:Sub-pattern When Trade-off (a) Real container-level failure
The test cares about systemic DB unavailability (connection refused, cluster failover, network partition). Bring up devstack,
docker compose stop postgres, run the test, observe/readyzflip + handler 503 + outbox row stays unpublished.Cannot inject selective failures (one query fails, others succeed). Cannot scope failures to a specific test invocation. Slow (5-10s setup + teardown).
(b) Repository / store trait at the codepath under test
The test wants to inject a failure at this specific store function. Define
pub trait FinalizeIdempotencyStore { async fn finalize_succeeded(…) → Result<…>; … }. Production impl wraps&PgPooland callssqlx::query. Tests passFaultyFinalizeIdempotencyStorethat errors on demand. Worker / handler holdsArc<dyn FinalizeIdempotencyStore>(withasync_traitper §D2.0c below) or generic<S: FinalizeIdempotencyStore>if codegen overhead matters.One trait + one prod impl + one test impl per faultable codepath. ~3-5 traits per service in scope (FinalizeIdempotencyStore, AttachmentStore, SendJobStore, OutboxStore, InboxStore). Adds an indirection at every call site that wants to participate in fault injection.
(c) No injection at this layer
The test’s failure mode is at a different layer (cipher, RMQ, object store) and DB calls are not the seam. Use the cipher/RMQ/object-store injectors instead and let DB calls run real.
None — the right answer when DB is not the failure point.
Decision rule: every Phase B test that wants a DB failure declares which sub-pattern it uses in its test file’s module-level doc comment. Reviewer enforces. If sub-pattern (b) is chosen, the trait is added in the same MR that lands the test (per the failing-test-first protocol §D2.0 last paragraph). Speculative trait extraction without a test target is rejected.
Concrete trait inventory expected (each ships in a Phase A.2 sub-MR alongside its first failing-test-first user):
-
FinalizeIdempotencyStore— for platform-stab-2 §D1 finalize-failure → live-response Phase B test 3. -
AttachmentStore— for platform-stab-2 §D4 commit-then-blob-fails Phase B test 6. -
SendJobStore— for platform-stab-2 §D5 mid-send-failure Phase B test 7. -
OutboxStore— for platform-stab-2 §D2 publish-then-update-fails (single-process simulation). -
InboxStore— for platform-stab-2 §D3 retry-after-failure Phase B test 5.There is no
Dbnewtype. Theinner() → &PgPoolaccessor pattern is removed from this plan.
-
-
Publisher(RabbitDownInjector + LatencyInjector + ConsumerBackpressureInjector): production storesPublisherdirectly inOutboxWorker,ExchangeSendWorker, etc. Pattern: trait-object viaasync_trait(or generic structs — see below). Nativeasync fnin dyn-compatible traits is technically supported on Rust 1.75+, but the dyn-compatibility rules aroundSend/'staticfutures make a hand-rolledArc<dyn PublishLike>brittle in the worker context (workers spawn the future onto a runtime, so the future must beSend + 'static). Two object-safe paths:// Path 1: async_trait macro (most ergonomic; small per-call boxing cost). #[async_trait::async_trait] pub trait PublishLike: Send + Sync { async fn publish(&self, env: &EventEnvelope) -> Result<(), PublishError>; async fn publish_in_tx( &self, tx: &mut sqlx::PgConnection, env: &EventEnvelope, ) -> Result<(), PublishError>; async fn publish_dlx( &self, env: &EventEnvelope, original_queue: &str, retry_count: i32, last_error: &str, ) -> Result<(), PublishError>; } // Worker holds Arc<dyn PublishLike + Send + Sync>; `async_trait` produces // the BoxFuture indirection automatically.// Path 2: generic worker struct (zero per-call cost; codegen monomorphizes). pub struct OutboxWorker<P: Publisher + Send + Sync + 'static> { pool: PgPool, publisher: Arc<P>, } // Production: OutboxWorker<RealPublisher>. Tests: OutboxWorker<RabbitDownInjector>. // Trade-off: each worker is a separate concrete type; harder to swap at runtime, // fine at compile-time.Decision rule: use Path 1 (
async_trait) when the worker is constructed inmain.rsand the choice of publisher impl varies per environment. Use Path 2 (generic) when the publisher impl is fixed at compile time and the worker is hot-path. Default proposal: Path 1 for the consolidation — single rebuild swaps the impl in tests, matches the existingBox<dyn ObjectStore>precedent incraig-store, andasync_traitis already a workspace dep (verify viaCargo.toml[workspace.dependencies]; if absent, addasync-trait = "0.1").Implement on
Publisher(production) and on each injector wrapper. Change worker fields toArc<dyn PublishLike>(withasync_trait’s `Send + Sync + 'staticbounds satisfied by the macro-generated impl). -
Store(object_store-backed) (ObjectStoreErrorInjector + LatencyInjector):craig-storealready exposes aStorestruct that wrapsArc<dyn object_store::ObjectStore>. Pattern: leverage the existingdynboundary. Each injector wrapsArc<dyn ObjectStore>and delegates to inner;Store::wrap_for_test(injected_arc)constructor. -
PublishInTxFailureInjector(free fncraig_mq::stage_event): free fn signature ispub async fn stage_event(executor, envelope) → Result<…>. Pattern: wrap in a smallEventStagertrait insidecraig-test-liband route test-mode calls through the trait; production keeps the free fn (no production change). Tests opt in via a thread-local override or per-testEventStager::install_for_test(…).
Refactor cost summary:
-
Trait-object additions: ~20 LOC of trait + impl per dep family + ~5 callsite signature updates per dep family.
-
Worker struct fields: change
publisher: Publisher→publisher: Arc<dyn PublishLike>in 7 services × 1-2 worker structs each = ~10 sites. -
Dbnewtype rollout: ~15 sites acrossApp*structs + ~50 store-fn callsites that today passapp.db.inner()already use the right shape — only theApp*field type changes. -
Total: ~3 LOC × ~80 callsites + ~150 LOC of new trait machinery = bounded; one focused MR per dep family. Fits as the first commit of Step 3 before any new injector lands.
Failing-test-first protocol for §D2: every new injector ships with (a) the seam refactor, (b) the injector wrapper, (c) one test that uses the injector against a real codepath. If (c) requires a codepath that doesn’t yet have the seam, file a follow-up issue and skip the injector until the seam lands. Do not merge an injector whose test target is uninjectable.
D2.0a. Initial state (epic &21 Step 2 inline injectors)
Epic &21 Step 2 introduces 5 minimal injectors:
-
CipherErrorInjector— wrapsFieldEncryptor;inject_error_on_next_n(n) -
DbErrorInjector— wrapsPgPool;inject_error_after(n)— brittle by design (per §D2.0 above): wrappingPgPoolwith a type that exposes&PgPoolcannot intercept queries that flow downstream intosqlx::query(). Platform-stab-2 Step 2 ships this as a placeholder so its dependent fixes have something compiling, but the inlineDbErrorInjectoris structurally limited to scenarios where the test happens to use the wrapped pool’s surface methods directly. Phase A.2 (Step 3) of THIS plan replaces the inlineDbErrorInjectorwith the codepath-specific store traits enumerated in §D2.0:FinalizeIdempotencyStore,AttachmentStore,OutboxStore,InboxStore,SendJobStore. Tests landed alongside platform-stab-2 fixes that today depend onDbErrorInjectorwill be migrated to the corresponding store trait as the trait lands — typically in the same MR, using the failing-test-first protocol (§D2.0 last paragraph). -
RabbitDownInjector— wrapsPublisher;inject_unavailable_for(duration). ThePublisherAPI is a small surface (publish,publish_in_tx,publish_dlx) so the wrapper genuinely intercepts. This pattern stays valid. -
ObjectStoreErrorInjector— wrapsStore;inject_put_failure_for_keys(prefix).Storealready wrapsArc<dyn ObjectStore>so the dyn boundary makes interception trivial. This pattern stays valid. -
PublishInTxFailureInjector— wrapscraig_mq::stage_event;fail_for_event_type(event_type). Free fn, intercepted via thread-local override. This pattern stays valid.
Each is a thin AtomicBool/AtomicUsize/Mutex<Instant> wrapper with
a Drop impl that asserts no remaining injected-but-not-consumed
faults. They cover the single-shot patterns: "fail once", "fail N",
"fail for duration", "fail keys with prefix".
Failure-path tests in epic &21 Phase 2 (#274–#280) and the broader Phase B adoption (#291) need richer scenarios that cannot be expressed with single-shot counters:
-
Latency injection — "this DB query takes 4s instead of 4ms" — needed to assert
await_winnerceiling-409 path (D1 Path C in the &21 plan) actually triggers under controlled conditions. -
Partial-write simulation — "the blob
putsucceeds but the status-promotion UPDATE fails" — needed to assertAttachmentScannerreaps orphans correctly (&21 D4.5). -
Intermittent flapping — "cipher fails on attempts 1, 3, 5; succeeds on 2, 4, 6" — needed to assert the inbox three-state retry’s backoff schedule actually retries to cap (&21 D3 Path C).
-
Timestamp-controlled scenarios — "at t=0 RMQ is up; at t=2s RMQ is down; at t=5s RMQ is up again" — needed for outbox-during-RMQ-blip tests.
-
Subscriber back-pressure — "consumer accepts the deliver but takes 60s to ack" — needed for any test that wants to assert outbox-depth alarms or DLX surface under slow-consumer conditions. No injector exists for this today.
D2.1. Decision: in-place extension via a module promotion
Same trade-off as D1: option (a) extend crates/craig-test-lib/src/fault_injection.rs
in place, or (b) split into a fault/ subdirectory.
Unlike D1, the natural shape of the extension is multiple files —
the existing 5 injectors (~50 LOC each) plus 3 new ones (~80 LOC each)
plus a scenarios.rs registry plus a harness.rs trait module gives
~600 LOC total. That crosses the threshold where one file is unwieldy.
Decision: option (b) — promote fault_injection.rs to a fault/
module directory, keep pub use re-exports at the existing path:
// crates/craig-test-lib/src/fault_injection.rs (post-promotion)
//! Compatibility re-exports. All concrete types now live under
//! `crate::fault::*`. Will be removed in a future release once
//! callers migrate.
pub use crate::fault::cipher::CipherErrorInjector;
pub use crate::fault::rabbit::RabbitDownInjector;
pub use crate::fault::object_store::ObjectStoreErrorInjector;
pub use crate::fault::publish_in_tx::PublishInTxFailureInjector;
// `DbErrorInjector` is intentionally NOT re-exported. Per §D2.0,
// the `PgPool`-wrapping pattern is replaced by codepath-specific
// store traits (FinalizeIdempotencyStore, AttachmentStore, etc.).
// Tests using the inline DbErrorInjector are migrated to the
// matching trait as the seam lands; see §D2.0 sub-pattern (b).
The new layout:
crates/craig-test-lib/src/fault/
├── mod.rs # FaultInjector trait + Attempt + ScenarioGuard
├── cipher.rs # CipherErrorInjector (migrated from inline)
├── rabbit.rs # RabbitDownInjector
├── object_store.rs # ObjectStoreErrorInjector
├── publish_in_tx.rs # PublishInTxFailureInjector
├── stores/ # NEW — codepath-specific store traits + faulty impls
│ ├── mod.rs # trait definitions per §D2.0 sub-pattern (b)
│ ├── finalize_idempotency.rs # FinalizeIdempotencyStore + Faulty impl
│ ├── attachment.rs # AttachmentStore + Faulty impl
│ ├── outbox.rs # OutboxStore + Faulty impl
│ ├── inbox.rs # InboxStore + Faulty impl
│ └── send_job.rs # SendJobStore + Faulty impl
├── latency.rs # NEW — LatencyInjector wrapper
├── flap.rs # NEW — IntermittentFlapInjector
├── backpressure.rs # NEW — ConsumerBackpressureInjector
└── scenarios.rs # NEW — named pre-built Scenario constants
(The inline db.rs from platform-stab-2 Step 2 is retired in this consolidation; its callers migrate to a stores/ trait per §D2.0a above. The shim file keeps the *valid &21 Phase 2 cipher/rabbit/object-store/publish-in-tx injector imports unchanged.)
D2.2. The FaultInjector trait
// crates/craig-test-lib/src/fault/mod.rs
/// Common shape every fault injector implements.
pub trait FaultInjector: Send + Sync + 'static {
/// The `Scenario` type drives this injector's behavior. Distinct
/// per injector — `CipherScenario`, `DbScenario`, etc.
type Scenario: Send + Sync + 'static;
/// Configure the injector with a scenario. Replaces any prior
/// scenario; returns a `ScenarioGuard` that, when dropped,
/// asserts the scenario was *consumed* (consume == "the wrapped
/// callsite was invoked under the scenario at least the expected
/// number of times"). Use `scenario_guard.disarm()` to opt out
/// of the assertion (e.g., for pre-condition setup that didn't
/// actually need the injection to fire).
#[must_use]
fn inject(&self, scenario: Self::Scenario) -> ScenarioGuard<'_>;
/// Audit log: every wrapped-callsite invocation under the active
/// scenario, in invocation order. Tests use this to assert "the
/// codepath was reached N times" without coupling to internal
/// counters.
fn recorded_attempts(&self) -> Vec<Attempt>;
}
/// One invocation of a fault-injector-wrapped callsite.
#[derive(Debug, Clone)]
pub struct Attempt {
/// 1-based attempt number under the current scenario.
pub n: u32,
/// Wall-clock instant the attempt started.
pub at: Instant,
/// Outcome: did the injector return Err (fault injected) or pass
/// the call through to the underlying real impl (success)?
pub injected: bool,
/// Optional per-injector context (e.g., for `ObjectStoreErrorInjector`,
/// the object key the call was for).
pub context: Option<String>,
}
/// RAII handle returned by `FaultInjector::inject`. On drop, asserts
/// the scenario's expected-consumption invariant. `disarm()` opts out.
pub struct ScenarioGuard<'a> {
inner: Box<dyn FnOnce() + 'a>, // the assertion closure
armed: bool,
}
impl<'a> ScenarioGuard<'a> {
pub fn disarm(mut self) { self.armed = false; }
}
impl Drop for ScenarioGuard<'_> {
fn drop(&mut self) {
if self.armed && !std::thread::panicking() {
// Move the closure out via Option-trick.
// Asserts "scenario was consumed as advertised".
}
}
}
The Drop assertion catches a class of false-positive tests that the &21
minimal injectors already attempt to catch: a test sets up
inject_error_after(3) but the codepath only invokes the wrapped
callsite twice, so the injection never fires, the test passes, and the
"failure-path coverage" claim is structurally false. The
ScenarioGuard makes that explicit — drop fires the assertion at
test-teardown time, and the panic-checking guard means the assertion
is suppressed if the test was already failing for an unrelated reason.
D2.3. LatencyInjector — wraps any of the 5 base injectors
// crates/craig-test-lib/src/fault/latency.rs
/// Wrap any `FaultInjector` to add a configurable latency before and/or
/// after each wrapped-callsite invocation. The wrapped injector's
/// scenario fires unchanged; the latency is purely additive.
pub struct LatencyInjector<I: FaultInjector> {
inner: I,
before: Option<Duration>,
after: Option<Duration>,
}
impl<I: FaultInjector> LatencyInjector<I> {
pub fn new(inner: I) -> Self;
pub fn with_pre_delay(mut self, d: Duration) -> Self;
pub fn with_post_delay(mut self, d: Duration) -> Self;
}
// Re-implements FaultInjector by delegating + sleeping.
impl<I: FaultInjector> FaultInjector for LatencyInjector<I> {
type Scenario = I::Scenario;
fn inject(&self, scenario: I::Scenario) -> ScenarioGuard<'_> { /* ... */ }
fn recorded_attempts(&self) -> Vec<Attempt> { /* ... */ }
}
When-to-use: the &21 D1 Path C "loser observes processing for 5s,
ceiling-409" test needs to make the winner’s handler genuinely take ≥5s.
Wrap a FinalizeIdempotencyStore impl with LatencyInjector::new(real_store).with_pre_delay(Duration::from_secs(6))
on the finalize_succeeded call (the §D2.0 sub-pattern (b) faultable trait
identified for §D1’s idempotency Path C). That makes the winner stall inside
the trait method, the loser poll, and the ceiling fire deterministically.
Smoke test (crates/craig-test-lib/tests/fault_latency.rs): wrap a
no-op FinalizeIdempotencyStore impl (or any CipherErrorInjector /
ObjectStoreErrorInjector instance — they all implement the
FaultInjector trait) with with_pre_delay(50ms); invoke the wrapped
callsite; assert the recorded Attempt.at deltas are ≥ 50ms.
(Per §D2.0, there is no DbErrorInjector wrapping PgPool directly.
DB fault injection happens at codepath-specific store traits — these
traits each implement FaultInjector and compose with LatencyInjector
via the same generic wrapper.)
D2.4. IntermittentFlapInjector — alternating Ok/Err on cadence
// crates/craig-test-lib/src/fault/flap.rs
/// Configurable cadence pattern of pass-through / inject-error.
pub struct IntermittentFlapInjector<I: FaultInjector> {
inner: I,
pattern: Vec<bool>, // true = inject, false = pass-through
counter: AtomicUsize,
}
impl<I: FaultInjector> IntermittentFlapInjector<I> {
/// `pattern.len() >= 1`. The pattern repeats indefinitely:
/// `[true, false]` = inject every odd attempt;
/// `[false, false, true]` = inject every 3rd attempt;
/// `[true, true, true, false]` = "fails 3-of-4".
pub fn new(inner: I, pattern: Vec<bool>) -> Self;
/// Convenience: alternating fail-then-succeed.
pub fn alternating(inner: I) -> Self {
Self::new(inner, vec![true, false])
}
}
impl<I: FaultInjector> FaultInjector for IntermittentFlapInjector<I> {
type Scenario = I::Scenario;
/* ... */
}
When-to-use: inbox three-state retry test (epic &21 D3) needs to assert
"handler fails attempt 1, succeeds attempt 2; processed_at IS NOT NULL,
error_count == 1". The minimal CipherErrorInjector from &21 Step 2
takes inject_error_on_next_n(1), which works for that test.
Phase B test #283 (the &22 adoption bundle’s "inbox flap then
recover") needs pattern = [true, true, false] — fail twice, succeed
on third — to exercise backoff schedule positions 1 and 2, then
success.
Smoke test: wrap a CipherErrorInjector with [true, false, true];
invoke 6 times; assert attempts 1, 3, 5 errored and 2, 4, 6 succeeded
(via recorded_attempts()).
D2.5. ConsumerBackpressureInjector — slow consumer simulation
The 6th injector. No equivalent exists in &21 Step 2; this is fully new in Phase A.2.
// crates/craig-test-lib/src/fault/backpressure.rs
/// Simulate a slow RabbitMQ consumer that delays its `delivery.ack()`.
/// Wraps a real `lapin::Consumer`-driven loop and inserts a configurable
/// pre-ack sleep; can also drop deliveries silently to simulate hung
/// consumers.
pub struct ConsumerBackpressureInjector {
/// Per-delivery pre-ack delay. Default `Duration::ZERO`.
pub ack_delay: Duration,
/// If `Some(n)`, every nth delivery is *not acked* (simulates a
/// permanently hung consumer slot).
pub drop_every_nth: Option<u32>,
counter: AtomicU32,
recorded: Mutex<Vec<Attempt>>,
}
impl ConsumerBackpressureInjector {
pub fn new() -> Self;
pub fn with_ack_delay(mut self, d: Duration) -> Self;
pub fn drop_every(mut self, n: u32) -> Self;
/// Wrap a delivery handler closure to apply the configured back-pressure.
/// Used by tests that drive a real or fake `lapin::Consumer` stream
/// through this injector before asserting outbox-depth / DLX behavior.
pub async fn handle_delivery<F, Fut>(
&self,
delivery: lapin::message::Delivery,
handler: F,
) -> anyhow::Result<()>
where
F: FnOnce(lapin::message::Delivery) -> Fut,
Fut: Future<Output = anyhow::Result<()>>;
}
impl FaultInjector for ConsumerBackpressureInjector {
type Scenario = BackpressureScenario;
/* ... */
}
pub enum BackpressureScenario {
Steady { ack_delay: Duration },
SpikeFor { duration: Duration, ack_delay: Duration },
HangEveryNth { n: u32 },
}
When-to-use:
-
DLX surface under slow consumer — Phase B test for #291: stage 100 events, run a
ConsumerBackpressureInjectorwithack_delay = 5s, assert RabbitMQ’s per-consumer prefetch hits the cap, outbox-depth alarm fires. -
Inbox claim-recovery under hung consumer —
HangEveryNth { n: 3 }drops every 3rd delivery; redelivery surfaces; the inbox row’sattempt_started_atis past the recovery window; the next worker re-claims. -
/livez vs /readyz semantic-differentiation (Phase B test for #291): RMQ-side back-pressure should not flip /livez to false, but should flip /readyz to false. Without this injector, the test cannot induce the condition.
Smoke test (crates/craig-test-lib/tests/fault_backpressure.rs): drive
a fake lapin::message::Delivery stream of 5 items through
with_ack_delay(50ms); assert per-delivery handling time ≥ 50ms;
assert recorded_attempts().len() == 5.
D2.6. craig_test_lib::fault::scenarios — named pre-built scenarios
// crates/craig-test-lib/src/fault/scenarios.rs
//! Named pre-built scenarios for the most common failure-path tests.
//! Tests should prefer these named constants over open-coding scenarios
//! — keeps test intent legible and centralizes "what does 'flaky DB'
//! mean for this codebase" in one place.
use super::*;
pub mod db {
/// Pool exhaustion after `n` successful queries. Mirrors the
/// production failure where pgbouncer hits its connection cap.
pub fn pool_exhausted_after_n(n: u32) -> DbScenario { /* ... */ }
/// Connection refused on every query. Mirrors the production
/// failure where the DB is unreachable (e.g., during a Postgres
/// failover).
pub fn connection_refused() -> DbScenario { /* ... */ }
/// Serialization-failure on every query. Mirrors the production
/// failure where row-level locking + concurrent updates trigger
/// `serializable_isolation_failure`.
pub fn serialization_failure() -> DbScenario { /* ... */ }
}
pub mod rmq {
/// RMQ unreachable for `dur`, then comes back. Models a
/// short-lived broker outage / failover.
pub fn flap_for_5s() -> RabbitScenario { flap_for(Duration::from_secs(5)) }
pub fn flap_for(dur: Duration) -> RabbitScenario { /* ... */ }
/// Permanently down. Models broker-side cluster failure with no
/// recovery in test runtime.
pub fn permanently_down() -> RabbitScenario { /* ... */ }
}
pub mod cipher {
/// `FieldEncryptor` panics once. Models the corrupted-key case
/// where the cipher impl panics on bad key material.
pub fn panic_once() -> CipherScenario { /* ... */ }
/// `FieldEncryptor` returns Err on next 3 attempts (matches the
/// inbox max_retries default of 5 attempts so the test can
/// exercise "3 fails, then succeed" — under-cap retry).
pub fn err_3_then_succeed() -> CipherScenario { /* ... */ }
}
pub mod object_store {
/// `Store::put` returns Err for any key matching the prefix.
pub fn put_failure_for_prefix(prefix: &'static str) -> ObjectStoreScenario { /* ... */ }
/// `Store::head` returns 404 (key not found) for any key — models
/// the post-Phase-2-orphan condition where DB has a `pending` row
/// but the blob never landed.
pub fn head_404_all() -> ObjectStoreScenario { /* ... */ }
}
When-to-use: every Phase B failure-path test should reach for a named scenario first; falls back to open-coding only if the named registry doesn’t cover the case. New named scenarios are added here as post-merge follow-ups whenever a test introduces an open-coded scenario that’s likely to recur.
D2.7. Test descriptions
For each new injector, smoke tests live at
crates/craig-test-lib/tests/fault_<name>.rs:
-
fault_latency—with_pre_delay(50ms)adds ≥50ms;with_post_delaysimilarly; both compose;recorded_attempts()shows correct timing. -
fault_flap_alternating— pattern[true, false], 6 invocations, attempts 1/3/5 errored, 2/4/6 succeeded (viarecorded_attempts()). -
fault_flap_repeating_pattern— pattern[true, true, true, false], 8 invocations, attempts 1/2/3, 5/6/7 errored, 4/8 succeeded. -
fault_backpressure_ack_delay— drive 5 fake deliveries,ack_delay=50ms; per-delivery ≥50ms. -
fault_backpressure_drop_every_nth—drop_every(3)over 9 deliveries; deliveries 3, 6, 9 are recorded asinjected=true(no ack); rest are recorded asinjected=false. -
fault_scenario_guard_drop_asserts— set updb.inject(pool_exhausted_after_n(2)), invoke wrapped callsite once, drop the guard; the test should panic with "scenario not consumed". Wrap instd::panic::catch_unwindto assert. -
fault_scenario_guard_disarm_does_not_assert— same setup,guard.disarm(), drop; no panic. -
fault_scenario_named_constants_compile— call every named scenario inscenarios::*to ensure the registry is wired up (compile-time check; runtime assertions optional per-scenario). -
fault_scenario_named_db_pool_exhausted—pool_exhausted_after_n(3)applied to a wrapped no-op pool; 4 invocations; first 3 succeed, 4th errors withsqlx::Error::PoolTimedOut(or chosen variant).
Path A/B/C structure for the most safety-critical case (the Drop assertion):
-
Path A — guard drops with consumed scenario: scenario fires the expected number of times; drop is silent.
-
Path B — guard drops with under-consumed scenario, no test panic in flight: drop panics with assertion message; test catches via
catch_unwindand asserts the panic message. -
Path C — guard drops during an unrelated test panic: simulate a test panicking before the guard drops (
std::panic::panic_any); the guard’sDropchecksstd::thread::panicking()and skips its assertion so the original panic surfaces unmuddled. Assert the surfaced panic carries the original payload, not the guard’s.
D3. Multi-replica devstack profile (Phase A.3 — Step 4)
Today the devstack is single-instance for every CRAIG application
service. docker-compose.yml declares each service exactly once with a
fixed --ports 800X:800X host-port mapping. Multi-replica behavior —
two OutboxWorker`s draining the same `event_outbox, two
craig-cases instances racing the idempotency-claim middleware — is
simulated single-process in epic &21’s tests by spawning multiple
worker tasks against one PgPool. That works for the per-test
verification of &21 Steps 4 and 5, but it does not exercise:
-
Two real OS processes contending on
FOR UPDATE SKIP LOCKEDagainst a shared Postgres connection (separate connection pools, separate prepared-statement caches, separate Rust task schedulers). -
Two real HTTP listeners behind a load-balancer answering same-key POSTs — needed to assert idempotency atomic-claim works across replicas, not just across tasks.
-
Two real RabbitMQ consumers competing for the same queue — needed for the inbox three-state-retry test under realistic broker prefetch and ack semantics.
Phase B failure-path tests (#291) — specifically "Multi-replica outbox" and "Concurrent same-key idempotency" — need a real multi-replica mode. This step builds it.
D3.1. Open-question default: docker-compose --scale
The plan’s existing Open Questions section flags this as decision-pending
between --scale and custom xtask logic, with default proposal
--scale. This step adopts the default. Rationale:
| Path | Verdict |
|---|---|
docker-compose |
Chosen. Compose-native, zero new orchestration code, every CRAIG service is already 12-factor-stateless (state in shared Postgres / RabbitMQ / Garage). The only friction is host-port pinning — fixed by switching scaled services to ephemeral ports + a Docker-internal hostname (which already exists for cross-service traffic via the compose network DNS). |
Custom xtask N-instance launcher with manual port allocation |
Rejected. Re-implements docker-compose’s load-balancer-pattern
hostname resolution. ~300 LOC of orchestration code with no
capability gain over |
Kubernetes / k3d local cluster |
Rejected. Multiple-orders-of-magnitude more setup cost; CI runners would need k3d in their image; docs would diverge from production manifests anyway. |
The trade-off: --scale requires the host-port mapping to not be
fixed for the scaled service, since Docker can’t bind 8002:8002 twice.
Compose handles this by binding <port> (no host pin) and assigning
ephemeral host ports — and tests reach replicas by name via the
internal compose-network DNS, not by host port.
D3.2. xtask interface
Extend xtask/src/cmd/dev.rs::DevCommand::Start with a --replicas
arg. Builds on the existing Start { force: bool } shape:
// xtask/src/cmd/dev.rs
#[derive(Subcommand)]
pub enum DevCommand {
/// Start/rebuild all services.
Start {
/// Force full rebuild, skip staleness check.
#[arg(long)]
force: bool,
/// Scale specific services. Format: `SERVICE=N[,SERVICE=N…]`.
/// Each entry must name a service from CRAIG_SERVICES;
/// N must be ≥1. Repeats with `=1` are no-ops.
///
/// Example: `--replicas craig-cases=2,craig-exchange=2`
///
/// Services not listed default to scale 1 (current behavior).
///
/// Limitations:
/// * Only stateless CRAIG application services are scalable
/// (the 8 in CRAIG_SERVICES). Postgres/RabbitMQ/Keycloak/
/// Garage scale to 1 — sharing state requires external
/// replication this devstack doesn't model.
/// * `craig-web` (BFF) is scalable but the 8080 host port
/// pins to the first replica. Use `craig-web=1` for now.
/// * Scaled services lose their fixed host ports; reach them
/// by Docker-internal DNS (`craig-cases-1`, `craig-cases-2`)
/// from inside the compose network, or by the ephemeral
/// host ports surfaced via `docker compose port`.
#[arg(long, value_parser = parse_replicas)]
replicas: Vec<ReplicaSpec>,
},
// ... rest unchanged
}
#[derive(Debug, Clone)]
pub struct ReplicaSpec {
pub service: String,
pub count: u32,
}
fn parse_replicas(raw: &str) -> Result<Vec<ReplicaSpec>, String> {
let mut out = Vec::new();
for entry in raw.split(',') {
let (svc, n) = entry.split_once('=')
.ok_or_else(|| format!("expected SERVICE=N, got {entry}"))?;
if !CRAIG_SERVICES.contains(&svc) {
return Err(format!(
"unknown service '{svc}'. Allowed: {}",
CRAIG_SERVICES.join(", ")
));
}
let count: u32 = n.parse().map_err(|_| format!("non-integer count: {n}"))?;
if count == 0 {
return Err(format!("count must be >= 1, got 0 for {svc}"));
}
out.push(ReplicaSpec { service: svc.to_string(), count });
}
Ok(out)
}
D3.3. Threading replicas through devstack::bring_up
xtask/src/cmd/dev.rs::run passes the replicas list to
devstack::ensure_ready, which threads it through to bring_up:
// xtask/src/devstack.rs
pub fn ensure_ready(force: bool, replicas: &[ReplicaSpec]) -> Result<()> { /* ... */ }
pub fn bring_up(no_cache: bool, replicas: &[ReplicaSpec]) -> Result<()> {
// ... existing port-reservation, compose-up logic ...
let mut up_args: Vec<String> = vec!["up".into(), "-d".into(), "--build".into()];
for spec in replicas {
up_args.push("--scale".into());
up_args.push(format!("{}={}", spec.service, spec.count));
}
docker::compose(&up_args.iter().map(String::as_str).collect::<Vec<_>>())?;
// health-wait (next section) ...
}
When replicas is empty, the args degenerate to the current
["up", "-d", "--build"] — zero behavior change for the default case.
D3.4. docker-compose.yml adjustments
Three categories of change. All are minimal and --scale=1-compatible
(no behavior change when the flag is absent):
-
Drop fixed
container_name— none of the CRAIG services declarecontainer_nametoday, so this is a no-op verification step. Confirm viaGrep "container_name:" docker-compose.yml. -
Switch host-port pins from
"8002:8002"to"8002"for scalable services — only when scaling. The default-scale-1 case keeps the current8002:8002mapping because tests that connect from the host (integration tests, k6, ZAP) need a stable port. The cleanest path: introduce a docker-compose override filedocker-compose.scale.ymlthat re-declares the scalable services with floating-port mappings, loaded only when--replicasis passed. -
Confirm shared-state-only — Postgres, RabbitMQ, Garage, Keycloak remain scale=1; verify each scaled service’s environment block references shared infra (postgres/rabbitmq/keycloak/garage hostnames) and not per-instance state.
# docker-compose.scale.yml — loaded only when --replicas is non-empty.
# Removes host-port pins from scalable services so docker-compose can
# bind the container ports to ephemeral host ports across replicas.
services:
craig-rules:
ports: !override
- "8001"
craig-cases:
ports: !override
- "8002"
craig-placement:
ports: !override
- "8003"
craig-exchange:
ports: !override
- "8004"
craig-financial:
ports: !override
- "8005"
craig-reporting:
ports: !override
- "8006"
craig-security:
ports: !override
- "8007"
craig-intake:
ports: !override
- "8008"
The !override tag (compose v2.20+) replaces the parent’s ports
list rather than appending. Confirmed against the compose version
pinned in .gitlab-ci.yml.
D3.5. Compose-override vs in-place compose edit
Two strategies for managing the scale-mode port adjustments:
| Path | Verdict |
|---|---|
Compose-override file ( |
Chosen. Loaded conditionally via
|
In-place edit of |
Rejected. Breaks every test/script that connects to host
|
Programmatic compose-rendering inside xtask |
Rejected. Re-implements compose-merge semantics. Overrides are the built-in mechanism; no benefit to recreating them in Rust. |
D3.6. Health-wait under scaling
devstack::wait_for_service_health(svc) (in xtask/src/devstack.rs)
calls docker::is_healthy("craig-cases"). Under --scale=2 there are
two containers — craig-cases-1 and craig-cases-2 — and the helper
needs to wait for both. Strategy: enumerate replica container names
via docker compose ps --services --filter status=running, then poll
each. Returns Ok only when every replica reports healthy.
// xtask/src/devstack.rs (additions)
fn replica_names(svc: &str) -> Result<Vec<String>> {
// `docker compose ps -a <svc> --format {{.Name}}` returns
// one line per replica. Trim, filter empties.
let raw = docker::compose_output(&["ps", "-a", svc, "--format", "{{.Name}}"])?;
Ok(raw.lines().filter(|l| !l.is_empty()).map(String::from).collect())
}
fn wait_for_service_health_all_replicas(svc: &str) -> Result<()> {
let replicas = replica_names(svc)?;
if replicas.is_empty() {
bail!("no running replicas of {svc} to wait for");
}
for name in replicas {
wait_for_named_health(&name)?;
}
Ok(())
}
docker::is_healthy already accepts a container name; the
adjustment is per-replica iteration only.
D3.7. MultiReplicaCluster test helper
For tests that need multi-replica directly (not via the global devstack that pre-push runs), expose a builder:
// crates/craig-test-lib/src/multi_replica.rs (new)
/// Programmatic multi-replica devstack control for tests that drive a
/// dedicated cluster (separate from the pre-push devstack). Tests that
/// run against the *existing* devstack should use the existing
/// `TestHarness` and reach replicas by their compose-network names.
pub struct MultiReplicaCluster {
workspace_root: std::path::PathBuf,
replicas: Vec<ReplicaSpec>,
compose_files: Vec<std::path::PathBuf>,
}
pub struct MultiReplicaClusterBuilder { /* … */ }
impl MultiReplicaCluster {
pub fn builder() -> MultiReplicaClusterBuilder;
/// Bring up the cluster. Idempotent — if already up with the same
/// scale config, no-op.
pub async fn start(&mut self) -> anyhow::Result<()>;
/// Tear down the cluster. Always wipes volumes.
pub async fn stop(self) -> anyhow::Result<()>;
/// Discover replica host:port endpoints. Iterates `docker compose
/// port --index N <svc> <port>` for each `index in 1..=count` —
/// the `--index` flag is **mandatory** here; without it,
/// `docker compose port <svc> <port>` returns the FIRST replica's
/// host port only (a silent ambiguity that would make tests pass
/// against one replica while claiming to test both).
/// Returns a Vec of `http://localhost:<ephemeral>` URLs, one per
/// replica of `svc`.
pub async fn replica_urls(&self, svc: &str) -> anyhow::Result<Vec<String>>;
/// Reach a specific replica by 1-based index (matching docker
/// compose's `--index` numbering). Useful for asserting which
/// replica handled which request (e.g., outbox claim coverage).
pub async fn replica_url(&self, svc: &str, index: usize) -> anyhow::Result<String>;
}
impl MultiReplicaClusterBuilder {
pub fn replica(mut self, svc: &str, count: u32) -> Self;
pub fn extra_compose_file(mut self, path: impl Into<std::path::PathBuf>) -> Self;
pub fn build(self) -> MultiReplicaCluster;
}
When-to-use: integration tests that require multi-replica (Phase B
test for #275 and the verification test below). Tests that don’t need
multi-replica continue to use the global TestHarness against the
default scale=1 devstack.
Implementation note: start() shells to docker compose -p <unique-project> -f …
--scale … up -d. Health waits use the same poll loop as the xtask
path, calling out to docker compose -p <unique-project> ps … --format {{.Name}}.
Port discovery uses docker compose -p <unique-project> port --index <N> <svc> <internal-port>,
iterating N from 1 to the configured replica count.
The helper does not re-implement compose orchestration in Rust — it
shells out so the production-path cargo xtask dev start --replicas
and the test-path MultiReplicaCluster exercise the same compose
invocation.
Project isolation (critical): every MultiReplicaCluster MUST construct
a unique --project-name (-p) per test instance — proposal:
format!("craig-mrc-{}", uuid::Uuid::now_v7().simple()). Without this,
the test cluster shares the docker-compose project name with the
developer’s running cargo xtask dev devstack, and cluster.stop()
(which always wipes volumes via down -v) would destroy the
developer’s local DB / RMQ / Garage state. The start()/stop()
implementations bail with a clear error if the project name collides
with craig (the default xtask devstack project name) or any other
running project visible via docker compose ls --format json.
D3.8. Verification test
A new integration test at
crates/craig-test-lib/tests/multi_replica_idempotency.rs exercises
the full path:
#[tokio::test]
async fn concurrent_same_key_post_to_two_replicas() {
// Path A: spin up dedicated 2-replica cluster.
let mut cluster = MultiReplicaCluster::builder()
.replica("craig-cases", 2)
.build();
cluster.start().await.unwrap();
let urls = cluster.replica_urls("craig-cases").await.unwrap();
assert_eq!(urls.len(), 2);
let harness = TestHarness::new().await.unwrap();
let token = harness.tokens.get_token("admin", "password").await.unwrap();
let body = serde_json::json!({ "case_number": "FULTON-20260503-DEAD" });
let key = uuid::Uuid::now_v7().to_string();
// Fire 5 same-key POSTs across both replicas — alternate by index.
let report = concurrent_fire_collect_ordered(5, |i| {
let url = urls[i % 2].clone();
let token = token.clone();
let body = body.clone();
let key = key.clone();
async move {
shared_http_client()
.post(format!("{url}/v1/cases"))
.bearer_auth(&token)
.header("Idempotency-Key", &key)
.json(&body)
.send()
.await
.unwrap()
.status()
.as_u16()
}
}).await;
let statuses = report.into_results().unwrap();
assert_eq!(statuses.len(), 5);
// Exactly one 200 (winner), four 200s as replays, none as 409
// (assuming the winner's handler completes inside the 5s ceiling).
assert!(statuses.iter().all(|s| *s == 200));
// Path B: query both replicas; assert there's exactly one row in
// craig-cases' DB matching the case_number. (Single Postgres
// backs both replicas; if the atomic-claim worked, there's
// exactly one INSERT.)
let admin_client = harness.admin_cases_client().await.unwrap();
let listed = admin_client.list_cases(/* filter by number */).await.unwrap();
assert_eq!(listed.total, 1);
cluster.stop().await.unwrap();
}
Three paths under verification:
-
Path A — concurrent fire across replicas: 5 POSTs split between 2 replicas, all return 200, all return identical body (winner
replays; loser path runs against the other replica’s handler-side effect by reading from the shared Postgres). -
Path B — single handler-side effect lands: assert
casestable has exactly one row matching the test’s case_number, regardless of which replica wrote it. -
Path C — replicas are genuinely contending: instrument with a per-request response header
X-Served-By: craig-cases-{1|2}(added via tower middleware in the test build) and assert the 5 responses span both replica hostnames. Without this, the test could pass with a load-balancer that pinned all 5 to one replica — which would silently degrade the test’s coverage.
D4. State-machine matrix-testing pattern (Phase A.4 — Step 5)
CRAIG ships 16 hand-rolled state machines across 6 services. Each one is a validate_<entity>_transition(from, to) → Result<(), String> wrapper over an enum’s can_transition_to(target) matches! arm. Today the only coverage is hand-picked positive + a handful of negative cases per machine (see services/craig-financial/src/transitions.rs:141-272 for the densest example). That pattern guarantees nothing about the invalid half of the state space — a regression that adds an unintended (Cleared, Pending) arm to PaymentStatus::can_transition_to would not be caught by any existing test, because no test enumerates the rejection set.
This step lands a generic helper that, given the enumeration of valid transitions and a rejection function, asserts every (from, to) pair not in the input list is rejected. First adoption is the financial-payment state machine.
D4.1. Decision: hand-rolled vs proptest
| Option | Rejected because |
|---|---|
|
|
|
Adds another macro to debug. Same finite-alphabet objection. |
Hand-rolled |
Selected. Bounded enums + a single helper crate-internal to |
This is the same hand-rolled-default already flagged in platform-stab-2 §Open Questions item 5 and in the parent epic plan; this section formalizes it.
D4.2. Helper module
New file crates/craig-test-lib/src/state_machine_matrix.rs. Re-export from crates/craig-test-lib/src/lib.rs as pub mod state_machine_matrix.
Public API (builder pattern, not trait — every CRAIG state machine is a closed enum, no need for generic StateMachine plumbing):
/// One row of the truth-table input.
#[derive(Debug, Clone, Copy)]
pub struct ValidTransition<S: Copy + Eq + std::fmt::Debug + 'static> {
pub from: S,
pub to: S,
}
/// Outcome bucket for a single (from, to) pair.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum Outcome {
/// `reject_fn` returned `Ok(())` and the pair was in the valid list.
AcceptedValid,
/// `reject_fn` returned `Err(_)` and the pair was NOT in the valid list.
RejectedInvalid,
/// **Bug:** valid pair rejected.
AcceptedExpectedReject,
/// **Bug:** invalid pair accepted.
RejectedExpectedAccept,
/// `reject_fn` panicked. Always a bug — transition validators must
/// never panic on bad input strings.
Panicked(String),
}
/// Result of running a full matrix.
#[derive(Debug, Default)]
pub struct MatrixReport<S> {
pub total: usize,
pub accepted_valid: Vec<(S, S)>,
pub rejected_invalid: Vec<(S, S)>,
pub mismatches: Vec<(S, S, Outcome)>,
pub panics: Vec<(S, S, String)>,
}
impl<S> MatrixReport<S>
where
S: Copy + Eq + std::fmt::Debug,
{
/// Convenience: `assert!()` no mismatches and no panics. Pretty-prints
/// the offending pairs into the assertion message.
pub fn assert_clean(&self);
}
/// Drive a transition predicate against the full Cartesian square of an
/// enum's variants and assert the rejection set is exactly the
/// complement of `valid`.
///
/// `all` enumerates every variant; `valid` lists every accepted ordered
/// pair; `reject_fn` is the function under test (returns `Ok(())` on
/// accept, `Err(_)` on reject; must never panic on well-formed input).
///
/// `reject_fn` runs inside `std::panic::catch_unwind` so a panic on any
/// pair is captured as `Outcome::Panicked` rather than aborting the
/// suite.
pub fn assert_only_valid_transitions_accepted<S, F>(
all: &[S],
valid: &[ValidTransition<S>],
reject_fn: F,
) -> MatrixReport<S>
where
S: Copy + Eq + std::fmt::Debug + std::panic::UnwindSafe + 'static,
F: Fn(S, S) -> Result<(), String> + std::panic::UnwindSafe + Copy;
D4.3. First adoption: PaymentStatus
The financial-payment state machine lives at services/craig-financial/src/transitions.rs:11-32 (PaymentStatus, PaymentStatus::can_transition_to, validate_payment_transition). The user spec referenced a transitions/payment.rs submodule path; current layout is single-file. No submodule split here — keep the diff minimal; future decomposition is out of scope.
New test file: services/craig-financial/tests/transitions_matrix.rs.
use craig_financial::transitions::{PaymentStatus, validate_payment_transition};
use craig_test_lib::state_machine_matrix::{
ValidTransition as VT, assert_only_valid_transitions_accepted,
};
const ALL: &[PaymentStatus] = &[
PaymentStatus::Pending,
PaymentStatus::Approved,
PaymentStatus::Issued,
PaymentStatus::Cleared,
PaymentStatus::Voided,
];
const VALID: &[VT<PaymentStatus>] = &[
VT { from: PaymentStatus::Pending, to: PaymentStatus::Approved },
VT { from: PaymentStatus::Approved, to: PaymentStatus::Issued },
VT { from: PaymentStatus::Issued, to: PaymentStatus::Cleared },
// any -> voided
VT { from: PaymentStatus::Pending, to: PaymentStatus::Voided },
VT { from: PaymentStatus::Approved, to: PaymentStatus::Voided },
VT { from: PaymentStatus::Issued, to: PaymentStatus::Voided },
VT { from: PaymentStatus::Cleared, to: PaymentStatus::Voided },
];
#[test]
fn payment_state_machine_matrix_is_exhaustive() {
let report = assert_only_valid_transitions_accepted(ALL, VALID, |from, to| {
validate_payment_transition(&from.to_string(), &to.to_string())
});
report.assert_clean();
// 5 × 5 = 25 pairs. 7 valid (above). 18 must reject — including 5 self-loops.
assert_eq!(report.total, 25);
assert_eq!(report.accepted_valid.len(), 7);
assert_eq!(report.rejected_invalid.len(), 18);
}
validate_payment_transition accepts string args (&str → Result<(), String>); the closure adapts the strongly-typed enum back to that shape via Display (already derived on the enum). Future migration that takes typed enum args at the validator boundary is a follow-up — out of scope for this step.
D4.4. Failing-test-first protocol
To prove the matrix actually catches a regression, the MR sequence is:
-
Land
state_machine_matrix.rswith the helper. -
Write
payment_state_machine_matrix_is_exhaustiveper D4.3 but omit(Issued, Cleared)fromVALID. Expect: assertion fires withOutcome::AcceptedExpectedRejectfor the omitted pair, listing(Issued, Cleared)inMatrixReport::mismatches. Commit a screenshot of the failure into the MR description. -
Restore
(Issued, Cleared)toVALID. Test passes. Land the MR. -
Subsequent state-machine adoption MRs (D4.5) repeat the same dance.
This protocol is the on-ramp for the Process Rule encoded in §D7.4.
D4.5. Cross-references — all 16 documented state machines
Future Phase B adoption issue (#291 per the parent epic; the bundle issue tracking the matrix rollout across all 16) covers each. Itemized so a contextless agent can locate them:
| # | Service | File:Symbol | Variants |
|---|---|---|---|
1 |
craig-financial |
|
5 |
2 |
craig-financial |
|
3 |
3 |
craig-financial |
|
3 |
4 |
craig-cases |
|
TBD via local read |
5 |
craig-cases |
|
TBD |
6 |
craig-cases |
|
TBD |
7 |
craig-cases |
|
TBD |
8 |
craig-placement |
|
TBD |
9 |
craig-placement |
|
TBD |
10 |
craig-exchange |
|
TBD |
11 |
craig-exchange |
|
TBD |
12 |
craig-exchange |
|
TBD |
13 |
craig-reporting |
|
TBD |
14 |
craig-reporting |
|
TBD |
15 |
craig-security |
|
TBD |
16 |
craig-security |
|
TBD |
Each of items 4–16 is a follow-up MR under Phase B. Step 5 lands the helper + item 1 only; the remaining 15 are tracked but not blocking this step’s completion.
D4.6. Test-coverage description
The helper itself ships with two unit tests in crates/craig-test-lib/tests/state_machine_matrix.rs:
-
Happy path: a 3-variant enum (
enum E { A, B, C }),valid = &[(A, B), (B, C)], predicate consistent withvalid. AssertMatrixReport::mismatches.is_empty(),total == 9,accepted_valid.len() == 2,rejected_invalid.len() == 7. -
Mismatch detection: same enum, predicate that erroneously also accepts
(C, A). Assertmismatchescontains(C, A, Outcome::AcceptedExpectedReject). -
Panic capture: predicate that
panic!()`s on `(B, C). Assertpanicscontains the pair with the panic message;assert_cleanreturns failure rather than aborting.
Plus the financial-payment integration test from D4.3.
D5. JWT mutation library (Phase A.5 — Step 6)
ADR-021 commits craig-auth to a strictness policy: tokens must carry aud matching the service name, typ: Bearer, valid iss, signing alg in the JWKS-advertised allowlist, present-and-recent exp/iat, optional azp/scope checked when service config requires. The current test surface for that policy is exactly two cases per service (a passing token + "invalid.token.here" per crates/craig-auth/src/middleware.rs:178-205). Strict-validator regressions — accidentally widening Validation::set_audience to multiple values, dropping the alg-pin, accepting none-alg tokens — pass that surface trivially.
This step lands a single-axis mutation library: a builder that mints valid baseline tokens, plus a mutation_matrix iterator yielding one variant per axis with one field tampered. Each variant is asserted to fail with a specific ApiError variant.
D5.1. Helper module
New file crates/craig-test-lib/src/jwt_mutation.rs. Re-export from lib.rs as pub mod jwt_mutation.
D5.2. Test signing-key fixture
craig-auth/test-fixtures/test_rsa_private.der already exists (used by crates/craig-auth/src/middleware.rs:104). Reuse it. Expose a JwksFixture API on the auth crate so the test-lib doesn’t have to reach into private fixtures:
// crates/craig-auth/src/jwks.rs (additive)
#[cfg(any(test, feature = "test-fixtures"))]
pub mod test_fixtures {
use super::*;
use jsonwebtoken::jwk::{Jwk, JwkSet};
use jsonwebtoken::{Algorithm, EncodingKey};
/// The shared RSA private key that test middlewares + the
/// `craig-test-lib::jwt_mutation` helpers sign tokens with.
/// Identical bytes to `crates/craig-auth/test-fixtures/test_rsa_private.der`.
pub fn test_encoding_key() -> EncodingKey {
let der = include_bytes!("../test-fixtures/test_rsa_private.der");
EncodingKey::from_rsa_der(der)
}
/// JWKS containing the public half of [`test_encoding_key`], with
/// the given `kid`. Mount this on a [`JwksProvider`] via
/// [`JwksProvider::inject_keys`] (test-only fn — make `pub(crate)`
/// → `pub` under the same `test-fixtures` feature).
pub fn test_jwk_set(kid: &str) -> JwkSet {
let key = test_encoding_key();
let mut jwk = Jwk::from_encoding_key(&key, Algorithm::RS256)
.expect("build JWK from RSA test key");
jwk.common.key_id = Some(kid.to_string());
JwkSet { keys: vec![jwk] }
}
}
Cargo.toml: add a test-fixtures feature on craig-auth that re-exports the same module to consumers (craig-test-lib, integration tests of every service that mounts auth_middleware). This is the minimum visibility change; the alternative (a separate craig-auth-test crate) is a heavier refactor for marginal isolation gain.
D5.3. ClaimsBuilder
/// Mutable JWT claim set under construction. All getters/setters are
/// chainable; finalize with [`mint`].
///
/// Defaults (set by `ClaimsBuilder::baseline`): valid sub, future exp
/// (1h), past iat, `iss = TEST_ISSUER`, `aud = ["craig-cases"]` (caller
/// overrides for other services), `typ = "Bearer"`, no scope.
#[derive(Debug, Clone)]
pub struct ClaimsBuilder {
sub: String,
iss: String,
aud: Option<Vec<String>>,
azp: Option<String>,
typ: Option<String>,
scope: Option<String>,
exp: i64,
nbf: Option<i64>,
iat: i64,
realm_roles: Vec<String>,
extra: serde_json::Map<String, serde_json::Value>,
alg: jsonwebtoken::Algorithm,
kid: String,
}
impl ClaimsBuilder {
/// Construct a baseline that, when minted, validates against a
/// JwksProvider configured for `service_aud`. All single-axis
/// mutators below derive from this.
pub fn baseline(service_aud: &str) -> Self;
/// Override the audience claim. Pass `&["craig-cases"]` to scope
/// to a single service; pass `&[]` then `.without_aud()` to drop
/// the claim entirely.
pub fn with_aud(mut self, aud: &[&str]) -> Self;
/// Drop the `aud` claim. Used to assert the validator rejects
/// audience-less tokens (ADR-021).
pub fn without_aud(mut self) -> Self;
/// Override the `typ` header/claim. ADR-021 requires `"Bearer"`;
/// passing `"Refresh"` or `"ID"` must be rejected.
pub fn with_typ(mut self, typ: &str) -> Self;
/// Set the scope claim. RFC 6749 space-separated string.
pub fn with_scope(mut self, scope: &str) -> Self;
/// Set the authorized-party claim. ADR-021 surface; for `craig-ui`
/// vs `craig-api` distinction.
pub fn with_azp(mut self, azp: &str) -> Self;
/// Construct an expired token. `expired(Duration::from_secs(60))`
/// sets `exp = now - 60s`.
pub fn expired(mut self, ago: std::time::Duration) -> Self;
/// Construct a not-yet-valid token via `nbf`.
pub fn not_yet_valid(mut self, ahead: std::time::Duration) -> Self;
/// Override the signing algorithm. Pass `Algorithm::HS256` to
/// assert RS256-only validators reject. Note: `jsonwebtoken` 10.3
/// does NOT expose `Algorithm::None`. The "alg=none" attack is
/// covered by `with_unsigned()` below, which manually constructs
/// the compact JWT with header `{"alg":"none"}` and an empty
/// signature segment per RFC 7515 §6.1.
pub fn with_alg(mut self, alg: jsonwebtoken::Algorithm) -> Self;
/// Manually construct an unsigned JWT (`{"alg":"none"}` header +
/// claims + empty signature segment). Cannot use `jsonwebtoken::encode`
/// because the crate intentionally drops `Algorithm::None` support.
/// Mint path:
/// header_b64 = base64url("{\"alg\":\"none\",\"typ\":\"JWT\"}")
/// payload_b64 = base64url(serde_json::to_vec(&self.claims))
/// token = format!("{header_b64}.{payload_b64}.") // empty signature
/// Validators that fail to enforce an algorithm allowlist will
/// accept this token; the test asserts they reject.
pub fn with_unsigned(mut self) -> Self;
/// Sign with a kid that does NOT exist in the JWKS — exercises
/// kid-mismatch rejection.
pub fn with_unknown_kid(mut self) -> Self;
/// Inject an additional top-level claim (negative-test for unknown
/// claim acceptance, or positive-test for service-specific claims).
pub fn with_extra_claim(mut self, key: &str, value: serde_json::Value) -> Self;
/// Mint the token. Fails only on RSA signing errors (i.e. never
/// in supported configs).
pub fn mint(&self) -> String;
}
D5.4. Mutation matrix
/// Identifies the single axis a mutation perturbs. Used in test
/// failure messages so a 1-of-N parametric test pinpoints which
/// strictness rule the validator broke.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum MutationKind {
AudMissing,
AudWrongRealm,
AudWrongService,
TypMissing,
TypNotBearer,
AzpWrongClient,
ScopeMissing, // only emitted when baseline declares a required scope
ScopeInsufficient,
Expired1s,
Expired1h,
NotYetValid1m,
AlgNone, // alg=none attack
AlgConfusionHs256, // HS256 against an RS256-pinned validator
UnknownKid,
SubMissing,
IssWrongRealm,
}
pub struct MutatedToken {
pub kind: MutationKind,
pub jwt: String,
/// The expected `craig_common::error::ApiError` variant. Matched
/// stringly by the harness because the variants are large and
/// `PartialEq` is not derived.
pub expected_api_error: &'static str,
}
/// Yield one [`MutatedToken`] per single-axis mutation against the
/// given baseline. The base must already mint a valid token against
/// the service under test.
pub fn mutation_matrix(base: ClaimsBuilder) -> impl Iterator<Item = MutatedToken>;
mutation_matrix returns ~16 items today. Each is constructed by cloning the baseline, applying one mutator, and minting. The expected ApiError is hard-coded per kind and aligned to ADR-021 (e.g. AudMissing/AudWrongService → "Unauthorized: invalid audience"; AlgNone → "Unauthorized: alg not allowed"). When the validator is later widened to accept a previously-rejected variant, that test must be deleted and an ADR updated — making the strictness regression surface in code review.
D5.5. Cross-cuts with craig-auth
The mutator covers axes ADR-021 already names. To stay in sync:
-
New strictness fields in ADR-021 → new
MutationKindvariant. Reviewer must check. -
The
expected_api_errorstring is the public error message; ifcraig_common::error::ApiErroris restructured (e.g. variant renamed), this file is one of the call-out review locations. Loose coupling is acceptable here because the test failure message is what the developer reads, not a programmatic match. -
JwksProvider::with_audienceis the validator hook. If a future ADR-021 amendment adds requiredazp(currently optional per ADR),JwksFixturegains anexpect_azpbuilder and the matrix grows accordingly.
D5.6. Test description — parametric rejection across 8 services
New file crates/craig-test-lib/tests/jwt_mutation_smoke.rs:
use craig_test_lib::jwt_mutation::{ClaimsBuilder, mutation_matrix};
#[test]
fn baseline_token_validates() {
// Sanity check: the base case mints a token the validator accepts.
let token = ClaimsBuilder::baseline("craig-cases").mint();
let provider = test_provider_for_audience("craig-cases").await;
assert!(provider.validate_token(&token).await.is_ok());
}
#[test]
fn every_single_axis_mutation_is_rejected_by_craig_cases_validator() {
let base = ClaimsBuilder::baseline("craig-cases");
let provider = test_provider_for_audience("craig-cases");
let mut failures = Vec::new();
for m in mutation_matrix(base) {
match provider.validate_token(&m.jwt).await {
Ok(_) => failures.push(format!("{:?} unexpectedly accepted", m.kind)),
Err(e) if e.to_string().contains(m.expected_api_error) => {} // pass
Err(e) => failures.push(
format!("{:?} rejected with wrong error: got {e}, want {}",
m.kind, m.expected_api_error)),
}
}
assert!(failures.is_empty(), "JWT mutation regressions: {failures:#?}");
}
A per-service variant of the second test lives under each service’s tests/api/auth_strictness.rs (8 files), each iterating mutation_matrix(ClaimsBuilder::baseline("<service-aud>")) against a router built with that service’s auth_middleware. The iteration replaces every present hand-rolled "invalid token" check.
The eight per-service test files are owned by Phase B follow-ups; this step lands the helper + the smoke test only.
D6. Magic-byte upload-spoofing fixtures (Phase A.6 — Step 7)
Pairs with platform-stab-2 §D8.1, where validate_upload grows a content-sniffing arm via the infer crate. That work implements magic-byte rejection. This step ships the test corpus — a catalogued set of byte sequences that exercise every spoofing class plus a parallel set of valid happy-path companions. Without the corpus, every uploader test re-invents fixtures inline and the cases drift between sites.
D6.1. Fixture directory
New tree: crates/craig-test-lib/fixtures/upload_spoofing/. Every file is checked into git (each <2KB). LICENSE-SPDX header appended as a sibling README.md (binaries themselves can’t carry SPDX in-band). The directory is loaded via include_bytes! at compile time so tests don’t depend on cwd.
| File | Size | Description |
|---|---|---|
|
64B |
|
|
64B |
|
|
1.2KB |
Pre-baked tiny ZIP archive whose decompressed size is ~10MiB (10 nested deflate streams of zeros). Declared: |
|
32B |
|
|
8B |
Exactly |
|
384B |
XXE bomb ( |
|
256B |
Real |
|
134B |
Minimal valid JPEG ( |
|
67B |
Minimal valid PNG ( |
|
64B |
|
The valid-companion fixtures exist so happy-path tests don’t have to re-invent fixtures for every adopter; they are the negative-control half of every spoof test pair.
D6.2. Fixture loader
New file crates/craig-test-lib/src/upload_fixtures.rs. Re-export from lib.rs as pub mod upload_fixtures.
/// Stable identifier for a fixture file. Variants map 1:1 to D6.1 entries.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum FixtureName {
PeDisguisedAsPdf,
ElfDisguisedAsJpeg,
ZipBombDisguisedAsCsv,
ScriptDisguisedAsPng,
TruncatedPdfFirst8Bytes,
XmlWithExternalEntity,
ValidPdf,
ValidJpeg,
ValidPng,
ValidCsv,
}
/// Static fixture bytes. Compiled into the test binary via `include_bytes!`.
pub fn fixture(name: FixtureName) -> &'static [u8];
/// One spoof case: the bytes, the lying declared MIME, and the
/// `StoreError` variant the validator must surface.
pub struct SpoofFixture {
pub name: FixtureName,
pub declared_mime: &'static str,
pub body: &'static [u8],
/// Stringly matched against `StoreError::to_string()` because
/// `StoreError` does not implement `PartialEq`.
pub expected_rejection: &'static str,
}
/// Iterate every spoofing case (excludes the valid-companion fixtures).
/// 6 cases today; grows as new spoof classes are added.
pub fn enumerate_spoofs() -> impl Iterator<Item = SpoofFixture>;
/// Iterate every valid-companion fixture for happy-path tests.
pub fn enumerate_valid() -> impl Iterator<Item = SpoofFixture>; // body + true MIME
D6.3. Filename-attack fixtures
A second axis. Today crates/craig-store/src/validation.rs::sanitize_filename strips path separators, NUL, and trims leading/trailing whitespace + dots. Coverage at crates/craig-store/src/validation.rs:124-163 is good for those cases; missing: CRLF, RTLO, UNC, fullwidth-Unicode confusables, surrogate-pair edge cases, RFC-5987 round-trip through Content-Disposition (which platform-stab-2 §D8.1 also lands).
New crates/craig-test-lib/src/filename_attack.rs:
/// One filename-attack case. `expected_sanitized` is the exact bytes
/// `sanitize_filename` must return; `expected_disposition` is the exact
/// header value `RFC 5987` Content-Disposition encoding must produce
/// (None when the case is rejected outright).
pub struct FilenameCase {
pub name: &'static str,
pub raw: &'static str,
pub expected_sanitized: Result<&'static str, &'static str>,
pub expected_disposition: Option<&'static str>,
}
pub const CASES: &[FilenameCase] = &[
FilenameCase {
name: "double_quote",
raw: r#"my"file.pdf"#,
// Quote is preserved by sanitize_filename today. The download
// path Content-Disposition encoder must escape it.
expected_sanitized: Ok(r#"my"file.pdf"#),
expected_disposition: Some(
r#"attachment; filename="my\"file.pdf"; filename*=UTF-8''my%22file.pdf"#),
},
FilenameCase {
name: "crlf_injection",
raw: "ok.pdf\r\nSet-Cookie: x=y",
// CRLF is not stripped today; the encoder is responsible for
// % -encoding it. If sanitize_filename later starts rejecting
// CRLF, flip expected_sanitized to Err and drop disposition.
expected_sanitized: Ok("ok.pdfSet-Cookie: x=y"),
expected_disposition: Some(
r#"attachment; filename="ok.pdfSet-Cookie: x=y"; filename*=UTF-8''ok.pdf%0D%0ASet-Cookie%3A%20x%3Dy"#),
},
FilenameCase {
name: "nul_byte",
raw: "ok\0.pdf",
expected_sanitized: Ok("ok.pdf"),
expected_disposition: Some(r#"attachment; filename="ok.pdf"; filename*=UTF-8''ok.pdf"#),
},
FilenameCase {
name: "backslash",
raw: r"a\b\c.pdf",
expected_sanitized: Ok("abc.pdf"),
expected_disposition: Some(r#"attachment; filename="abc.pdf"; filename*=UTF-8''abc.pdf"#),
},
FilenameCase {
name: "forward_slash",
raw: "a/b/c.pdf",
expected_sanitized: Ok("abc.pdf"),
expected_disposition: Some(r#"attachment; filename="abc.pdf"; filename*=UTF-8''abc.pdf"#),
},
FilenameCase {
name: "leading_trailing_whitespace",
raw: " spaced.pdf ",
expected_sanitized: Ok("spaced.pdf"),
expected_disposition: Some(r#"attachment; filename="spaced.pdf"; filename*=UTF-8''spaced.pdf"#),
},
FilenameCase {
name: "length_256",
raw: "<256 'a' chars>.pdf",
// 255-byte char-boundary truncation — see validation.rs
// sanitize_truncates_at_char_boundary_for_multibyte_input.
expected_sanitized: Ok("<255 'a' chars>"),
expected_disposition: Some(r#"attachment; filename="<255 'a' chars>"; filename*=UTF-8''<...>"#),
},
FilenameCase {
name: "unicode_resume",
raw: "résumé.pdf",
expected_sanitized: Ok("résumé.pdf"),
expected_disposition: Some(r#"attachment; filename="r__sum__.pdf"; filename*=UTF-8''r%C3%A9sum%C3%A9.pdf"#),
},
FilenameCase {
name: "rtlo_override",
raw: "fdp.\u{202E}exe", // U+202E RIGHT-TO-LEFT OVERRIDE — renders "exe.pdf" in many UIs
// RTLO must be stripped: it has no semantic in a filename and
// the rendering surface gets to lie to the user.
expected_sanitized: Ok("fdp.exe"),
expected_disposition: Some(r#"attachment; filename="fdp.exe"; filename*=UTF-8''fdp.exe"#),
},
FilenameCase {
name: "unc_path",
raw: r"\\evil.example.com\share\f.pdf",
expected_sanitized: Ok("evil.example.comsharef.pdf"),
expected_disposition: Some(
r#"attachment; filename="evil.example.comsharef.pdf"; filename*=UTF-8''evil.example.com%5Cshare%5Cf.pdf"#),
},
FilenameCase {
name: "all_dots",
raw: "...",
expected_sanitized: Err("InvalidFilename"),
expected_disposition: None,
},
FilenameCase {
name: "fullwidth_dot",
raw: "f\u{FF0E}exe", // U+FF0E FULLWIDTH FULL STOP — visually `.` but not the path-separator char
expected_sanitized: Ok("f\u{FF0E}exe"),
expected_disposition: Some(r#"attachment; filename="f.exe"; filename*=UTF-8''f%EF%BC%8Eexe"#),
},
];
The expected outputs are frozen at the table — every change to sanitize_filename or the disposition encoder must also touch CASES (and therefore the MR diff), making sanitization-rule changes auditable.
D6.4. Test-coverage description
crates/craig-test-lib/tests/upload_fixtures_smoke.rs:
-
every_spoof_is_rejected_by_validate_upload: parametric loop overenumerate_spoofs, each callsvalidate_upload(case.declared_mime, case.body.len(), &v)(wherevincludes the §D8.1 magic-byte check) and asserts the resultingStoreErrorto_string()containscase.expected_rejection. -
every_valid_companion_passes_validate_upload: parametric loop overenumerate_valid, each must returnOk(()). -
filename_attack_round_trip: for eachFilenameCase,sanitize_filename(case.raw)is matched againstcase.expected_sanitized; ifOk, the result is fed through the (platform-stab-2 §D8.1) disposition encoder and matched againstcase.expected_disposition. -
Failing-test-first protocol (mirrors §D4.4): land the helper, write the test with
truncated_pdf_first_8_bytes.bindeliberately omitted fromenumerate_spoofs, observe the spoof rejection coverage drop, restore.
These tests run without devstack — pure in-process validation — and gate at the workspace cargo nextest run --workspace --lib level. Phase B uploader-handler integration tests (4 sites: contact_attachments, court_orders, ICPC, report_attachments) consume these fixtures via craig_test_lib::upload_fixtures::enumerate_spoofs and POST each through their respective endpoint, asserting 415 / 422 per platform-stab-2 §D8.1.
D7. testing.md "Failure-path testing" + Test Categories taxonomy + Process rule (Phase A.7 — Step 8)
This is the meta-deliverable. It is the only step in this plan that is documentation-only — no Rust, no migration. It encodes the test-categorization taxonomy that today is implicit in the parent epic plan, formalizes per-category file-naming conventions so a contextless agent can place a new test correctly, links every helper landed in §D4–§D6 (and the Phase 2 / Step 2 helpers from platform-stab-2) into a usage cheat-sheet, and writes down the Process Rule that closes the loop on the two-pass platform-invariant audit.
D7.1. New "Test Categories" section in .claude/docs/testing.md
Inserted after "Critical Rules" / "Mandatory Rules" and before "Migration Naming Convention". Eight categories. The descriptions below are the exact text to land verbatim:
| Category | Description |
|---|---|
Functional |
Single-fire request → single response → assert status + body. The status quo. Catches "does this endpoint exist and return the right shape." Example: |
Invariant |
Asserts a property that must hold across many states or across many calls. Distinct from functional in that the property is a universal claim, not a single observation. Example: "every successful POST |
Concurrency |
Spawns N parallel calls against the same endpoint or store function and asserts a no-double / no-drop / consistent-state property. Uses |
Fault-injection |
Wraps a dependency (DB, RMQ, cipher, object store, stage_event) in a wrapper that errors on demand, then asserts the error path is reached and behaves correctly. Uses |
Recovery |
Like fault-injection but the test continues past the fault and asserts the system recovers — orphans get reaped, in-flight jobs get reclaimed, retries succeed. Example: "kill craig-cases between Phase 1 commit and Phase 2 put in attachment upload; restart; assert scanner reaps orphan within 5 min" (platform-stab-2 §D4). |
Contract |
End-to-end typed-DTO round-trip through a service’s HTTP boundary. The test client uses the same DTO type the service handler imports. A breaking schema change to either side fails compilation rather than passing through a |
Property-based |
Generators ( |
(s: String) { assert!(sanitize_filename(&s).map( |
n |
!n.contains('\0')).unwrap_or(true)) })`. Available but not required; introduce on the rare functions whose input space is too large to enumerate. |
Mutation |
The taxonomy is descriptive of how tests are written across the codebase, not prescriptive of a category quota per service. Most tests are Functional today; the goal of epic &22 is to make Concurrency / Fault-injection / Recovery the second-most-common classes, with Invariant tests added as platform invariants get encoded.
D7.2. New "File-naming convention for test categories" section
| Category | Path convention |
|---|---|
Functional |
|
Invariant |
|
Concurrency |
|
Fault-injection |
|
Recovery |
|
Contract |
typed-client tests live in |
Property-based |
|
Mutation |
No per-test file. Configured at the workspace level via |
Existing tests are not mass-relocated. The convention applies to net-new test files per Phase B and per future MRs.
D7.3. New "Failure-path testing helpers" section
A usage cheat-sheet pointing at every helper landed by this plan + by platform-stab-2 Step 2:
| Helper | Imported as | Use-case |
|---|---|---|
|
|
Concurrency tests — spawn N parallel tasks, await all. |
|
|
Fault-injection on |
Codepath-specific store traits + faulty impls (e.g. |
|
DB fault injection — at the codepath under test (§D2.0 sub-pattern b). For systemic DB failures use container-level (sub-pattern a) instead. There is no |
|
|
Fault-injection on |
|
|
Fault-injection on |
|
|
Fault-injection on |
|
|
State-machine matrix testing (§D4). |
|
|
JWT strictness rejection tests (§D5). |
|
|
Upload-spoofing rejection tests (§D6). |
|
|
Filename-sanitization round-trip tests (§D6.3). |
|
|
Phase A.3 (sibling step in this plan, parent epic Phase A.3). |
|
dev-dependency on |
Property-based tests (introduce per-test-file). |
|
|
Mutation testing (periodic, not per-push). |
Each row carries a one-line example invocation in the actual testing.md so it is grep-able.
D7.4. New "Process rule" section
Verbatim text to land:
Process rule (epic &21 / &22 onward): every concurrency-relevant fix, fault-tolerance fix, or platform-invariant fix MUST ship with at least one Concurrency, Fault-injection, Invariant, or Recovery test — in any combination — in the same MR as the production-code change. The choice of category(ies) is the author’s; the existence of some failure-path test is non-negotiable. Failing-test-first (commit failing test, then commit fix in the same MR) is preferred — it makes the regression demonstrable from the diff alone — but not mandated; reviewer-requested addition is acceptable.
Reviewer responsibility: when reviewing an MR labeled concurrency, platform-invariant, durability, or any sub-label of those, the reviewer must explicitly confirm in CR that a test of one of the four categories exists. The MR is blocked until either (a) the test is added or (b) the author argues — and the reviewer accepts — that the change is structurally incapable of regressing the invariant (e.g. a comment-only edit). Discipline-only enforcement; no pre-commit hook (default proposal in the parent epic Open Questions item 4).
Linkage: this rule is the code-and-test embodiment of the two-pass platform-invariant audit's storage-shape-vs-semantics lesson. An audit that verifies "table exists" without the matching Concurrency / Recovery test does not constitute platform-invariant verification.
D7.5. Cross-references to update
-
.claude/docs/delivery-protocol.md— add a one-paragraph pointer in the "Plan completion audit" section: "When the audit is verifying a platform invariant, the auditor must follow the two-pass platform-invariant audit — name the failure mode in plain English, cite (or write) the failing test that catches it, then verify the fix. Storage-shape verification alone is insufficient." -
CONTRIBUTING.adoc— add a new section "Failure-path testing" with a one-paragraph pointer to the testing.md sections D7.1–D7.4 above. Keep CONTRIBUTING.adoc the public-facing entry; testing.md is the agent-facing detail. -
.claude/CLAUDE.md"Required Reading" table — testing.md row already exists; no change. (No new MUST-READ; this is documentation expansion within an existing required doc.)
D7.6. Quality budgets — available, not enforced
A separate sub-section of testing.md, after Process rule. Eight numeric budgets, each implemented as a script in xtask/src/budget.rs (Phase B follow-up — not this plan’s step):
| Budget | Threshold (proposed) | Command |
|---|---|---|
Max route module length |
500 LOC (post platform-stab-2 §D10.1 decomposition) |
|
Max function length |
80 LOC |
|
Max |
declining baseline (start: current count) |
|
Max |
declining baseline |
|
Max |
declining baseline (post platform-stab-2 §D7) |
|
New dependency duplicates |
0 (post platform-stab-2 §D10.2 cleanup) |
|
New untyped test clients |
0 among trusted-write clients post-§D8; |
|
Mutation-test surviving-mutant ratio on critical modules |
0 in |
|
Threshold + command shape are reserved here; no enforcement in this plan. The pre-push CI gates listed by the user (cargo xtask reliability, contracts, mutants --smoke) are deliberately deferred — the plan documents the call-shape so the future enforcement MR has a stable target name, but the gate itself is not part of any step in this plan or epic &22 Phase A.
D8. Typed DTOs across 8 test clients (Phase A.8 — Step 9)
Every typed test client today (crates/craig-test-lib/src/clients/*.rs) uses serde_json::Value for both request bodies and response bodies. 211 occurrences across 8 files. The cost: a hand-rolled JSON shape mismatch in a test does not fail at compile time; it fails at the assertion site with a null field, days after the actual schema regression in the service-side handler. A typed client would have caught every "this field renamed" / "field type changed" / "required field made optional" regression in a single recompile.
D8.1. Decision: full per-client rewrite, not gradual
| Option | Outcome |
|---|---|
Gradual within each client (mix |
Rejected. A half-typed client carries the worst of both: review still has to read the JSON shape for the un-typed methods, and the type system has to be re-asserted at every typed/untyped boundary. The migration would stall. |
Full per-client rewrite, 8 small MRs |
Selected. Each MR is independently reviewable, the partial state across the codebase is "this client typed, that one not yet" (clean) instead of "this method typed, the next not" (unclean), and the order can be rebalanced as bandwidth allows. Default per the parent epic Open Questions item 3. |
Single big-bang rewrite |
Rejected. Diff size unreviewable. |
D8.2. DTO ownership: per-service *-contracts crates, not central re-export
A previous draft had test-client typed DTOs use-ing the service crate directly: craig-test-lib would [dev-dependencies] craig-cases = { workspace = true } and use craig_cases::api::dto::*. This creates a Cargo dependency cycle: services already dev-depend on craig-test-lib (e.g. services/craig-cases/Cargo.toml:34). Adding a back-edge from craig-test-lib → craig-cases produces:
craig-cases → [dev-dep] → craig-test-lib → [dep] → craig-cases # cycle
Cargo’s resolver routes around this for the test target (dev-deps don’t propagate via the normal graph), but the cycle is brittle: any non-dev import path breaks cargo build workspace-wide. Production-safety lives in not creating the cycle in the first place.
| Option | Trade-off |
|---|---|
Service crate exposes |
Rejected. Creates the cycle described above. Worked while the prior draft assumed services don’t dev-depend on test-lib; that assumption is false today. |
New |
One canonical home, easy |
Per-service |
Selected. No cycle (contracts crates are leaf nodes; nothing depends back on test-lib). DTO change is a 1-MR diff that the contracts crate compiles, the service compiles, the test client compiles — three crates, one MR, mechanical change. Adds 8 new crates to the workspace; each is ~50-200 LOC of pure data types. Mitigates the cycle without paying the central-bottleneck cost of a single |
Move typed clients into each service’s own test tree ( |
Eliminates the test-lib boundary entirely — each service owns both its DTOs and its typed test client. But: breaks the "test-lib has the cross-service test infrastructure" pattern that’s already established for the 8 untyped clients. Cross-service tests (e.g. integration tests in |
Per-service -contracts crate shape*:
# crates/craig-cases-contracts/Cargo.toml (new)
[package]
name = "craig-cases-contracts"
version = "0.1.0"
edition = "2024"
[dependencies]
serde = { workspace = true, features = ["derive"] }
serde_json = { workspace = true } # only if any DTO field needs Value
chrono = { workspace = true }
uuid = { workspace = true, features = ["serde"] }
utoipa = { workspace = true, features = ["chrono", "uuid"] }
craig-reference = { workspace = true } # if DTOs use shared enums
# NO dependency on craig-cases or craig-test-lib
// crates/craig-cases-contracts/src/lib.rs (new)
pub mod cases;
pub mod persons;
pub mod referrals;
// ... one module per api submodule with public DTOs
// services/craig-cases/Cargo.toml — add a non-dev dep
[dependencies]
craig-cases-contracts = { workspace = true }
// services/craig-cases/src/api/dto.rs (existing)
pub use craig_cases_contracts::cases::*;
pub use craig_cases_contracts::persons::*;
// internal re-export so existing handler code at `crate::api::dto::*` keeps working
// crates/craig-test-lib/Cargo.toml — add per-service contracts dev-deps only
[dev-dependencies]
craig-cases-contracts = { workspace = true }
craig-placement-contracts = { workspace = true }
craig-exchange-contracts = { workspace = true }
craig-financial-contracts = { workspace = true }
craig-reporting-contracts = { workspace = true }
craig-rules-contracts = { workspace = true }
craig-security-contracts = { workspace = true }
craig-intake-contracts = { workspace = true }
// crates/craig-test-lib/src/clients/cases.rs (rewritten in Step 9)
use craig_cases_contracts::cases::{CreateCaseRequest, CaseResponse};
use craig_cases_contracts::persons::{CreatePersonRequest, PersonResponse, ListPersonsResponse};
// no `use craig_cases::*;` — no cycle.
Migration ordering: each per-service -contracts crate is filed as the *first commit of its respective Phase A.8 sub-MR. Step 9 of THIS plan files the CasesClient pilot, which means: Step 9 lands craig-cases-contracts/ first, then migrates the existing DTOs into it (one move-per-file), then re-exports from the service, then rewrites the test client to use the contracts crate. 7 follow-up MRs each repeat the pattern for one service.
Cycle-detection guard: add a cargo deny bans.deny rule name = "craig-cases", deny-from = ["craig-test-lib"] (and similarly for the other 7 services). Any future PR that introduces the back-edge fails CI. Lightweight; pure prevention.
D8.3. Inventory — current state of the 8 clients
| # | Client (file) | Method count | Body type today |
|---|---|---|---|
1 |
|
~60 methods |
|
2 |
|
~32 |
same |
3 |
|
~22 |
same |
4 |
|
~17 |
same |
5 |
|
~15 |
same |
6 |
|
~7 |
same |
7 |
|
~46 |
same |
8 |
|
~12 |
same (mix of authenticated + unauthenticated; both use |
Total: ~211 methods, 1:1 with the grep count. Migration order is not alphabetical:
| Order | Client | Why this position |
|---|---|---|
1 |
|
Highest churn (60 methods); piloting here surfaces every wrinkle (path-param escaping, multipart upload, query param threading) before the smaller clients. |
2 |
|
Second-largest, similar concurrency-test load coming in Phase B. |
3 |
|
Heavy in epic &21 Step 7 send-worker — typed responses pay off there. |
4 |
|
First full state-machine matrix consumer (§D4); typed |
5 |
|
Lower endpoint count, lower test churn — schedule before security to keep momentum. |
6 |
|
Tiny client; lands fast, provides confidence-builder mid-migration. |
7 |
|
46 methods; large diff but isolated from the epic &21 P0/P1 timeline. |
8 |
|
The mixed authenticated/unauthenticated split means this one needs a builder method per auth posture; saves the trickiest for last when the pattern is firm. |
Each migration is a single MR with branch prefix feat/test-framework-typed-clients-<svc>.
D8.4. Worked example: CasesClient::list_persons
Today (crates/craig-test-lib/src/clients/cases.rs:41):
pub async fn search_persons(&self, params: &[(&str, &str)]) -> Result<ApiResponse<Value>> {
self.inner
.get_with_query("/v1/cases/persons", ¶ms.to_vec())
.await
}
Tomorrow:
use craig_cases::{ListPersonsResponse, ListPersonsQuery};
pub async fn search_persons(&self, query: &ListPersonsQuery) -> Result<ApiResponse<ListPersonsResponse>> {
self.inner
.get_with_query("/v1/cases/persons", &query.to_query_pairs())
.await
}
Where ListPersonsQuery is a new typed struct in services/craig-cases/src/api/persons.rs (deserialize from URL query via serde_urlencoded already in the deps tree) re-exported from the lib root. ListPersonsQuery::to_query_pairs(&self) → Vec<(&'static str, String)> lives next to the type.
ApiResponse<T> (in crates/craig-test-lib/src/client.rs) is already generic; this migration is a body-type swap, not a client-shape change.
For create_person, the diff:
// today
pub async fn create_person(&self, body: &Value) -> Result<ApiResponse<Value>> {
self.inner.post("/v1/cases/persons", body).await
}
// tomorrow
use craig_cases::{CreatePersonRequest, PersonResponse};
pub async fn create_person(&self, body: &CreatePersonRequest) -> Result<ApiResponse<PersonResponse>> {
self.inner.post("/v1/cases/persons", body).await
}
ServiceClient::post<T: Serialize, R: DeserializeOwned> already supports generic body and response — the wrapper at crates/craig-test-lib/src/client.rs is generic at the inner layer, only the per-client typed wrappers were Value-specialized.
Test impact: every existing test calling cases_client.create_person(&json!({…})) now writes cases_client.create_person(&CreatePersonRequest { … }). The json! body is replaced with a struct literal; serde inference does the rest. The diff is mechanical except for two patterns:
-
Pre-existing JSON malformation as test input. A few tests deliberately POST malformed JSON to assert 422. These tests bypass the typed client and call
ServiceClient::post_raw(existing escape hatch — preserved). -
Open-shape responses. A handful of endpoints return
{ …heterogeneous fields… }based on a discriminator; these get a#[serde(tag = "kind")]enum DTO on the service side and the test client benefits identically.
D8.5. Quality-budget tie-in (warnings-only)
Per §D7.6, a cargo xtask budget --metric value-count walks crates/craig-test-lib/src/clients//.rs and counts serde_json::Value. Budget threshold: declining baseline. After all 8 clients migrate, the budget is 0 in clients/. Value remains valid in client.rs (the boundary layer that handles raw response bytes / problem+json) — those occurrences are excluded by path. The check is warnings-only* in this plan; promotion to a hard gate is filed as a future MR after the 8 MRs land.
D8.6. Test-coverage description — per-client smoke
Each per-client migration MR ships one Contract test under crates/craig-test-lib/tests/typed/<svc>_client.rs:
// tests/typed/cases_client.rs (lands with MR 1)
use craig_cases::{CreatePersonRequest, PersonResponse};
use craig_test_lib::{TestHarness, devstack_available};
#[tokio::test]
async fn create_person_typed_round_trip() {
if !devstack_available().await { return; }
let harness = TestHarness::new().await;
let client = harness.caseworker_cases_client();
let req = CreatePersonRequest {
first_name: "Round".into(),
last_name: "Trip".into(),
// … every required field. Compile fails if a field is missing
// or renamed — caught in CI rather than in a flaky `null` deref.
};
let resp = client.create_person(&req).await.unwrap();
assert_eq!(resp.status, reqwest::StatusCode::CREATED);
let person: &PersonResponse = resp.body.as_ref().unwrap();
assert_eq!(person.first_name, "Round");
// No more `body["first_name"].as_str().unwrap()` — direct field access.
}
The test pins one round-trip per client. A subsequent service-side rename to first_given_name fails at craig-cases::CreatePersonRequest’s definition; the test client fails to compile; the test client’s caller (`harness.create_person(&req)) fails to compile; the regression is caught at cargo build rather than at integration runtime.
This is the only Contract test landed by the migration; deeper Contract coverage is per-endpoint and accumulates as Phase B tests adopt the new clients.
D9. proptest scaffolding + first 7 properties (Phase A.9 — Step 10)
The eight Phase-A foundation pieces enumerated in the existing draft are imperative helpers (one helper at a time, hand-coded scenarios). Property-based testing is the declarative counterpart: you describe the input space and the invariant the code must satisfy, and the runner generates+shrinks counter-examples. Adding it now closes the gap between the small-case integration suite and the unbounded input space the production services actually see.
The user steered the existing Step 5 open question ("proptest crate vs hand-rolled state-machine matrix") toward hand-rolled for the bounded state-machine alphabet. That conclusion stands for state machines. Property testing — by contrast — is exactly where proptest’s strength lies: filename sanitization, JSON canonicalization, idempotency-fingerprinting, RFC-encoded headers all live in unbounded input spaces where a hand-curated case list is structurally inadequate.
D9.1 Workspace dependency
# Cargo.toml [workspace.dependencies]
# Property-based testing — used in `tests/properties/<prop>.rs` per testing.md.
proptest = "1.7" # latest stable as of 2026-05; verify on commit via `cargo search proptest`
Per-crate adoption opts in via [dev-dependencies] only — proptest never reaches a release artefact:
# crates/craig-store/Cargo.toml
[dev-dependencies]
proptest = { workspace = true }
D9.2 Module convention
testing.md (extended in Step 8 of this plan with the "Failure-path testing" section) gains a sibling subsection: "Property tests". The convention:
| Layout | Rule |
|---|---|
File path |
|
Module index |
|
Per-property test name |
|
Cases override |
Default |
Shrinking budget |
Use proptest defaults. If a property repeatedly times out shrinking on CI, raise |
|
The Step 12 subcommand globs |
The Step 8 testing.md edit also documents the fail-first discipline: introduce a deliberate one-character bug in the unit-under-test, run the prop, watch shrinking land on a minimal counter-example, then revert. Land each property with a one-line PR comment quoting the shrunk input that the un-bugged code now passes. This is the property-test analogue of the failing-then-passing rule for concurrency tests.
D9.3 Property 1 — state-machine round-trip (PaymentStatus)
File: services/craig-financial/tests/properties/payment_status.rs
The state-machine matrix helper from Step 5 (Phase A.4) covers reject-set verification by enumeration. The property version covers a different invariant: that an enum’s can_transition_to agrees with the validator function used at the API boundary, across the full cross-product.
use proptest::prelude::*;
// PaymentStatus actually lives at services/craig-financial/src/transitions.rs:11,
// NOT in craig-reference. Five variants only — no Draft, Disbursed, Reversed, Cancelled.
use craig_financial::transitions::{PaymentStatus, validate_payment_transition};
fn any_status() -> impl Strategy<Value = PaymentStatus> {
prop_oneof![
Just(PaymentStatus::Pending),
Just(PaymentStatus::Approved),
Just(PaymentStatus::Issued),
Just(PaymentStatus::Cleared),
Just(PaymentStatus::Voided),
]
}
proptest! {
#![proptest_config(ProptestConfig::with_cases(512))]
#[test]
fn prop_payment_can_transition_matches_validator(
from in any_status(),
to in any_status(),
) {
let enum_says = from.can_transition_to(to);
let validator_result = validate_payment_transition(from.as_str(), to.as_str());
prop_assert_eq!(enum_says, validator_result.is_ok(),
"enum/validator disagree for {from:?} -> {to:?}");
}
}
Fail-first sketch: temporarily flip one can_transition_to arm (e.g. allow Cleared → Pending); proptest shrinks to exactly that pair.
Expected runtime: 25 ordered pairs × 512 cases = 12,800 calls; <50ms total.
D9.4 Property 2 — pagination
File: crates/craig-api/tests/properties/pagination.rs
Pagination invariants are heavily relied on across all 8 services and have escaped review in the past (see Phase 10 Web UI work that surfaced the per_page=25 vs per_page=10 interaction). Property protects three invariants simultaneously:
-
Each page contains at most
per_pageitems. -
Items are stably ordered (the sort key is monotonic across pages).
-
No item appears on two pages.
use proptest::prelude::*;
use craig_api::pagination::{PageRequest, paginate};
fn any_dataset() -> impl Strategy<Value = Vec<u64>> {
proptest::collection::vec(0u64..1_000_000, 0..500)
.prop_map(|mut v| { v.sort(); v.dedup(); v })
}
proptest! {
#[test]
fn prop_pagination_no_dupes_no_drops(
data in any_dataset(),
per_page in 1usize..50,
) {
let total = data.len();
let pages: Vec<_> = (1..=total.div_ceil(per_page).max(1))
.map(|p| paginate(&data, PageRequest { page: p, per_page }))
.collect();
// each page len <= per_page
for p in &pages { prop_assert!(p.items.len() <= per_page); }
// ordering monotonic across pages
let flat: Vec<_> = pages.iter().flat_map(|p| p.items.iter().copied()).collect();
for w in flat.windows(2) { prop_assert!(w[0] <= w[1]); }
// no duplicates across pages
let mut seen = std::collections::HashSet::new();
for x in &flat { prop_assert!(seen.insert(*x)); }
// total reachable equals input
prop_assert_eq!(flat.len(), total);
}
}
Expected runtime: 256 cases × ~10 page builds = 2.5k calls; <100ms.
D9.5 Property 3 — filename sanitization
File: crates/craig-store/tests/properties/sanitize_filename.rs
craig_store::validation::sanitize_filename (signature pub fn sanitize_filename(&str) → Result<String, StoreError> at crates/craig-store/src/validation.rs:58) is on the upload hot path and has been touched repeatedly (most recently by platform-stab-2 §D8.1’s RFC 5987 work). Property locks four invariants on the Ok arm (the Err arm fires for inputs that sanitize to empty, which is its own correctness requirement):
-
Output never contains
/,\, NUL, or other ASCII controls. -
Output is non-empty (the only return path;
Erris the empty case). -
Idempotent:
sanitize_filename(sanitize_filename(s).unwrap())? == sanitize_filename(s)?. -
Output byte length ⇐ 255 (the function’s documented truncation cap).
use proptest::prelude::*;
use craig_store::validation::sanitize_filename;
proptest! {
#![proptest_config(ProptestConfig::with_cases(1024))]
#[test]
fn prop_sanitize_filename_invariants(s in ".{0,256}") {
let Ok(out) = sanitize_filename(&s) else {
// Empty-after-sanitization → Err is correct behavior; skip this case.
return Ok(());
};
for ch in out.chars() {
prop_assert!(!matches!(ch, '/' | '\\' | '\0'),
"forbidden char {ch:?} survived sanitization of {s:?}");
prop_assert!(!ch.is_control(),
"control char {ch:?} survived sanitization");
}
prop_assert!(!out.is_empty(), "Ok arm returned empty string");
let twice = sanitize_filename(&out)
.expect("Ok output must itself be sanitizable to Ok");
prop_assert_eq!(twice, out, "non-idempotent: f(f(x)) != f(x)");
prop_assert!(out.len() <= 255, "output exceeds 255-byte cap: {}", out.len());
}
}
Fail-first sketch: replace the c != '/' filter arm in the chain at validation.rs:60 with c != ';' so / re-appears in output; proptest shrinks to a single-character input "/".
Expected runtime: 1024 cases × short string ops; <50ms.
D9.6 Property 4 — Content-Disposition encoding round-trip
File: crates/craig-store/tests/properties/content_disposition.rs
Pairs with platform-stabilization-2 §D8.1’s RFC 6266 encoder. Property: the encoded header round-trips through a parser back to the original filename, modulo the documented ASCII-fallback substitution.
use proptest::prelude::*;
use craig_store::validation::content_disposition_attachment;
/// Reference parser using the `headers` crate (or hand-rolled per RFC 6266 §5).
fn parse_filename_star(header_value: &str) -> Option<String> { /* ... */ }
proptest! {
#![proptest_config(ProptestConfig::with_cases(512))]
#[test]
fn prop_content_disposition_round_trip(name in "[\\PC]{1,200}") {
// \PC = any non-control char, including non-ASCII for the filename* field.
let header = content_disposition_attachment(&name);
prop_assert!(header.starts_with("attachment;"),
"header missing attachment disposition: {header}");
prop_assert!(header.contains("filename*=UTF-8''"),
"header missing RFC 5987 extended form: {header}");
let recovered = parse_filename_star(&header)
.expect("parser must recover filename*");
prop_assert_eq!(recovered, name);
}
}
Expected runtime: 512 cases × encode+parse; <100ms.
D9.7 Property 5 — canonical JSON signing
File: crates/craig-signing/tests/properties/canonical_json.rs
craig-signing exposes two primitives at crates/craig-signing/src/lib.rs: canonicalize_json(&serde_json::Value) → String (line 40 — recursive sort-keys + minimal whitespace) and hash_payload(&str) → String (line 73 — SHA-256 hex of the canonical string). Together they form the signing input for inter-service envelopes. Insignificant whitespace and key-order shouldn’t change the canonical output (and therefore must not change the hash). Property:
-
Canonicalize
v, hash; re-serializevwithserde_json::to_string_pretty(adds whitespace + reshuffles HashMap key order), parse back, canonicalize, hash — hashes must match. -
Replace any leaf value with a different leaf — canonical strings must differ (and thus hashes must differ).
use proptest::prelude::*;
use craig_signing::{canonicalize_json, hash_payload};
use serde_json::Value;
fn any_json_leaf() -> impl Strategy<Value = Value> {
prop_oneof![
Just(Value::Null),
any::<bool>().prop_map(Value::Bool),
(-1_000_000i64..1_000_000).prop_map(|n| Value::from(n)),
"[\\PC]{0,32}".prop_map(Value::from),
]
}
fn any_json() -> impl Strategy<Value = Value> {
let leaf = any_json_leaf();
leaf.prop_recursive(4, 16, 8, |inner| {
prop_oneof![
proptest::collection::vec(inner.clone(), 0..6).prop_map(Value::Array),
proptest::collection::hash_map("[a-z]{1,8}", inner, 0..6)
.prop_map(|m| serde_json::to_value(m).unwrap()),
]
})
}
proptest! {
#![proptest_config(ProptestConfig::with_cases(256))]
#[test]
fn prop_canonical_hash_invariant_to_whitespace_and_key_order(v in any_json()) {
let canonical_a = canonicalize_json(&v);
let hash_a = hash_payload(&canonical_a);
// Round-trip through pretty-printer reshuffles HashMap iteration order
// and inserts whitespace; canonicalize must recover identical bytes.
let pretty = serde_json::to_string_pretty(&v).unwrap();
let v_round: Value = serde_json::from_str(&pretty).unwrap();
let canonical_b = canonicalize_json(&v_round);
let hash_b = hash_payload(&canonical_b);
prop_assert_eq!(canonical_a, canonical_b,
"canonical string changed under whitespace/key-order normalization");
prop_assert_eq!(hash_a, hash_b,
"hash changed under whitespace/key-order normalization");
}
}
Fail-first sketch: replace canonicalize_json recursion arm for Value::Object with serde_json::to_string(&map).unwrap() (HashMap iteration order is non-deterministic in Rust); pretty-printed input shrinks to a 2-key object that hashes differently on different runs.
Expected runtime: 256 cases × recursive json (depth 4); <500ms (canonicalization dominates).
D9.8 Property 6 — idempotency body fingerprint
File: crates/craig-api/tests/properties/idempotency_fingerprint.rs
Pairs with platform-stab-2 §D1’s idempotency claim. The body-fingerprint function gates the loser-side fingerprint comparison; if it ever returns the same fingerprint for two different bodies, two requests with mismatched bodies are quietly accepted as duplicates.
Visibility note: body_fingerprint is currently a private free function at crates/craig-api/src/idempotency.rs:105. Two paths to make it testable:
| Option | Trade-off |
|---|---|
Promote to |
Keeps the function private to non-test consumers; one-line addition. |
Promote to |
Makes the fingerprint algorithm a public API contract, which it arguably already is (since the value is persisted in |
Adopt option 2: change fn body_fingerprint(…) at idempotency.rs:105 to pub fn body_fingerprint(…), document it as part of the crate’s public API. Step 10 of THIS plan includes the visibility flip as a one-line change in crates/craig-api/src/idempotency.rs.
Two complementary property assertions:
-
Equality preservation:
fingerprint(b1) == fingerprint(b2)iffb1 == b2. -
Small-flip distinctness: differ-by-one-byte inputs produce different fingerprints (catches a checksum that ignores high-order bits or has a small fingerprint domain).
use proptest::prelude::*;
use craig_api::idempotency::body_fingerprint; // requires the pub-visibility flip noted above
proptest! {
#![proptest_config(ProptestConfig::with_cases(512))]
#[test]
fn prop_fingerprint_equal_iff_bytes_equal(
a in proptest::collection::vec(any::<u8>(), 0..2048),
b in proptest::collection::vec(any::<u8>(), 0..2048),
) {
let fa = body_fingerprint(&a);
let fb = body_fingerprint(&b);
prop_assert_eq!(fa == fb, a == b);
}
#[test]
fn prop_fingerprint_one_byte_flip_differs(
bytes in proptest::collection::vec(any::<u8>(), 1..1024),
idx in any::<prop::sample::Index>(),
) {
let i = idx.index(bytes.len());
let mut flipped = bytes.clone();
flipped[i] = flipped[i].wrapping_add(1);
prop_assert_ne!(body_fingerprint(&bytes), body_fingerprint(&flipped));
}
}
Expected runtime: 1024 cases × two SHA-256-ish hashes; <300ms.
D9.9 Property 7 — reference-data validation never panics
File: crates/craig-reference/tests/properties/parse_never_panics.rs
The 48-test reference suite covers the happy paths for each enum. Property complements: parsing arbitrary garbage strings against any reference enum returns Ok() or Err() — never panics, never out-of-bounds, never overflows.
use proptest::prelude::*;
use std::str::FromStr;
use craig_reference::{
PaymentStatus, PlacementStatus, ContactType, ReportStatus,
AdminUnit, FipsCode, EventType,
};
macro_rules! prop_no_panic {
($t:ty, $name:ident) => {
proptest! {
#![proptest_config(ProptestConfig::with_cases(2048))]
#[test]
fn $name(s in ".{0,64}") {
// Catch panics; success or failure is fine, panic is not.
let _ = std::panic::catch_unwind(|| <$t>::from_str(&s));
}
}
};
}
prop_no_panic!(PaymentStatus, prop_payment_status_no_panic);
prop_no_panic!(PlacementStatus, prop_placement_status_no_panic);
prop_no_panic!(ContactType, prop_contact_type_no_panic);
prop_no_panic!(ReportStatus, prop_report_status_no_panic);
prop_no_panic!(EventType, prop_event_type_no_panic);
prop_no_panic!(FipsCode, prop_fips_code_no_panic);
Fail-first sketch: introduce s.split_at(s.len() + 1) into one parser; proptest shrinks to a single character.
Expected runtime: 6 enums × 2048 cases × cheap parse; <500ms total.
D9.10 Aggregate budget
| Property | Cases | Wall (ms) | Crate |
|---|---|---|---|
D9.3 PaymentStatus round-trip |
512 |
<50 |
craig-financial |
D9.4 Pagination |
256 |
<100 |
craig-api |
D9.5 sanitize_filename |
1024 |
<50 |
craig-store |
D9.6 Content-Disposition |
512 |
<100 |
craig-store |
D9.7 Canonical JSON signing |
256 |
<500 |
craig-signing |
D9.8 Body fingerprint (×2) |
1024 |
<300 |
craig-api |
D9.9 Reference parse no-panic (×6) |
12,288 |
<500 |
craig-reference |
Total |
~16k |
<2s |
— |
The full property suite fits in cargo nextest run --workspace -E 'test(::properties::)' under 2 seconds. Reasonable to run on every developer push under cargo xtask reliability (Step 12) without making the existing pre-push gate slower.
D9.11 Test description (per property)
Each property carries:
-
Strategy — the
impl Strategy<Value = T>quoted verbatim above. -
Invariant — the
prop_assert!body. -
Fail-first sketch — the one-character flip in production code that demonstrates the property catches the bug, with the expected shrunk counter-example documented in a
// FAIL-FIRST:comment retained in the test file. -
Cases-override — explicit when not the default 256.
-
Runtime budget — measured on the contributor’s machine; recorded in a
// PROPTEST_BUDGET_MS:comment so reviewers spot regressions.
D10. cargo-mutants smoke on 5 critical crates (Phase A.10 — Step 11)
Mutation testing answers a question coverage cannot: do the existing tests actually catch wrong code, or do they just execute it? cargo-mutants mutates production code (flips comparisons, deletes statements, replaces Ok)) with Ok(( no-ops, etc.) and re-runs the test suite per mutant. A mutant that survives (test suite still passes) is a hard signal that the test suite has a real semantic gap on that line — exactly the storage-shape-vs-semantics gap the 2026-05-03 review surfaced for the platform invariants.
Per the user-stated constraints, this is a smoke deployment: the tool is available via cargo xtask mutants --smoke, the baseline report is committed, and surviving mutants are catalogued — but mutation pass-rate is not a CI gate. Tightening the budget happens in a future plan.
D10.1 Tool installation + version pin
# .cargo/mutants.toml — workspace-level config (committed)
# cargo-mutants 25.x is the actively-maintained line as of 2026-05.
# Pin via `cargo install cargo-mutants --version "^25.0" --locked`.
timeout = 300 # per-mutant cap (seconds); kills hangs
test_tool = "nextest" # use the same runner as the rest of CRAIG
test_package = ["--locked"]
exclude_globs = [
"**/tests/**",
"**/examples/**",
"**/benches/**",
"**/build.rs",
"**/main.rs", # boot/wiring code; mutants here surface as test-harness gaps, not invariants
]
CONTRIBUTING.adoc gains a one-line "Optional dev tools" section pointing at cargo install cargo-mutants --version "^25.0" --locked. No-tool installation does not break any existing workflow — cargo xtask mutants is opt-in.
D10.2 The five critical crates (rationale)
| Crate | Why it’s critical | Mutant surface |
|---|---|---|
|
Hosts idempotency middleware (platform-stab-2 §D1), problem-details mapper, auth wrappers. Test failures here have correctness blast-radius across all 8 services. |
|
|
Outbox + inbox durability (§D2 + §D3 of platform-stab-2). Concurrency-correctness pivot point. |
|
|
JWT validation — typ/aud/azp/scope/exp/nbf strictness. Direct attacker-facing surface. |
|
|
Upload validation, encryption boundaries, object-store interactions. |
|
|
5 state machines, the platform’s largest cross-product of permitted transitions. |
|
The 5 are deliberately the highest-leverage crates — not the most-touched. Other crates remain runnable via cargo xtask mutants --full but are not in the smoke baseline.
D10.3 Smoke-profile bounds
-
Per crate: cap mutant count at ~50 via
cargo mutants --in-place --no-shuffle --jobs 1 --shard 0/NwithNchosen so each shard runs <5min on the contributor machine. -
Per-mutant timeout: 300s (above).
-
Total wall-time budget: <30 minutes for the full smoke. Documented in CONTRIBUTING as "expect a coffee".
-
Reports written to
target/mutants-<crate>/(cargo-mutants default), then a single summary copied totarget/mutants-summary.jsonby the xtask wrapper (D11.3).
D10.4 Surviving-mutant cataloguing
Each smoke run produces three lists per crate:
| Bucket | Definition | Action |
|---|---|---|
Caught |
Mutant produced a test failure. Healthy. |
None. |
Missed (survived) |
Test suite passed unchanged. Real test gap. |
Catalogue in |
Unviable / timeout |
Mutant didn’t compile, or hit the 300s timeout. |
Skip from the baseline; cargo-mutants reports them but they’re not test-quality signal. |
The baseline file is committed as the current floor. A future plan can tighten the budget by requiring:
-
New CRs may not increase the missed count (delta=0).
-
Each
// reason: tracked-issue-#NNNline must reference a real backlog issue.
The first iteration intentionally does not enforce either — per the user’s "available, not gated" constraint.
D10.5 Test description
Treat the smoke run itself as the verification:
-
cargo install cargo-mutants --version "^25.0" --locked -
cargo xtask mutants --smoke(Step 12) -
Inspect
target/mutants-summary.json; assert at least onecaughtmutant per crate (sanity floor — if a crate has zero caught mutants the smoke is misconfigured). -
Commit
xtask/mutants-baseline.tomlwith the missed list. -
Fail-first sketch: on a clean working tree, manually delete one
prop_assert!from a property in §D9; rerun smoke; observe the corresponding mutant survive where it previously didn’t. Document this as the regression-detection demonstration.
D11. xtask subcommands: reliability + contracts + mutants + coverage-matrix + invariants (Phase A.11 — Step 12)
Five thin orchestration subcommands surface the new tooling. Each follows the existing xtask/src/cmd/*.rs pattern verbatim — clap Args struct, pub fn run(args) → Result<()>, sectioned === … === output, OK / FAIL / SKIP markers per item, no panics.
The shared design point — non-negotiable per user constraint — is that none of these subcommands run as part of the pre-push hook today. .githooks/pre-push is unchanged. Each subcommand prints, on entry, a one-liner identifying itself as opt-in.
D11.1 Subcommand registration
Extend xtask/src/cmd/mod.rs:
pub mod reliability;
pub mod contracts;
pub mod mutants;
pub mod coverage_matrix;
pub mod invariants;
pub mod quality_budgets; // §D12
#[derive(Subcommand)]
pub enum Command {
Dev(dev::DevArgs),
Test(test::TestArgs),
E2e(e2e::E2eArgs),
Validate(validate::ValidateArgs),
CheckDocs(check_docs::CheckDocsArgs),
Perf(perf::PerfArgs),
Security(security::SecurityArgs),
Sbom,
ApiDocs(api_docs::ApiDocsArgs),
Reconcile(reconcile::ReconcileArgs),
/// Run reliability suite (concurrency / recovery / fault / invariants / properties)
Reliability(reliability::ReliabilityArgs),
/// Run typed-client contract tests
Contracts(contracts::ContractsArgs),
/// Run cargo-mutants on the 5 critical crates (smoke) or workspace (--full)
Mutants(mutants::MutantsArgs),
/// Generate the route × test-axis coverage matrix
CoverageMatrix(coverage_matrix::CoverageMatrixArgs),
/// Run platform-invariant tests (atomic-claim, FUSL, processed-state, ...)
Invariants(invariants::InvariantsArgs),
/// Report or enforce code-quality budgets (§D12)
QualityBudgets(quality_budgets::QualityBudgetsArgs),
}
xtask/src/main.rs gets six new match arms wired identically to the existing pattern.
D11.2 reliability subcommand
File: xtask/src/cmd/reliability.rs
Glob nextest filter expression covers all four reliability axes from the testing.md taxonomy added in Step 8 of this plan, plus the property suite from §D9:
use anyhow::{Context, Result, bail};
use clap::Args;
use std::process::Command;
const ENTRY_BANNER: &str =
"Reliability suite (opt-in; not part of pre-push gate). \
See plans/test-framework-hardening.adoc §D11.";
const NEXTEST_FILTER: &str = "test(::concurrency::) \
| test(::recovery::) \
| test(::fault::) \
| test(::invariants::) \
| test(::properties::)";
#[derive(Args)]
pub struct ReliabilityArgs {
/// Optional nextest filter to AND with the default reliability filter.
#[arg(long)]
pub filter: Option<String>,
}
pub fn run(args: ReliabilityArgs) -> Result<()> {
println!("=== {ENTRY_BANNER} ===");
let mut filter = NEXTEST_FILTER.to_string();
if let Some(extra) = args.filter {
filter = format!("({filter}) & ({extra})");
}
let status = Command::new("cargo")
.args(["nextest", "run", "--workspace", "--locked", "-E", &filter])
.status()
.context("spawn cargo nextest")?;
if !status.success() { bail!("reliability suite failed with {status}"); }
println!("=== Reliability OK ===");
Ok(())
}
D11.3 mutants subcommand
File: xtask/src/cmd/mutants.rs
use anyhow::{Context, Result, bail};
use clap::Args;
use std::process::Command;
const SMOKE_CRATES: &[&str] = &[
"crates/craig-api",
"crates/craig-mq",
"crates/craig-auth",
"crates/craig-store",
"services/craig-cases",
];
#[derive(Args)]
pub struct MutantsArgs {
/// Run full workspace mutation testing (no per-crate filter)
#[arg(long, conflicts_with = "smoke")]
pub full: bool,
/// Run the smoke profile against the 5 critical crates (default)
#[arg(long)]
pub smoke: bool,
}
pub fn run(args: MutantsArgs) -> Result<()> {
println!("=== cargo-mutants (opt-in; not part of pre-push gate). \
Surviving mutants catalogued in xtask/mutants-baseline.toml. ===");
if !is_installed() {
bail!("cargo-mutants is not installed. Install with:\n\
cargo install cargo-mutants --version \"^25.0\" --locked");
}
let crates: Vec<&str> = if args.full { vec![] } else { SMOKE_CRATES.into() };
if crates.is_empty() {
run_one(None)?;
} else {
for c in &crates { run_one(Some(c))?; }
}
summarize()?;
println!("=== Mutants OK (see target/mutants-summary.json) ===");
Ok(())
}
fn is_installed() -> bool {
Command::new("cargo").args(["mutants", "--version"]).output()
.map(|o| o.status.success()).unwrap_or(false)
}
fn run_one(scope: Option<&str>) -> Result<()> { /* ... cargo mutants --in-place --jobs 1 [--in-dir scope] ... */ Ok(()) }
fn summarize() -> Result<()> { /* aggregate per-crate target/mutants-*/outcomes.json into target/mutants-summary.json */ Ok(()) }
D11.4 contracts subcommand
File: xtask/src/cmd/contracts.rs
The typed-client tests (Phase A.8 in the existing draft) live under tests/typed/<service>/ per the testing.md convention. The subcommand pins them as a callable axis.
use anyhow::{Context, Result, bail};
use clap::Args;
use std::process::Command;
#[derive(Args)]
pub struct ContractsArgs {
/// Restrict to a specific service (cases, exchange, ...)
#[arg(long)]
pub service: Option<String>,
}
pub fn run(args: ContractsArgs) -> Result<()> {
println!("=== Typed-client contract tests (opt-in; not part of pre-push gate). ===");
let filter = match args.service.as_deref() {
Some(s) => format!("test(::typed::{s}::)"),
None => "test(::typed::)".to_string(),
};
let status = Command::new("cargo")
.args(["nextest", "run", "--workspace", "--locked", "-E", &filter])
.status().context("spawn cargo nextest")?;
if !status.success() { bail!("contracts suite failed"); }
Ok(())
}
D11.5 coverage-matrix subcommand
File: xtask/src/cmd/coverage_matrix.rs — full implementation belongs to §D13 (Step 14). The subcommand here is a one-line shell-out.
use anyhow::Result;
use clap::Args;
#[derive(Args)]
pub struct CoverageMatrixArgs { /* see §D13 */ }
pub fn run(args: CoverageMatrixArgs) -> Result<()> {
println!("=== Coverage matrix (opt-in; report-only, no enforcement gate). ===");
crate::cmd::coverage_matrix::generate(args) // module-private fn from §D13
}
D11.6 invariants subcommand
File: xtask/src/cmd/invariants.rs — wraps a nextest filter targeting the platform-invariant tests added across this plan and platform-stab-2.
pub fn run(_: InvariantsArgs) -> Result<()> {
println!("=== Invariant tests (opt-in; not part of pre-push gate). ===");
let status = Command::new("cargo")
.args(["nextest", "run", "--workspace", "--locked",
"-E", "test(::invariants::)"])
.status()?;
if !status.success() { bail!("invariant suite failed"); }
Ok(())
}
D11.7 Documentation surface
-
CONTRIBUTING.adoc— new "Optional reliability tooling" section enumerating the five subcommands as available tools (not required for MR landing). -
.claude/docs/testing.md— Step 8’s edits already document the file-naming convention; this step appends a "Local subcommands" subsection cross-referencing the xtask entry-points. -
README.adoc— one-paragraph note in the Development section.
D11.8 Test description
Per subcommand, a tests/ integration test exercises the subcommand once against a known-good fixture and asserts:
-
Exit 0 on a passing project.
-
Exit non-zero on an artificially-failed project (delete one
prop_assert!, runreliability, expect failure). -
Output prints the entry banner verbatim — guards against the "available, not gated" message ever drifting.
D12. Code-quality budget warnings (Phase A.12 — Step 13)
User-stated budgets ship as a report, never as a pre-push gate. The mechanism is opt-in: contributors run cargo xtask quality-budgets --report and see a table; reviewers read the same table on each MR by manually invoking it. The --fail-on-regression flag exists for future CI activation but is not wired into pre-push or CI today.
D12.1 Budget catalogue
| ID | Budget | Initial threshold | Scope |
|---|---|---|---|
|
Max LOC per route module |
500 |
|
|
Max LOC per function |
100 |
All |
|
|
0 in |
All |
|
|
20 across workspace |
All |
|
|
50 |
|
|
Duplicate dependency versions |
0 (zero new dupes vs baseline) |
|
|
Untyped methods on core test clients |
0 new (paired with Phase A.8) |
|
Each budget has a per-path exception list (xtask/quality-budgets.toml); the budget compares the count outside the exception list against the threshold. Exception entries require a reason = "…" per the workspace lint convention.
D12.2 Config file shape
# xtask/quality-budgets.toml
# Code-quality budgets — report-only as of test-framework-hardening Step 13.
# `--fail-on-regression` reads thresholds; `--report` emits a table.
[B1.route_module_loc]
threshold = 500
glob = ["services/*/src/api/*.rs", "services/craig-web/src/routes/*.rs"]
exceptions = [
# Examples — empty initially; new exceptions require a reason field
# { path = "services/craig-cases/src/api/reports.rs", reason = "tracked: split-pending #282" },
]
[B2.function_loc]
threshold = 100
glob = ["**/*.rs"]
exclude = ["**/tests/**"]
[B3.serde_json_value]
threshold_src = 0
threshold_tests = 5
src_glob = ["**/src/**/*.rs"]
src_exclude = ["**/api/dto.rs", "**/api/dto/**/*.rs"]
tests_glob = ["**/tests/**/*.rs"]
[B4.allow_attributes]
threshold = 20
[B5.unwrap_or_default]
threshold = 50
glob = ["**/src/**/*.rs"]
[B6.duplicate_deps]
threshold = 0
# Baseline captured in xtask/quality-budgets.lock; B6 reports delta vs lock.
[B7.untyped_test_client_methods]
threshold = 0
# Counts methods returning `serde_json::Value` from craig-test-lib clients.
glob = ["crates/craig-test-lib/src/clients/*.rs"]
D12.3 Lockfile / baseline
# xtask/quality-budgets.lock — generated by `cargo xtask quality-budgets --report --write-lock`
# Tracks the COUNT (not threshold) at the time of the lock. Regression check
# under `--fail-on-regression` reads BOTH the threshold and the lock and fails
# if `count > max(threshold, lock)`.
[counts]
B1_route_module_loc = 0 # LOC over threshold; populated on first run
B2_function_loc = 0
B3_serde_json_value_src = 0
B3_serde_json_value_test = 0
B4_allow_attributes = 0
B5_unwrap_or_default = 0
B6_duplicate_deps = 0
B7_untyped_test_methods = 0
generated_at = "2026-05-03T00:00:00Z"
generated_from_sha = "<git-rev>"
The lock is committed. New CRs that increase a count must justify in CR description (reviewer discipline; the lockfile diff is the obvious signal).
D12.4 Subcommand shape
File: xtask/src/cmd/quality_budgets.rs
use anyhow::Result;
use clap::Args;
#[derive(Args)]
pub struct QualityBudgetsArgs {
/// Print the budget table (default mode)
#[arg(long, conflicts_with = "fail_on_regression")]
pub report: bool,
/// Exit non-zero if any budget exceeded (NOT run in pre-push today)
#[arg(long)]
pub fail_on_regression: bool,
/// Update xtask/quality-budgets.lock with current counts
#[arg(long, requires = "report")]
pub write_lock: bool,
}
#[derive(Debug)]
struct BudgetReport {
id: &'static str,
name: &'static str,
threshold: usize,
locked: usize,
actual: usize,
}
pub fn run(args: QualityBudgetsArgs) -> Result<()> {
println!("=== Quality budgets (report-only; not a pre-push gate). ===");
let cfg = read_config()?;
let lock = read_lock()?;
let reports = vec![
run_b1_route_module_loc(&cfg, &lock)?,
run_b2_function_loc(&cfg, &lock)?,
run_b3_serde_json_value(&cfg, &lock)?,
run_b4_allow_attributes(&cfg, &lock)?,
run_b5_unwrap_or_default(&cfg, &lock)?,
run_b6_duplicate_deps(&cfg, &lock)?,
run_b7_untyped_test_methods(&cfg, &lock)?,
];
print_table(&reports);
if args.write_lock { write_lock(&reports)?; }
if args.fail_on_regression {
let regressed: Vec<_> = reports.iter()
.filter(|r| r.actual > r.threshold.max(r.locked))
.collect();
if !regressed.is_empty() {
anyhow::bail!("{} budget(s) regressed", regressed.len());
}
}
Ok(())
}
fn print_table(rows: &[BudgetReport]) { /* tabled — same crate already in workspace */ }
fn read_config() -> Result<Config> { /* parse xtask/quality-budgets.toml */ Ok(Default::default()) }
fn read_lock() -> Result<Lock> { /* parse xtask/quality-budgets.lock */ Ok(Default::default()) }
fn write_lock(_reports: &[BudgetReport]) -> Result<()> { Ok(()) }
fn run_b1_route_module_loc(_: &Config, _: &Lock) -> Result<BudgetReport> { todo!() }
// ... B2..B7
The implementations are mostly directory walks (walkdir = "2" already in workspace via several deps) plus regex counts (regex = "1"). B6 shells out to cargo tree -d. B7 uses syn to parse client files and inspect return types — already a workspace dep via cargo-deny derivations.
D12.5 Sample output (target shape)
=== Quality budgets (report-only; not a pre-push gate). ===
+-----+----------------------------------+--------+--------+--------+--------+
| ID | Name | Thresh | Locked | Actual | Status |
+-----+----------------------------------+--------+--------+--------+--------+
| B1 | Route module LOC | 500 | 893 | 893 | LOCKED |
| B2 | Function LOC | 100 | 142 | 142 | LOCKED |
| B3a | serde_json::Value in src | 0 | 12 | 12 | LOCKED |
| B3b | serde_json::Value in tests | 5 | 187 | 187 | LOCKED |
| B4 | #[allow(...)] count | 20 | 34 | 34 | LOCKED |
| B5 | unwrap_or_default in production | 50 | 101 | 101 | LOCKED |
| B6 | Duplicate dependency versions | 0 | 2 | 2 | LOCKED |
| B7 | Untyped test-client methods | 0 | 56 | 56 | LOCKED |
+-----+----------------------------------+--------+--------+--------+--------+
Status: LOCKED = at-or-below current lockfile baseline.
OVER = above lockfile baseline; explain in CR description.
D12.6 Test description
-
Unit: each
run_b<N>_*function has a fixture-directory test asserting it counts deterministically. -
Snapshot: integration test runs the subcommand against a fixed fixture, snapshots the rendered table; visible diff on regression.
-
Lockfile-init: first
--write-lockproduces a lockfile that round-trips parse → render.
D13. Coverage Matrix tooling — xtask coverage-matrix (Phase A.13 — Step 14)
Coverage tools historically conflate two different signals: line coverage (which lines executed during tests) and risk coverage (which behaviors of the system were tested). Line coverage is what cargo-tarpaulin and cargo-llvm-cov produce. Risk coverage — the matrix of "for each route, which adversarial axes have we tested?" — has no off-the-shelf tool. This subcommand builds it.
Per the user’s "report-only" directive: the output is a Markdown table at target/coverage-matrix.md, plus a single percentage at the end. No threshold, no gate.
D13.1 Inputs
| Source | Used for | Pull mechanism |
|---|---|---|
OpenAPI specs (each service exposes |
Route inventory: |
|
Test source files |
Test inventory by axis |
|
Test names within those files |
Axis tagging via |
Lightweight regex on |
D13.2 Six axes
| Axis | What it asserts | How counted |
|---|---|---|
|
Documented success path |
Test name ends |
|
Documented failure path (4xx) |
Test name ends |
|
Adversarial input (attack-shaped) |
Test name ends |
|
Concurrent same-key behavior |
Test file matches |
|
Crash-and-restart / redelivery semantics |
Test file matches |
|
Dependency error handling |
Test file matches |
For each (service, method, path, axis) cell, the cell is ✓ if at least one test references the route — reference meaning either the URL appears as a string literal in the test source, or the typed-client method name corresponding to the route is invoked. The typed-client mapping comes from a single xtask/route-to-client-method.toml table maintained alongside the typed clients (Phase A.8 deliverable; this plan adds an empty starter file with the manually-mapped seed entries).
D13.3 Output format
Markdown table at target/coverage-matrix.md:
# CRAIG Coverage Matrix
Generated: 2026-05-03T12:00:00Z
Route: 247 endpoints across 8 services
Risk axes: happy / sad / evil / concurrency / replay / fault
| Service | Method | Path | happy | sad | evil | conc | replay | fault |
|---------|--------|------|-------|-----|------|------|--------|-------|
| craig-cases | GET | /v1/cases | ✓ | | | | | |
| craig-cases | POST | /v1/cases | ✓ | ✓ | | ✓ | | |
| craig-cases | GET | /v1/cases/{id} | ✓ | | | | | |
| ...
**Risk-coverage score: 412 / 1482 cells (27.8%)**
The percentage is total_cells_filled / total_cells. total_cells = endpoints × 6. Score is reported but not gated.
D13.4 Subcommand shape
File: xtask/src/cmd/coverage_matrix.rs
use anyhow::{Context, Result};
use clap::Args;
use std::collections::BTreeMap;
use std::path::PathBuf;
#[derive(Args)]
pub struct CoverageMatrixArgs {
/// Output path (default: target/coverage-matrix.md)
#[arg(long, default_value = "target/coverage-matrix.md")]
pub output: PathBuf,
/// Base URL for OpenAPI scraping (default: http://localhost)
#[arg(long, default_value = "http://localhost")]
pub base_url: String,
/// Skip OpenAPI fetch; use cached `target/openapi-cache.json`
#[arg(long)]
pub use_cache: bool,
}
#[derive(Debug, Clone)]
struct Route { service: String, method: String, path: String }
#[derive(Debug)]
struct Cell { axes: [bool; 6] } // happy, sad, evil, conc, replay, fault
pub fn generate(args: CoverageMatrixArgs) -> Result<()> {
println!("=== Coverage matrix (report-only). ===");
let routes = if args.use_cache { read_cache()? }
else { fetch_openapi(&args.base_url)? };
let tests = scan_tests()?;
let matrix = build_matrix(&routes, &tests);
write_markdown(&args.output, &routes, &matrix)?;
let (filled, total) = score(&matrix);
println!("=== Risk-coverage: {filled}/{total} ({:.1}%) — see {}",
100.0 * filled as f64 / total as f64,
args.output.display());
Ok(())
}
fn fetch_openapi(_base: &str) -> Result<Vec<Route>> { /* see api_docs.rs:46-72 */ Ok(vec![]) }
fn scan_tests() -> Result<Vec<TestRef>> { /* walkdir + regex */ Ok(vec![]) }
fn build_matrix(_: &[Route], _: &[TestRef]) -> BTreeMap<(String,String,String), Cell> { Default::default() }
fn write_markdown(_: &std::path::Path, _: &[Route], _: &BTreeMap<(String,String,String), Cell>) -> Result<()> { Ok(()) }
fn score(m: &BTreeMap<(String,String,String), Cell>) -> (usize, usize) {
let total = m.len() * 6;
let filled: usize = m.values().map(|c| c.axes.iter().filter(|x| **x).count()).sum();
(filled, total)
}
#[derive(Debug)] struct TestRef { /* axis, route_match */ }
fn read_cache() -> Result<Vec<Route>> { Ok(vec![]) }
D13.5 Decision: regex tagging, not proc-macro
The user explicitly directed not to introduce a #[test_category(…)] proc-macro. Rationale:
| Approach | Cost | Benefit |
|---|---|---|
Proc-macro |
New crate ( |
Compile-time guarantee that an axis tag is one of the six known. |
Regex over name suffix ( |
Trivial; lives entirely inside xtask. |
No compile-time guarantee — typo |
The footer-block mitigation is enough at the current scale (~1259 Rust tests). Revisit if untagged-test count climbs over a threshold.
D13.6 Initial baseline commit
The Step 14 MR commits both the subcommand and a first-run target/coverage-matrix.md snapshot under docs/modules/ROOT/pages/operations/coverage-matrix.adoc (the markdown table is wrapped in [source,markdown]). Subsequent CRs do not update this file; it’s a one-shot baseline. A future plan revisits "live coverage matrix in CI" once the foundation has bedded in.
D13.7 --diff main flag for scorecard delta
Reviewer-discipline lift: the scorecard at docs/modules/ROOT/pages/test-coverage-scorecard.adoc lives in the repo and changes per MR. Without tooling, reviewers must eyeball cell flips. Add --diff <REF> (default main) to the subcommand:
#[derive(Args)]
pub struct CoverageMatrixArgs {
/// Compare against a base ref (default: main). Prints filled/empty cell deltas.
#[arg(long)]
pub diff: Option<String>,
// ...other args from D13.4
}
Behavior:
-
Run the matrix against the current tree →
current_matrix. -
git stash(preserve uncommitted) →git checkout <REF>→ re-run subcommand →base_matrix→git checkout -→git stash pop. -
Print:
=== Coverage-matrix delta vs main === + POST /v1/cases :: evil (filled by tests/api/cases_evil.rs::create_case_oversize_name_evil) + GET /v1/cases :: replay (filled by tests/recovery/cases_list_replay.rs) - DELETE /v1/cases/:id :: happy (regression: previously filled, no longer matched) === Risk-coverage delta: +2 cells, -1 cell, net +1 === -
Exit 0 always — report-only, even on regressions, per the no-CI-pressure directive.
This is the lightest possible "pre-commit hint": developers run cargo xtask coverage-matrix --diff main before pushing; reviewers can run it during review. No pre-commit hook, no CI gate.
D13.8 Test description
-
Fixture: a tiny mock OpenAPI spec (3 routes) + 3 mock test files, one tagged each axis. Subcommand should produce a 3-row matrix with the expected cells filled.
-
Untagged warning: extra test file with no recognized suffix; subcommand prints it under "Untagged tests" in the footer.
-
Score arithmetic: 3 routes × 6 axes = 18 cells; 5 cells filled → score string
5/18 (27.8%).
D14. DB Constraint Evil Tests — hand-curated harness (Phase A.14 — Step 15)
The user’s framing is correct: auto-deriving negative tests from migrations is brittle. CHECK constraints are arbitrary boolean expressions; UNIQUE constraints are tuples; FK constraints have ON DELETE semantics that vary; NOT NULL is the only constraint where the negative space is trivial. Even when the SQL parser succeeds, the interesting assertion is at the API boundary ("does the handler return 409 Conflict with the right RFC 9457 type URL?") rather than at the DB layer ("does the INSERT raise SQLSTATE 23514?"). The DB-layer signal is necessary but not sufficient.
The deliverable is therefore a hand-curated harness: a small helper module + a per-service convention for placing constraint tests + a seed of 2 representative constraints per service (16 tests total, 8 services). The remainder is tracked as a backlog issue and grown alongside future migrations.
D14.1 Helper module
File: crates/craig-test-lib/src/constraints.rs (new; exported from lib.rs)
//! Hand-curated DB constraint test harness.
//!
//! For each constraint we want to verify, write one test in
//! `services/<svc>/tests/constraints/<constraint>.rs` that uses
//! `assert_constraint_violation_returns(...)` to drive an API request
//! known to violate the constraint and assert the resulting 4xx +
//! RFC 9457 problem-type URL.
use crate::client::TypedClient;
use reqwest::StatusCode;
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum ExpectedRejection {
BadRequest400,
UnprocessableEntity422,
Conflict409,
NotFound404,
}
impl From<ExpectedRejection> for StatusCode {
fn from(e: ExpectedRejection) -> Self {
match e {
ExpectedRejection::BadRequest400 => StatusCode::BAD_REQUEST,
ExpectedRejection::UnprocessableEntity422=> StatusCode::UNPROCESSABLE_ENTITY,
ExpectedRejection::Conflict409 => StatusCode::CONFLICT,
ExpectedRejection::NotFound404 => StatusCode::NOT_FOUND,
}
}
}
/// Drive `request` against `client`; assert the response carries
/// `expected` status and a problem+json body whose `type` field equals
/// `problem_type_url`.
///
/// `request` is a closure so the harness can rerun it (state-machine
/// constraint tests sometimes need a setup step before triggering).
pub async fn assert_constraint_violation_returns<R, Fut>(
client: &TypedClient,
request: R,
expected: ExpectedRejection,
problem_type_url: &str,
) where
R: FnOnce(TypedClient) -> Fut,
Fut: std::future::Future<Output = reqwest::Response>,
{
let resp = request(client.clone()).await;
let status = resp.status();
let body: serde_json::Value = resp.json().await.expect("problem+json");
assert_eq!(status, StatusCode::from(expected),
"expected {expected:?} got {status}; body={body}");
assert_eq!(body.get("type").and_then(|v| v.as_str()), Some(problem_type_url),
"wrong problem type; full body: {body}");
}
D14.2 Per-service file convention
| Service | Test directory | Module index |
|---|---|---|
|
|
|
|
|
same |
|
|
same |
|
|
same |
|
|
same |
|
|
same |
|
|
same |
|
|
same |
The pattern matches the existing tests/api.rs + tests/api/*.rs taxonomy from testing.md.
D14.3 Constraint inventory (CRAIG-wide)
A first pass via Grep "(CHECK|UNIQUE|REFERENCES|NOT NULL)" services//migrations/.sql yields the constraints worth exercising. Indicative, not exhaustive — full enumeration is a backlog issue tracked under epic &22:
| Service | Table.constraint | Kind | Negative request |
|---|---|---|---|
craig-mq (shared) |
|
CHECK |
not API-driven; covered by §D2 outbox tests |
craig-mq (shared) |
|
CHECK (after platform-stab-2 §D1) |
POST with hand-crafted state isn’t reachable through the handler — this is a DB-layer integrity guard; cover via a focused |
craig-cases |
|
CHECK (after platform-stab-2 §D4) |
same as above |
craig-financial |
|
CHECK |
|
craig-financial |
|
FK (cross-service; reconcile-walked, not DB-enforced) |
|
craig-placement |
|
FK |
|
craig-placement |
|
CHECK |
|
craig-exchange |
|
FK ON DELETE RESTRICT |
|
craig-security |
|
UNIQUE |
second |
craig-rules |
|
UNIQUE |
second |
craig-reporting |
|
CHECK |
not API-reachable (handler validates first) |
craig-intake |
|
FK |
submission with rotated/deleted key → 401 problem+json type |
Step 1 of this plan files the full enumeration as a tracking issue; the seed below picks 2 representative API-reachable constraints per service.
D14.4 Seed — 16 tests, 2 per service
| File | Constraint | ExpectedRejection | Problem-type URL |
|---|---|---|---|
|
regex CHECK on |
|
|
|
CHECK on |
|
|
|
CHECK on |
|
|
|
RESTRICT FK |
|
|
|
RESTRICT FK |
|
|
|
CHECK |
|
|
|
CHECK |
|
|
|
UNIQUE |
|
|
|
UNIQUE |
|
|
|
CHECK |
|
|
|
UNIQUE |
|
|
|
CHECK |
|
|
|
UNIQUE |
|
|
|
CHECK regex |
|
|
|
FK with |
|
|
|
settings-validation (paired §D8.2) |
|
D14.5 Sample test body
// services/craig-financial/tests/constraints/payment_amount_positive.rs
use craig_test_lib::{
constraints::{assert_constraint_violation_returns, ExpectedRejection},
TestHarness,
};
use craig_test_lib::clients::caseworker_financial_client;
#[tokio::test]
async fn payment_amount_must_be_positive_evil() {
let h = TestHarness::new().await;
if !h.devstack_available().await { return; }
let client = caseworker_financial_client(&h);
assert_constraint_violation_returns(
&client,
|c| async move {
c.create_payment_raw(serde_json::json!({
"case_id": h.fixtures.case_id,
"placement_id": h.fixtures.placement_id,
"amount": -1,
"kind": "ive_maintenance",
})).await
},
ExpectedRejection::BadRequest400,
"https://docs.craig/problems/invalid-amount",
).await;
}
D14.6 Test description
-
Initial seed verification: all 16 tests pass against a fresh devstack on first run.
-
Regression marker: the test names include
_evilso they’re picked up by §D13’s coverage matrix as theevilcell for the relevant routes. -
Pushback assurance: no migration parsing, no auto-generation. The harness is 50 lines of Rust + 16 hand-written tests; debugging a failure points directly at the API boundary, not at a generator.
D15. Central Evil Input Corpus consolidation (Phase A.15 — Step 16)
Steps 6 (D5 — JWT mutation library) and 7 (D6 — magic-byte + filename-attack fixtures) ship as standalone fixture modules first, because the platform-stab-2 sibling fixes need them inline. D15 consolidates those two seed corpora into a single craig_test_lib::evil module and extends the corpus across nine new attack categories so every parametric API test, fuzz harness, contract-test, and security regression draws from one canonical iterator. Two motivations, in order:
-
Single source of truth for "what does a hostile input look like". Today the magic-byte fixtures, the JWT mutations, and the ad-hoc 422 tests scattered across 8 services each carry their own corpus. When a reviewer adds "we should also reject CRLF in filenames", that addition lands in one fixture file and is silently absent from the other 7. A consolidated corpus means one
EvilCaseadded once is exercised against every public-ingestion endpoint that opts into the unified contract test (§D15.6). -
Discovery via iteration, not enumeration. The current pattern asks the test author to list which inputs are evil. The corpus inverts the question: for every endpoint, the test loops
all_evil_cases()and asserts each is rejected with a clean 4xx. New evil categories propagate to all endpoints automatically.
D15.1 Consolidation: from two seeds to one module
The Step-6 JWT mutation library lives at crates/craig-test-lib/src/jwt_mutations.rs (filed by the prior agent). The Step-7 upload fixtures live at crates/craig-test-lib/src/upload_fixtures.rs. Both are kept callable as-is (their callers in platform-stab-2 §D5 / §D8.1 don’t move). D15 re-homes them under crates/craig-test-lib/src/evil/:
crates/craig-test-lib/src/evil/
├── mod.rs — re-exports + EvilCase / EvilCategory / EvilPayload / ExpectedRejection
├── jwt.rs — wraps Step-6 jwt_mutations; emits EvilCase::Jwt(JwtMutation)
├── upload.rs — wraps Step-7 upload_fixtures; emits EvilCase::Upload(UploadFixture)
├── string.rs — overlong / invalid-UTF-8 / NUL / CRLF / RTLO / ZWSP (NEW)
├── uuid.rs — wrong-version / wrong-length / all-zeros / non-hex (NEW)
├── json.rs — depth bomb / int overflow / extra fields / type swap (NEW)
├── path.rs — `../`, double-encoded, NUL-truncated, UNC, ADS (NEW)
├── html.rs — script tags, javascript: URLs, polyglot SVG/HTML (NEW)
├── unicode.rs — NFC vs NFD, homoglyph, BiDi override, BOM (NEW)
├── enum_value.rs — unknown / case-mismatched / boundary-adjacent (NEW)
├── date.rs — leap-second, year-9999, negative, fractional overflow (NEW)
├── multipart.rs — missing boundary, conflicting Content-Length, … (NEW)
└── signature.rs — wrong-length, wrong-key, replay, alg-confusion (NEW)
jwt.rs and upload.rs are thin wrappers — they import the existing seed modules and emit EvilCase enum variants. The seed modules continue to expose their direct iterator API for callers that want raw JwtMutation/UploadFixture types (e.g. the platform-stab-2 §D5 JWT-strictness test does not need the unified EvilCase envelope; it iterates jwt_mutations::all() directly).
D15.2 Public types
// crates/craig-test-lib/src/evil/mod.rs
#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)]
pub enum EvilCategory {
Jwt, Upload, String, Uuid, Json, Path, Html, Unicode,
EnumValue, Date, Multipart, Signature,
}
#[derive(Clone, Debug)]
pub struct EvilCase {
/// Stable, human-meaningful identifier. Used as the test-block name when
/// fed into `parametric_evil_test!`. Examples:
/// "string.overlong_1mb"
/// "json.depth_bomb_4096"
/// "path.dot_dot_double_url_encoded"
pub name: &'static str,
pub category: EvilCategory,
pub payload: EvilPayload,
pub expected: ExpectedRejection,
}
#[derive(Clone, Debug)]
pub enum EvilPayload {
/// String body — fed as the field value in a request DTO.
Str(String),
/// Raw bytes — fed as a multipart body or as a typed Bytes argument.
Bytes(Vec<u8>),
/// JSON value — fed as a request body, typically with a wrapper DTO.
Json(serde_json::Value),
/// Pre-built JWT (header.payload.signature) — fed via Authorization header.
BearerToken(String),
/// Multipart fragment — already encoded; injected as the request body.
MultipartRaw { content_type: String, body: Vec<u8> },
}
#[derive(Clone, Copy, Debug)]
pub enum ExpectedRejection {
/// Any 4xx status acceptable — used for "reject with a clean error".
AnyClientError,
/// Specific 4xx status — used when the contract pins it (e.g. JWT → 401).
Status(u16),
/// 4xx with a specific RFC 9457 type pointer.
ProblemType(&'static str),
}
pub fn all_evil_cases() -> impl Iterator<Item = EvilCase> { /* chain all categories */ }
pub fn evil_cases_for(category: EvilCategory) -> impl Iterator<Item = EvilCase> {
all_evil_cases().filter(move |c| c.category == category)
}
The choice of &'static str for name is deliberate: every case must be a compile-time literal so test names are stable across runs (snapshot-testing parity, JUnit history matching).
D15.3 Corpus content per new category
| Category | Cases (default count) | Coverage rationale |
|---|---|---|
|
1 KB / 1 MB / 16 MB overlong; invalid UTF-8 (lone surrogate, truncated continuation byte); NUL injection mid-string; CRLF injection in filename / display-name fields; leading + trailing whitespace; RTLO |
Hits length bounds, encoder strictness, log-injection, RTLO header spoofing. |
|
Wrong version (v4 token where v7 expected); 31 hex chars; 33 hex chars; all-zeros ( |
Hits UUID parser strictness + idempotency-key strictness. |
|
Depth bomb at 1024, 4096, 16384 nesting; ref cycle attempt ( |
Hits serde_json depth limits + DTO |
|
|
Hits path-sanitization in upload + download handlers. |
|
|
Hits any field that’s later rendered without escape. |
|
Same string in NFC vs NFD (deduplication test); homoglyph — Cyrillic |
Hits normalization-mismatch person-match collisions + display-name strictness. |
|
Unknown variant name; case-mismatched ( |
Hits string-typed enum boundaries before the §D10.7 type-system fix lands; protects post-fix. |
|
Leap second |
Hits chrono/jiff parser quirks + audit-log timestamp invariants. |
|
Missing |
Hits multer / axum-extra multipart parsing. |
|
Wrong-length JWS (truncated); signed with wrong key; replay (timestamp from 6 hours ago); alg-confusion (HS256 token forwarded to RS256 verifier); |
Hits craig-signing canonical-JWS path + JWT validator strictness. |
Default total corpus size: ~120 cases. Cheap (no I/O, no async); a single endpoint loop costs <2s.
D15.4 Helper macro: parametric_evil_test!
Boilerplate-free adoption from a service test file:
// services/craig-cases/tests/api/evil_corpus.rs
use craig_test_lib::evil::{parametric_evil_test, EvilCategory};
parametric_evil_test! {
name: cases_post_rejects_evil_inputs,
endpoint: |client, case| async move {
client.post_case_with_evil(case).await
},
skip_categories: [EvilCategory::Multipart, EvilCategory::Signature],
}
The macro expands to one #[tokio::test] per EvilCase, naming each cases_post_rejects_evil_inputs<case.name>. Names are stable, individually addressable in cargo nextest run -E 'test(/path\.dot_dot/)'. skip_categories is the escape hatch for endpoints where a category is structurally inapplicable (POST /v1/cases doesn’t accept multipart, so skip it rather than expecting a useless reject).
D15.5 Reusability test: the global ingestion-rejection contract
A single integration test in crates/craig-test-lib/tests/evil_corpus_global_contract.rs opts the most-public endpoint of every API service into the contract:
#[tokio::test]
async fn every_evil_case_is_rejected_at_public_intake() {
if !devstack_available().await { return; }
let h = TestHarness::new().await;
let intake = h.public_intake_client();
for case in craig_test_lib::evil::all_evil_cases() {
let res = intake.submit_with_evil(&case).await;
match (res.status().as_u16(), case.expected) {
(s, ExpectedRejection::AnyClientError) if (400..500).contains(&s) => {}
(s, ExpectedRejection::Status(want)) if s == want => {}
// ProblemType: parse RFC 9457 body, compare type field
(s, ExpectedRejection::ProblemType(t)) if (400..500).contains(&s) => {
assert_problem_type(res, t).await;
}
(s, exp) => panic!(
"evil case {:?} (cat={:?}) returned status {} (expected {:?}) — \
server accepted hostile input or 5xx-ed instead of clean reject",
case.name, case.category, s, exp,
),
}
}
}
The contract: for every EvilCase, the public ingestion endpoint returns a 4xx with the expected shape — no 500s, no 200s, no silent acceptance. A 500 is a test failure even if the input was hostile, because hostile input is a normal-business-hours occurrence and the service must classify it cleanly. This is the single test that establishes the boundary contract globally; per-endpoint parametric_evil_test! invocations harden each service-specific surface.
D15.6 Adoption: where the corpus is wired in
| Site | What changes |
|---|---|
|
New file: |
|
New file: same shape against |
|
New file each: same pattern. Aggregate ~50 new test blocks per service. |
|
Existing file extended — replaces the ad-hoc seed corpus with the unified module. Public-intake is the most hostile surface. |
|
New file: §D15.5 global contract. |
|
New file: feeds the corpus into the BFF’s typed clients to assert the BFF doesn’t propagate hostile values upstream. (Defends platform-stab-2 §D7.) |
D15.7 Test coverage
Self-test for the corpus: crates/craig-test-lib/tests/evil_corpus_self.rs asserts (a) every category contributes at least one case; (b) every case has a non-empty name; (c) names are unique across all categories; (d) the iterator is deterministic across runs (sort + diff against a snapshot file).
Cross-reference: the two-pass platform-invariant audit's lesson reified as data — the corpus is the structural-vs-semantic boundary check.
D16. Restart/Replay light helpers (Phase A.16 — Step 17)
A heavyweight restart-replay harness was considered (full process spawn, signal-based crash, supervisor reattach) and rejected: it conflates testing with init systems, introduces flakiness from PID racing, and the only behavior we need is "drive the recovery code path of a service that crashed at a chosen semantic point". A small set of helpers, paired with the existing test-pattern docs, gives 100% of the value with <200 LOC and zero process-management state.
D16.1 Module shape
crates/craig-test-lib/src/restart.rs (new):
/// A semantic point at which the service simulates a crash. The injector at
/// this point hard-errors with a panic-equivalent (returns Err that the
/// surrounding code unwraps in test mode), forcing the in-flight tx to abort
/// and the calling fn to return.
#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)]
pub enum CrashPoint {
/// Inside an attachment upload, after Phase 1 commit and BEFORE Phase 2 put.
AttachmentMidUpload,
/// Inside an attachment upload, after Phase 2 put and BEFORE status promote.
AttachmentPostPut,
/// Inside outbox drain, after publish() returns Ok and BEFORE post-publish UPDATE.
OutboxPostPublish,
/// Inside inbox handler, between handler success and processed_at UPDATE.
InboxPostHandle,
/// Inside exchange send, after adapter.send() returns Ok and BEFORE finalize tx.
ExchangePostSend,
/// Inside placement update, after update_placement succeeds and BEFORE event publish.
PlacementPostUpdate,
/// Generic — caller-supplied predicate (for ad-hoc test scenarios).
Custom(&'static str),
}
/// Result of a simulated crash: the service was driven up to the crash point
/// and then forcibly errored. The DB / blob / queue state is whatever the
/// simulated point left behind.
pub struct SimulatedCrash {
pub at: CrashPoint,
pub error: anyhow::Error,
/// Useful diagnostics — e.g. the row id that was being processed.
pub artifact: Option<serde_json::Value>,
}
/// Drive a builder closure that constructs and exercises the service-under-test
/// up to `point`. Uses the Phase A.2 fault-injection harness to deliver the
/// crash at exactly the named site.
pub async fn simulate_crash_after<F, Fut>(
point: CrashPoint,
builder: F,
) -> SimulatedCrash
where
F: FnOnce(CrashHarness) -> Fut,
Fut: std::future::Future<Output = anyhow::Result<()>>;
/// Re-run a service's `main_inner(args, env)` (see §Open Questions) to drive
/// recovery sweeps after a simulated crash. Mocks the binary entrypoint without
/// spawning a process. Returns a handle that the test can await/cancel.
pub fn restart_service<F, Fut>(svc_main: F) -> SimulatedRestart
where
F: FnOnce() -> Fut + Send + 'static,
Fut: std::future::Future<Output = anyhow::Result<()>> + Send + 'static;
/// Poll until the system reaches the expected recovery state, or timeout.
/// `state` is a closure that queries DB / blob / queue and returns Ok(true)
/// when the invariant holds.
pub async fn assert_recovers_to<F, Fut>(
state: F,
timeout: Duration,
) -> anyhow::Result<()>
where
F: Fn() -> Fut,
Fut: std::future::Future<Output = anyhow::Result<bool>>;
The CrashHarness passed to the builder closure is a thin façade exposing the same fault injectors from Phase A.2 plus a crash_now(point) method that the test author calls at the chosen semantic point in service code (or, more commonly, the crash injector is armed before exercising the production handler, and the production handler hits the injector mid-execution because the injector is wired into the dependency the handler depends on — DB pool, publisher, store).
D16.2 How the crash points are wired
Each CrashPoint variant maps to a specific Phase A.2 injector configuration. The mapping table is the contract:
| CrashPoint | Underlying injector | Activation |
|---|---|---|
|
|
Phase 1 succeeds, Phase 2 |
|
|
Status-promote UPDATE errors |
|
|
Post-publish UPDATE errors |
|
|
|
|
|
Finalize tx errors after partner accepted |
|
|
Tx aborts; whole update rolls back |
|
Caller passes their own |
Caller-defined site |
(Per §D2.0, there is no DbErrorInjector::inject_error_after(N) operating on a generic PgPool wrapper. Every DB fault is scoped to a named store trait whose method the production codepath actually depends on; faulty impls of those traits are what the harness wires.)
The contract for each CrashPoint is also encoded as a doc-test in restart.rs — a compile-time check that every variant has a wiring match in the harness setup.
D16.3 Adoption pattern
// services/craig-cases/tests/api/contact_attachments_recovery.rs
#[tokio::test]
async fn upload_recovers_after_post_put_crash() {
if !devstack_available().await { return; }
let crash = simulate_crash_after(CrashPoint::AttachmentPostPut, |h| async move {
let client = h.cases_client();
client.upload_contact_attachment(case_id(), b"hello world").await?;
Ok(())
}).await;
assert!(crash.error.to_string().contains("simulated"));
// Now drive recovery — the AttachmentScanner re-runs and promotes the row.
restart_service(|| async {
craig_cases::main_inner(test_args(), test_env()).await
});
assert_recovers_to(|| async {
let row = query_attachment_row(crash.artifact.as_ref().unwrap()).await?;
Ok(row.object_status == "present")
}, Duration::from_secs(30)).await.unwrap();
}
The pattern is: arm the crash → exercise the handler → restart the service (or just spawn the recovery worker; for many cases a full restart isn’t needed, only the scanner) → assert the post-recovery state.
D16.4 Adoption sites — paired with platform-stab-2 fixes
Each platform-stab-2 fix that ships a recovery sweep gets a paired test under the new module:
| Platform-stab-2 fix | Crash point exercised | Test file |
|---|---|---|
§D2 Outbox FUSL |
|
|
§D3 Inbox three-state |
|
|
§D4 Atomic blob+DB+event upload |
|
|
§D5 Exchange send worker |
|
|
Each test asserts: (a) the crash genuinely aborted the in-flight operation, (b) the DB / blob / queue state matches the documented post-crash invariant, (c) restart drives the recovery sweep to completion, (d) the post-recovery state matches the success-path invariant for the same operation.
D16.5 Documentation
The pattern is documented in .claude/docs/testing.md under "Restart/replay testing" (Phase A.7 already adds the parent section). Convention: every platform-invariant fix that has a recovery sweep ships a paired restart.rs test as the failing-test-first artifact. Reviewer responsibility — a fix without a paired test fails the reviewer-discipline gate from coding-conventions.md Plan Authoring rules.
D16.6 Test description (parametric over CrashPoint)
A meta-test in crates/craig-test-lib/tests/restart_helpers_self.rs: for each CrashPoint variant (excluding Custom), simulate a crash + restart against an in-memory minimal harness, assert the helper observes the crash error and the recovery completes within a 5s timeout. Catches drift between the variant set and the wiring table from §D16.2.
D17. Invariant Sweeper — xtask invariants (Phase A.17 — Step 18)
After cargo xtask test or cargo xtask e2e runs, the dev DB is in some terminal state. Today nothing checks whether that terminal state is consistent. Steps 3-8 of platform-stab-2 introduce a new class of multi-row invariants — "no orphan attachment rows", "no stuck in_flight send jobs", "no `processed_at IS NOT NULL AND failed_at IS NOT NULL`" — that no individual test asserts because the invariants are cross-test. The invariant sweeper turns these into post-test assertions.
D17.0 DB connection scope
CRAIG runs 7 stateful services, each with its own Postgres DB (craig_cases, craig_placement, craig_exchange, craig_financial, craig_reporting, craig_security, craig_rules). The invariant sweeper iterates per-service-DB rather than against a unified read replica or per-service xtask invocation. Rationale:
-
Catalog files at
crates/craig-test-lib/sql/invariants/<svc>/*.sqlare physically partitioned per service (§D17.3) — one xtask process loads all 8 buckets and visits each. -
Per-service xtask invocation would multiply CI / pre-push runtime by 8× and complicate aggregation of the pass/fail summary.
-
A unified read replica does not exist in CRAIG’s deployment topology (each service owns its DB; no cross-DB read access by design).
DSN discovery: reuse the existing devstack-bringup environment variables (CRAIG_CASESDATABASE_URL, CRAIG_PLACEMENTDATABASE_URL, …) that cargo xtask dev already populates in .env. The xtask reads them via std::env::var(); if any required DSN is missing the subcommand bails with a clear message ("Run cargo xtask dev start first" or "Set CRAIG_<SVC>__DATABASE_URL"). Use a thin helper enumerate_services() that returns a Vec<ServiceDb { name: &str, dsn: String }> for the 7 stateful services; the helper lives in xtask/src/dsn.rs so it can be reused by other subcommands later.
Connection pool sizing: open a small pool per service (max_connections=2) inside connect_to(&dsn) since the workload is read-only and short-lived; close after each service is done.
D17.1 Output format and contract
cargo xtask invariants is a report-only subcommand by default. It prints a per-invariant pass/fail summary, exits 0 on all-pass, and exits 1 on any-fail. It is wired into cargo xtask test --post-check as opt-in (NOT a pre-push gate per the user directive that all xtask reliability/contracts/mutants/coverage-matrix/invariants/quality-budgets commands ship report-only).
A failed invariant surfaces as:
[invariants] craig-cases :: orphan_pending_attachments
query: SELECT id, object_status, created_at FROM contact_attachments
WHERE object_status='pending' AND created_at < now() - INTERVAL '24 hours'
expected: 0 rows
got: 3 rows
sample: [
{ id: 0193…, object_status: "pending", created_at: "2026-05-01T…" },
…
]
remediation: see plans/platform-stabilization-2.adoc §D4.5 — scanner should
have reaped this row; investigate scanner logs / DB connectivity.
The remediation text is part of each invariant’s metadata (see §D17.4) — never a free-form panic; always a pointer to the doc that specifies the fix.
D17.2 Implementation
xtask/src/cmd/invariants.rs (new):
pub async fn run(args: InvariantsArgs) -> anyhow::Result<()> {
let services = enumerate_services(&args)?; // default: all 8 stateful
let mut all_passed = true;
for svc in services {
let pool = connect_to(&svc.dsn).await?;
let invariants = load_invariants_for(&svc)?; // §D17.3
for inv in invariants {
let rows = sqlx::query(&inv.sql)
.fetch_all(&pool)
.await
.with_context(|| format!("running invariant {} on {}", inv.name, svc.name))?;
if rows.is_empty() {
println!("[invariants] {} :: {} OK", svc.name, inv.name);
} else {
all_passed = false;
report_failure(&svc, &inv, &rows);
}
}
}
if !all_passed { std::process::exit(1); }
Ok(())
}
D17.3 Invariant catalog file layout
Each service ships SQL files under crates/craig-test-lib/sql/invariants/<svc>/ (chosen over per-service service trees because the catalog is a test artifact, not a service artifact, and the test-lib already owns cross-service test infra):
crates/craig-test-lib/sql/invariants/
├── craig-cases/
│ ├── orphan_pending_attachments.sql
│ ├── duplicate_active_contact_persons.sql
│ └── audit_row_per_state_mutation.sql
├── craig-exchange/
│ ├── stuck_pending_transactions.sql
│ ├── stuck_in_flight_send_jobs.sql
│ └── orphan_pending_attachments.sql — ICPC mirror
├── craig-mq/ — global cross-service
│ ├── unpublished_outbox_beyond_grace.sql
│ └── inbox_processed_with_failed.sql
├── craig-financial/…
├── craig-placement/…
├── craig-reporting/…
├── craig-rules/…
└── craig-security/…
Each .sql file is a single SELECT that must return zero rows. The first comment line of the file is metadata in TOML-fragment form:
-- name = "orphan_pending_attachments"
-- service = "craig-cases"
-- grace = "24h"
-- remediation = "plans/platform-stabilization-2.adoc#D4.5"
-- severity = "high"
SELECT id, object_status, created_at, file_name
FROM contact_attachments
WHERE object_status = 'pending'
AND created_at < now() - INTERVAL '24 hours';
xtask reads the comment header at load time, strips it from the SQL it actually executes, and embeds the metadata in the report.
D17.4 The full invariant catalog (initial set)
| # | Service | Name | Query (one-line summary) |
|---|---|---|---|
1 |
craig-cases / craig-exchange |
|
|
2 |
every soft-delete table |
|
Per-table: no two rows with the same business key and |
3 |
craig-exchange |
|
|
4 |
craig-exchange |
|
|
5 |
craig-mq (per service DB) |
|
|
6 |
craig-mq (per service DB) |
|
|
7 |
craig-api (per service DB) |
|
|
8 |
every state-mutating endpoint per service |
|
Per-entity: every row in |
D17.5 Grace windows as constants
To prevent drift between the SQL grace literal and the semantic constant the codepath uses:
// crates/craig-common/src/constants.rs
use std::time::Duration;
pub const ORPHAN_PENDING_ATTACHMENT_GRACE: Duration = Duration::from_secs(24 * 60 * 60);
pub const OUTBOX_PENDING_GRACE: Duration = Duration::from_secs(5 * 60);
pub const STUCK_EXCHANGE_TX_GRACE: Duration = Duration::from_secs(6 * 60 * 60);
pub const STUCK_SEND_JOB_GRACE: Duration = Duration::from_secs(60);
pub const IDEMPOTENCY_CLAIM_TTL: Duration = Duration::from_secs(30);
pub const IDEMPOTENCY_PROCESSING_GRACE_PAST_TTL: Duration = Duration::from_secs(60);
A unit test in xtask/src/cmd/invariants.rs parses each .sql file, extracts the INTERVAL '<n> <unit>' literal, and asserts it matches the corresponding constant. Drift fails the test.
D17.6 Per-service audit-row catalog (invariant #8)
Built progressively. Each service contributes a crates/craig-test-lib/sql/invariants/<svc>/audit_catalog.sql listing (entity_table, audit_table, mutation_event_types) rows that the invariant joins against. Initial seed: cases, placements, exchange transactions, financial payments, security partners. The remaining ~10 entities ship in W3 hardening passes.
D17.7 Wiring into cargo xtask test --post-check
// xtask/src/cmd/test.rs (extension)
if args.post_check {
super::invariants::run(InvariantsArgs::default()).await?;
}
--post-check is opt-in. CI does not gate on it. Pre-push does not gate on it. Local developers run it on demand. Quarterly cadence: a scheduled CI job runs cargo xtask invariants --against staging-db for trend tracking.
D17.8 Test description
xtask/tests/invariants.rs: parses every .sql file in the catalog, validates header metadata, asserts every name is unique, asserts every grace literal matches the corresponding craig_common::constants::*_GRACE value, runs each query against an empty test DB and asserts zero rows. Catches catalog drift at unit-test time without devstack.
DB. Phase B — adoption bundle (Step 19)
Phase B of this plan does not duplicate the platform-stab-2 sibling plan’s per-step concurrent/fault/recovery tests. Steps 3 (idempotency), 4 (outbox), 5 (inbox), 6 (attachments), 7 (exchange send), 8 (placement) of platform-stab-2 each ship their own concurrent/fault/recovery test inline, using the platform-stab-2 §D2 minimal helpers. Phase B of this plan adopts the consolidated helpers from Phase A (which extend those minimal helpers) for the additional tests on the user’s "Specific Tests I’d Add First" list that platform-stab-2 does not already cover.
The user’s two-list overlap analysis: the original 10 Phase B tests in the current draft and the 10 "Specific Tests I’d Add First" share 6 tests. The deduplicated list of unique adoption tests is 10 tests + a risk-coverage scorecard. Each one is a paired failing-test-first artifact for either a platform-stab-2 fix (cross-referenced) or a hardening item that has no platform-stab-2 counterpart.
DB.1 The 10 adoption tests
| # | Service | Test | Helpers used | Sibling fix |
|---|---|---|---|---|
1 |
craig-api |
Concurrent same-key Idempotency-Key + same body → exactly one handler side effect |
Phase A.1 |
platform-stab-2 §D1 (Step 3) |
2 |
craig-api |
Same key + different body → 422 + does NOT execute the second handler |
Phase A.1 |
platform-stab-2 §D1 (Step 3) |
3 |
craig-api |
Idempotency storage failure during finalize → returns the live response body, not empty success |
Phase A.2 |
platform-stab-2 §D1 (Step 3) |
4 |
craig-mq |
Two outbox workers cannot publish same row twice |
Phase A.1 + Phase A.3 multi-replica devstack OR single-process simulation |
platform-stab-2 §D2 (Step 4) |
5 |
craig-mq |
Inbox handler failure doesn’t permanently suppress redelivery (under-cap retry) |
Phase A.2 fault injector |
platform-stab-2 §D3 (Step 5) |
6 |
craig-cases / craig-placement |
Object-store write succeeds, DB insert fails → no untracked blobs |
Phase A.2 fault injector + Phase A.17 invariant sweep + Phase A.16 restart |
platform-stab-2 §D4 (Step 6) |
7 |
craig-exchange |
Crash/failure between transaction creation and adapter completion → leaves recoverable state |
Phase A.16 |
platform-stab-2 §D5 (Step 7) |
8 |
craig-web |
Malformed upstream JSON → renders explicit service error, NOT default empty data |
Phase A.2 fault injector for upstream |
platform-stab-2 §D7 (Step 9) |
9 |
craig-store |
Filename + Content-Disposition fuzzing: quotes, CRLF, Unicode, long names, path tricks |
Phase A.6 + Phase A.15 |
platform-stab-2 §D8.1 (Step 10) — FIXTURES SIDE |
10 |
craig-test-lib + craig-cases |
Typed-client pilot for craig-cases — proves Phase A.8 migration runs end-to-end |
Phase A.8 typed DTOs |
n/a (this plan only) |
Each test file lives under the service’s tests/ directory, named <test>_evil.rs / _recovery.rs / _concurrent.rs per its dominant axis. Naming convention matches the regex tagging discussion in §Open Questions — suffixes _happy, _sad, _evil, _concurrent, _fault, _replay are claimed as test-category markers consumed by the coverage-matrix xtask.
DB.2 Risk-coverage scorecard
docs/modules/ROOT/pages/test-coverage-scorecard.adoc is a living artifact tracking, for each of the top-20 critical workflows, which of six test-category cells is covered:
= Test Coverage Scorecard
:description: Risk-based coverage for the top 20 critical workflows.
[cols="3,1,1,1,1,1,1"]
|===
| Workflow | Happy | Sad | Evil | Concurrency | Replay | Fault
| Idempotent POST handler (cross-cutting)
| ✅ | ✅ | ✅ | ✅ | ✅ | ✅
| Outbox publish (cross-cutting)
| ✅ | ✅ | ⏳ | ✅ | ✅ | ✅
| Inbox handler (cross-cutting)
| ✅ | ✅ | ⏳ | ⏳ | ✅ | ✅
| Attachment upload (4 sites)
| ✅ | ✅ | ⏳ | ⏳ | ✅ | ✅
| Exchange send (1 path)
| ✅ | ✅ | ⏳ | ⏳ | ✅ | ✅
| Placement transition guard
| ✅ | ✅ | ⏳ | ✅ | ⏳ | ⏳
| BFF page render under upstream failure
| ✅ | ✅ | ⏳ | ⏳ | ⏳ | ✅
| Person match
| ✅ | ⏳ | ⏳ | ⏳ | ⏳ | ⏳
| Rules evaluation
| ✅ | ⏳ | ⏳ | ⏳ | ⏳ | ⏳
| Public intake submit
| ✅ | ✅ | ✅ | ⏳ | ⏳ | ⏳
| (10 more — see source) | | | | | | |
|===
Legend: ✅ covered, ⏳ planned, ❌ gap.
The 6 cells are the canonical test-category taxonomy enforced via reviewer discipline (Phase A.7 — Step 8). Each Phase B test from §DB.1 ticks one or more cells; the scorecard is updated in the same MR that lands the test (delivery checklist gate).
The scorecard is generated and hand-maintained: a script-section at the top is auto-populated by cargo xtask coverage-matrix --report --emit-scorecard (Phase A.11/A.12 deliverable) by reading test-name-suffix tags; the workflow rows are hand-curated from the operational risk register. Drift between the auto-section and the curated section flags a stale scorecard.
DB.3 Step 19 deliverable list
-
10 adoption test files (one per row of §DB.1).
-
docs/modules/ROOT/pages/test-coverage-scorecard.adoc(new — top-20 workflows × 6-cell grid). -
cargo xtask coverage-matrix --emit-scorecardflag (extension to A.11 deliverable). -
nav.adocentry under Architecture & Design pointing at scorecard. -
CHANGELOG entry per landed test + a single entry for the scorecard introduction.
Steps
Step 1: Plan adoc + nav.adoc Active + CHANGELOG entry + 9 new GitLab issues
Files:
-
docs/modules/ROOT/pages/plans/test-framework-hardening.adoc(new — content of this entire plan) -
docs/modules/ROOT/pages/plans/platform-stabilization-2.adoc(sibling — already written fromsnappy-napping-puzzle.md) -
docs/modules/ROOT/nav.adoc(Active section: add both plans) -
CHANGELOG.adoc(single combined entry under== Unreleasedcovering both plan adoptions)
Implementation:
-
Replace the existing thin
test-framework-hardening.adoc(137 lines) with this plan’s full content (~5100 lines). The thin draft on the current branch (chore/platform-stab-2-and-test-framework-plans) was an earlier scratch that the plan-mode review supersedes. Use a single git operation:git rmthen re-add the new content, OR overwrite-in-place. Either way, the diff isreplace file contents. -
Verify
platform-stabilization-2.adocis the sibling-plan content already written fromsnappy-napping-puzzle.md(the prior plan-mode review). No edits needed in this MR. -
Update
nav.adocActive section to include both plans (already done in this branch’s working tree; verify still in place). -
Append the combined CHANGELOG entry under
== Unreleasedcovering both plan adoptions (already done in this branch’s working tree; verify still in place). -
File 9 new GitLab issues under epic &22 for Phase A.9–A.17 not covered by the existing #283–#291. Each issue title matches the Status-table description; body cross-references the corresponding
§Dsection. Filed viaglab issue create --epic 22 --title "…" --description "…" --label "P1-high,test-framework"(one per issue):-
A.9 — proptest scaffolding + first 7 properties
-
A.10 — cargo-mutants smoke on 5 critical crates
-
A.11 — xtask subcommands (reliability / contracts / mutants / coverage-matrix / invariants / quality-budgets, REPORT-only)
-
A.12 — Code-quality budget warnings + lock-file baseline
-
A.13 — Coverage Matrix tooling (xtask coverage-matrix, REPORT-only)
-
A.14 — DB Constraint Evil Tests (hand-curated harness)
-
A.15 — Central Evil Input Corpus consolidation
-
A.16 — Restart/Replay light helpers
-
A.17 — Invariant Sweeper (xtask invariants)
-
No code, no migration, no tests. Pure docs + issue-tracker bookkeeping.
Verification:
-
glab epic view 22 --with-issues— confirms 9 new issues are linked under &22 alongside the existing #283–#291 (total 18 issues + 1 Phase B bundle = 19) -
Antora docs render the new
.adocwithout errors (verifiable via local Antora build orcargo xtask api-docs) -
nav.adocActive section visibly shows both plans (grep -A2 "** Active" docs/modules/ROOT/nav.adocshows both xrefs) -
CHANGELOG
== Unreleasedsection lists the combined entry as the topmost entry -
Pre-push gate passes — no Rust code touched
Cross-reference: top-matter Status table.
Step 2: Phase A.1 — concurrent_fire helper consolidation
Files:
-
crates/craig-test-lib/src/concurrent.rs(extend in place — addsconcurrent_fire_collect,concurrent_fire_collect_ordered,concurrent_fire_synchronized(auto-barrier),concurrent_fire_with_barrier(caller-managed sync point),concurrent_fire_until_first_success,concurrent_fire_first_n,ConcurrentRunReport,TaskOutcome,TaskResult) -
crates/craig-test-lib/src/lib.rs(re-exports + deprecatedconcurrent_firealias) -
crates/craig-test-lib/Cargo.toml(no new deps;tokio::sync::Barrieris in the existingtokioworkspace dep) -
crates/craig-test-lib/tests/concurrent.rs(extend with the 12 smoke tests from §D1.4) -
.claude/docs/testing.md(update the Failure-path testing helpers section added by epic &21 Step 2 with the 4 new helpers + when-to-use table) -
(No service-side or migration changes; pure test infra.)
Implementation:
-
Land §D1.2’s full surface in
crates/craig-test-lib/src/concurrent.rs, alphabetized, with rustdoc on every public item including the when-to-use guidance from §D1.2. -
Move existing minimal
concurrent_firebody intoconcurrent_fire_collect. Addpub use self::concurrent_fire_collect as concurrent_fire;with#[deprecated(note = "renamed to concurrent_fire_collect; will be removed in next minor version")]. -
Implement
ConcurrentRunReport,TaskOutcome,TaskResult;into_results()collects into Vec on Ok and Vec<String> on any panic;ok_count()/panic_count()are simple iteration. -
concurrent_fire_collect_ordered: usetokio::task::JoinSet; recordInstant::now()at task entry; sort outcomes bystarted_atbefore returning. -
concurrent_fire_with_barrier: buildArc::new(Barrier::new(n)); pass to each spawned future;started_atis recorded afterbarrier.wait().await. The caller’s builder receives the barrier and is expected to call.wait().awaitat its semantic sync point. -
concurrent_fire_synchronized: buildArc::new(Barrier::new(n))internally; spawn each task as a wrapper that callsbarrier.wait().awaitbefore invoking the caller’sbuilder(i)closure. Caller never sees the barrier; cannot leak the synchronization. Recommended default for "all start together" tests. -
concurrent_fire_until_first_success+concurrent_fire_first_n: useJoinSet::join_next; abort on threshold viaJoinSet::abort_all; collect errors from naturally-completed Err results only. -
Panic capture per §D1.3:
JoinError::is_panic()→JoinError::into_panic()→Any::downcast_ref::<&str>thenAny::downcast_ref::<String>then fallback message. -
Add the 12 smoke tests from §D1.4 to
tests/concurrent.rs. Existing &21 Step 2 tests remain green (calling deprecated alias). -
Update
.claude/docs/testing.mdFailure-path section: list 5 helpers + when-to-use cell per row + one-line code sketch per helper. -
Errata entry in this plan: "Existing
concurrent_firerenamed toconcurrent_fire_collect; back-compat alias retained for one minor version. Reason: consolidate naming family for 4 new variants."
Verification:
-
cargo nextest run -p craig-test-lib --test concurrent— 12 smoke tests pass (the 1 from &21 Step 2 against the alias, plus 11 new). -
cargo clippy -p craig-test-lib --tests --locked — -D warnings— clean, including#[deprecated]warning suppression in the alias callsite. -
cargo build --workspace --locked— confirms no caller ofconcurrent_fire(alias) breaks; confirms one deprecation warning fires per existing call site so the migration scope is visible. -
Manual: count call sites via
Grep "concurrent_fire\b" --type rust— expected to match the &21 Step 2 idempotency / outbox / inbox tests; each call site is candidate for a follow-up rename in epic &21’s MR series.
Cross-reference: §D1.
Step 3: Phase A.2 — Fault-injection harness consolidation
Files:
-
crates/craig-test-lib/src/fault/(new module directory) withmod.rs,cipher.rs,db.rs,rabbit.rs,object_store.rs,publish_in_tx.rs(migrated from epic &21 Step 2fault_injection.rs),latency.rs,flap.rs,backpressure.rs,scenarios.rs(new) -
crates/craig-test-lib/src/fault_injection.rs(replaced with re-export shim per §D2.1) -
crates/craig-test-lib/src/lib.rs(addpub mod fault;alongside existingpub mod fault_injection;) -
crates/craig-test-lib/Cargo.toml(no new deps; uses existingtokio,lapin,sqlxworkspace deps) -
crates/craig-test-lib/tests/fault_latency.rs,crates/craig-test-lib/tests/fault_flap.rs,crates/craig-test-lib/tests/fault_backpressure.rs,crates/craig-test-lib/tests/fault_scenarios.rs(new smoke tests) -
crates/craig-test-lib/tests/fault_injection.rs(existing &21 Step 2 smoke tests; remain green via the shim) -
.claude/docs/testing.md(extend Failure-path section with the 6 injectors + named-scenarios registry +FaultInjectortrait recipe; epic &21 Step 2 added the initial 5; this step extends)
Implementation:
-
Migration step — move every existing injector body from
fault_injection.rsinto its sibling underfault/. Source diff is pure relocation; no behavior change; all &21 Step 2 tests should still pass against the shim. Runcargo test -p craig-test-lib --test fault_injectionbetween this step and the next as the fast-feedback gate. -
Define
FaultInjectortrait +Attemptstruct +ScenarioGuardper §D2.2 infault/mod.rs. Implement on each migrated injector by promoting their existing internal state to fit the trait shape (each already has counter / Mutex internals; the trait just formalizes the access pattern). -
Implement
LatencyInjectorper §D2.3. Compose-style: it holds an innerI: FaultInjectorand forwardsinject/recorded_attemptsunchanged, but insertstokio::time::sleepbefore/after each wrapped call. -
Implement
IntermittentFlapInjectorper §D2.4. The pattern is aVec<bool>; theAtomicUsizecounter isfetch_add(1)modulo the pattern length; each wrapped call indexes the pattern. -
Implement
ConsumerBackpressureInjectorper §D2.5. Thehandle_deliveryhelper wraps alapin::message::Deliveryhandler withtokio::time::sleep (ack_delay)+ optional drop.recorded_attempts()records each delivery withinjected = ack_delay > 0 || dropped. -
Build out
scenarios.rsper §D2.6 —db::*,rmq::*,cipher::*,object_store::*,publish_in_tx::*,backpressure::*modules with named constructors. Each named constructor returns the right Scenario enum variant. Test withcargo doc -p craig-test-lib --no-depsto verify rustdoc rendering. -
Add the smoke tests from §D2.7 — 9 named cases across 4 new test files (
fault_latency,fault_flap,fault_backpressure,fault_scenarios). The existingfault_injection.rstest file remains untouched. -
Drop-assertion path: implement Path A/B/C tests from §D2.7’s closing block. Use
std::panic::catch_unwindto verify behavior in Path B; usestd::panic::panic_anyfor Path C; ensurestd::thread::panicking()check survives. -
Update
.claude/docs/testing.md. Sections to add: "FaultInjector trait", "Latency / Flap / Backpressure wrappers", "Named scenarios in `craig_test_lib::fault::scenarios`", and a "Picking the right injector" decision table.
Verification:
-
cargo nextest run -p craig-test-lib --test fault_injection— &21 Step 2 tests pass via shim. -
cargo nextest run -p craig-test-lib --test fault_latency— 1 case. -
cargo nextest run -p craig-test-lib --test fault_flap— 2 cases. -
cargo nextest run -p craig-test-lib --test fault_backpressure— 2 cases. -
cargo nextest run -p craig-test-lib --test fault_scenarios— 4 cases including drop-assertion Path A/B/C. -
cargo clippy -p craig-test-lib --tests --locked — -D warnings— clean. -
cargo doc -p craig-test-lib --no-deps— generates without warnings; manual eyeball offault::scenariosmodule page confirms named scenarios are documented with when-to-use. -
Manual:
Grep "use craig_test_lib::fault_injection"— every &21 Step 2 caller still imports cleanly (proves the shim is tight).
Cross-reference: §D2.
Step 4: Phase A.3 — Multi-replica devstack profile
Files:
-
xtask/src/cmd/dev.rs(extendDevCommand::Startwith--replicasper §D3.2; addReplicaSpec+parse_replicas) -
xtask/src/devstack.rs(threadreplicas: &[ReplicaSpec]throughensure_ready+bring_upper §D3.3; addreplica_names
wait_for_service_health_all_replicasper §D3.6) -
xtask/src/docker.rs(addcompose_with_overrideshelper that takes an&[&str]of-f <file>args;composeand friends delegate) -
docker-compose.scale.yml(new, root of repo, per §D3.4 — 8 scalable services with!overridefloating-port lists) -
crates/craig-test-lib/src/multi_replica.rs(new —MultiReplicaCluster+ builder per §D3.7) -
crates/craig-test-lib/src/lib.rs(addpub mod multi_replica;
re-exportMultiReplicaCluster) -
crates/craig-test-lib/Cargo.toml(no new deps — usestokio
existing process-shell pattern) -
crates/craig-test-lib/tests/multi_replica_idempotency.rs(new — the §D3.8 verification test) -
.claude/docs/local-dev.md(new section: "Multi-replica devstack") -
.claude/docs/testing.md(update withMultiReplicaClusterrecipe + when-to-use) -
docs/modules/ROOT/pages/dev/devstack.adoc(or equivalent developer-guide page) — describe--replicasflag
Implementation:
-
Add
ReplicaSpec+parse_replicastoxtask/src/cmd/dev.rsper §D3.2.parse_replicasis unit-testable in isolation; add 4#[cfg(test)]cases (happy path, missing=, unknown service, zero count). -
Thread
replicas: &[ReplicaSpec]throughcmd::dev::run,devstack::ensure_ready,devstack::bring_up,devstack::reload,devstack::reseed. Default at every other entry point (Stop, Restart, Clean, etc.) is&[]. -
In
devstack::bring_up: whenreplicasis non-empty, splice-f docker-compose.scale.ymlinto the compose invocation (docker compose -f docker-compose.yml -f docker-compose.scale.yml up -d --build --scale craig-cases=2 …). When empty, the existingdocker compose up -d --buildpath runs unchanged. -
Implement
replica_names+wait_for_service_health_all_replicasper §D3.6. Updatewait_for_health(the all-services loop) to call the new replica-aware helper. Default-scale-1 case still returns one replica name and terminates immediately on health. -
Author
docker-compose.scale.ymlper §D3.4. Verify the!overridetag behavior against the compose version pinned in.gitlab-ci.yml— local test:docker compose -f docker-compose.yml -f docker-compose.scale.yml config | grep -A2 'craig-cases:' | grep portsshould show only the floating port, not the merged list. -
Implement
MultiReplicaCluster+ builder per §D3.7. Sub-implementation detail:start()chooses an isolated compose project name (e.g.,craig-test-mrc-<uuid>) so the dedicated cluster does not collide with the developer’s persistent default-name devstack.replica_urlsiteratesdocker compose -p <project> port --index N craig-cases 8002forNin1..=count, parsing each output line as a singlehost:port. The--index Nflag is mandatory: without it,docker compose port craig-cases 8002returns only the FIRST replica’s host port (a silent ambiguity that would make a 2-replica test pass against a 1-replica subset). -
Author the §D3.8 verification test. Covers Path A (concurrent fire across replicas), Path B (one handler-side effect lands), Path C (both replicas served traffic via the
X-Served-Byheader trick). Note: the test is gated byif !devstack_available().await { return; }style guard fromcraig-test-lib, but it spins its own dedicated cluster — the global devstack is not affected and the test can run in parallel with other integration tests. -
Update
.claude/docs/local-dev.mdwith a new section: how to invoke--replicas, thedocker-compose.scale.ymlrationale, the ephemeral-port discovery recipe (docker compose port), and the current limitations (postgres/keycloak/garage stay scale=1; craig-web’s host port pins to first replica). -
Update
.claude/docs/testing.mdwithMultiReplicaClusterusage: "Use this when …" table + 1 minimal code snippet.
Verification:
-
cargo build -p xtask --locked— clean build with new args. -
cargo nextest run -p xtask --lib—parse_replicasunit tests pass. -
cargo xtask dev start --replicas craig-cases=2,craig-exchange=2on a clean dev box — both services come up healthy with 2 replicas each;cargo xtask dev statusshows replica names; ephemeral port range shows two distinct host ports per scaled service. -
cargo xtask dev startwithout--replicas— default scale=1 behavior, host ports8002:8002etc. unchanged. Pre-push hook unaffected. -
cargo nextest run -p craig-test-lib --test multi_replica_idempotency— Path A/B/C all green. Test runtime ≤ 90s (cluster boot dominates). Also passes when run aftercargo xtask dev restart(cluster boots in isolation regardless of global devstack state). -
Manual:
docker compose -f docker-compose.yml -f docker-compose.scale.yml config— render output shows scalable services with floating ports and shared-state services unchanged. -
Manual: kill one replica of
craig-casesmid-test (docker compose stop craig-cases-2); rerun the verification test; assert it still passes (the surviving replica handles all 5 POSTs; atomic-claim still wins). This proves the test asserts correctness, not just load distribution.
Cross-reference: §D3.
Step 5: Phase A.4 — State-machine matrix-testing pattern + Payment first adoption
Files:
-
crates/craig-test-lib/src/state_machine_matrix.rs(new) -
crates/craig-test-lib/src/lib.rs(pub mod state_machine_matrix;) -
crates/craig-test-lib/tests/state_machine_matrix.rs(new — helper unit tests per §D4.6 items 1–3) -
services/craig-financial/tests/transitions_matrix.rs(new — first adoption per §D4.3) -
.claude/docs/testing.md(testing.md gets the matrix helper added to the §D7.3 cheat-sheet — folded into Step 8 unless that step is in flight)-
Implement
ValidTransition,Outcome,MatrixReport,assert_only_valid_transitions_acceptedper §D4.2.std::panic::catch_unwindwraps each predicate call so a panic in one pair does not abort the suite. -
Implement
MatrixReport::assert_cleanwith a multi-line message listing every mismatch + every panic. -
Add the three helper unit tests per §D4.6.
-
Land the financial-payment matrix test per §D4.3, omitting
(Issued, Cleared)fromVALIDfirst: confirm assertion fires; commit the failing-test screenshot/log into the MR description; restore the omission; confirm green; merge. -
Cross-link in §D4.5 table → file an issue per remaining state machine (15 issues) under the parent epic. Each is a one-file MR following the same pattern. Phase B adoption work, not blocking this step.
-
Run:
cargo nextest run -p craig-test-lib --test state_machine_matrix— green;cargo nextest run -p craig-financial --test transitions_matrix— green.
-
Cross-reference: §D4.
Step 6: Phase A.5 — JWT mutation library
Files:
-
crates/craig-test-lib/src/jwt_mutation.rs(new — per §D5.3 + §D5.4) -
crates/craig-test-lib/src/lib.rs(pub mod jwt_mutation;) -
crates/craig-test-lib/Cargo.toml(addjsonwebtokendev-dep if absent at workspace-pin level; today craig-auth carries it — re-export rather than duplicate) -
crates/craig-auth/src/jwks.rs(additive:pub mod test_fixturesgated onfeature = "test-fixtures"per §D5.2) -
crates/craig-auth/Cargo.toml(addtest-fixturesfeature;craig-test-libdeclares the feature in its dev-dep edge) -
crates/craig-test-lib/tests/jwt_mutation_smoke.rs(new — per §D5.6)-
Implement
ClaimsBuilderper §D5.3. Defaults: now-relativeiat/exp,iss = TEST_ISSUER,kid = "test-kid"(matches the existingcrates/craig-auth/src/middleware.rs:108fixture). -
Implement
mint(&self) → Stringusingjsonwebtoken::encodewithEncodingKey::from_rsa_derfrom §D5.2. TheAlgorithm::None"alg=none" attack vector lives in a separate helperwith_unsigned()that manually constructs the compact JWT (base64url header + base64url payload + empty signature segment), sincejsonwebtoken10.3 does NOT exposeAlgorithm::Nonein its enum. -
Implement
mutation_matrixper §D5.4 — single-axis variants only; combinatoric explosion is out of scope. -
Add the smoke test from §D5.6 (
baseline_token_validates+every_single_axis_mutation_is_rejected_by_craig_cases_validator). -
Open the 8 per-service follow-up issues (one per service that mounts
auth_middleware) under epic &22 Phase B. Each lands atests/api/auth_strictness.rsconsumingmutation_matrix(ClaimsBuilder::baseline("<svc>")). Not blocking this step’s completion. -
Run:
cargo nextest run -p craig-test-lib --test jwt_mutation_smoke— green;cargo nextest run -p craig-auth— green (existing tests survive thetest-fixtures-feature reshuffle).
-
Cross-reference: §D5.
Step 7: Phase A.6 — Magic-byte upload-spoofing fixtures
Files:
-
crates/craig-test-lib/fixtures/upload_spoofing/(new dir, 10 binary files per §D6.1) -
crates/craig-test-lib/fixtures/upload_spoofing/README.md(new — SPDX header + table mapping each.binto its description per §D6.1) -
crates/craig-test-lib/src/upload_fixtures.rs(new — per §D6.2) -
crates/craig-test-lib/src/filename_attack.rs(new — per §D6.3) -
crates/craig-test-lib/src/lib.rs(two newpub modlines) -
crates/craig-test-lib/tests/upload_fixtures_smoke.rs(new — per §D6.4) -
crates/craig-test-lib/tests/filename_attack_smoke.rs(new — round-trip table-driven test per §D6.4 item 3)-
Mint each spoof fixture per §D6.1 byte-by-byte. Use
python3 -c 'import sys; sys.stdout.buffer.write(bytes.fromhex("…"))' > pe_disguised_as_pdf.bin(or the equivalent xxd-roundtrip). Verify each.binsize matches §D6.1; verify each magic-byte prefix withinfer::get(…)in a one-shot script before commit. -
Mint the 4 valid-companion fixtures (
valid_pdf.bin,valid_jpeg.bin,valid_png.bin,valid_csv.bin). -
Implement
FixtureName,fixture(),SpoofFixture,enumerate_spoofs,enumerate_validper §D6.2. Useinclude_bytes!for the static byte slices. -
Implement
filename_attack::CASESper §D6.3. The 12-row table is hand-frozen; subsequent additions go in the same file alphabetically byname. -
Add
upload_fixtures_smoke.rsper §D6.4 items 1–2 — both run as workspace unit tests (no devstack needed). Note: item 1 currently depends on platform-stab-2 §D8.1 having landed; if §D8.1 has not landed, the test asserts on the (existing) MIME-allowlist arm only and gets a// TODO(epic-21-D8.1)comment near the magic-byte assertion. Tracked. -
Add
filename_attack_smoke.rsper §D6.4 item 3. Theexpected_dispositionhalf is gated on platform-stab-2 §D8.1’s RFC 5987 encoder shipping; same// TODOpattern applies if not yet landed. -
Failing-test-first protocol: deliberately omit
truncated_pdf_first_8_bytes.binfromenumerate_spoofs’s array, observe the smoke test’s coverage drop in the assertion (the test asserts `enumerate_spoofs().count() == 6); restore; merge.
-
Cross-reference: §D6 (and platform-stab-2 §D8.1 for the validator side).
Step 8: Phase A.7 — testing.md "Failure-path testing" + Test Categories taxonomy + Process rule
Files:
-
.claude/docs/testing.md(4 new sections per §D7.1, §D7.2, §D7.3, §D7.4 — and §D7.6 if the budget shape lands in this MR; the budget script is deferred per §D7.6) -
.claude/docs/delivery-protocol.md(one-paragraph audit-rule pointer per §D7.5) -
CONTRIBUTING.adoc(new "Failure-path testing" section per §D7.5)-
Insert §D7.1 "Test Categories" section into
.claude/docs/testing.mdafter "Mandatory Rules". Eight categories, exact verbatim text. Cross-link each to its first example file. -
Insert §D7.2 "File-naming convention for test categories" section immediately after §D7.1.
-
Insert §D7.3 "Failure-path testing helpers" section after §D7.2. Include one-line example invocation per row (grep-able).
-
Insert §D7.4 "Process rule" section immediately after §D7.3, marked as a
[NOTE]block in AsciiDoc-style emphasis (testing.md is markdown, so equivalent: a> Process ruleblockquote). -
Insert §D7.6 "Quality budgets — available, not enforced" section after §D7.4. Mark every command as future (italics in the doc) so a contextless agent doesn’t try to invoke them.
-
Update
.claude/docs/delivery-protocol.md"Plan completion audit" section per §D7.5 — single paragraph. -
Update
CONTRIBUTING.adocper §D7.5 — single section, one paragraph, link to testing.md anchor. -
No code, no tests, no migration. Pre-push gate runs as usual; only Antora docs need to render cleanly. Run:
cargo xtask api-docs --check(or local Antora build).
-
Cross-reference: §D7.
Step 9: Phase A.8 — Typed DTOs across 8 test clients (8 MRs, this step lands MR 1)
This step is the long-running migration umbrella. Step 9 in this plan lands MR 1 (CasesClient pilot). Each subsequent MR is filed and tracked under epic &22 but does not block the parent plan’s "Step 9" status from flipping to Complete — Complete here means "the migration pattern is proven on CasesClient and the remaining 7 issues are filed."
The contracts-crate pattern is mandatory. Per §D8.2, the prior draft had craig-test-lib dev-depending on craig-cases directly, which would create a cycle with the existing services/craig-cases/Cargo.toml:34 dev-dep on craig-test-lib. This step instantiates the per-service contracts-crate pattern; do not dev-dep the service crate from the test-lib.
Files (MR 1 — CasesClient):
-
crates/craig-cases-contracts/Cargo.toml(new — leaf crate, no deps on services/* or craig-test-lib) -
crates/craig-cases-contracts/src/lib.rs(new — module per api submodule:pub mod cases; pub mod persons; pub mod referrals;…) -
crates/craig-cases-contracts/src/cases.rs(new — DTOs moved fromservices/craig-cases/src/api/cases.rs) -
crates/craig-cases-contracts/src/persons.rs(new — same) -
crates/craig-cases-contracts/src/referrals.rs(new — same) -
Cargo.tomlworkspace[workspace.dependencies]— addcraig-cases-contracts = { path = "crates/craig-cases-contracts" } -
services/craig-cases/Cargo.toml— addcraig-cases-contracts = { workspace = true }to[dependencies] -
services/craig-cases/src/api/cases.rs— delete the moved DTO definitions, replace withpub use craig_cases_contracts::cases::*;so handler code atcrate::api::cases::CreateCaseRequestkeeps working unchanged -
services/craig-cases/src/api/persons.rs(same shape) -
services/craig-cases/src/api/referrals.rs(same shape) -
crates/craig-test-lib/Cargo.toml— addcraig-cases-contracts = { workspace = true }to[dev-dependencies](NOTcraig-cases— that would reintroduce the cycle the design rejects) -
crates/craig-test-lib/src/clients/cases.rs(~60-method rewrite per §D8.4) -
crates/craig-test-lib/tests/typed/cases_client.rs(new — Contract round-trip per §D8.6) -
crates/craig-test-lib/tests/typed.rs(module index file declaringmod cases_client;) -
deny.toml— add abans.denyentry preventing any future PR from addingcraig-casesas acraig-test-libdependency (cycle guard)-
Contracts crate first: create
crates/craig-cases-contracts/as a leaf crate. Contents: pure data types (request/response DTOs). Add only the deps the DTOs need (serde, chrono, uuid, utoipa, craig-reference where shared enums apply). Do not depend onservices/craig-casesorcraig-test-lib. -
Move DTOs: move every
Create*Request,Update*Request,Response, paged-list response shape fromservices/craig-cases/src/api/{cases,persons,referrals,…}.rsinto the matchingcrates/craig-cases-contracts/src/<module>.rs. Each DTO retains its#[derive(Deserialize, Serialize, Debug, Clone, PartialEq, ToSchema)]derives. The service-internalapi::::*modules thenpub use craig_cases_contracts::*::*;so existing handler code compiles unchanged. -
Service-side audits:
-
Every DTO touched by a test-client method needs
#[derive(Serialize)](today most are deserialize-only); add. -
Many response types are constructed inline from store rows and never named at the API boundary; lift those into named structs (
PersonResponse, etc.) and move to the contracts crate. -
Rule of thumb: every test-client method’s response type must be a
#[derive(Deserialize, Serialize, Debug, Clone, PartialEq)]named struct in the contracts crate; anonymous JSON shapes get a name during this step.
-
-
Test-client-side: rewrite
crates/craig-test-lib/src/clients/cases.rsper §D8.4.use craig_cases_contracts::cases::*; use craig_cases_contracts::persons::*; …— note the_contractssuffix on the crate. Methods that return non-JSON (e.g.list_personsreturns paged metadata +data: Vec<Person>) get aListPersonsResponse { data: Vec<PersonResponse>, total: u64, … }DTO in the contracts crate first. -
Test-callers: every
tests/api/*.rsfile underservices/craig-cases/tests/currently using the JSON-y forms breaks at compile. Rewrite each call-site:json!({"first_name": "Foo"})→CreatePersonRequest { first_name: "Foo".into(), … }. Estimated touch: ~30 test files (count viaGrep "cases_client\." services/craig-cases/tests/). -
Smoke: add
tests/typed/cases_client.rsper §D8.6. The first round-trip test is the canary for the migration — failing here means the service-side DTOs and the test-client DTOs disagree. -
Quality budget: §D7.6 budget tracker checks
crates/craig-test-lib/src/clients/cases.rsreports zeroserde_json::Valuehits. Warning, not failure (per §D8.5). -
File 7 follow-up issues under epic &22 Phase A.8 for the remaining 7 clients per the §D8.3 order (Placement → Exchange → Financial → Reporting → Rules → Security → Intake). Each issue links its own §D8.3 row, its own MR branch (
feat/test-framework-typed-clients-<svc>), and the same MR template as MR 1. -
Run:
cargo nextest run -p craig-cases— every call-site recompiles + passes;cargo nextest run -p craig-test-lib --test typed— green;cargo xtask budget --metric value-count(when shipped) —clients/cases.rsreports 0.
-
Cross-reference: §D8.
Step 10: proptest scaffolding + first 7 properties
Files:
-
Cargo.toml(workspace) — addproptest = "1.7"to[workspace.dependencies] -
crates/craig-store/Cargo.toml— addproptest = { workspace = true }under[dev-dependencies] -
crates/craig-api/Cargo.toml— same -
crates/craig-signing/Cargo.toml— same -
crates/craig-reference/Cargo.toml— same -
services/craig-financial/Cargo.toml— same -
services/craig-financial/tests/properties.rs(new) +tests/properties/payment_status.rs(new) -
crates/craig-api/tests/properties.rs(new) +tests/properties/{pagination,idempotency_fingerprint}.rs(new) -
crates/craig-store/tests/properties.rs(new) +tests/properties/{sanitize_filename,content_disposition}.rs(new) -
crates/craig-signing/tests/properties.rs(new) +tests/properties/canonical_json.rs(new) -
crates/craig-reference/tests/properties.rs(new) +tests/properties/parse_never_panics.rs(new) -
.claude/docs/testing.md— append "Property tests" subsection under the Step 8 "Failure-path testing" section
Cross-reference: §D9.
-
Workspace Cargo.toml: add proptest dep verbatim (verify version with
cargo search proptest --limit 1). -
Per crate listed: add
[dev-dependencies] proptest = { workspace = true }. -
Author the seven property files per §D9.3–§D9.9. Each carries the verbatim Strategy + invariant block + the FAIL-FIRST and PROPTEST_BUDGET_MS comments.
-
Add the
tests/properties.rsmodule-index file per crate (mirrorstests/api.rspattern). -
Append the "Property tests" subsection to testing.md per §D9.2.
-
Run
cargo nextest run --workspace -E 'test(::properties::)'; assert all pass and total wall-time <2s. -
Fail-first verification (per property): introduce one-character bug in the unit-under-test; run prop; record shrunk counter-example; revert.
Step 11: cargo-mutants smoke on 5 critical crates
Files:
-
.cargo/mutants.toml(new) — workspace mutants config per §D10.1 -
xtask/mutants-baseline.toml(new) — committed missed-mutant catalogue, populated on first run -
CONTRIBUTING.adoc— new "Optional dev tools" subsection with cargo-install line -
.claude/docs/testing.md— new "Mutation testing" subsection cross-referencing the xtask subcommand
Cross-reference: §D10. (Subcommand wiring lives in Step 12 / §D11.3.)
-
Author
.cargo/mutants.tomlper §D10.1. Verify version compatibility withcargo install cargo-mutants --version "^25.0" --locked. -
Document install line in CONTRIBUTING under "Optional dev tools".
-
Run
cargo xtask mutants --smoke(after Step 12 lands the subcommand) on a clean working tree. -
Inspect each crate’s
target/mutants-<crate>/outcomes.json; for each"missed"mutant, append an entry toxtask/mutants-baseline.toml:[[missed]] crate = "crates/craig-mq" file = "src/inbox.rs" line = 142 mutant_kind = "replace == with !=" reason = "tracked: inbox-retry coverage gap, see #28X" -
Commit baseline.
-
Append testing.md "Mutation testing" subsection: tool purpose, when to run, what surviving mutants mean, how to interpret the baseline.
-
Fail-first verification: delete one
prop_assert!from the §D9 properties; rerun smoke; observe the corresponding mutant survive that previously didn’t. Document this demonstration in the MR description, then revert.
Step 12: xtask subcommands: reliability + contracts + mutants + coverage-matrix + invariants
Files:
-
xtask/src/cmd/mod.rs— register six new subcommands per §D11.1 -
xtask/src/main.rs— match arms for the six new commands -
xtask/src/cmd/reliability.rs(new) — §D11.2 -
xtask/src/cmd/contracts.rs(new) — §D11.4 -
xtask/src/cmd/mutants.rs(new) — §D11.3 -
xtask/src/cmd/coverage_matrix.rs(new, stub) — §D11.5; full impl in Step 14 -
xtask/src/cmd/invariants.rs(new) — §D11.6 -
xtask/src/cmd/quality_budgets.rs(new, stub) — §D11.1 registration only; full impl in Step 13 -
CONTRIBUTING.adoc— "Optional reliability tooling" subsection §D11.7 -
README.adoc— one-paragraph note in Development section -
.claude/docs/testing.md— "Local subcommands" subsection §D11.7 -
xtask/tests/subcommand_smoke.rs(new) — per-subcommand smoke per §D11.8
Cross-reference: §D11.
-
Edit
cmd/mod.rsper §D11.1 — registers six new modules and six newCommandenum variants. -
Author
cmd/reliability.rsverbatim from §D11.2. -
Author
cmd/contracts.rsverbatim from §D11.4. -
Author
cmd/mutants.rsverbatim from §D11.3 (depends on Step 11 baseline file). -
Author
cmd/coverage_matrix.rsper §D11.5 — stub that delegates to the §D13 implementation (which lands in Step 14). -
Author
cmd/invariants.rsverbatim from §D11.6. -
Author
cmd/quality_budgets.rsper §D11.1 registration; full impl lands in Step 13. -
Edit
main.rsto add six match arms. -
Append "Optional reliability tooling" subsection to CONTRIBUTING.adoc enumerating each subcommand + the entry banner each prints.
-
Add one-paragraph Development-section note to README.adoc.
-
Append "Local subcommands" subsection to testing.md cross-referencing each subcommand and noting they’re opt-in (not pre-push gates).
-
Author
xtask/tests/subcommand_smoke.rsper §D11.8 — three integration tests per subcommand (passing, deliberately-failed, banner-text). -
Verification: run each subcommand once on a clean tree; observe the exact entry-banner text from §D11.
Step 13: Code-quality budget warnings
Files:
-
xtask/src/cmd/quality_budgets.rs— full implementation per §D12.4 -
xtask/quality-budgets.toml(new) — config per §D12.2 -
xtask/quality-budgets.lock(new) — initial baseline per §D12.3 -
xtask/Cargo.toml— addwalkdir = "2",regex = "1",syn = "2"to[dependencies]if not already present -
CONTRIBUTING.adoc— new "Quality budgets" subsection -
xtask/tests/quality_budgets_test.rs(new) — fixtures + snapshot per §D12.6
Cross-reference: §D12.
-
Replace the Step 12 stub in
cmd/quality_budgets.rswith the full implementation per §D12.4. Eachrun_b<N>_*is a directory walk + regex count. -
Author
xtask/quality-budgets.tomlverbatim from §D12.2 (empty exception lists). -
Run
cargo xtask quality-budgets --report --write-lockonce; commit the resultingxtask/quality-budgets.lock. This captures the current counts as the baseline. -
Document in CONTRIBUTING: how to run, what regression means, where to add an exception (with a reason field).
-
Author fixture-based unit tests per §D12.6 covering each budget.
-
Append snapshot test of the rendered table.
-
Verification: run
cargo xtask quality-budgets --reporton a clean tree; observe table matches lockfile values; intentionally add one extraunwrap_or_default()in a test; rerun with--fail-on-regression; observe non-zero exit; revert.
Step 14: Coverage Matrix tooling
Files:
-
xtask/src/cmd/coverage_matrix.rs— full implementation per §D13.4 (replacing Step 12 stub) -
xtask/route-to-client-method.toml(new) — initial mapping per §D13.2; mostly empty, populated alongside Phase A.8 typed-client work -
target/coverage-matrix.md(committed once as baseline) — first-run output snapshot -
docs/modules/ROOT/pages/operations/coverage-matrix.adoc(new) — wraps the markdown table in[source,markdown]; linked fromnav.adocOperations section -
xtask/tests/coverage_matrix_test.rs(new) — fixture per §D13.7
Cross-reference: §D13.
-
Replace the Step 12 stub in
cmd/coverage_matrix.rswith the full implementation per §D13.4. -
Author the empty-but-shaped
route-to-client-method.toml; document the column meanings + the convention for new entries. -
Run
cargo xtask coverage-matrixon a healthy devstack; commit the resultingtarget/coverage-matrix.mdas the baseline (copied intodocs/modules/ROOT/pages/operations/coverage-matrix.adoc). -
Add nav entry under Operations.
-
Author the three fixture tests per §D13.7.
-
Decision documented in the MR description: name-suffix regex tagging, not proc-macro (per §D13.5).
-
Verification: run subcommand against a 3-route mock OpenAPI fixture; observe expected matrix rows and score.
Step 15: DB Constraint Evil Tests — hand-curated harness
Files:
-
crates/craig-test-lib/src/constraints.rs(new) — helper module per §D14.1 -
crates/craig-test-lib/src/lib.rs—pub mod constraints; -
services/craig-cases/tests/constraints.rs+tests/constraints/{case_number_format,contact_type_enum}.rs -
services/craig-placement/tests/constraints.rs+tests/constraints/{placement_status_enum,home_in_use_on_delete}.rs -
services/craig-exchange/tests/constraints.rs+tests/constraints/{partner_in_use_on_delete,icpc_status_enum}.rs -
services/craig-financial/tests/constraints.rs+tests/constraints/{payment_amount_positive,rate_table_unique_name}.rs -
services/craig-reporting/tests/constraints.rs+tests/constraints/{report_run_unique_name,report_kind_enum}.rs -
services/craig-rules/tests/constraints.rs+tests/constraints/{rule_set_unique_name,jurisdiction_enum}.rs -
services/craig-security/tests/constraints.rs+tests/constraints/{partner_unique_code,admin_unit_fips_format}.rs -
services/craig-intake/tests/constraints.rs+tests/constraints/{api_key_revoked,captcha_required}.rs -
.claude/docs/testing.md— append "Constraint tests" subsection -
Backlog GitLab issue (filed in Step 1 of this plan) — full constraint enumeration grown from §D14.3 over time
Cross-reference: §D14.
-
Author the helper module verbatim from §D14.1; export from
lib.rs. -
For each of the 8 services: create
tests/constraints.rsmodule-index file declaring the two#[path = …] modlines. -
Author each of the 16 test files following the §D14.5 sample shape. Each test:
-
Skips if devstack absent (
if !h.devstack_available().await { return; }) -
Boots
TestHarness -
Constructs the typed client
-
Calls
assert_constraint_violation_returns(…)with the constraint-specific request body, expected rejection, and problem-type URL from §D14.4.
-
-
Append testing.md "Constraint tests" subsection: discoverability convention, expected RFC 9457 problem types, link to backlog issue for full enumeration.
-
File backlog issue under epic &22 with the §D14.3 enumeration as the seed; note that future migrations land with their constraint test in the same MR.
-
Verification:
cargo nextest run --workspace -E 'test(::constraints::)'; all 16 pass; runtime <30s under devstack. -
Pushback documentation: the MR description quotes §D14’s first paragraph verbatim — auto-generation rejected, hand-curation chosen.
Step 16: Phase A.15 — Central Evil Input Corpus
Files:
-
crates/craig-test-lib/src/evil/{mod,jwt,upload,string,uuid,json,path,html,unicode,enum_value,date,multipart,signature}.rs(new — 12 modules + mod) -
crates/craig-test-lib/src/lib.rs(addpub mod evil) -
crates/craig-test-lib/Cargo.toml(add deps:inferalready added by platform-stab-2 §D8.1;proptest = "1"for the BiDi/RTLO/Unicode generators;unicode-normalization = "0.1"for NFC/NFD pairs) -
crates/craig-test-lib/tests/evil_corpus_self.rs(new) -
crates/craig-test-lib/tests/evil_corpus_global_contract.rs(new) -
services/craig-{cases,exchange,financial,placement,reporting,rules,security,intake}/tests/api/evil_corpus.rs(new — 8 files, one per service) -
services/craig-web/tests/upstream_evil_corpus.rs(new) -
.claude/docs/testing.md("Evil Input Corpus" section)
Implementation:
-
Create the 12-module tree under
crates/craig-test-lib/src/evil/per §D15.1.jwt.rsandupload.rsimport the existing Phase A.5/A.6 seed modules and emitEvilCaseenum variants — no changes to the seeds. -
Implement the 9 NEW category modules per §D15.3 corpus content. For each:
pub fn cases() → impl Iterator<Item = EvilCase>. Compile-time constants where possible; onlymultipart.rsandsignature.rsneed runtime construction (they wrap mints with timestamps). -
Implement
mod.rs::all_evil_cases()as a chained iterator.evil_cases_for(category)is a thin filter. -
Implement
parametric_evil_test!macro per §D15.4. Macro lives incrates/craig-test-lib/src/evil/mod.rswith#[macro_export]. -
Implement self-test (
tests/evil_corpus_self.rs) per §D15.7. Asserts non-empty per category, unique names, deterministic iteration. -
Implement global contract test (
tests/evil_corpus_global_contract.rs) per §D15.5. Skips if devstack unavailable. -
Wire
parametric_evil_test!into 8 service test trees per §D15.6. Each service file is ~30 LOC.
Verification:
-
cargo nextest run -p craig-test-lib --test evil_corpus_self— passes -
cargo nextest run -p craig-test-lib --test evil_corpus_global_contract— passes (devstack required) -
Per-service:
cargo nextest run -p craig-cases --test evil_corpus— every case rejected with 4xx -
.claude/docs/testing.mdshows new "Evil Input Corpus" section -
cargo clippy -p craig-test-lib --tests --locked — -D warningsclean
Step 17: Phase A.16 — Restart/Replay light helpers
Files:
-
crates/craig-test-lib/src/restart.rs(new) -
crates/craig-test-lib/src/lib.rs(addpub mod restart) -
crates/craig-test-lib/tests/restart_helpers_self.rs(new — meta-test per §D16.6) -
services/craig-{cases,exchange,placement}/src/main.rs(extractpub async fn main_inner(args, env) → anyhow::Result<()>; existingmain()becomestokio::mainwrapper around it — see §Open Questions) -
crates/craig-mq/tests/{outbox_post_publish_crash,inbox_post_handle_crash}.rs(new) -
services/craig-cases/tests/api/attachments_recovery.rs(new) -
services/craig-exchange/tests/api/exchange_send_recovery.rs(new) -
.claude/docs/testing.md("Restart/replay testing" section)
Implementation:
-
Implement
CrashPointenum +CrashHarness+simulate_crash_after+restart_service+assert_recovers_toper §D16.1. -
Implement crash-point wiring table per §D16.2. Each variant maps to a specific Phase A.2 injector configuration. Doc-test enforces wiring exists for every variant.
-
Refactor 8 service
main.rsfiles to extractmain_inner(args, env). The refactor is small (~10 LOC per service) — wrap existing main body, exposing the dependency-injection seam to tests.main()becomes a 5-linetokio::mainwrapper. -
Implement 4 paired tests (one per platform-stab-2 fix with a recovery sweep) per §D16.4.
-
Implement self-test per §D16.6.
-
Document the pattern in testing.md.
Verification:
-
cargo nextest run -p craig-test-lib --test restart_helpers_self— passes -
cargo nextest run -p craig-mq --test outbox_post_publish_crash— passes -
cargo nextest run -p craig-mq --test inbox_post_handle_crash— passes -
cargo nextest run -p craig-cases --test attachments_recovery— passes -
cargo nextest run -p craig-exchange --test exchange_send_recovery— passes -
Each service still builds;
main_innerextraction is binary-compatible -
.claude/docs/testing.mdshows new "Restart/replay testing" section
Step 18: Phase A.17 — Invariant Sweeper xtask
Files:
-
xtask/src/cmd/invariants.rs(new) -
xtask/src/cmd/mod.rs(export + dispatch) -
xtask/src/cmd/test.rs(add--post-checkflag) -
xtask/Cargo.toml(no new deps — sqlx + clap already present) -
crates/craig-test-lib/sql/invariants/<svc>/*.sql(new — initial 8 invariants seeded across 8 services per §D17.4) -
crates/craig-common/src/constants.rs(extend with grace constants from §D17.5) -
xtask/tests/invariants.rs(new — catalog parser self-test per §D17.8) -
.claude/docs/testing.md("Invariant Sweeper" section)
Implementation:
-
Implement
xtask/src/cmd/invariants.rsper §D17.2. SQL loaded at runtime; metadata header parsed via simple string splits (no TOML dep — header is a tiny dialect). -
Seed initial 8-invariant catalog per §D17.4. Each is a single
.sqlfile with comment header. The "audit_row_per_state_mutation" invariant ships seeded for cases / placements / exchange / financial / security only; remaining 3 services ship aTODO.sqlplaceholder that returns 0 rows. -
Add grace constants to
crates/craig-common/src/constants.rsper §D17.5. -
Wire
cargo xtask test --post-checkto invoke the sweeper after the test battery returns. -
Implement xtask self-test per §D17.8: parse every
.sql, validate header metadata, assert grace literals match constants, run each against empty DB. -
Document in testing.md, including the report-only contract.
Verification:
-
cargo nextest run -p xtask --test invariants— catalog self-test passes -
cargo xtask invariantsagainst a clean dev DB — exits 0, prints all-pass -
cargo xtask invariantsagainst a DB seeded with one orphan row — exits 1, prints failure with sample row + remediation pointer -
cargo xtask test --post-checkruns the test battery + sweeper end-to-end -
.claude/docs/testing.mdshows "Invariant Sweeper" section
Step 19: Phase B — adoption bundle (10 tests)
Files:
-
services/craig-api/tests/idempotency_concurrent_same_body.rs(new — test 1) -
services/craig-api/tests/idempotency_different_body_422.rs(new — test 2) -
services/craig-api/tests/idempotency_finalize_failure_returns_live.rs(new — test 3) -
crates/craig-mq/tests/outbox_two_workers_no_double_publish.rs(new — test 4) -
crates/craig-mq/tests/inbox_failure_doesnt_suppress_redelivery.rs(new — test 5) -
services/craig-cases/tests/api/upload_db_failure_no_blob.rs(new — test 6) -
services/craig-exchange/tests/api/send_crash_recoverable_state.rs(new — test 7) -
services/craig-web/tests/upstream_malformed_renders_error.rs(new — test 8) -
services/craig-store/tests/filename_disposition_evil.rs(new — test 9) -
services/craig-cases/tests/api/typed_client_pilot.rs(new — test 10) -
docs/modules/ROOT/pages/test-coverage-scorecard.adoc(new — top-20 workflows × 6 cells) -
docs/modules/ROOT/nav.adoc(Architecture & Design entry pointing at scorecard) -
xtask/src/cmd/coverage_matrix.rs(extend with--emit-scorecardflag) -
CHANGELOG.adoc(per-test + scorecard entries)
Implementation:
-
Each test follows the helper pairings from §DB.1.
-
Each test file’s first
#[tokio::test]is the failing-test-first artifact; the test name mirrors the GitLab issue title for one-grep traceability. -
The scorecard is hand-curated for the workflow rows; the cell values are auto-emitted from
cargo xtask coverage-matrix --emit-scorecardreading test-name suffix tags. -
Cross-reference each test to its sibling platform-stab-2 step in the test file’s module-level doc comment:
//! Paired with platform-stab-2.adoc §D1 (Step 3) — failing-test-first artifact.
Recommended sub-commit ordering (single MR, but per-commit reviewer-friendly):
-
Commit 1: Test 1 (concurrent same-body idempotency — Phase A.1 canary). Validates the consolidated
concurrent_firehelper end-to-end against the platform-stab-2 §D1 fix. -
Commit 2: Test 2 (different-body 422). Tightens the §D1 contract.
-
Commit 3: Test 3 (idempotency finalize-failure returns live response). Pairs with §D1 error mapping.
-
Commit 4: Test 4 (two outbox workers, no double publish). Pairs with platform-stab-2 §D2.
-
Commit 5: Test 5 (inbox handler-failure-then-retry). Pairs with platform-stab-2 §D3.
-
Commit 6: Test 6 (object-store + DB-failure → no blob orphan). Pairs with platform-stab-2 §D4.
-
Commit 7: Test 7 (exchange crash → recoverable state). Pairs with platform-stab-2 §D5; uses Phase A.16 restart helpers.
-
Commit 8: Test 8 (BFF malformed-upstream renders error). Pairs with platform-stab-2 §D7.
-
Commit 9: Test 9 (filename + Content-Disposition evil corpus). Pairs with platform-stab-2 §D8.1; uses Phase A.15 evil corpus.
-
Commit 10: Test 10 (typed-client pilot). Pairs with Phase A.8.
-
Commit 11: Top-20 risk-coverage scorecard authoring + nav.adoc xref + xtask
--emit-scorecardflag. -
Commit 12: CHANGELOG entries (one per test + one for scorecard) + final scorecard run capturing the 10 new tests' cell flips.
Verification:
-
cargo nextest run --workspaceincludes all 10 new test files; each passes against the post-platform-stab-2 binaries -
cargo xtask coverage-matrix --emit-scorecardproduces a non-empty auto section -
cargo xtask coverage-matrix --diff mainshows +N filled cells (one per Phase B test) and zero regressions -
Scorecard renders in Antora without errors
-
nav.adoc Architecture & Design section shows the scorecard link
-
Manual: review scorecard against ops risk register; confirm top-20 workflows are accurate
-
CHANGELOG shows one entry per test + one for the scorecard
Step 20: Plan completion audit + archive
Files: docs/modules/ROOT/pages/plans/test-framework-hardening.adoc (Status table → all Complete), docs/modules/ROOT/nav.adoc (move from Active to archive), docs/modules/ROOT/pages/plans/archive.adoc (new row under Infrastructure & Reliability), .claude/CLAUDE.md (Phase Status — add "Test Framework Hardening" row), CHANGELOG.adoc (wrap-up entry).
-
Spawn
plan-completion-auditsubagent per delivery-protocol.md -
Verify all 19 prior steps complete via MR list
-
Flip Status table all-Complete
-
Move plan from nav.adoc Active to archive.adoc Infrastructure & Reliability section
-
Update CLAUDE.md Phase Status (add "Test Framework Hardening" row with helper count + test count + scorecard link)
-
CHANGELOG wrap-up entry
-
Close epic &22
Files Touched
| File | Change |
|---|---|
|
Plan file (this content). Step 1 expansion of the prior thin draft. |
|
Active section: confirm test-framework-hardening xref present (already added in prior MR). |
|
Step 20 — new row under Infrastructure & Reliability with all step MR numbers. |
|
Step 19 — new file. Top-20 workflows × 6-cell grid. |
|
Entry per step (Steps 1–20). |
|
Step 2 — |
|
Step 3 — extends platform-stab-2 inline injectors with |
|
Step 5 — pattern helper for state-machine matrix testing; first adoption: Payment. |
|
Step 6 — JWT mutation library (kept callable as-is by platform-stab-2 §D5). |
|
Step 7 — magic-byte + filename-attack fixtures (kept callable as-is by platform-stab-2 §D8.1). |
|
Step 16 — central evil input corpus (12 modules + mod). Wraps Step 6/7 seeds. |
|
Step 17 — restart/replay light helpers. |
|
Step 14 — DB constraint evil-test harness (curated; sibling to D17 sweeper). |
|
Re-exports for every new module (Steps 2/3/5/6/7/16/17). |
|
Deps: |
Workspace |
Add |
|
Step 18 — per-service invariant catalog (initial 8 invariants). |
|
Step 18 — grace constants used both by codepaths and by the invariant sweeper SQL literals. |
|
Step 11 (deliverable) — surfaces concurrency/fault/recovery test results in a per-axis report. Report-only. |
|
Step 11 — contract-tests harness (consumer-driven contracts). Report-only. |
|
Step 12 — |
|
Step 12 — coverage-matrix report keyed by test-name-suffix tags + |
|
Step 18 — invariant sweeper. Report-only. |
|
Step 13 — quality-budgets aggregator (LOC, cyclomatic, fn count). Report-only. |
|
Wiring for the 6 new subcommands; |
|
Step 16 — adoption: |
|
Steps 17 + 19. |
|
Steps 17 + 19. |
|
Steps 17 + 19. |
|
Step 19. |
|
Steps 16 + 19. |
|
Step 19. |
|
Step 17 — extract |
|
Steps 8 + 16 + 17 + 18 — Test Categories taxonomy + helper docs + Evil Input Corpus + Restart/replay testing + Invariant Sweeper sections. |
|
Step 8 — failure-path-test process rule. |
|
Step 8 — pointer to testing.md test-categories. Step 11 — "Optional dev tools" cargo-install line for cargo-mutants. Step 12 — "Optional reliability tooling" subsection enumerating the 6 new xtask subcommands. Step 13 — "Quality budgets" subsection. |
|
Step 20 — Phase Status row added. |
|
Step 19 — Architecture & Design entry pointing at test-coverage-scorecard. Step 20 — move plan to archive. |
| Step 1 of THIS plan files 9 GitLab issues for the new Phase A pieces (A.9 proptest; A.10 cargo-mutants smoke; A.11 xtask subcommands reliability/contracts/mutants/coverage-matrix/invariants/quality-budgets; A.12 code-quality budget warnings; A.13 coverage matrix tooling; A.14 DB constraint evil-test harness; A.15 central evil corpus; A.16 restart/replay light helpers; A.17 invariant sweeper). These join the existing 8 Phase A issues + 1 Phase B bundle (#283–#291) for a total of 18 step issues + 1 audit issue under epic &22. |
Verification
-
cargo nextest run --workspace— full unit + integration battery passes including the 10 Phase B adoption tests + 8 service evil-corpus suites + restart-helpers self-test + invariant sweeper self-test. -
cargo xtask reliability— surfaces concurrency / fault / recovery test results per axis; report-only. -
cargo xtask contracts— consumer-driven contract tests pass; report-only. -
cargo xtask mutants --smoke— surviving-mutant report; report-only (defaults flagged in §Open Questions). -
cargo xtask coverage-matrix --report— per-category coverage report produced from test-name-suffix tags. -
cargo xtask invariants— exits 0 on a clean dev DB after a successful e2e run; exits 1 with a remediation pointer when an invariant is violated. -
cargo xtask quality-budgets --report— LOC / cyclomatic / fn-count per crate; report-only. -
Manual: kill craig-cases mid-blob-write (paired with platform-stab-2 §D4 manual verification step); restart; assert scanner reaps within 5 min.
-
Manual: stage 200 concurrent same-key POSTs against
POST /v1/cases(paired with platform-stab-2 §D1 manual step); assert exactly one handler invocation via the side-effect counter. -
cargo xtask coverage-matrix --emit-scorecardruns and produces a non-empty auto-section indocs/modules/ROOT/pages/test-coverage-scorecard.adoc; the page renders in Antora without errors. -
cargo xtask api-docs— Antora docs build without error including new pages (plan adoc, scorecard, archive row). -
cargo xtask validate --skip-docker— fmt + clippy + cargo-deny clean.
Documentation Updates
-
.claude/docs/testing.md— Test Categories taxonomy (happy / sad / evil / concurrency / replay / fault) + per-helper usage docs + reviewer-discipline process rule (Step 8 deliverable). Plus Evil Input Corpus section (Step 16), Restart/replay testing section (Step 17), Invariant Sweeper section (Step 18). -
.claude/docs/delivery-protocol.md— failure-path-test process rule under Delivery Checklist: "every concurrency-relevant fix ships with a failing-test-first artifact paired with the failing-test-first directory in test-coverage-scorecard.adoc". -
CONTRIBUTING.adoc— pointer to.claude/docs/testing.mdTest Categories. -
CHANGELOG.adoc— entry per step (20 entries; the audit step is a wrap-up summary). -
docs/modules/ROOT/pages/test-coverage-scorecard.adoc— top-20 workflows × 6-cell grid (Phase B Step 19 deliverable). -
docs/modules/ROOT/pages/plans/archive.adoc— Infrastructure & Reliability row added on Step 20 with all step MR numbers. -
docs/modules/ROOT/nav.adoc— Architecture & Design entry for scorecard (Step 19); plan moved from Active to archive (Step 20). -
.claude/CLAUDE.md— Phase Status table: add "Test Framework Hardening" row (Step 20).
Open questions
-
Coverage matrix tagging convention. Two viable options:
-
Regex on test name — suffix tags
_happy,_sad,_evil,_concurrent,_replay,_fault. Coverage-matrix xtask scanscargo test — --listoutput and bins by suffix. No proc-macro dep, no compile-time machinery. Loose discipline — typo’d suffix → silently uncategorized. -
#[test_category(…)]proc-macro — strict, IDE-discoverable, compile-time-checked. Addscraig-test-macrosproc-macro crate dep.Default proposal: regex tagging. Reason: keeps deps lean; the looseness is acceptable because the scorecard is a living artifact maintained by reviewer discipline, not a CI gate. Revisit if the suffix-typo rate exceeds 1% in W1.
-
-
Quality-budgets enforcement. Initially report-only (warnings in xtask output, not pre-push gate). The user directive is explicit: D11/D12 + invariants + quality-budgets ship report-only. Default proposal: defer enforcement to a future plan after 90 days of report-only operation. Concretely: a follow-up issue filed in W3 of this plan to revisit at the 90-day mark; promotion to enforced gate requires a second plan with explicit Status-table entry, not a silent flip.
-
proptest case count default.
proptest’s own default is 256 cases per property. Default proposal: keep 256 globally, raise to 1024 specifically for canonical-JSON signing properties (where the bug surface is wide and determinism is high). Configurable per `proptest! { #![proptest_config(ProptestConfig::with_cases(1024))] }on individual properties; do not override the global default inproptest.toml. -
Mutation testing budget.
cargo-mutantsproduces a "surviving mutants" count per crate. Where do we draw the line? Default proposal: 0 surviving mutants incraig-authandcraig-mqcore paths (the security + concurrency boundaries); ≤5 elsewhere. The budget is a report-only warning in W3, not an enforced gate. Revisit at the 90-day post-report-only checkpoint together with §Open Question 2. -
Restart helpers vs full mock-binary. How does a test drive a service’s
main()without spawning a real process? Default proposal: extractpub async fn main_inner(args: Args, env: Env) → anyhow::Result<()>from each service’smain.rs(~10 LOC refactor per service), so testsrestart_service(|| async { craig_cases::main_inner(test_args(), test_env()).await })directly with mocked dep injection. The existingtokio::mainbecomes a 5-line wrapper aroundmain_inner. Binary-compatible; no behavioral change in production. Filed as part of Step 17. -
Phase B test 4 — multi-replica vs single-process simulation. Two outbox workers can be exercised by either (a) Phase A.3 multi-replica devstack (
cargo xtask dev start --replicas craig-cases=2) or (b) single-process simulation (twoOutboxWorkerinstances against a sharedPgPoolin one test process — the platform-stab-2 §D2 model). Default proposal: prefer single-process simulation as the primary test for fast feedback; multi-replica devstack ships as the secondary end-to-end smoke that runs only undercargo xtask e2e. Two tests, one fast and locally runnable, one slow and CI-only. -
proptest version pin — 1.7 was current as of 2026-05; verify against the registry on commit and bump in this MR if 1.8 has shipped.
-
cargo-mutants 25.x release line — confirm
^25.0resolves to a stable release; fall back to a hard-pin (=25.X.Y) if patch-version churn proves disruptive in the smoke run. -
Coverage-matrix axis tagging at scale — name-suffix regex is fine at 1259 tests; revisit when untagged-test count exceeds (e.g.) 200 untagged-in-
tests/api/. -
Quality-budgets B6 (duplicate dependencies) — current zen-engine + jsonwebtoken dupes (per platform-stab-2 §D10.2) are the current state; the lockfile captures them as the floor. Confirm with reviewer that "current dupe count = floor" is the correct semantics for the seed lock.
-
Constraint test backlog seeding — §D14.3 enumeration is indicative. The Step 1 backlog issue must be filed before Step 15 lands so seed contributors can append rather than fork the canonical list.
-
xtask subcommand naming —
reliabilityvsreliability-suitevsrel. Default proposal:reliability(matches existing one-word subcommand convention:dev,test,e2e,validate,perf,security,reconcile).
Potential improvements
Documented per pre-commit Q6 — concrete improvements identified during implementation that are NOT in scope for this plan but worth tracking:
-
cargo xtask reliability— concurrency-and-fault selectivity flags. Today the suite runs every category in one shell-out (concurrency | recovery | fault | invariants | properties). Once Phase B adoption (Step 19) lands enough category-specific tests, splitting into--concurrency-only/--fault-only/--recovery-onlyflags would let contributors target the slowest category for iterative debugging without rerunning the rest. File a follow-up after Step 19 ships. -
cargo xtask mutants --smoke— per-crate parallelism.run_oneinvokescargo mutants --jobs 1per crate sequentially. The 5 critical crates could fan out across cores if mutation testing sees regular use; deferred until the Step 11 follow-up clean-tree population run shows the actual wall-time delta. -
cargo xtask contracts --service <name>— CSV form. Today--service casesfilters to one service; a CSV (--service cases,exchange) would let CI dashboards group services per pipeline shard. Trivial nextest-filter join; deferred until contract test counts cross ~50 per service (Phase B). -
cargo xtask quality-budgets— exception lists not yet enforced. §D12.1 specifies that each budget has a per-path exception list (xtask/quality-budgets.tomlalready declaresexceptions = []arrays). The catalogue parser reads them but the counters do not yet honour exception entries — every match within the configured glob counts. Adding a path-glob exception walker is straightforward but unnecessary today (zero exceptions are populated). Implement when the first reviewer-approved exception lands so the schema and the enforcement arrive together. -
B6 distinct-package count vs. version-pair count. B6 today is "distinct package names with multiple versions" (67). An alternative metric is "total duplicate (name, version) pairs minus distinct names" — i.e. the number of redundant installs. Both are defensible; the current choice is more stable across cargo updates because added/removed minor versions of one package only swing the count by 1. Worth re-evaluating if a future MR consolidates a duplicated package and the count drops by an unintuitive amount.
-
cargo xtask coverage-matrix --diff <REF>. §D13.7’s deferred flag. Re-running the matrix againstmainand printing cell flips (+ POST /v1/cases :: evil,- DELETE /v1/cases/{id} :: happy) gives reviewers a per-MR delta view that no other tool produces. Implementation needs a cleangit stash+git checkout+ re-run dance — workable, but the working-tree contract pushed it out of Step 14. File a follow-up once the typed-method bridge populates non-zero cells. -
Per-service evil-corpus adoption files (§D15.6) — RETIRED. Step 16 originally deferred the 8 service-side
tests/api/evil_corpus.rsfiles; the post-Step-17 review-fix MR !212 implemented all 8 with appropriateskip_categorieslists where the handler doesn’t validate today. Each skipped category is a documented finding tracked under #311 (RFC 9457 standardization initiative). The BFFservices/craig-web/tests/upstream_evil_corpus.rsis the one piece still deferred — it depends on platform-stab-2 §D7’s BFF-upstream rendering work and will land as part of that initiative. Net: this §Potential improvements bullet is closed; the BFF upstream test remains as a distinct platform-stab-2 follow-up. -
Constraint test seed expansion to 2 tests/service (16 total) — RETIRED. §D14.4 listed 16 tests; Step 15 originally shipped 8. The post-Step-17 review-fix MR !212 implemented the second 8 (one per service), bringing the total to 16/16 passing — the original §D14.4 promise is now fully delivered.
-
RFC 9457 problem-type URL standardization. Today only 2 problem-type URLs are emitted across the entire codebase (
idempotency-conflict,payload-too-large— both incraig-api/idempotency.rs). The §D14.4 design asks each constraint test to assert a per-violation type URL; that requires defining ~20 type URLs and wiring each handler’s rejection path to emit them. This is its own multi-MR initiative and was scoped out of Step 15. The constraints helper supports both modes (status-only today, type-URL when handlers ship them) so adoption is incremental once the URLs are defined. -
cargo xtask coverage-matrixtyped-method bridge. The committedxtask/route-to-client-method.tomldeclares the entry shape but has zero entries; thescan_typed_calls()consumer is unimplemented. Today the matrix only matches literal URL strings in test source, which the typed-client migration removed from most tests — hence the honest 0/1308 baseline. Implementing the typed-method bridge requires (a) populating the TOML alongside Phase A.8 typed-DTO migrations and (b) adding a syn-based call-site scanner to the subcommand. Land both together when the typed migration crosses ~50% adoption.
Errata
Implementation deltas vs. the original plan, captured for reviewer clarity:
-
Step 2 — built from scratch, not extended. §D1 framed Phase A.1 as "in-place extension of platform-stab-2 Step 2’s minimal helper" because the sibling plan was expected to land its inline
concurrent_firefirst. Per the user’s directive to do test-framework-hardening before platform-stab-2 (so platform-stab-2’s P0 fixes ship with proper coverage from the start), there was no minimal prefix to extend; the consolidated surface was authored from scratch incrates/craig-test-lib/src/concurrent.rs. The deprecatedconcurrent_firealias is retained per §D1.1 to keep platform-stab-2’s authored-but-unmerged tests forward-compatible. No semantic difference — same module, same surface, same test set. -
Step 19 — 3 of 10 adoption tests shipped; 7 platform-stab-2-dependent tests deferred. §DB.1 lists 10 specific tests. Tests 1-7 each require a platform-stab-2 fix or store-trait seam to exist before the test has anything to exercise: Tests 1-3 need the persistent idempotency middleware (§D1); Test 4 needs the multi-replica outbox infrastructure (§D2); Test 5 needs the three-state inbox (§D3); Test 6 needs the atomic blob+DB+event upload semantics (§D4); Test 7 needs the exchange send-job state machine (§D5). All 7 are filed as separate GitLab issues. Tests 8-10 ship today: Test 8 (BFF malformed upstream JSON → service error not silent empty data) lands as 3 unit tests in
services/craig-web/src/api_client.rs::testsexercising malformed JSON + empty body + upstream 5xx via wiremock; Test 9 (filename + Content-Disposition fuzzing) lands as 3 integration tests incrates/craig-store/tests/filename_evil_corpus.rslooping the central evil corpus’s String/Path/Unicode categories againstsanitize_filenameand asserting post-sanitize invariants (no/, no\, no\0, no..segment, idempotent); Test 10 (typed-client pilot) lands as 7 round-trip + boundary-pinning tests incrates/craig-cases-contracts/tests/typed_dto_pilot.rsexercising every public DTO across persons/cases/contacts/referrals plus thedeny_unknown_fields=offandunknown-string-acceptedposture pins. The risk-coverage scorecard atdocs/modules/ROOT/pages/test-coverage-scorecard.adoc+ the newcargo xtask coverage-matrix --emit-scorecardflag close out §DB.2/§DB.3. -
Step 18 — initial invariant set is 4 (3 cross-service + 1 exchange-specific); §D17.4’s full table is platform-stab-2-dependent and ships incrementally. §D17.4 lists 8 invariants in the initial catalog. Of those, 4 reference platform-stab-2 schema that doesn’t exist today:
orphan_pending_attachmentsneedscontact_attachments.object_status(platform-stab-2 §D4);stuck_in_flight_send_jobsneeds asend_jobstable that doesn’t exist;inbox_processed_with_failedneeds anevent_inbox.failed_atcolumn that doesn’t exist (today’s table has onlyprocessed_at);idempotency_processing_past_ttlneedsidempotency_responses.{status, claim_expires_at}that don’t exist (today’s table has justexpires_atfor cleanup). Step 18 ships the framework + the 4 invariants that map to today’s actual schema:outbox_unpublished_beyond_grace,inbox_unprocessed_beyond_grace,idempotency_expired_not_cleaned(cross-service incraig-mq/), andstuck_pending_transactions(incraig-exchange/). The runtime SKIP-on-42P01handling means as platform-stab-2 lands the missing schema, new SQL files dropped into the catalog light up automatically without runner changes. Audit-row invariant (§D17.6) is unimplemented; flagged in §D17.6 as "ships in W3 hardening passes" — deferred. The live sweep against the running devstack already paid for itself: it surfacedcraig-placementmissing theevent_inboxmigration that the other 6 stateful services received — filed as a separate platform finding (the other services'20260430173943_event_inbox.sqlmigration is absent underservices/craig-placement/migrations/). The sweeper’s runtime SKIP correctly classified this as "feature-rollout gap" rather than a 1-failure ceiling violation. -
Step 17 — both
CrashHarnessand a bare-closure entry shipped; named CrashPoint variants pair with the 3 fault injectors that exist today, with the 4 store-trait injectors lighting up as platform-stab-2’s §D2.0 seam refactor lands. §D16.1 specified one entry point passing aCrashHarnessto the builder. Step 17 ships two entry points:simulate_crash_after_with_harness(point, builder)matches §D16.1 (builder receivesCrashHarness, requires live devstack) and is the canonical adoption shape for §D16.4’s platform-stab-2-paired tests;simulate_crash_after(point, builder)is a bare-closure entry that the helper’s own self-tests use to exercise the orchestration logic without devstack.CrashHarnesswrapsTestHarnessfor client construction, captures diagnostic artifacts viarecord_artifact, and exposescrash_now(msg)for manual aborts plusarm(injector, scenario)for fault-injection. The 3 wrapper injectors that ship today (Latency,IntermittentFlap,ConsumerBackpressure) are armable throughCrashHarness::armimmediately; the 4 store-trait injectors §D16.2 references (FaultyAttachmentStore,FaultyOutboxStore,FaultyInboxStore,FaultySendJobStore) light up as platform-stab-2’s §D2.0a/§D2.0b store-seam refactor introduces them. Earlier draft of this MR shipped only the bare-closure entry on the (lazy) reasoning that the named-variant injectors don’t exist yet; corrected after user pushback because the façade itself — TestHarness wrapping, artifact capture, manualcrash_now— is useful regardless of which injectors back the named variants. 10 self-tests cover both entry points (8 bare + 2 harness-integration that skip without devstack). -
Step 16 — corpus + global contract + macro shipped; per-service adoption files deferred to backlog. §D15.6 lists 8 service-side adoption files (
services/craig-<svc>/tests/api/evil_corpus.rs) plus the BFF upstream test (services/craig-web/tests/upstream_evil_corpus.rs). Each adoption file is mechanical fan-out using theparametric_evil_test!macro, but the per-service "endpoint:" closure needs raw-JSON-accepting client paths that don’t uniformly exist (cases is fully typed-DTO post Step-9 redo; financial / reporting / placement / security have a Value-accepting method per major resource). Authoring 8 files of ~30 LOC each + verifying each against a live devstack would have doubled the MR’s footprint without adding new contract surface — the global contract test (evil_corpus_global_contract.rs) already exercises every JSON/string/UUID/path/html/unicode/enum/date case end-to-end against the public-intake endpoint, which §D15.5 calls "the single test that establishes the boundary contract globally." Per-service fan-out is tracked in §Potential improvements. Three tactical adjustments inside the corpus itself: (1)string.overlong_16mbreduced tostring.overlong_4mbto keep self-test wall-time under 1s while preserving the "above-cap-but-below-DoS" alarm shape; (2)json.depth_bomb_*cases emit pre-serialized bytes rather than constructedserde_json::Valuetrees because Value’sDroprecurses and overflowed the test-thread stack at depth 4096; (3) theparametric_evil_test!macro skips applicable-category checks in-test rather than gating expansion at compile-time, because category applicability depends on which Axum extractor the endpoint uses (Json vs Multipart vs raw Bytes) and is more honestly expressed as runtime configuration. -
Step 15 — seed reduced from 2 tests/service (16) to 1 test/service (8); RFC 9457 type-URL portion of the helper deferred; signature simplified. §D14.4 listed 16 specific tests with §D14.1’s
assert_constraint_violation_returns(client, request_closure, ExpectedRejection, problem_type_url)— closure-based, requiring a genericTypedClienttrait that doesn’t exist in the codebase, and assuming RFC 9457 problem-type URLs that also don’t exist (only 2 such URLs are emitted across the entire codebase today, both incraig-api/idempotency.rs). The signature was simplified to two response-shape helpers:assert_constraint_violation_status(actual_status, expected, context)for status-only assertion andassert_constraint_violation_returns(actual_status, body_type, expected, problem_type_url, context)for the full-fidelity assertion. The seed shipped is 1 test per service (8 tests total) against real handler rejection paths verified against a live devstack. Each test surfaces a real boundary (FK lookup failure, UNIQUE-conflict mapping to 409, typed-Json-extractor 422 on bad enum, etc.). The §D14.4 list assumed several validation behaviors that don’t exist in the codebase today — e.g.,cases.admin_unitis not validated against the FIPS registry on create;financial.payments.amount > 0is not enforced as a CHECK constraint at the DB layer (financial has zero user-driven DB constraints);cases.case_numberUNIQUE collision is auto-retried server-side and not surfaceable through the API. The second-test-per-service expansion + the type-URL standardization initiative are tracked in §Potential improvements; both are real platform-improvement candidates that this plan should not have absorbed. -
Step 14 —
--diff <REF>flag deferred; route-to-client-method bridge ships as a stub config file. §D13.7 specified a--diff mainflag that runs the matrix twice (current + base) and prints filled/empty cell deltas. Implementing it requires a cleangit stash+git checkout+ re-run dance from inside the subcommand, which conflicts with the user’s working-tree convention (running tools while holding uncommitted changes). The flag is not mission-critical for the baseline — the per-MR review signal it provides can be obtained by hand-diffing two runs. Deferred to a follow-up MR; tracked under §Potential improvements. Separately, §D13.2 specified aroute-to-client-method.tomlmapping that lets the matrix count typed-method invocations (not just literal URL strings). The mapping’s file ships in this MR with the agreed-upon entry shape and zero entries; the consumer (ascan_typed_calls()step alongsidescan_tests()) lands when the Phase A.8 typed-DTO migration is far enough along to populate the mapping with real entries. Today the matrix only matches literal"/v1/…".to_string()references in test source, which is why the baseline reports 0/1308 cells — current tests use typed-client method calls, not URL literals. The 0% baseline is honest signal, not a tooling bug. -
Step 13 —
synspan locations swapped for a streaming brace-balance scan in B2; B7 regex broadened to bareValue. §D12.4 sketched B2 (function-LOC over threshold) usingsynto walkItemFn/ImplItemFnand readingSpan::start()/end(). Those positions requireproc-macro2’s `span-locationsfeature, which the workspace doesn’t enable. Implemented B2 with a streaming text scan that tracks brace depth, skips strings/chars/line+block comments/raw strings, and identifies fn-signature braces by walking backward from{looking for thefnkeyword. Three unit tests cover the basic case, brace-in-string false-positive guard, and nested-closure handling. B7 (untyped test-client methods) was originally aserde_json::Valueregex on return types, but the repository convention isuse serde_json::Value;at the top of each clients/*.rs file followed by bareValuein signatures. B7 now also matches bareValuewhen an import is in scope — which is what bumped the actual count from 0 (false negative) to 194 (true count). B6 (duplicate dep versions) also hit a counting bug — the originally specifiedcargo tree -d --prefix none(without--depth 0) returned the full reverse-dep tree under each duplicate, inflating counts; switched to--depth 0which returns only the duplicate roots, then dedup’d by package name in aBTreeSet. -
Step 12 —
coverage-matrixandquality-budgetsship as stubs;mutants-baseline.tomlpopulation still deferred to a follow-up clean-tree run. §D11 frames Step 12 as wiring 6 stable entry points; the two report-generators (coverage-matrixandquality-budgets) belong to later steps (§D13 / §D12 respectively) and are intentionally registered as stubs in this MR so callers (CI dashboards, contributors, scripts) can pin a stable command name today. Each stub prints aSTUB:line naming the step that finishes it. The four runner subcommands (reliability,invariants,contracts,mutants) are fully functional. Themutants-baseline.tomlpopulation that Step 11’s errata deferred to "after Step 12 lands" is still pending — it requires a clean working tree and a singlecargo xtask mutants --smokerun; that’s a separate housekeeping commit, not a code change, and is filed against the Step 11 follow-up backlog rather than rolled into this MR. -
Step 11 — config + empty baseline shipped; population deferred until Step 12’s xtask subcommand exists. §D10.5 Step 5 calls for running
cargo xtask mutants --smokeand committing the populatedxtask/mutants-baseline.toml. The xtask subcommand itself is Step 12 work (per §D10’s own cross-reference: "Subcommand wiring lives in Step 12 / §D11.3"). Step 11 ships the infrastructure:.cargo/mutants.tomlworkspace config (timeout, nextest runner, exclude_globs),xtask/mutants-baseline.tomlas an empty template with the entry-shape spec,CONTRIBUTING.adoc"Optional dev tools" install line,.claude/docs/testing.mdMutation testing sub-section. The baseline gets populated immediately after Step 12 lands (a singlecargo xtask mutants --smokerun on a clean working tree, committed as the floor). Documenting this seam keeps the plan honest about the dependency. -
Step 10 — Content-Disposition property deferred; sanitize_filename + pagination reframed against existing fns; reference-enum list corrected; surfaced + fixed a real
sanitize_filenamenon-idempotency bug. §D9.6 (Content-Disposition encoding round-trip) targetscraig_store::validation::content_disposition_attachmentwhich is platform-stab-2 §D8.1 work and not yet shipped — deferred. §D9.4 wrote the pagination property against apaginate(&data, PageRequest)helper that doesn’t exist in the codebase; reframed againstPageRequest::offset/limitarithmetic (the actual SQL-layer commitment), same invariants. §D9.9 enumeratedPaymentStatus,PlacementStatus,ReportStatus,AdminUnit,FipsCode,EventType— none of those types exist incraig-reference; replaced with 11 actualEnumString-derived enums (Gender, Race, Ethnicity, ReporterType, Priority, AbuseType, Disposition, ContactType, CourtOrderType, PlacementType, PermanencyGoal). The §D9.5prop_sanitize_filename_invariantsproperty surfaced a real non-idempotency bug on first run: leading-.-then-\tsequences had\tsurvive the firstsanitize_filenamecall (becausetrim()ran beforetrim_matches('.'), while the dot was still leading). Fixed in the same MR by replacing the two-pass trim with a single combined predicate (c.is_whitespace() || c == '.') and adding a post-truncation re-trim pass for the case where 255-byte truncation creates a trailing whitespace. Property bit; production code corrected.body_fingerprintincraig-api/src/idempotency.rswas promoted fromfntopub fnper §D9.8 so out-of-crate property tests can lock the equality and one-byte-flip invariants — the value was already part of the cross-service contract via the persistedidempotency_responses.body_fingerprintcolumn. -
§D8 batch — IntakeClient (#307) excluded from the typed-DTO test-client migration by design (2026-06-20). §D8.3’s inventory listed 8 clients to type, row 8 being
IntakeClient, on the assumption it just needed "a builder method per auth posture." Implementing the sibling migrations (#303 financial / #304 reporting / #305 rules / #306 security — all landed) revealed a category error in that framing:IntakeClientis the test suite’s untrusted-ingress simulator, not a trusted-write client. Itssubmit_public_reportis the anonymous public-intake endpoint and the target of the evil-corpus contract test — theEvilCategory::Jsoncase (crates/craig-test-lib/src/evil/intake_body.rs) sends arbitrary non-object JSON that a typed&SubmitReportRequeststructurally cannot carry, and the partner submit methods are likewise probed with malformed bodies + JWS byte-fidelity (jws_error_uniformity,sanitization,cross_partner_bola). Typing the client would delete the capability those security tests exist to exercise. The correct architecture is already in place: the server owns the typed contract —submit_report(Json(_): Json<SubmitReportRequest>)deserializes-and-validates untrusted bytes (or 400s) — and the test client, like a real external caller, correctly speaks rawValue. A typed-happy-path + raw-_raw(&Value)-escape-hatch split was considered and rejected as a fig leaf. Scope correction: the §D8.3 "8 clients" inventory and the §D8.5 "0 untyped test clients post-migration" budget stand corrected to trusted-write clients only —IntakeClientremains intentionally&Value. Dead intake client methods (register_signer_key/get_signer_key_public/list_organizations, targeting/keysroutes retired to craig-security per ADR-017 Step 6) and the craig-web↔intakeSubmitReportRequestshape-duplication are filed as #636 / #637. (#307 closed not-applicable-by-design.) -
Step 9 redo — full DTO lift; CasesClient method migration scoped to follow-up #308. The original Step 9 MR (!200) shipped only a persons-module skeleton — 1 of 12 DTO modules, 0 method swaps, 0 test-callsite rewrites. Per the user’s pushback ("are you being lazy and deferring things that don’t need to be deferred?"), Step 9 was redone: this MR lifts all 28 request-side DTOs spanning 9 modules (case_plans, cases, contacts, court_orders, investigations, persons, referrals, report_persons, reports) into
crates/craig-cases-contracts/, andservices/craig-cases/src/api/.rsretains the API surface viapub use craig_cases_contracts::::*;. Response types tied tostore::models::*(ReferralDetail,ReportDetail) andcraig_matching::JsonbEntry(EntrySuggestions,PersonSuggestionsResponse) remain in craig-cases — lifting them requires moving the underlying store models, which is a separate refactor outside the typed-DTO scope. The CasesClient method-signature migration (60 methods) + ~30 test-callsite rewrites underservices/craig-cases/tests/api/*.rsare tracked as #308 (not blocking Step 9 completion). The 7 sibling-client migrations are #301–#307. This MR’s "Step 9" completion now means: contracts crate has the full DTO surface, the cycle-free dev-dep edge from craig-test-lib works, and downstream method/test migration is mechanical lift from typed DTOs that already exist. -
Step 9 — architectural pilot only; per-method migration deferred to 8 follow-up issues. (Original errata for !200; superseded by the redo above.) §D8.1 frames Step 9 as "MR1 of an 8-MR migration umbrella" and §D8.4 walks through ~60 method signatures that need typed-DTO swaps in
crates/craig-test-lib/src/clients/cases.rs, plus ~30 test files atservices/craig-cases/tests/api/.rswhosejson!({…})call-sites would all break at compile. Step 9 ships the architectural skeleton: a leafcrates/craig-cases-contracts/crate with the persons-module DTOs lifted (PersonSearchQuery,CreatePersonRequest,UpdatePersonRequest); craig-cases retains the API surface viapub use craig_cases_contracts::persons::;so handler code is unchanged; craig-test-lib gains[dev-dependencies] craig-cases-contractsand a typed_contracts smoke test that round-trips aCreatePersonRequestthrough serde without pulling craig-cases. The cycle guard the plan §D8.2 reserved fordeny.tomlis documented in deny.toml comments rather than enforced today: craig-cases is currently bin-only, so the structural guard is "no lib target — nothing CAN depend on it." The actualentries withwrappers = […]allowlists land when the per-module migrations introduce real consumers. The remaining 11 craig-cases DTO modules + the actual CasesClient method-signature migration + the 7 sibling-client migrations are 8 follow-up issues filed under epic &22. -
Step 8 —
delivery-protocol.mdcross-reference dropped (Tier 1 doc). §D7.5 prescribed adding a one-paragraph audit-rule pointer to.claude/docs/delivery-protocol.md. That file is a Tier 1 quickstart-template doc enforced bycargo xtask check-docs; downstream edits fail validation. The storage-shape-vs-semantics rule lives in the Process rule sub-section of.claude/docs/testing.mdinstead, which delivery-protocol.md’s existing "Plan Completion Audit" section already cross-references via its generaltesting.mdpointer. Same outcome, no Tier 1 drift. -
Step 7 cleanup — partial filename-attack table shipped. The original Step 7 errata deferred the entire
filename_attacktable to platform-stab-2 §D8.1 because some cases (CRLF, RTLO, fullwidth) depend on a sanitizer extension that hasn’t shipped. The user pushed back that this was lazy — a subset of cases works against the currentsanitize_filename. Shippedcraig_test_lib::filename_attack::CASESwith 8 cases the current sanitizer handles correctly (forward_slash,backslash,nul_byte,leading_trailing_whitespace,leading_trailing_dots,all_dots,empty_string,mixed_separators) plus a round-trip smoke test + count-pin. The CRLF / RTLO / fullwidth /length_256/unicode_resume/double_quote/unc_pathrows + theexpected_dispositionhalf of every row still ship with platform-stab-2 §D8.1 — the harness exists today so that MR is purely additive (append rows, bump the count-pin, add disposition assertions). -
Step 7 — upload corpus + loader shipped; filename-attack table + magic-byte rejection assertion deferred to §D8.1. (Original errata; partially superseded by the cleanup above for the filename-attack table portion. Magic-byte rejection assertion remains §D8.1-dependent.)
-
Step 6 cleanup —
nbfenforcement added; mutation restored. The original Step 6 errata droppedNotYetValidOneMinutebecauseValidation::validate_nbfdefaults tofalseand the validator didn’t opt in. The user pushed back that this was lazy ("ADR-021 is about strictness; that’s in-spirit"). Restored:craig_auth::jwks::JwksProvider::validate_tokennow setsvalidation.validate_nbf = true(defense-in-depth — tokens withoutnbfare unaffected since jsonwebtoken treats absentnbfas immediately valid; tokens with futurenbfpast the default 60s leeway now reject). The mutation is back in the matrix, renamedNotYetValidand using a +5-minute offset to clear the leeway window. TheAlgNonesubstring narrowing remains (upstream library reality — jsonwebtoken’s header parser refusesalg: noneat deserialization time, surfacing as a serde"unknown variant \`none\`"error rather than a clean alg-allowlist rejection). -
Step 6 —
nbfmutation dropped, alg=none expected-error substring narrowed. (Original errata; superseded by the cleanup above.) -
Step 5 — minimal
lib.rsadded tocraig-financialfor integration-test access. §D4.3 specifiesservices/craig-financial/tests/transitions_matrix.rsas an integration test, but craig-financial was bin-only (nolib.rs), sotests/.rscould not importcraig_financial::transitions::. Shipped a minimalservices/craig-financial/src/lib.rsexposingpub mod transitions;— the bin target keeps itsmod transitions;declaration in main.rs (separate compilation unit). This is the minimum change that unblocks the §D4.5 sibling adoptions across the other 5 services with state machines (each will land an analogous lib.rs in the MR that adopts the matrix). Errata documents the pattern. -
Step 4 —
--replicasinfrastructure shipped; full §D3.8 verification test deferred. §D3.8’s verification test asserts "exactly one row in cases" given concurrent same-key POSTs across replicas — that’s the strong-claim assertion of the platform-stab-2 #274 atomic-claim fix. Without #274 the current Postgres-backed idempotency middleware may not handle the race correctly under contention, so the test as specified would be testing unfixed behavior. Step 4 ships the infrastructure (--replicasxtask flag,docker-compose.scale.ymloverride, per-replica health-wait,MultiReplicaClustertest helper with unique-project-name + start/stop/replica_urls) plus 3 lifecycle / project-name-isolation tests gated byCRAIG_MULTI_REPLICA_SMOKE=1(opt-in, since spinning up a dedicated cluster takes ~30-60s and would slow pre-push). The full §D3.8 verification test ships in platform-stab-2 #274 alongside the atomic-claim fix it actually validates. The X-Served-By tower middleware (Path C in §D3.8) is a service-side change outside test-framework scope; ships when needed by Phase B tests. -
Step 3 — harness foundation only; base injectors deferred per failing-test-first protocol. §D2.0 mandates that every new injector ships with (a) the production seam refactor, (b) the injector wrapper, and (c) one test that uses the injector against a real codepath. Step 3 ships the harness foundation (
FaultInjectortrait +Attempt+ScenarioGuardwith Drop assertion), the 3 new generic wrappers (LatencyInjector,IntermittentFlapInjector,ConsumerBackpressureInjector), thescenarios::backpressure::*named-constructor sub-module, and 10 smoke tests. The 4 base injectors named in §D2.0a (CipherErrorInjector,RabbitDownInjector,ObjectStoreErrorInjector,PublishInTxFailureInjector) require production seam refactors and have no current test consumer; they are deferred — each ships in the MR that lands its first failing-test consumer (platform-stab-2 P0/P1 fixes for the seams the fixes need; Phase B Step 19 for the rest). Thedb::*,rmq::*,cipher::*,object_store::*sub-modules ofscenarios.rsare populated as their base injectors land.ConsumerBackpressureInjector::handle_deliveryis generic over an opaque message type rather than concretelapin::message::Deliveryso smoke tests can pass any value while production passes the real type.
After this plan lands
-
8 test categories operational across the workspace: happy / sad / evil / concurrency / replay / fault / state-machine / property
-
The happy/sad/evil convention is enforced via reviewer discipline + the risk-coverage scorecard’s living top-20 grid — every code-changing MR’s diff is checked against the scorecard’s pending cells before merge
-
Race conditions surface in CI / pre-push instead of by external review — the platform-stab-2 sibling fixes are the last cohort to require external-review surfacing
-
Mutation testing surfaces test-without-assertion gaps quarterly via
cargo xtask mutantsreports; a dashboard graph tracks surviving mutants per crate over time -
The invariant sweeper catches post-test-run state violations that no individual test asserts — orphan rows, stuck pending tx, missing audit rows — turning silent inconsistencies into actionable failures
-
Sibling epic &21 (Platform Stabilization Phase 2) ships with proper coverage; the platform-stab-2 plan’s "minimum inline test infrastructure" (Step 2) is folded into the consolidated Phase A modules without breaking callers
-
The CRAIG test framework is structurally and semantically positioned to catch the next class of correctness gap before it reaches an external reviewer
This is foundational work. Treat it as a dependency for everything safety-critical that follows.