Test Coverage Scorecard

On this page

This is the single living scorecard for CRAIG’s risk coverage — the matrix of "for each critical workflow, which adversarial axes have we tested?" Distinct from line coverage (which cargo-tarpaulin and cargo-llvm-cov produce); risk coverage answers "would a regression in this workflow be caught by the existing test suite?"

Plan: test-framework-hardening.adoc §DB.2.

The 6 axes

Every workflow row tracks coverage across the canonical Test Categories taxonomy from Testing Reference (CRAIG):

Axis What it asserts

Happy

Documented success path returns 2xx + the expected envelope

Sad

Documented failure path returns the documented 4xx (e.g. 404 for unknown id, 422 for invalid body, 409 for state conflict)

Evil

Adversarial input — Unicode, RTLO, NUL injection, oversized strings, depth bombs — is rejected cleanly

Concurrency

Concurrent same-key behavior matches the documented invariant (one winner; no double-execution; no lost write)

Replay

Crash-and-restart / message redelivery semantics produce the documented post-recovery state

Fault

Dependency error handling — DB down, broker down, object-store error, upstream 5xx, malformed JSON

Legend: ✅ covered, ⏳ planned (issue filed), ❌ gap (no plan).

Top-20 workflows

Workflow Happy Sad Evil Concurrency Replay Fault

Convert-class POST claims (cross-cutting; crates/craig-api/src/request_claims.rs — ADR-062 §B replaced the deleted idempotency.rs middleware, epic &77)

✅

✅

✅

✅

✅

✅

Outbox publish (cross-cutting; crates/craig-mq::Publisher::publish_in_tx)

✅

✅

⏳

⏳

⏳

⏳

Inbox handler dedup (cross-cutting; crates/craig-mq::handle_idempotently — the ADR-062 §A single-transaction attempt machine, epic &77)

✅

✅

✅

✅

✅

✅

Attachment upload (4 sites — cases contacts, cases reports, exchange ICPC, placement homes; ADR-062 §U generation-fenced attempts, epic &77)

✅

✅

⏳

✅

✅

✅

Exchange send (services/craig-exchange/src/api/transactions.rs)

✅

✅

⏳

✅

✅

⏳

Placement transition guard (services/craig-placement/src/transitions.rs)

✅

✅

⏳

⏳

⏳

⏳

BFF page render under upstream failure (services/craig-web/src/api_client.rs)

✅

✅

⏳

⏳

⏳

✅

Person match (services/craig-cases/src/api/persons.rs::search_persons)

✅

⏳

⏳

⏳

⏳

⏳

Rules evaluation (services/craig-rules/src/api.rs::evaluate_rule_set)

✅

⏳

⏳

⏳

⏳

⏳

Public intake submit (services/craig-intake/src/api/public.rs::submit_report)

✅

✅

✅

⏳

⏳

⏳

Partner intake submit (services/craig-intake/src/api/partner.rs)

✅

✅

⏳

⏳

⏳

⏳

JWT validation across 8 services (crates/craig-auth::JwksProvider::validate_token)

✅

✅

✅

⏳

n/a

⏳

Cross-service FK reconciliation (cargo xtask reconcile)

✅

⏳

⏳

n/a

n/a

⏳

Health endpoints — /livez vs /readyz differentiation + worker/MQ gates (crates/craig-api unit readiness + tests/fault/, &76 U2/U3; Concurrency = the U1 supervisor unit tests' deterministic order-serialization + duplicate-rejection pins, which feed the gate)

✅

✅

n/a

✅

⏳

✅

Invariant sweep — outbox/inbox transport drains (cargo xtask invariants; the idempotency-table invariant was deleted with its table in epic &77 B2/BF — the request_claims beyond-grace watchdog replaced it)

✅

✅

n/a

⏳

⏳

✅

State-machine matrix on all 16 machines (crates/craig-test-lib::state_machine_matrix)

✅

✅

⏳

⏳

⏳

⏳

Soft-delete activation flip (every soft-delete table — ~20 tables across services)

✅

⏳

⏳

⏳

⏳

⏳

Audit log emission per state mutation (services/craig-security/src/api/audit.rs)

✅

⏳

⏳

⏳

⏳

⏳

Filename + Content-Disposition sanitization (crates/craig-store::sanitize_filename)

✅

✅

✅

n/a

n/a

⏳

Typed-DTO contract round-trip (crates/craig-cases-contracts)

✅

✅

n/a

n/a

n/a

n/a

Reading this scorecard

  • Cell at ✅ means there is at least one test that exercises this workflow on this axis. Test counts per cell are NOT tracked here — go to cargo xtask coverage-matrix for cell-by-cell test inventory.

  • Cell at ⏳ means a tracking issue exists and the plan has a slot for the test. Click through to the issue tracker for the specific blocker.

  • Cell at ❌ means no test, no plan. These are the cells that should drive the next plan-authoring round.

  • n/a means the axis is structurally inapplicable (e.g. a typed-DTO crate has no concurrency axis because it’s a pure-Rust serde test).

When to update

  • Per-MR: when a test lands that ticks a previously-empty cell, flip the cell from ⏳/❌ to ✅ in the same MR. This is part of the delivery checklist (the delivery-protocol standard).

  • Plan-authoring: when a new plan adds tests that fill cells, update the scorecard before the plan ships so reviewers see the projected post-plan state.

  • Quarterly: a maintainer rebuilds the scorecard from scratch by walking the 20 workflows and re-evaluating each cell. This catches drift where ✅ cells decay into stale tests that no longer exercise the axis they claim.

Drift detection

cargo xtask coverage-matrix --emit-scorecard prints an auto-derived version of the matrix from test-name suffix tags (_happy, _sad, _evil, etc.) plus the directory-prefix tags (tests/concurrency/, tests/recovery/, tests/fault/). Compare its output against the hand-curated rows above; significant drift is a signal that test naming and the curated scorecard have diverged.

The auto-derivation is a signal, not a replacement: it counts test files, not coverage quality. A workflow can have 5 tests on the Happy axis that all hit the same code path while leaving important branches untested. That’s exactly the kind of gap the curated scorecard surfaces.

Line coverage (Plan K F-044)

The risk-coverage matrix above is the primary scorecard. Line coverage is a complementary signal — useful for spotting whole files / branches that no test reaches, useless for arguing that a workflow is "covered".

cargo xtask coverage wraps cargo llvm-cov nextest --workspace --summary-only for local measurement. #1208 moved the runner onto nextest: plain cargo test (same-process test threads) races the process-global-state tests that are safe only under nextest’s process-per-test model (craig-common’s Prometheus-instrument OnceLock pins were the observed casualty), and nextest is the mandated runner everywhere else.

Invocation Effect

cargo xtask coverage

Measure + print summary, then enforce the baseline floor (#1208): FAILS when workspace line coverage drops more than 0.5 points below the tracked .coverage-baseline.json — the test-authoring floor. With no baseline recorded it measures, prints, and points at --baseline.

cargo xtask coverage --baseline

Measure + write .coverage-baseline.json at the workspace root (TRACKED — commit it; the diff review is the ratchet’s consent step). Refresh when a change legitimately moves coverage.

cargo xtask coverage --threshold N

Additional ABSOLUTE floor. Adds --fail-under-lines N to the underlying llvm-cov invocation; bails when workspace line coverage drops below N, independent of the baseline delta.

Local prerequisites: cargo install cargo-llvm-cov --locked (or cargo binstall cargo-llvm-cov) + cargo-nextest (already the mandated test runner).

Promotion checklist

#1208 ended the soft start LOCALLY: cargo xtask coverage now enforces the baseline-relative floor by default (>0.5-point drop fails), and checklist item 1 is DONE — .coverage-baseline.json (45.68% lines, 2026-08-09) is committed at the workspace root. What remains is CI promotion, in a follow-up MR only after ALL of the following are true:

  1. [x] A .coverage-baseline.json committed at the workspace root, captured via cargo xtask coverage --baseline (#1208, 2026-08-09).

  2. [ ] Per-crate baselines recorded below (one row per workspace member) and held steady for at least a quarter.

  3. [ ] A CI job wired to run cargo xtask coverage (the default baseline-floor mode) and the workspace value not regressed across 4+ consecutive merges.

At that point: keep the baseline-relative floor as the primary gate and optionally add --threshold <baseline - 2%> as the absolute backstop (canopy precedent), then re-flag this section’s status row to "Enforced".

Per-crate baselines

To be populated from a --baseline-era report. Format: crate-name | lines | covered | percent. (The tracked baseline file is deliberately totals-only — per-file entries carry absolute checkout paths and churn ~360 KB per refresh; per-crate rows belong HERE, curated.) The promotion to hard CI gate is blocked on this table being non-empty + held steady for a quarter per the checklist above.

Crate Lines Covered Percent

(awaiting baseline run)

–

–

–
Edit this page · latest