Testing

On this page

Philosophy

  • NEVER dismiss test failures as transient. Investigate root cause; classify as an edge case only after thorough analysis.

  • All checks must pass before push — enforced by the pre-push hook.

  • Test results are saved to test-results/ at the repo root — always check it for failure context.

Test Output

All test results go in test-results/ at the repo root. Non-negotiable — do not mount, write, or look for results anywhere else.

test-results/
├── unit/           # cargo-nextest unit test JUnit XML
├── integration/    # cargo-nextest integration test JUnit XML
├── e2e/            # Playwright JUnit XML + traces/screenshots
└── ci/             # CI-only artifacts (coverage, SAST, container scan reports)

  • test-results/ is gitignored — never committed.

  • All test types produce JUnit XML in their subdirectory; CI consumes these exact paths.

  • When reviewing failures, read the full XML — do not tail/head/partial-read.

Test Runner

  • cargo-nextest is the standard runner for all Rust tests (unit + integration).

  • Config .config/nextest.toml; CI profile writes JUnit XML; all profiles fail-fast = false.

E2E Framework

  • Playwright is mandatory for all projects with a web UI — no Cypress, no Selenium.

  • E2E tests run inside Docker, never on the host.

  • Playwright config includes a JUnit reporter to test-results/.

  • Projects without a web UI delete the E2E sections entirely (no empty placeholders).

Pre-Push Hook

Location .githooks/pre-push; activate git config core.hooksPath .githooks && chmod +x .githooks/. The *sole functional-correctness gate — it runs the full local battery (CI runs only security/supply-chain scans).

Step 1 — cargo xtask validate --skip-docker (all must pass): public-visibility check; commit-signature verification; mandatory project docs exist; SPDX headers on .rs; cargo fmt --check --all; cargo clippy --all-targets — -D warnings; cargo nextest run --workspace --profile integration (120s + JUnit).

Step 2 — cargo xtask e2e: Docker build + containers + Playwright (when the project has a web UI / docker-compose), then teardown.

Step 3 — cargo doc --workspace --no-deps: rustdoc compiles, with broken intra-doc links denied (RUSTDOCFLAGS="-D rustdoc::broken_intra_doc_links") so a broken link hard-blocks rather than only warning.

Step 4 — cargo xtask perf --profile smoke: k6 smoke when configured; soft-skips (exit 4) when there is no load suite (opt out with SKIP_PERF=1).

What CI runs (it does NOT duplicate the battery)

CI is security + supply-chain + release only: cargo-audit/cargo-deny/ cargo-machete (audit-tools image); GitLab SAST + secret-detection; advisory plan-lint, check-docs, secrets-yaml-lint, fn-shape-report, audit-memory; release (tags only) sbom + release-xtask.

Nextest Profiles

Profile Use Timeout JUnit Output

default

Local dev (cargo nextest run)

60s

None

integration

Pre-push hook

120s

test-results/integration/results.xml

ci

CI unit tests

60s

test-results/unit/results.xml

ci-integration

CI full devstack tests

120s (8 threads)

test-results/integration/results.xml

All profiles fail-fast = false.

Deterministic Seed-Based Test Data

  • Generators take a --seed for deterministic output (same seed = same entities, UUIDs, relationships).

  • No seed → generate a random one and print it to stderr so failures reproduce.

  • E2E tests consume typed manifests generated from the seed, not hardcoded values.

  • Document the minimum seed size needed for pagination/boundary tests.

cargo run -p project-seed -- --seed 42 --families 12   # deterministic
cargo run -p project-seed                              # random; prints seed to stderr

Integration Test Guards

Infrastructure-dependent tests (databases, Docker, external services) must guard:

if !devstack_available().await { return; }  // skip if infra isn't running

Never fail with "connection refused" — guard first (health endpoint, ~3s timeout). A skip is not a failure.

Test Harness Pattern

  • A TestHarness struct manages authenticated HTTP clients, cleanup stacks, and lifecycle.

  • Builder patterns for entities (PersonBuilder, CaseBuilder) — don’t hand-build JSON.

  • Typed service clients with bearer-token injection; cleanup tears down created entities on drop.

Performance Testing

  • Criterion (micro) — add benchmarks when performance is a stated requirement or a hot path is identified; >10% regression on a hot path warrants investigation. On-demand (cargo bench), not in pre-push/CI by default.

  • k6 (macro) — cargo xtask perf; profiles smoke (1 VU ~5s, in pre-push), load, stress, soak; threshold assertions on p95 latency + error rate; results to test-results/k6/; runs in Docker (grafana/k6).

Test Categories (Taxonomy)

Name a test for the property it asserts, not the endpoint/function it calls.

Category Purpose Location

Functional

Happy-path correctness

tests/api/ or #[cfg(test)] mod tests

Invariant

Cross-cutting properties (uniqueness, referential integrity, drift)

tests/invariants/

Concurrency

Races, deadlocks, lock ordering, parallel safety

tests/concurrency/

Fault-injection

Behavior under failures (network/broker/partial-write)

tests/fault_injection/

Recovery

Behavior after failure (retry, idempotency, rollback, replay)

tests/recovery/

Contract

API/wire-format pinning (response shapes, error schemas, CSP)

tests/contract/

Property-based

Universally-quantified properties via proptest/quickcheck

tests/property/

Mutation

Uncovered-path detection via cargo mutants

driven by cargo xtask mutants

Process rule: concurrency / fault-injection / platform-invariant fixes MUST ship with a test in the corresponding category — a race-condition bug gets a concurrency test, not just a functional one.

Constraint Tests Are Hand-Curated

API-boundary rejection tests (UNIQUE/FK violations, enum validation, duplicate-create) are hand-written, not auto-derived from SQL schemas or OpenAPI. Auto-generation produces shallow tests that miss the real boundary cases — write the test for the rejection behavior you want.

Invariant Tests with Drift Contract

For projects with rulesets, migrations, or per-service catalogs, maintain an invariant test that sweeps the catalog at test time: every file parses, names are unique, constants match the code. Examples: all migrations apply against a fresh DB; all rulesets/*.json names unique; rulesets reference only code-defined enums.

Evil Input Corpus

For services accepting user-controlled input, maintain a hostile-input corpus per category, run via a parametric macro (adding an endpoint becomes a one-line subscription):

Category Example payloads

jwt

Tampered signature, expired, unsigned alg: none, oversized header

upload

Zip-bomb, polyglot, MIME mismatch, path-traversal filename

string

NULL bytes, oversized (10MB), homoglyph, RTL override, control chars

uuid

Wrong version, zero UUID, non-canonical, oversized

json

Deeply nested, duplicate keys, integer overflow, NaN/Infinity

path

../, ..\\, URL-encoded, absolute, symlink loop, reserved names

html

XSS variants, mXSS, SVG-embedded JS, data: URLs, on-attribute handlers

unicode

Bidi override, zero-width joiners, normalization mismatches

enum

Unknown variant, case-mismatch, oversized, wrong type

date

Pre-epoch, year 9999, leap second, timezone gap, non-ISO

multipart

Boundary in payload, missing boundary, oversized/malformed part

signature

Truncated, wrong algorithm, key confusion (HS256 vs RS256), replay

Each evil-input test asserts: rejected with 4xx, no PII in the error, no internal state mutation, no log spam.

Property-Based Testing

Property-based testing (proptest / quickcheck) is mandatory for parsers, deserializers, serializers, and numerical/math logic, and anything with universally-quantified invariants. State the invariant; let the tool find counter-examples. Example-based tests with hand-picked values miss edge cases (empty/max-size inputs, surrogate pairs, overflows). Prefer proptest (better shrinking); ProptestConfig { cases: 1000, .. } for slow targets.

Coverage Floor Gate

cargo xtask coverage [--threshold N] [--baseline] wraps cargo-llvm-cov (LCOV
JSON to test-results/coverage/), compares against .coverage-baseline.json, and fails on a >0.5% drop. --baseline refreshes; --threshold N enforces a minimum line coverage (CI-gateable). Tracking baselines prevents silent test decay.

Mutation Testing

cargo xtask mutants [--smoke] wraps cargo-mutants to find uncovered paths. --smoke runs a 1/20 shard (~minutes); full runs in scheduled CI. Accepted mutants are documented in mutants-baseline.toml with per-entry justification.

Critical Rules

  1. Never dismiss test failures as transient — investigate root cause.

  2. All checks must pass before push — fmt, clippy, tests (hook-enforced).

  3. Test results in test-results/ — check it before asking questions.

  4. Docker-based tests run in Docker — never run Playwright/E2E on the host.

  5. Never weaken a test to make code pass — fix the code, not the test. No delete/skip/#[ignore]; no loosened assertions; no changed expected values. The only legitimate test edit is a genuinely-incorrect test, explained in the commit.

  6. All test types produce JUnit XML to test-results/.

Project-specific test types, commands, the CI pipeline, and E2E setup live in the project’s project testing page.

Edit this page · latest