Plan: Dynamic E2E Seed Manifest Generation

On this page

Status

Step Description Status

1

Plan + nav.adoc + GitLab issue

Not started

2

Verify craig-seed --manifest is DB-free

Not started

3

Add manifest regeneration to xtask e2e

Not started

4

Fix RNG-dependent E2E assertions

Not started

5

Verify with 3 different seeds

Not started

6

Update docs (testing.md, local-dev.md)

Not started

7

Restore default seed, final pre-push verification

Not started

Issue: #TBD
Branch: feature/e2e-dynamic-seed

Context

The E2E test suite is coupled to a static tests/e2e/lib/seed.ts manifest generated for CRAIG_SEED=42 CRAIG_FAMILIES=12. Running with any other seed/family values causes 29+ test failures because the hardcoded UUIDs, entity names, and counts don’t match.

This blocks:

  • CI seed rotation (catch assumptions about specific data)

  • Fuzz testing with random seeds

  • Developer freedom to adjust seed parameters

  • Pre-push hook with rotating seeds

The craig-seed binary already supports --manifest <path> to emit a TypeScript manifest. Currently the manifest is generated once and committed. This plan makes it runtime-generated.

Scope

In scope:

  • Runtime manifest regeneration in cargo xtask e2e

  • Fixing E2E assertions that depend on RNG-derived values

  • Multi-seed verification (3 different seed/family combinations)

  • Documentation updates

Out of scope:

  • CI seed rotation policy (follow-up)

  • Pre-push seed randomization (follow-up)

  • Changing the seed generator itself

Steps

Step 1: Plan + nav.adoc + GitLab issue

Create plan file (this file), link in docs/modules/ROOT/nav.adoc under Active Plans, create GitLab issue.

Step 2: Verify craig-seed --manifest is DB-free

File: tools/craig-seed/src/main.rs, tools/craig-seed/src/manifest.rs

Confirm that --manifest mode is purely deterministic (RNG-only computation, no database access). Read the manifest generation code path:

  • If RNG-only: the host command approach works directly — cargo run -p craig-seed — --manifest

  • If it queries DB: need to set DATABASE_URL env var pointing to devstack Postgres (localhost:5432)

The current code generates data from the RNG seed without DB access. Both --output-dir (SQL) and --manifest (TypeScript) are pure computation. Verify this is still true.

Step 3: Add manifest regeneration to xtask e2e

File: xtask/src/cmd/e2e.rs

Before the Playwright docker compose run command, add manifest regeneration:

println!("==> Regenerating E2E seed manifest...");
let seed = std::env::var("CRAIG_SEED").unwrap_or_else(|_| "42".into());
let families = std::env::var("CRAIG_FAMILIES").unwrap_or_else(|_| "12".into());
println!("    Seed: {seed}, Families: {families}");
let status = std::process::Command::new("cargo")
    .args(["run", "-p", "craig-seed", "--",
           "--seed", &seed, "--families", &families,
           "--manifest", "tests/e2e/lib/seed.ts"])
    .status()?;
if !status.success() {
    bail!("Failed to regenerate seed manifest");
}

Keep seed.ts tracked in git (committed for seed=42 default). The pre-push hook runs with defaults, so it regenerates for seed=42 and there are no unstaged changes. Developers who run with custom seeds understand the override is temporary.

Verification: After regeneration, confirm tests/e2e/lib/seed.ts starts with // Generated by craig-seed and contains export const SEED.

Step 4: Fix RNG-dependent E2E assertions

Review all SEED.* references in E2E specs. Tests must NOT assert on:

  • Specific county names (self.pick(GEORGIA_ADMIN_UNITS) — RNG-dependent)

  • Specific person names (self.pick(FIRST_NAMES) — RNG-dependent)

  • Specific entity counts beyond minimums

  • Specific UUIDs

Tests SHOULD assert on:

  • Static seed values (hardcoded in datagen.rs: "2025-Q4", "AC-2", "biennial", "physical_exam")

  • Structural properties (table has rows, badge exists, form renders)

  • Entity relationships via SEED.* keys (case has contacts, referral has allegations)

  • SEED. indexed references (SEED.cases.case0_, SEED.fosterHomes.home0_*) — these work with any seed because the index (0, 1, 2) is stable even when the county suffix changes

The assertion quality audit (completed earlier this session) already fixed most RNG-dependent assertions. Verify no regressions by running with seed=7777.

Likely fixes needed:

  • cases.spec.ts: findInTable calls that search for RNG-dependent county names

  • bobsmith.spec.ts: Any assertion on specific case names

  • placement.spec.ts: findInTable for specific home names

  • financial.spec.ts: Assertions on specific payment amounts (if amount depends on RNG-chosen rate)

Step 5: Verify with 3 different seeds

Run E2E with 3 different seed/family combinations. All must pass with 0 failures:

# Default (should always work)
cargo xtask dev restart && cargo xtask e2e

# Different seed, fewer families
CRAIG_SEED=7777 CRAIG_FAMILIES=8 cargo xtask dev restart && CRAIG_SEED=7777 CRAIG_FAMILIES=8 cargo xtask e2e

# Different seed, more families
CRAIG_SEED=1 CRAIG_FAMILIES=20 cargo xtask dev restart && CRAIG_SEED=1 CRAIG_FAMILIES=20 cargo xtask e2e

Minimum family count: CRAIG_FAMILIES >= 8 (pagination tests need >10 referrals at per_page=10; 8 families produces ~12 referrals).

Step 6: Update documentation

Files:

  • .claude/docs/testing.md — Add note that cargo xtask e2e regenerates seed.ts at runtime, document CRAIG_SEED / CRAIG_FAMILIES env vars, document minimum families requirement

  • .claude/docs/local-dev.md — Add seed customization section: CRAIG_SEED=99 CRAIG_FAMILIES=15 cargo xtask dev restart && CRAIG_SEED=99 CRAIG_FAMILIES=15 cargo xtask e2e

  • CHANGELOG.adoc — Add entry

Step 7: Restore default seed, final pre-push verification

cargo xtask dev restart  # defaults to seed=42, families=12
git push  # pre-push regenerates seed.ts for default, all tests pass

Verify git status shows no unstaged changes to seed.ts after push.

Files Modified

File Change

xtask/src/cmd/e2e.rs

Add manifest regeneration before Playwright run

tests/e2e/lib/seed.ts

Regenerated at runtime (remains tracked for seed=42 default)

tests/e2e/specs/*.spec.ts

Fix any RNG-dependent assertions (likely 3-5 files)

.claude/docs/testing.md

Document runtime manifest, seed env vars, minimum families

.claude/docs/local-dev.md

Add seed customization section

CHANGELOG.adoc

Add entry

Risks

  • Build time: cargo run -p craig-seed adds ~2-3 seconds to E2E startup (already compiled). Acceptable.

  • Manifest format drift: If craig-seed --manifest output changes shape, E2E specs break. Mitigated by existing manifest integration tests in tools/craig-seed/tests/integration.rs.

  • FAMILIES minimum: Pagination tests need ≥8 families. Document this constraint. Fail fast in xtask if CRAIG_FAMILIES < 8.

Edit this page · latest