Plan: Scalable, Deterministic Seed Data Generator

On this page

Status

COMPLETE

Context

Current seed data is ~250 hand-crafted records in static SQL files (devstack/seed/sql/*.sql) with hardcoded UUIDv7s. E2E tests reference these UUIDs via tests/e2e/lib/seed.ts. This approach doesn’t support randomized testing, reproducible bug reports, or stress-test scale.

The goal is a generator that is random by default, deterministic with a seed number, and scales to arbitrary size via a families argument.

Approach: New tools/craig-seed Workspace Crate

A Rust binary that generates SQL files to stdout/directory. Replaces the static SQL files entirely.

CLI Interface

craig-seed [OPTIONS]

  --seed <u64>          RNG seed (random if omitted; always printed for reproduction)
  --families <N>        Number of family units to generate (default: 9)
  --output-dir <DIR>    Write per-database SQL files here (required)
  --manifest <PATH>     Write TypeScript seed.ts manifest for E2E tests (optional)

Examples:

# Random seed, 9 families (current baseline equivalent)
craig-seed --output-dir /tmp/seed

# Deterministic, scaled up
craig-seed --seed 42 --families 500 --output-dir /tmp/seed

# With E2E manifest
craig-seed --seed 42 --families 9 --output-dir /tmp/seed --manifest /tmp/seed/seed.ts

The seed used is always printed to stderr: Seed: 42 (or Seed: 8374619283 (random)).

Output Files

One SQL file per service database, matching current convention:

  • craig_cases.sql — persons, referrals, allegations, investigations, safety_assessments, cases, case_household, case_plans, case_plan_tasks, contacts, court_orders

  • craig_placement.sql — foster_homes, foster_home_training, placements, kinship_options

  • craig_exchange.sql — exchange_partners, data_sharing_agreements, exchange_transactions, icpc_requests, icpc_home_studies

  • seed.ts (optional) — TypeScript manifest with named references to key entities

Rule set loading stays as-is (from rulesets/*.json via seed.sh). Rules are authored content, not generated data.

Data Generation Model

Per-Family Unit

Each family generates a realistic cascade. The RNG determines the family’s "profile":

Entity Count per family Notes

Parents/caregivers

1-2

Shared last name, realistic ages

Children

1-3

Ages 0-17, shared last name

Extended family

0-2

Grandparents, aunts/uncles

Referrals

1-2

~80% screened in, various reporter types

Allegations

1-3 per referral

Various abuse types

Investigations

0-1 per screened-in referral

~70% closed, status distribution

Cases

0-1 per closed investigation

~60% open, rest closed with reasons

Household members

all family persons

Roles: child, parent, caregiver

Case plans

0-1 per open case

Draft/active/completed distribution

Tasks

1-3 per plan

Various statuses

Contacts

0-3 per case

Home visits, phone calls, office

Court orders

0-1 per case

Various order types

Probabilistic flags per family:

  • ~10% ICWA-eligible (tribal affiliation populated)

  • ~50% of cases result in out-of-home placement

  • ~10% involve cross-county or interstate elements

Shared/Global Entities

Generated proportionally to family count:

Entity Generation rule

Foster homes

~1 per 3 placed children, mixed types (foster/kinship/group/therapeutic)

Training records

2-4 per foster home

Placements

1 per placed child, ~20% have ended+active (transfer history)

Kinship options

0-2 per placed child

Exchange partners

Fixed small set: 3-5 state agencies

Agreements

1-2 per partner

Transactions

1-3 per active agreement

ICPC requests

~10% of cases with out-of-state needs

ICPC home studies

1 per non-draft ICPC request

Realistic Data Pools

Using the fake crate with seeded RNG for:

  • First/last names (culturally diverse, matching race/ethnicity)

  • Addresses (Georgia streets, cities, ZIP codes)

  • Phone numbers, SSN last-four

  • Georgia county names (Fulton, DeKalb, Gwinnett, Cobb, Clayton, Cherokee, Bibb, etc.)

  • Reporter types: mandated, professional, anonymous, self_report, law_enforcement

  • Abuse types: physical_abuse, neglect, sexual_abuse, emotional_abuse, medical_neglect

  • Case numbers: {COUNTY}-{DATE}-{HASH} format

Deterministic UUIDs

UUIDv7 embeds a timestamp + random bits. For deterministic generation:

  • Use a monotonic millisecond counter starting from a fixed epoch (e.g., 2024-01-01T00:00:00Z)

  • Random bits from the seeded StdRng

  • Result: valid UUIDv7s that are time-ordered and fully reproducible from a seed

pub struct DeterministicUuidGenerator {
    rng: StdRng,
    counter_ms: u64,  // monotonic, increments per UUID
}

impl DeterministicUuidGenerator {
    pub fn next(&mut self) -> Uuid {
        self.counter_ms += 1;
        // Build 16 bytes: 48-bit timestamp | 4-bit version(7) |
        //   12-bit rand_a | 2-bit variant | 62-bit rand_b
        // timestamp from counter_ms, rand bits from self.rng
    }
}

Crate Structure

tools/craig-seed/
+-- Cargo.toml
+-- src/
    +-- main.rs        # CLI (clap), orchestration
    +-- lib.rs         # Public API (for unit testing the generator)
    +-- uuid.rs        # DeterministicUuidGenerator
    +-- model.rs       # In-memory structs: Person, Referral, Case, FosterHome, etc.
    +-- gen.rs         # Generation engine: SeedGenerator
    +-- sql.rs         # SQL rendering: model structs -> INSERT statements
    +-- manifest.rs    # TypeScript seed.ts rendering

Dependencies (new to workspace)

[workspace.dependencies]
fake = { version = "4", features = ["derive", "chrono"] }
rand = "0.8"

Crate-level:

[dependencies]
fake = { workspace = true }
rand = { workspace = true }
clap = { workspace = true, features = ["derive"] }
chrono = { workspace = true }
uuid = { workspace = true }
serde = { workspace = true }
serde_json = { workspace = true }

Docker Integration

Dockerfile Changes

Add craig-seed to the multi-stage build:

# In builder stage, add:
COPY tools/craig-seed/Cargo.toml tools/craig-seed/Cargo.toml
# ... stub file for cache ...
RUN cargo build --release -p craig-seed  # (added to existing build command)

# New runtime stage:
FROM alpine:3.21 AS craig-seed
RUN apk add --no-cache postgresql16-client
COPY --from=builder /app/target/release/craig-seed /usr/local/bin/
COPY rulesets/ /seed/rulesets/
COPY devstack/seed/seed.sh /seed/seed.sh
RUN chmod +x /seed/seed.sh
ENTRYPOINT ["/seed/seed.sh"]

docker-compose.yml Changes

craig-seed:
  build:
    context: .
    dockerfile: Dockerfile
    target: craig-seed
  environment:
    PGHOST: postgres
    PGUSER: craig
    PGPASSWORD: craig
    CRAIG_SEED: ${CRAIG_SEED:-}
    CRAIG_FAMILIES: ${CRAIG_FAMILIES:-9}

Updated seed.sh

#!/bin/sh
set -e
echo "=== CRAIG Devstack Seed ==="

echo "Waiting for PostgreSQL..."
until pg_isready -h "$PGHOST" -U "$PGUSER" -q 2>/dev/null; do sleep 1; done

# Generate seed SQL
ARGS="--families ${CRAIG_FAMILIES:-9} --output-dir /tmp/seed"
[ -n "$CRAIG_SEED" ] && ARGS="$ARGS --seed $CRAIG_SEED"
craig-seed $ARGS

# Load rule sets from JSON (unchanged)
# ... (existing rule set loading loop from rulesets/georgia/*.json) ...

# Load generated SQL
for db in craig_cases craig_placement craig_exchange; do
    sql="/tmp/seed/${db}.sql"
    if [ -f "$sql" ]; then
        echo "Seeding ${db}..."
        psql -h "$PGHOST" -U "$PGUSER" -d "$db" -q -f "$sql"
    fi
done

echo "=== Seed complete ==="

Devstack Scripts

cargo xtask dev passes through env vars:

  • CRAIG_SEED — forwarded to craig-seed container

  • CRAIG_FAMILIES — forwarded to craig-seed container

E.g.: CRAIG_SEED=42 CRAIG_FAMILIES=100 cargo xtask dev start

E2E Test Compatibility

Manifest Generation

When --manifest is provided, the generator writes a TypeScript file with named references to key entities. The naming strategy:

  1. Each family gets a camelCase key derived from the last name (e.g., johnson, rivera)

  2. First 9 families (at seed 42) produce a well-known set matching current E2E coverage

  3. Manifest includes: persons, referrals, investigations, cases, casePlans, contacts, fosterHomes, placements, exchangePartners, agreements, icpcRequests

Workflow for E2E Seed Updates

  1. Run: cargo run -p craig-seed — --seed 42 --families 9 --output-dir /dev/null --manifest tests/e2e/lib/seed.ts

  2. Commit the generated seed.ts

  3. E2E tests use CRAIG_SEED=42 in docker-compose.yml (or cargo xtask e2e sets it)

The seed.ts file gets a header comment:

// AUTO-GENERATED by craig-seed --seed 42 --families 9
// Do not edit manually. Regenerate with:
//   cargo run -p craig-seed -- --seed 42 --families 9 --manifest tests/e2e/lib/seed.ts

Files to Modify

New Files

  • tools/craig-seed/Cargo.toml

  • tools/craig-seed/src/main.rs

  • tools/craig-seed/src/lib.rs

  • tools/craig-seed/src/uuid.rs

  • tools/craig-seed/src/model.rs

  • tools/craig-seed/src/gen.rs

  • tools/craig-seed/src/sql.rs

  • tools/craig-seed/src/manifest.rs

Modified Files

  • Cargo.toml — add tools/craig-seed to workspace members, add fake/rand to workspace deps

  • Dockerfile — add craig-seed build + runtime stage, add stubs for tools/craig-seed

  • docker-compose.yml — update craig-seed service (target, env vars)

  • devstack/seed/seed.sh — invoke craig-seed binary, keep rule set JSON loading

  • tests/e2e/lib/seed.ts — regenerated from --manifest output

Deleted Files

  • devstack/seed/Dockerfile — replaced by Dockerfile target

  • devstack/seed/sql/craig_cases.sql — replaced by generator

  • devstack/seed/sql/craig_placement.sql — replaced by generator

  • devstack/seed/sql/craig_exchange.sql — replaced by generator

  • devstack/seed/sql/craig_rules.sql — replaced by generator (rules still from JSON, but this 9-line file is unnecessary)

Testing Strategy

Unit Tests (in-crate)

  • Determinism: same seed + families = identical SQL output (byte-for-byte)

  • UUID validity: generated UUIDs pass v7 format validation

  • FK consistency: all foreign key references resolve within generated data

  • Scale: generate 1, 10, 100, 1000 families without panic

  • SQL validity: output parses as valid SQL (basic string checks)

Integration Test

  • Boot devstack with CRAIG_SEED=42 CRAIG_FAMILIES=9, verify services respond with expected data counts

  • E2E tests pass with generated seed data (same seed as manifest)

Verification Commands

# Generate and inspect locally
cargo run -p craig-seed -- --seed 42 --families 9 --output-dir /tmp/seed
wc -l /tmp/seed/*.sql

# Verify determinism
cargo run -p craig-seed -- --seed 42 --families 9 --output-dir /tmp/seed1
cargo run -p craig-seed -- --seed 42 --families 9 --output-dir /tmp/seed2
diff /tmp/seed1 /tmp/seed2  # should be empty

# Full devstack test
CRAIG_SEED=42 cargo xtask dev restart
cargo xtask e2e
Edit this page · latest