Devstack Script Durability Improvements

On this page

Status

COMPLETE

Context

During the reference data work, we discovered that cargo xtask dev restart (formerly devstack.ps1 restart / devstack.sh restart) doesn’t reliably invalidate Docker’s BuildKit layer cache. On Windows Docker Desktop, docker compose up -d --build reuses cached COPY layers even when source files have changed, causing stale binaries to be deployed. This required manual docker compose build --no-cache craig-seed workarounds. The restart command promises "wipe all data and start fresh" — it should deliver on that promise without cache surprises.

Additionally, seed failures (e.g., CHECK constraint violations) are completely silent — the seed container exits with an error, but services still start and report healthy, leading to empty databases and mysterious E2E failures.

Changes

1. Force --no-cache on restart builds

Files: xtask/src/main.rs (formerly devstack.sh, devstack.ps1)

Currently the start command always uses docker compose up -d --build, which relies on BuildKit cache. This is fine for start and reload (speed matters, code probably hasn’t changed). But restart means "wipe everything and start fresh" — it should guarantee no stale layers.

Approach: Add an optional no-cache parameter to the start function. When restart calls it, pass the flag. When start/reload call it, don’t.

  • cargo xtask dev start accepts an optional --no-cache arg. cargo xtask dev restart uses --no-cache internally. The function checks for the flag and runs docker compose build --no-cache && docker compose up -d instead of docker compose up -d --build.

2. Verify seed container success after startup

Files: xtask/src/main.rs (seed verification logic)

After health checks complete, verify that the seed container exited cleanly:

verify_seed() {
    echo "==> Verifying seed data..."
    local exit_code
    exit_code=$(docker compose ps -a craig-seed --format '{{.ExitCode}}' 2>/dev/null || echo "1")
    if [ "$exit_code" != "0" ]; then
        echo "ERROR: craig-seed exited with code $exit_code" >&2
        echo "--- seed container logs ---" >&2
        docker compose logs craig-seed >&2
        exit 1
    fi
    echo "    Seed completed successfully."
}

Called from the start function after health checks.

3. Fix health check elapsed counter (shared across services)

Files: xtask/src/main.rs

Bug: The elapsed counter starts at 0 and increments across all services. If craig-rules takes 60s, the remaining 7 services only get 240s combined, not 300s each. A slow first service starves the timeout budget for later services.

Fix: Reset elapsed=0 at the start of each service’s inner loop (move it inside the for loop).

4. Always rebuild E2E container

Files: xtask/src/main.rs

The e2e command uses docker compose --profile e2e run --rm craig-e2e which does NOT rebuild the image. If test files change, stale tests run.

Fix: Add --build to the run command:

docker compose --profile e2e run --build --rm craig-e2e "$@"

5. Add --rmi local to clean for restart path

Files: xtask/src/main.rs

Currently clean runs docker compose down -v --remove-orphans which removes containers and volumes but NOT images. This means restart (which calls clean then start) still has images in the local image cache that BuildKit may reference.

Approach: When called from restart, also pass --rmi local to remove locally-built images (not pulled base images). Don’t add --rmi to standalone clean since the user might want to keep images for a faster start later.

6. Update MEMORY.md

Update the devstack section to document:

  • restart uses --no-cache builds (guaranteed fresh)

  • restart removes locally-built images

  • Seed verification happens automatically on start/restart

  • E2E container auto-rebuilds on e2e command

  • Health timeout is per-service (300s each), not shared

Summary of changes per file

xtask/src/main.rs:

  1. dev start — accept optional --no-cache flag, conditional build logic

  2. Seed verification — check craig-seed exit code, fail loudly on error

  3. Health check — reset elapsed counter per service

  4. e2e — add --build to docker compose run

  5. dev clean — accept optional --rmi flag

  6. dev restart — pass --no-cache to start, --rmi to clean

Verification

  1. cargo xtask dev restart — builds with --no-cache, all 8 services healthy, seed verified

  2. Works identically on Linux, macOS, and Windows (cross-platform Rust binary)

  3. Modify a source file, cargo xtask dev restart → new binary is deployed (no stale cache)

  4. Break a seed CHECK constraint intentionally → restart fails loudly with seed error

  5. cargo xtask dev start — uses cached build (fast), still verifies seed

  6. cargo xtask e2e — rebuilds E2E container automatically

  7. cargo xtask dev clean — removes volumes but preserves images

  8. Health timeout: if a service is slow, later services still get the full 300s budget each

Edit this page · latest