DevStack Guide

On this page

The CRAIG devstack is a Docker Compose environment that runs every piece of infrastructure and every application service on your own machine. It gives you a complete, self-contained development and testing environment: ephemeral data, pre-configured authentication, and realistic seed data.

Prerequisites

  • Docker Desktop (Windows/macOS) or Docker Engine (Linux)

  • Docker Compose v2 (included with Docker Desktop)

  • At least 8 GB of RAM allocated to Docker

On Windows, use Docker Desktop with the WSL 2 backend for best performance.

Quick Start

cargo xtask dev start

The start command:

  1. Builds all Docker images

  2. Starts infrastructure (PostgreSQL, RabbitMQ, Keycloak, Garage)

  3. Starts CRAIG services (rules, cases, placement, exchange, financial, reporting, security, intake, web)

  4. Initializes the Garage S3-compatible object store (idempotent)

  5. Waits for all services to pass health checks (up to 5 minutes)

  6. Runs the craig-seed one-shot container to load mock data

Once complete you will see:

==> CRAIG devstack is ready.

Open http://localhost:8080 to access the web UI. Log in with one of the test user accounts.

Command Reference

The cargo xtask dev command supports the following subcommands. Running with no arguments or help prints usage information.

start

Build images, start all containers, initialize Garage, and wait for health checks.

cargo xtask dev start

This is the primary command for bringing up the devstack. If the images are already cached, subsequent starts are faster: only changed layers are rebuilt.

stop

Stop all containers and remove them. Data volumes (Garage, PostgreSQL) are preserved, so a subsequent start will resume with existing data.

cargo xtask dev stop

reload

Rebuild containers and restart without wiping data. Useful after making code changes: the new binaries are compiled into fresh Docker images, but databases and object storage keep their data.

cargo xtask dev reload

This is equivalent to stop followed by start.

restart

Wipe all data and start fresh. Removes all containers, volumes, and orphan resources, then performs a full start including Garage initialization and seed data loading.

cargo xtask dev restart

Use this when you want a completely clean environment (e.g., after schema changes or to reset seed data).

reseed

Wipe data volumes and restart the stack with cached images. This is faster than restart because the built images are kept: only the containers are recreated and craig-seed re-run. The pre-push hook uses it between validate and e2e, because the integration tests that validate runs leave data behind and the e2e suite needs a clean seed.

cargo xtask dev reseed

Typical duration: 30–60 seconds versus ~20 minutes for a full restart.

restart-service

Restart a single CRAIG application service without touching the rest of the stack. Data volumes are preserved; the command polls the service’s /healthz endpoint and waits for a healthy response before returning. Used by the E2E cookie-lifecycle suite to prove sessions survive a web-tier restart (ADR-013).

cargo xtask dev restart-service craig-web

The service name is validated against the CRAIG application services list (craig-rules, craig-cases, craig-placement, craig-exchange, craig-financial, craig-reporting, craig-security, craig-intake, craig-composition, craig-web, mock-server — CRAIG_SERVICES in xtask/src/docker.rs) to prevent arbitrary docker compose restart invocations.

clean

Stop everything and wipe all data without restarting. Like restart but without the subsequent start — leaves the environment fully torn down.

cargo xtask dev clean --confirm

status

Display a formatted table showing the state, health, ports, and uptime of every service.

cargo xtask dev status

Example output:

CRAIG Devstack Status
=====================

Infrastructure:
  SERVICE              STATE      HEALTH     PORT(S)        UPTIME
  -------              -----      ------     -------        ------
  postgres             running    healthy    5432           47 minutes
  postgres-test        running    healthy    5433           47 minutes
  rabbitmq             running    healthy    5672, 15672    47 minutes
  keycloak             running    healthy    8180           47 minutes
  garage               running    healthy    3900, 3903     47 minutes

Services:
  SERVICE              STATE      HEALTH     PORT(S)        UPTIME
  -------              -----      ------     -------        ------
  craig-rules          running    healthy    8001           47 minutes
  craig-cases          running    healthy    8002           47 minutes
  craig-placement      running    healthy    8003           47 minutes
  craig-exchange       running    healthy    8004           47 minutes
  craig-financial      running    healthy    8005           47 minutes
  craig-reporting      running    healthy    8006           47 minutes
  craig-security       running    healthy    8007           47 minutes
  craig-intake         running    healthy    8008           47 minutes
  craig-web            running    healthy    8080           10 minutes
  mock-server          running    healthy    9090           47 minutes

One-shot:
  SERVICE              STATE      HEALTH     PORT(S)        UPTIME
  -------              -----      ------     -------        ------
  craig-seed           exited     -          -              Exited (0)

Volumes:
  craig_garage-data
  craig_garage-meta

If nothing is running, it prints CRAIG devstack is not running (no containers found).

logs

Follow container logs in real time. With no argument, follows all services. With a service name, follows only that service.

# All services
cargo xtask dev logs

# Single service
cargo xtask dev logs craig-rules

Press Ctrl+C to stop following. If you pass an invalid service name, the script prints an error with the list of available services.

e2e

Run the Playwright end-to-end test suite against the running devstack. If the devstack is not running, it is started automatically.

# Run the full E2E suite (the Playwright report gives live counts;
# they drift too fast to pin here)
cargo xtask e2e

# Run only tests matching "rule" in the name
cargo xtask e2e --grep "rule"

# Run with a visible browser window
cargo xtask e2e --headed

# Step through tests interactively
cargo xtask e2e --debug

Any extra arguments are forwarded directly to npx playwright test inside the container. On failure, a hint is printed with the path to the HTML report:

Hint: open tests/e2e/playwright-report/index.html for the full report

help

Print the built-in usage text.

cargo xtask dev help

# Also shown when no arguments are given:
cargo xtask dev

Services and Ports

Infrastructure

Service Image Port(s) Purpose

PostgreSQL 18

postgres:18-alpine

5432

Database server; each CRAIG service gets its own database

PostgreSQL 18 (test plane)

postgres:18-alpine (same devstack/postgres build)

5433

postgres-test carries only throwaway scratch and template databases (#1403). It keeps its data directory on tmpfs and runs with fsync=off, so the forced checkpoints that constant database creation and teardown trigger never land on the shared instance the services use.

RabbitMQ 4.2

rabbitmq:4.2-management-alpine

5672, 15672

Message broker (speaking AMQP, the standard message-queue protocol) plus its management UI

Keycloak 26.5

quay.io/keycloak/keycloak:26.5

8180

OpenID Connect (OIDC) identity provider; imports the craig realm on startup

Garage

dxflrs/garage:v2.2.0

3900, 3903

S3-compatible object storage for file attachments

CRAIG Services

Service Port Database Purpose

craig-rules

8001

craig_rules

Rules and policy engine (rule sets in the JSON Decision Model, JDM)

craig-cases

8002

craig_cases

Case management (referrals, investigations, cases, plans)

craig-placement

8003

craig_placement

Placement and foster care (homes, placements, matching)

craig-exchange

8004

craig_exchange

Data exchange and ICPC (partners, agreements, transactions)

craig-financial

8005

craig_financial

Financial management (payments, rate tables, claims)

craig-reporting

8006

craig_reporting

Reporting and analytics (AFCARS, NCANDS, ad-hoc reports)

craig-security

8007

craig_security

Security, audit logging, compliance reviews, NIST controls

craig-intake

8008

 — (stateless edge per ADR-017; the pre-created craig_intake DB is never populated)

Public intake: the anonymous child abuse/neglect reporting API

craig-composition

8009

craig_composition

Composition layer engine: arranges plugin surfaces (ADR-035)

craig-web

8080

 — 

Web UI, built as a backend-for-frontend (BFF): it proxies to the backend services and handles authentication itself, so the browser never talks to them directly

mock-server

9090

 — 

A registry-driven stand-in for the 10 typed partner systems, so craig-exchange adapters can be exercised in the devstack without any real partner endpoint (Plan T2.3, per ADR-032 A5 and §2.5). Each partner crate ships its own mock module (crates/craig-partner-*/src/mock.rs) behind the mock Cargo feature, mounted via craig-state-default/mock at /partner/{adapter_kind}; DOE’s adapter token is doe_slds. Seed data points 7 of the 11 typed exchange partners at http://mock-server:9090/partner/{adapter_kind}; the other four (cwca, tanf, external_data, tribal) have no SHINES equivalent to mock and keep fake https URLs. Routes: GET /health returns the registry-derived system list; partner traffic goes to GET|POST /partner/{adapter_kind}/…​.

One-Shot Containers

Container Purpose

craig-seed

Generates and loads seed data into all databases once the services are healthy, using the craig-seed binary for scalable and optionally deterministic generation. Two environment variables control it: CRAIG_SEED picks the random seed (the compose file sets 42; leave it empty for a random run) and CRAIG_FAMILIES picks the dataset size (the compose file sets 12, the operational dataset; the binary’s own default of 9 is the byte-identity baseline from #1006). Exits with code 0 on success.

craig-e2e

Playwright browser tests. Only runs on demand via cargo xtask e2e or docker compose --profile e2e run --rm craig-e2e.

Databases

PostgreSQL is pre-configured with nine CRAIG databases:

CREATE DATABASE craig_rules;
CREATE DATABASE craig_cases;
CREATE DATABASE craig_placement;
CREATE DATABASE craig_financial;
CREATE DATABASE craig_exchange;
CREATE DATABASE craig_reporting;
CREATE DATABASE craig_security;
CREATE DATABASE craig_intake;
CREATE DATABASE craig_composition;

craig_intake is pre-created but never populated, because the intake edge is stateless per ADR-017. The databases are defined in devstack/postgres/init.sql, which runs on first startup and also creates the multibackend-profile IdP databases (authentik, zitadel) and the per-service migration-owner/runtime role pairs (#1310). Schema application happens in the per-service craig-<svc>-migrate one-shot gates (ADR-063); serving boot only verifies the schema.

Connection credentials: craig / craig (user/password), configured in docker-compose.yml.

Test Users (Keycloak)

The devstack imports a pre-configured craig realm with seven test users:

Username Password Roles

admin

password

admin

jane.doe

password

caseworker, supervisor

bob.smith

password

caseworker, eligibility_worker

carol.reader

password

readonly

dana.county

password

supervisor, county_director

rita.regional

password

supervisor, regional_director

sam.state

password

admin, state_office

The Keycloak admin console is available at http://localhost:8180 (login: admin / admin).

Two clients are configured:

  • craig-api: a public client with direct access grants (the resource-owner password credentials flow, ROPC), used by the CLI and integration tests

  • craig-ui: a public client using the authorization-code flow with PKCE (Proof Key for Code Exchange), used by the web UI

Token lifetime is 30 minutes. Long-running test sessions may hit 401 errors if tokens expire.

Object Storage (Garage)

Garage provides S3-compatible object storage for file attachments (case documents, exchange files). The start command initializes Garage automatically:

  1. Assigns the single node to a data center zone

  2. Applies the cluster layout

  3. Imports a static API key (GKdeadbeefdeadbeefdeadbeef)

  4. Creates the craig-dev bucket

  5. Grants read/write/owner permissions to the key

This initialization is idempotent: it skips if the bucket already exists.

Configuration

Environment Variables

Services are configured via CRAIG_<SERVICE>__* environment variables defined in docker-compose.yml. Key variables:

Variable Purpose

KC_HOSTNAME

Keycloak’s public URL (default http://host.docker.internal:8180). Keycloak 26 and later use hostname v2, which requires a full URL so the port is pinned in JWT iss claims.

OIDC_ISSUER

Full issuer URL for JWT validation. Override for CI environments where host.docker.internal doesn’t resolve.

WEB_EXTERNAL_URL

External URL for craig-web. Used by the OIDC redirect URI and e2e test base URL.

Split Keycloak URLs

Services use two Keycloak URLs:

This split is necessary because host.docker.internal resolves on the host but not always between containers, while keycloak:8080 resolves between containers but not on the host. (On Linux, host-side resolution itself needs a one-time /etc/hosts mapping — see Local Development.)

Dockerfiles (Not Bind Mounts)

Configuration files (RabbitMQ definitions.json, Keycloak realm export, Postgres init script, Garage config) are baked into images via COPY in their respective Dockerfiles under devstack/. This avoids a Docker Desktop on Windows bug where bind-mounted files are sometimes mounted as directories.

Development Workflow

Making Code Changes

After modifying Rust source code, you must rebuild the Docker containers for changes to take effect:

cargo xtask dev reload

This rebuilds all images (Docker layer caching makes unchanged services fast) and restarts containers while preserving database data.

Running Integration Tests

Integration tests run on the host against the devstack services. Make sure to rebuild containers first if you’ve changed code:

# Rebuild and restart
cargo xtask dev reload

# Run all tests
cargo nextest run --workspace --locked --profile integration

Running E2E Tests

The Playwright e2e test suite runs in a Docker container against the web UI:

# Run the full suite (starts devstack if needed)
cargo xtask e2e

# Filter by test name
cargo xtask e2e --grep "dashboard"

# Debug a specific test interactively
cargo xtask e2e --debug --grep "login"

Connecting the CLI

The CRAIG CLI connects to devstack services via a profile configuration. Create or update ~/.config/craig/profiles.toml (Linux/macOS) or %APPDATA%\craig\profiles.toml (Windows):

[devstack]
oidc_internal_url = "http://localhost:8180/realms/craig"
keycloak_client_id = "craig-api"
rules_url = "http://localhost:8001"
cases_url = "http://localhost:8002"
placement_url = "http://localhost:8003"
exchange_url = "http://localhost:8004"
financial_url = "http://localhost:8005"
reporting_url = "http://localhost:8006"
security_url = "http://localhost:8007"
intake_url = "http://localhost:8008"

Then use:

craig --profile devstack rules sets list

Troubleshooting

Service Fails to Start

Check the service logs for errors:

cargo xtask dev logs craig-rules

If a service fails its health check, the start command will time out after 5 minutes and print the last 20 log lines automatically.

Port Conflicts

If you see errors about ports already in use, another process is binding to one of the devstack ports. Check with:

# Linux/macOS
lsof -i :8080

# Windows (PowerShell)
Get-NetTCPConnection -LocalPort 8080

Common conflicts: another web server on 8080, a local PostgreSQL on 5432, or a previous devstack that wasn’t properly stopped. Use cargo xtask dev clean --confirm to remove any leftover containers and volumes.

RabbitMQ Authentication Errors

The RabbitMQ user is defined in devstack/rabbitmq/definitions.json with a SHA-256 password hash — not via the RABBITMQ_DEFAULT_USER environment variable. This is because the custom rabbitmq.conf (which loads definitions.json) overrides the Docker entrypoint’s generated config.

If you see AMQP authentication failures, verify that definitions.json contains a valid user entry.

Keycloak Token Expiry

Access tokens expire after 30 minutes. If integration tests or CLI commands start returning 401 errors after running for a while, re-authenticate:

craig --profile devstack auth login

Windows-Specific Issues

  • Git Bash path mangling: The xtask sets MSYS_NO_PATHCONV=1 to prevent Git Bash from converting /garage paths to Windows paths.

  • File bind-mount bug: Docker Desktop on Windows sometimes mounts files as directories. The devstack avoids this by baking config files into images via COPY in Dockerfiles. Do not switch to bind mounts.

  • Line endings: The craig-seed container runs a shell script that must have Unix line endings. The Dockerfile’s craig-seed stage runs sed -i 's/\r$//' to strip Windows carriage returns.

Starting Over

When in doubt, wipe everything and start fresh:

cargo xtask dev restart

This removes all containers, volumes, and orphan resources, then rebuilds and re-seeds from scratch.

File Layout

Path Purpose

docker-compose.yml

Service definitions, ports, environment variables, health checks

xtask/

Cargo xtask: the cross-platform home of cargo xtask dev and cargo xtask e2e

devstack/postgres/

PostgreSQL Dockerfile and init.sql (creates per-service databases)

devstack/rabbitmq/

RabbitMQ Dockerfile, rabbitmq.conf, and definitions.json (user/vhost)

devstack/keycloak/

Keycloak Dockerfile and craig-realm.json (realm import)

devstack/garage/

Garage Dockerfile and garage.toml (storage config)

devstack/seed/

seed.sh, the shell script that invokes the craig-seed binary and loads rule sets from JSON. The seed container image is built from the main Dockerfile (target: craig-seed).

tools/craig-seed/

Rust binary that generates deterministic, scalable seed SQL. Controlled via CRAIG_SEED and CRAIG_FAMILIES environment variables.

Dockerfile

Multi-stage build for all CRAIG Rust services and tools

tests/e2e/

Playwright test suite — Dockerfile, specs, page objects, fixtures

CI/CD

The GitLab CI pipeline (.gitlab-ci.yml) has its own devstack setup that does not use cargo xtask dev. It runs Docker-in-Docker (dind: a sidecar service container runs a second Docker daemon, which the CI job drives over the network) with environment variable overrides:

The CI pipeline boots the full devstack inline, waits for health, and runs the e2e tests with npx playwright test directly (not via the Docker container). See .gitlab-ci.yml for the full configuration.

Edit this page · latest