Developer Guide

On this page

Prerequisites

  • Rust 1.97.0: rust-toolchain.toml pins the exact version (and the workspace rust-version tracks it), so rustup installs the right toolchain automatically

  • Docker with Docker Compose v2

  • Node.js 18+ (for Antora docs and E2E tests)

  • Git with core.hooksPath configured (see below)

Quick Start

# Clone the repository
git clone https://gitlab.com/gadhs/application/ccwis/craig.git
cd craig

# Start the full devstack (Postgres, RabbitMQ, Keycloak, Garage, all services)
cargo xtask dev start

# Run the test suite
cargo nextest run --workspace --locked --profile integration

# Activate pre-push hooks
git config core.hooksPath .githooks

The devstack script builds all Docker images, starts infrastructure, initializes the S3 bucket (Garage), loads seed data, and waits for all services to become healthy.

Project Structure

craig/
├── Cargo.toml                  # Workspace root (66 members; living inventory in shared-crates.adoc)
├── Dockerfile                  # Multi-stage build for all services
├── docker-compose.yml          # Unified devstack config
├── xtask/                      # cargo xtask dev / e2e / validate / perf / security / sbom / check-docs
├── crates/                     # Shared libraries
│   ├── craig-common/           # Errors, IDs, pagination, settings, telemetry
│   ├── craig-auth/             # JWT claims, JWKS, auth middleware
│   ├── craig-db/               # PostgreSQL connection pool + migrations
│   ├── craig-mq/               # RabbitMQ publisher/subscriber
│   ├── craig-api/              # Axum server scaffold, CORS, Swagger UI
│   ├── craig-store/            # Object storage abstraction (S3/local)
│   ├── craig-reference/        # Domain enums, FIPS codes, AFCARS/NCANDS translations
│   ├── craig-intake-sdk/       # Public intake API client library
│   └── craig-test-lib/         # Integration test helpers (Tempest-inspired)
├── services/
│   ├── craig-rules/            # Rules engine (port 8001)
│   ├── craig-cases/            # Case management (port 8002)
│   ├── craig-placement/        # Placement & foster care (port 8003)
│   ├── craig-exchange/         # Data exchange & ICPC (port 8004)
│   ├── craig-financial/        # Financial & claims (port 8005)
│   ├── craig-reporting/        # Reporting & analytics (port 8006)
│   ├── craig-security/         # Security & compliance (port 8007)
│   ├── craig-intake/           # Public intake (port 8008; stateless edge — no DB/MQ, ADR-017)
│   ├── craig-composition/      # Composition layer engine (port 8009)
│   ├── craig-web/              # Web UI — BFF (port 8080)
│   └── craig-cli/              # CLI client
├── tools/
│   └── craig-seed/             # Deterministic seed data generator
├── rulesets/                   # JDM rule set files (georgia/, texas/)
├── tests/e2e/                  # Playwright E2E tests
├── devstack/                   # Dockerfiles for infra (postgres, rabbitmq, keycloak, garage)
└── docs/                       # Antora documentation site

Architecture Overview

CRAIG follows a modular microservice architecture inspired by OpenStack:

  • Each service owns its database and communicates via REST APIs and RabbitMQ events

  • A shared set of Rust crates provides common functionality (auth, DB, MQ, API scaffolding)

  • The CLI and Web UI consume the same public APIs

  • All services are stateless and horizontally scalable

Shared Crates

Crate Purpose

craig-common

ApiError (RFC 9457 Problem Details), Id (UUID v7 wrapper), PageRequest/PageResponse (pagination), ServiceSettings (config), telemetry init

craig-auth

JWT Claims extraction, JWKS fetching + auto-refresh, AuthLayer middleware

craig-db

DbPool wrapping sqlx::PgPool: connection management, health checks, migration runner

craig-mq

Publisher (publish to craig.events exchange), Subscriber (competing + exclusive consumers), EventEnvelope (routing key, payload, metadata)

craig-api

ApiServer: builds the Axum router with auth middleware, CORS, body limits, compression, tracing, Swagger UI. AppState holds DB pool + auth layer + the MQ readiness contract (MqRequirement) + the worker registry (WorkerHealth). Also provides bootstrap() (common service startup: settings, telemetry, DB, auth, MQ), the Supervisor worker-supervision module (#1186 / ADR-061), SecurityAddon (shared OpenAPI Bearer JWT modifier), and shutdown_signal().

craig-store

Store: the object storage abstraction (S3 in devstack/prod, local filesystem in tests), with upload validation and filename sanitization.

craig-reference

Domain enums (Gender, Race, PlacementType, etc.) via strum, FIPS codes (state + administrative unit), AFCARS/NCANDS field translations, validation functions. 48 unit tests.

craig-test-lib

TestHarness: boots a devstack client with auth tokens, provides builders for all entity types and CRUD helpers. Exports pinned Keycloak user UUIDs (ADMIN_SUB, SUPERVISOR_SUB, CASEWORKER_SUB).

Service Initialization Pattern

Every stateful service dispatches its non-serving process modes first (the ADR-063 migration gate + --print-openapi), then boots VERIFY-ONLY through craig_api::bootstrap_verified — data plane (env, settings, telemetry, database) → schema verdict (verify_schema under the compatibility floor, so a JWKS/RabbitMQ failure can never mask a schema refusal) → control plane (auth, MQ). Serving binaries never apply DDL; schema application belongs to the compose craig-<svc>-migrate gates (deployment guide § Migration gates):

use craig_api::{BootstrapResult, Supervisor, WORKER_DRAIN_DEADLINE};

// ADR-063: `<binary> migrate` (the gate) and --print-openapi exit here.
let migrator = sqlx::migrate!();
if craig_api::run_process_modes_if_requested("craig-cases", &migrator, &ApiDoc::openapi())
    .await?
{
    return Ok(());
}
// Verify-only boot: data plane -> schema verdict -> control plane.
let (settings, BootstrapResult { db, auth, publisher, subscriber, .. }) =
    craig_api::bootstrap_verified("CRAIG_CASES", "craig-cases", &migrator).await?;
let supervisor = Supervisor::install();

// Start event subscribers (service-specific bindings)
subscriber.subscribe("craig-cases.events", &["intake.created", ...], handler).await?;

// Build router + serve under the supervisor's token (#1186)
let router = ApiServer::router(state, service_routes, opts, Some(ApiDoc::openapi()));
ApiServer::serve(router, settings.port, supervisor.token()).await?;
supervisor.drain(WORKER_DRAIN_DEADLINE).await;
supervisor.check_exit()?;

bootstrap_verified() returns (ServiceSettings, BootstrapResult). The prefix ("CRAIG_CASES") determines environment variable names (e.g. CRAIG_CASESPORT, CRAIG_CASESDATABASE_URL); the plain bootstrap() composition remains for callers with no schema to verify against.

Canonical orchestrator shape

A stateful service’s main is an orchestrator, not a place for logic: it wires dependencies, spawns workers, builds the router, and serves — each substantive step delegated to a named helper so main itself stays small (≈50–80 LOC). All eight stateful services + intake + the BFF follow this shape:

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    // 0. non-serving process modes exit first (ADR-063: the migration
    //    gate `<binary> migrate`, then --print-openapi)
    let migrator = sqlx::migrate!();
    if craig_api::run_process_modes_if_requested("craig-cases", &migrator, &ApiDoc::openapi())
        .await?
    {
        return Ok(());
    }
    // 1. VERIFY-ONLY bootstrap (ADR-063 M3b): data plane -> schema
    //    verdict under the compat floor -> control plane. Serving
    //    binaries never apply DDL — the compose gates own that.
    let (settings, br) =
        craig_api::bootstrap_verified("CRAIG_CASES", "craig-cases", &migrator).await?;

    // 2. the worker supervisor owns THE shutdown token (#1186 / ADR-061);
    //    every worker, the router's cleanup task, and serve derive from it
    let supervisor = Supervisor::install();
    let shutdown = supervisor.token();
    let http_client = craig_bootstrap::build_shared_http_client(
        env!("CARGO_PKG_NAME"), env!("CARGO_PKG_VERSION"))?;

    // 3. authz engine (boot → retry → fall back; spawn invalidation + TTL refresh)
    let authz = boot_authz_engine(AuthzBootSpec { /* … */ }).await?;

    // 4. object store + any service-specific resources (encryptor, registries, …)
    let object_store = craig_bootstrap::init_object_store()?;

    // 5. background workers (one helper; outbox + service-specific subscribers/scanners)
    spawn_workers(&br, &settings, object_store.clone(), shutdown.clone());

    // 6. router (one helper layering AppState + service_routes + standard extensions) + serve.
    //    build_router takes &supervisor: AppState.workers = supervisor.health() — router-internal
    //    workers derive shutdown from that registry (WorkerHealth::token, #1233); one source.
    let router = build_router(&settings, &br, authz.engine.clone(), http_client, object_store, &supervisor);
    ApiServer::serve(router, settings.port, shutdown).await?;

    // 7. bounded worker drain, then exit NONZERO if a Critical worker died
    //    (fail-fast — the orchestrator restarts the pod).
    supervisor.drain(WORKER_DRAIN_DEADLINE).await;
    supervisor.check_exit()?;
    Ok(())
}

The fixed sequence is process modes → bootstrap_verified → (deps) → spawn_workers → build_router → ApiServer::serve → supervisor.drain → supervisor.check_exit. Workers are spawned in one spawn_workers() helper (always craig_bootstrap::spawn_outbox_worker plus the service’s own inbox/send/cleanup/upload-attempt-reconciler workers); the router is assembled in one build_router() helper that mounts the domain service_routes and calls craig_bootstrap::attach_standard_extensions (authz + jurisdiction + service-token + actor-issuer). build_router takes &supervisor and derives AppState { mq, workers } (the health handlers' MQ contract + worker registry) from it; the shared request-claims-retention pruner ApiServer::router spawns takes its shutdown from that same registry (WorkerHealth::token, #1233 — the retired ServerOptions.shutdown second channel is gone), so it breaks on SIGTERM/SIGINT rather than parking in its hourly sleep — and registers as an Observed worker while it’s at it. Service-specific extensions (encryptor, deadline config) are layered by the helper, not inlined in main. craig-exchange additionally boots the state-bundle registries (bundle_orchestrator::boot_exchange_registries) at step 4; craig-web’s wide protected_routes(state) route table is the sanctioned § Style carve-out to the small-helper rule.

How to Add an Endpoint

  1. Define the handler in src/routes/<module>.rs:

    #[utoipa::path(
    get,
    path = "/widgets",
    responses((status = 200, body = PageResponse<Widget>)),
    security(("bearer" = []))
)]
pub async fn list_widgets(
    State(state): State<AppState>,
    Extension(claims): Extension<Claims>,
    Extension(authz): Extension<Arc<dyn AuthzEngine>>,
    Extension(jurisdiction): Extension<Jurisdiction>,
    Query(page): Query<PageRequest>,
) -> Result<Json<PageResponse<Widget>>, ApiError> {
    // Plan A § Step 8+: every protected handler routes through the
    // authz engine. For LIST endpoints, `auto_scope_list` returns
    // the SQL filter to apply (All / AssignedWorker(sub) / Denied).
    let scope = authz
        .auto_scope_list(
            &claims,
            ResourceType::Widget,
            Action::List,
            jurisdiction.as_str(),
        )
        .await?;
    if matches!(scope, ListScope::Denied) {
        return Err(ApiError::Forbidden);
    }
    // Translate scope → SQL filter, query DB, return response.
}

For single-row endpoints, use authz.check(claims, resource_ref, Action::Read).await?. Per-resource policies live in rulesets/<jurisdiction>-authz-widget.json. Roles, scoping, and approval workflows are inputs to the JDM (JSON Decision Model) decision table; do not add claims.require_caseworker_or_above() calls.

Mutations whose authorization consumed a concurrently-mutable resource field (e.g. the assignment-gated placement/kinship updates, where a policy rule tests resource.assigned_worker_sub and an inbox handler rewrites it) must follow ADR-060: keep the decisive pre-tx authz.check (fail-fast, no locks), then inside the mutation transaction re-read the row FOR NO KEY UPDATE and compare the policy-consumed field against the pre-read value the decision consumed — refuse with 409 CONCURRENT_MODIFICATION on drift. Never re-run the authz engine under a held row lock (the evaluator is strictly serial and a cache miss can wait on HTTP + a second same-pool audit transaction — ADR-060 D2 names the mechanisms).

  1. Register the route in src/routes/mod.rs:

    pub fn routes() -> Router<AppState> {
    Router::new()
        .route("/widgets", get(list_widgets).post(create_widget))
        .route("/widgets/{id}", get(get_widget).put(update_widget))
}

  2. Add to OpenAPI in src/api.rs (import SecurityAddon from the shared crate):

    use craig_api::SecurityAddon;

#[derive(OpenApi)]
#[openapi(
    paths(routes::widgets::list_widgets, routes::widgets::create_widget),
    components(schemas(Widget, CreateWidgetRequest)),
    modifiers(&SecurityAddon),
)]
pub struct ApiDoc;

  3. Write integration tests in tests/api/widgets.rs using TestHarness.

How to Add a Service

  1. Create services/craig-<name>/ with Cargo.toml, src/main.rs, src/lib.rs

  2. Add to [workspace.members] in root Cargo.toml

  3. Create migrations/ directory with SQL files

  4. Add database to devstack/postgres/init.sql

  5. Add service block to docker-compose.yml

  6. Add Dockerfile stage in root Dockerfile

  7. Add CLI commands in services/craig-cli/

  8. Add web routes in services/craig-web/

  9. Update profiles.toml schema in CLI (all Profile URL fields are required)

Service-add checklist

The wiring that is easy to miss when standing up a new service (each item is a silent-failure trap if skipped):

  • Keycloak identity, on every realm client. Add a per-service client, a service:craig-<svc> role, and an aud-craig-<svc> audience mapper on every realm client — a missing audience mapper fails silently with InvalidAudience.

  • Config env vars. Set CRAIG_<SVC>CLIENT_ID / CRAIG_<SVC>CLIENT_SECRET, and the required CRAIG_<SVC>__ADMIN_UNIT_LABEL.

  • Runtime-file-reading services. A service that reads files at runtime must COPY those files into the image and set WORKDIR so the relative paths resolve.

  • Provisioning only happens on a volume wipe. The database and realm are provisioned only on cargo xtask dev reseed / restart (a volume wipe) — not on cargo xtask dev start.

  • New ResourceType = fleet-wide authz JSON. A new ResourceType requires a {jurisdiction}-authz-<resource>.json for every jurisdiction (georgia, texas, …).

  • routes() must not include /v1. The version prefix is mounted by the shared API scaffold; the service’s own routes() must not repeat it.

  • xtask CRAIG_SERVICES drives more than compose. The CRAIG_SERVICES list in xtask (docker.rs) also drives Keycloak-client and actor-key provisioning. A no-OIDC sidecar (e.g. the keyring; standalone/SHINES intake instances) belongs in PORT_MAPPINGS but not in CRAIG_SERVICES.

How to Add a Web UI Page

The web UI (craig-web) uses the BFF (Backend-for-Frontend) pattern:

  1. Create a template in services/craig-web/templates/<module>/:

    {% extends "base.html" %}
{% block title %}My Page{% endblock %}
{% block content %}
<h1>My Page</h1>
<!-- htmx for dynamic updates, Alpine.js for client-side state -->
{% endblock %}

  2. Add a route handler in services/craig-web/src/routes/<module>.rs:

    pub async fn my_page(
    session: Session,
    State(state): State<WebState>,
) -> Result<impl IntoResponse, WebError> {
    let ctx = build_context(&session).await?;
    let data = state.api_client.get("/v1/widgets", &ctx.token).await?;
    Ok(MyPageTemplate { ctx, data })
}

  3. Register in the router — add to protected_routes() in src/routes/mod.rs.

  4. Add nav link in templates/base.html (with appropriate role guard).

Key Patterns

  • Flash messages: set_flash(&session, "Widget created"), then redirect, then get_flash() in the template

  • Inline forms: Alpine.js x-data="{ showForm: false }" + toggle button for sub-resource creation

  • Detail fetches (#992): detail handlers fetch the primary record via fetch_detail (routes/mod.rs) — a genuine upstream 404 renders render_not_found(); every other failure (5xx/transport/deserialize) ?-propagates as BffApiError and renders the 502 upstream-failure page (401 redirects to /login). Never .ok()-swallow the primary fetch; secondary widget fetches may degrade silently with a // silent fallback: comment

  • RBAC: Route-level gates on the restricted surfaces (#476): the /security/* pages mount require_admin_or_supervisor (partner API-key/signer-key management mounts require_admin_only), /studio/composition mounts require_admin_only, and privileged actions (reporting approve/transmit, payment approve/issue/clear) mount require_admin_or_supervisor. /rules is deliberately ungated at the route level — admin-only actions are hidden in templates via ctx.user.is_admin(). require_auth authenticates every protected route; authorization is always re-enforced by the backend APIs via Bearer JWT.

How to Add Sort & Search to a List Endpoint

All list endpoints follow a standard pattern for server-side sort and search.

1. Backend store functions

Add validated_sort_column() to your store module (columns legitimately differ per table), call the SHARED craig_common::pagination::validated_sort_dir() for the direction (#996 — never a per-module copy: the direction allowlist is a security-relevant mitigation and lives in ONE tested place), then add search, sort_by, sort_dir parameters to both the list_*paged and count* functions:

fn validated_sort_column(sort_by: Option<&str>) -> &str {
    match sort_by {
        Some("name") => "name",
        Some("status") => "status",
        _ => "created_at",
    }
}

Use format!() with the validated values in the SQL query — search columns use parameterized ILIKE:

let col = validated_sort_column(sort_by);
let dir = craig_common::pagination::validated_sort_dir(sort_dir);
let sql = format!(
    "SELECT * FROM widgets
     WHERE ($3::TEXT IS NULL OR status = $3)
       AND ($4::TEXT IS NULL
            OR name ILIKE '%' || $4 || '%')
     ORDER BY {col} {dir}
     LIMIT $1 OFFSET $2"
);

2. API handler

Add search, sort_by, sort_dir as Option<String> to the query params struct and pass .as_deref() to store functions.

3. BFF route handler

Add the same fields to the web route’s params struct. Append them to the backend API URL and pass to the template context. Use extra_params() to build query strings for pagination/sort links.

4. Template

Add sortable headers and search toolbar:

<th class="th--sortable{% if sort_by == "name" %} th--{{ sort_dir }}{% endif %}">
  <a href="/widgets?sort_by=name&sort_dir={% if sort_by == "name" && sort_dir == "asc" %}desc{% else %}asc{% endif %}{{ extra }}">Name</a>
</th>

How to Add a New List View

A complete list view combines pagination, sort, search, column resize, and (optionally) cross-service name resolution.

1. Backend

Follow the sort & search pattern above. Ensure your list_*paged and count* store functions accept search, sort_by, sort_dir parameters.

2. BFF route handler

Use fetch_page(), total_pages(), extra_params(), and the pagination window variables (win_start, win_end). If your list references IDs from another service, use the batch-lookup pattern:

// Collect foreign-key UUIDs from the page
let case_ids: Vec<_> = resp.data.iter().map(|p| p.case_id).collect();
let lookup = batch_lookup(&state.api, &state.config.cases_url, token, &case_ids, &[]).await;

// Merge resolved names into view models
for item in &mut resp.data {
    if let Some(name) = lookup.cases.get(&item.case_id) {
        item.case_display = name.clone();
    }
}

Add #[serde(default)] on enrichment fields so deserialization from the API (which doesn’t include these fields) works.

3. Template

<table class="data-table data-table--resizable" x-data="tableResize('my-table')">
  <thead>
    <tr>
      <th class="th--sortable{% if sort_by == "name" %} th--{{ sort_dir }}{% endif %}">
        <a href="/my-list?sort_by=name&sort_dir=...{{ extra }}">Name</a>
      </th>
      <!-- more columns -->
    </tr>
  </thead>
  <tbody>...</tbody>
</table>

Key classes:

  • data-table—​resizable + x-data="tableResize('id')" — enables column resize with localStorage persistence

  • th—​sortable + th—​asc/th—​desc — sort indicators

  • .table-toolbar — search bar + filter tabs

  • .pagination-bar — numbered pagination with page windowing

For enriched fields, always include a fallback to truncated UUID:

{% if item.display_name.is_empty() %}{{ item.id.to_string()[..8] }}…{% else %}{{ item.display_name }}{% endif %}

How to Add a Rule Set

  1. Create the JDM file in rulesets/<jurisdiction>/<name>.json

  2. Use the Swagger UI at http://localhost:8001/swagger-ui or the CLI:

    # Create a new rule set
craig rules create --name "georgia-my-assessment" \
    --description "My custom assessment rules" \
    --content @rulesets/georgia/my-assessment.json

# Or import into an existing rule set
craig rules import <rule-set-id> --file rulesets/georgia/my-assessment.json

  3. Rule set names are prefixed with jurisdiction (set via CRAIG_RULES__JURISDICTION env var).

Testing

Unit Tests

cargo test --workspace --lib --locked

Integration Tests

Integration tests use craig-test-lib and require a running devstack:

cargo xtask dev start                                          # Start devstack
cargo nextest run --workspace --locked --profile integration   # Run all tests (unit + integration)

E2E Tests (Playwright)

cargo xtask e2e                      # Run E2E tests (starts devstack if needed)
cargo xtask e2e --grep "dashboard"   # Run specific tests

E2E tests use Page Object Model pattern. The cargo xtask e2e command automatically rebuilds the Playwright container before running tests, so no manual rebuild step is needed.

Pre-Push Hook

The .githooks/pre-push hook is the primary quality gate. All checks run sequentially and any failure rejects the push:

  1. cargo xtask validate --skip-docker: repository visibility check (Kerckhoffs enforcement), commit signing, mandatory Tier-3 docs present, Tier-1 doc drift check against the claude-quickstart template, SPDX headers on every .rs file, JDM ruleset schema validation, gitleaks secret scan (soft-fail if not installed), cargo deny check, cargo fmt --check --all, cargo clippy --workspace --locked — -D warnings, cargo build --workspace --locked, cargo nextest run --workspace --locked --profile integration.

  2. cargo xtask dev reseed: wipes data volumes and re-seeds, because the integration tests that validate runs leave data behind and the e2e suite needs a clean seed.

  3. cargo xtask e2e: the full Playwright suite against the live devstack (the Playwright report gives live counts; they drift too fast to pin here).

  4. cargo xtask perf --profile smoke: a k6 smoke test (1 virtual user, 29 requests across all 9 services, about 5 seconds).

  5. Security regression: cargo xtask security --skip-zap --skip-fuzz (auth boundary, injection, and infrastructure phases, about 5 seconds).

Activate with: git config core.hooksPath .githooks

The hook uses set -euo pipefail, so any single failure aborts the push.

Devstack Commands

Command Description

cargo xtask dev start

Start / resume the devstack (staleness-aware; skips rebuild if nothing changed since the last write_markers)

cargo xtask dev start --force

Force a full cold start regardless of staleness markers

cargo xtask dev reload

Cached rebuild and restart without wiping data (compose down + bring_up(false))

cargo xtask dev restart

Full wipe: tear_down(true) + bring_up(true) — removes built images; ~20 minutes

cargo xtask dev reseed

Wipe data volumes + restart with cached images; re-runs craig-seed. Used between validate (DB pollution) and e2e (clean seed) in the pre-push hook. ~30–60 seconds.

cargo xtask dev restart-service <name>

Restart a single CRAIG application service (preserves data) and wait for /healthz. Name is validated against docker::CRAIG_SERVICES.

cargo xtask secrets init / init --for-ci

Mint the dev age identity (persisted ~/.config/sops/age/keys.txt, shared with canopy) / an ephemeral CI or recovery keypair (nothing persisted; paste-one-line instructions). ADR-064.

cargo xtask secrets decrypt / check

Emit the store as validated dotenv lines / run the non-disclosing verification (schema v1, 32-byte key, .sops.yaml↔ciphertext recipient parity) — the same check the CI secrets-policy job runs.

cargo xtask secrets edit

Rotate or edit store values in $EDITOR inside the devtools container (terminal editors only); commit the result — every mutating devstack command then adopts it (teardown → publish → reseed).

cargo xtask secrets add-recipient / remove-recipient <age1…​>

Transactional recipient management: add re-wraps under a root-staged policy with byte-exact rollback; remove is REAL revocation (from-scratch re-encrypt = fresh sops data key + rotated field key; refuses self-removal and the 3-recipient floor).

cargo xtask secrets bootstrap-store / fork-rebootstrap [--force]

Create the initial store (refuses overwrite; ≥3 recipients) / the destructive fork path (a fork can never rewrap the upstream store).

cargo xtask dev stop

Stop containers, preserve volumes

cargo xtask dev clean --confirm

Stop and remove all volumes (--confirm required)

cargo xtask dev status

Show service health, ports, uptime, staleness report

cargo xtask dev logs [service]

Follow logs (all or specific service; name validated against compose config)

cargo xtask e2e [args]

Run Playwright E2E tests

cargo xtask import-subsidy-history --manifest <json> --records <jsonl> [--api-base <url>] [--batch-id <id>] [--out <dir>] [--throttle-rps <n>] [--yes] [--abort] [--bearer-token <t>]

The #1071/ADR-057 conversion operator tool: drives the craig-financial import surface through one batch lifecycle — create-or-resume → stage every JSONL record (throttled; replays are progress) → reconciliation report + rejection manifest into --out → finalize with resume → post-finalize verification + closeout checklist. Authenticates as the responsible STATE-OFFICE human via device-code OAuth (CRAIG_IMPORT_OIDC_ISSUER + CRAIG_IMPORT_OIDC_CLIENT; --bearer-token is the devstack/CI escape hatch — no service-principal mode exists). An absent manifest checksum is COMPUTED from the records file via the shared contracts canonicalization (a present one that disagrees bails before staging). --abort = dry-run closeout: stage + report, then ABORT instead of finalizing — nothing ever becomes generator-visible. The tool NEVER triggers payment generation (that authority stays with the deployment guide's SHINES-conversion-cutover runbook)

cargo xtask docs deadlinks

Dead-link check (#1299): external URLs via lychee against the checked-in .lychee.toml (AsciiDoc link macros de-glued by docs/extensions/lychee-preprocess.sh), plus a full Antora build with --log-failure-level=warn when antora is on PATH (soft-skip otherwise — the advisory docs-xref-check CI job enforces that leg). Non-zero on any dead link; requires cargo install lychee --locked

Operating the Subsidy Payment Generator (#1068 / ADR-053)

The monthly subsidy generator (craig-financial) is OFF by default (CRAIG_FINANCIAL__SUBSIDY_GENERATOR_INTERVAL_SECONDS=0) and Georgia-gated; the devstack/CI compose files enable it at 3600s. Operational notes:

  • Disable: set the interval to 0 (or remove the env) and restart craig-financial — the scheduler simply never spawns. The manual trigger (POST /v1/financial/subsidy-payments/generate, admin/supervisor in GA) keeps working for bounded historical repair ([current − 12, current]).

  • One run at a time: the deployment-wide lease (pg_try_advisory_lock on subsidy-gen) makes concurrent triggers/replicas a 409 (generation-already-running) — retry after the running batch. A crashed run frees the lease with its connection; no cleanup step exists or is needed.

  • Partial errors: a run returns 200 with errors > 0 when individual children failed (their transactions rolled back; the batch continued). Locate them via the WARN logs keyed by the report’s run_id, fix the cause, re-run the month — the idempotency key makes re-runs free (already_existed).

  • Corrections: backdated ledger edits self-heal UNDISBURSED months on the next run (void + regenerate, voided_stale); DISBURSED months are frozen and queue in the subsidy_disbursed_derivation_mismatch invariant sweep for manual remediation. The subsidy_perdiem_same_month_overlap sweep is the per-diem/subsidy handoff queue.

  • Metrics: the subsidy_generation_* counter family (generated / voided_stale / derivation_mismatch / skipped / errors) plus one INFO line per run (subsidy generation run completed, all counters + run_id).

Operating the Review Sweep (#1096 / ADR-054 U4)

The review-enforcement sweep (craig-financial) is OFF by default — the tick AND each write mode are separate consent knobs, all nested under CRAIG_FINANCIALSUBSIDY_REVIEW_SWEEP* and Georgia-gated. The devstack/test posture keeps the scheduler off entirely (sweep enforcement MUTATES agreement status, unlike the additive+idempotent generator); docker-compose.sweep-demo.yml demonstrates the full-consent shape.

  • Rollout is observe → materialize → enforce, knob by knob:

    1. __INTERVAL_SECONDS=3600 alone — a pure OBSERVE cadence: one run row + one completion event per tick, the two backlog gauges populate (subsidy_sweep_leg1_backlog / leg2_backlog), zero agreement writes. Watch the backlogs and the run records (GET /v1/financial/subsidy-reviews/sweep/runs or the craig-web Review Sweep page) until the numbers are understood.

    2. + __MATERIALIZE=true — repairs missing first review slots from term anchors (idempotent; converges to zero per run).

    3. + AUTO_SUSPEND=true, later + AUTO_TERMINATE=true — the enforcement legs, recorded as approval_level = "system" (the knob IS the authority, ADR-054).

  • Manual mass action is two-step: POST …/sweep/preview pins the actionable sets and mints a ONE-TIME execute token (15-minute expiry; only its SHA-256 digest persists); POST …/sweep/{run_id}/execute demands the regional SweepExecute office proof, CASes previewed → executing (single-use — replays get a 409 naming the state), enforces ONLY the pinned set with per-row in-tx re-verification (drift skips, never forces), and records the EXECUTOR’s office. The craig-web page drives exactly this flow.

  • Recovery from an enforcement: a sweep-suspended agreement’s overdue review stays completable/reschedulable (the ADR-054 state matrix legalizes work on suspended heads) — completing or rescheduling it removes the agreement from leg 1; reinstatement (terminated/suspended → active) is an admin-session transition bound by the office matrix. Suspension enqueues exactly the CURRENT month for the #1068 generator’s reconcile drain (the undisbursed month voids on its next run).

  • Run records are durable-first: every run persists before it processes and is finalized completed/failed (a crashed run is stamped failed, never left looking in-flight); expired marks a preview whose window elapsed. The per- agreement failure set on the run row is the triage queue.

  • Report-only invariant queues: subsidy_review_overdue_unsuspended (leg-1 twin, 30-day grace), subsidy_three_months_unterminated (leg-2 twin), subsidy_paper_anchor_missing, and subsidy_review_chain_broken run under cargo xtask invariants — the two non-leg queues have no API surface and remain xtask/operator territory (recorded U5 scope note).

  • Seed fuse: the devstack seed pins as-of-relative review dues (11/5 months out), so a fresh stack’s sweep queues are EMPTY by construction — a nonzero backlog on a fresh devstack means the seed grew overdue state (fix the seed, not the sweep).

  • Metrics: subsidy_sweep_* — run/suspension/termination/materialized/drift/error counters, both backlog gauges, run_duration_seconds, last_success_timestamp_seconds, and lease_skips_total (the subsidy-sweep lease is deployment-global; skips are contention, not failures).

Operating ERR Enrollment (#1069 / ADR-055)

The one-shot native ERR creation endpoint (POST /v1/financial/subsidy-agreements) is OFF by default and Georgia-gated: enabling CRAIG_FINANCIALSUBSIDY_ERRENABLED=true (F4) is the operator’s RECORDED consent to the ⁂ #1073 money policies — until then the endpoint refuses with a typed 403 naming the knob. Operational notes:

  • Creation never pays: the create transaction writes the agreement (identity
    parties + active interval + terms + review anchors) and ENQUEUES every birth month for the #1068 generator — zero payment rows. Payments appear on the next generator tick (or a manual POST /v1/financial/subsidy-payments/generate run), so payment visibility follows the generator cadence, not the enrollment.

  • The 120-day approval clock is report-only: foster_home_approval_due is recorded at creation (placement start + 120 calendar days, GA 22.8) and surfaced as approval_clock_overdue for active ERR; an overdue clock never auto-terminates. Termination is a manual transition with reason foster_home_approval_lapsed, witness-checked in the store (a recorded clock that has actually run out; no clock = fail closed).

  • Per-diem handoff is sweep leg 3: detection and the subsidy_sweep_leg3_backlog gauge run on every sweep regardless of knobs; CRAIG_FINANCIALSUBSIDY_REVIEW_SWEEPAUTO_PER_DIEM_HANDOFF=true consents ONLY the scheduled handoff TERMINATION (reason per_diem_begins, truth-dated to the per diem’s start). The manual preview/execute path always enforces all pinned legs — the execute body must acknowledge the per-leg counts the operator rendered (acknowledged_suspend/_terminate/_handoff; any mismatch with the pinned sets is a 409 sweep-preview-stale).

  • Moved children are report-only: a per diem beginning on a DIFFERENT placement than the agreement’s anchor is a move, not a handoff — counted (leg3_moved_children) and listed on the run record, never auto-terminated (manual child_no_longer_in_home).

Operating Guardianship (SG/NRSG) Enrollment (#1070 / ADR-056)

The two-step guardianship-subsidy flow (SG relative / UAS 552, NRSG non-relative / UAS 550) is OFF by default and Georgia-gated: enabling CRAIG_FINANCIALSUBSIDY_SGENABLED=true (ONE knob for the family, U4) is the operator’s RECORDED consent to the ⁂ #1073 money-policy readings. It gates the create arms + the witnessed activation ONLY — existing agreements' lifecycle (generator, ALL transitions including the corrective guardianship_finalized, reviews, sweep) is deliberately grandfathered, so a knob-off flip never strands owed money. The operator walk, in order:

  • Enroll (sign): New Guardianship on the subsidy-agreements list — visible to county_director/admin sessions (U6; the backend’s county-floor CreatePending proof remains the authority, the UI gate only keeps the UX honest). Pick the program (Relative 552 / Non-relative 550 — the caller’s classification; relationship evidence is REQUIRED for 552, the TANF degree of relationship, and optional documentation for 550), anchor on the child’s ACTIVE placement (the residence-under-supervision evidence — no placement-type constraint, no residence floor at create), and enter the A&A signature date — it must be strictly BEFORE the guardianship transfer (same-day signing is refused at activation). Submitting RECORDS the manual-preconditions attestation (citizenship/residency, child income eligibility, funding availability, the signing caregiver being the placement’s caregiver — 22.8 facts CRAIG does not verify; #1114 tracks hard checks). The form mints client_request_id once per RENDER, so a double submit replays to the SAME pending agreement (F8); success lands on the PENDING agreement’s detail page.

  • Finalize (end the placement): end the anchoring placement with reason guardianship. The end form runs the SS6 gate FIRST — if no pending SG/NRSG agreement is VISIBLE to your session (agreement reads are visibility-scoped, so the vocabulary is "none visible", never "none exists") or the lookup could not verify, the end is REFUSED with a create-it-first prompt; checking the acknowledgement box proceeds anyway (support stops — the two acknowledged outcomes stay distinguishable in the post-end flash). The acknowledgement NEVER suppresses a found agreement: that path proceeds and the text-only flash names the pending agreement ready to activate.

  • Hand off (terminate the open ERR): activation refuses while ANY other open agreement exists for the child (the refusal names it), so terminate the predecessor ERR first — the detail-page transition form, reason guardianship_finalized (typed into the reason field; county floor — guardianship_dissolved is the regional-floor reversal), with the business date EQUAL to the placement’s end date: the handoff termination is truth-dated, and any other date is refused naming the expected one.

  • Activate: the Activate form renders on the pending agreement’s detail page for FLOW-SHAPED native rows only (family program + pending + the F8 stamp + no import provenance; imported agreements never enter native activation — open-pending imports are UNREPRESENTABLE since #1071/ADR-057, so the refusal is belt-only). The court-order evidence key is REQUIRED; the TANF-terminated date is an OPTIONAL past-dated attestation that shifts the payments-begin month later — TANF still active ⇒ activate after it ends. The transfer date is DERIVED from the anchoring placement’s guardianship ending, never typed; every witness refusal flashes VERBATIM (that text IS the operator guidance — end the placement first, wrong reason, residence floor short, signing not before the transfer, …). A re-submit after success is a recognized 200 replay, not an error.

  • Money: activation stores payments_begin_month = max(month after transfer, month after TANF termination) and enqueues the window for the #1068 generator — activation itself writes NO payment rows, so payments appear on the next generator tick (or a manual generate run), from the boundary month on. A transfer ON the 1st leaves that month unpaid by BOTH programs (the truth-dated ERR is inactive at month start and SG sits behind its boundary — ⁂ #1073); mid-month transfers keep ERR’s full final month (standing whole-month semantics).

Keycloak Test Users

Username Password Roles

admin

password

admin

jane.doe

password

caseworker, supervisor

bob.smith

password

caseworker, eligibility_worker

carol.reader

password

readonly

dana.county

password

supervisor, county_director

rita.regional

password

supervisor, regional_director

sam.state

password

admin, state_office

Admin console: http://localhost:8180 (admin/admin)

Conventions

  • UUID v7 for all primary keys

  • RFC 9457 Problem Details for all API errors

  • All create endpoints return HTTP 200 (Axum Json<T> default)

  • Soft-deletes set active=false (most tables also stamp deleted_at; craig-rules and craig-exchange carry only the flag); since #1579 the GET-by-id paths answer 404 for a tombstone (the #1215 "uniformly absent" posture) — the row survives only for the ADR-062 claim-replay channel, which reads through explicit _any store getters

  • Config via CRAIG_<SERVICE>__* environment variables

  • Competing consumers (subscribe()) for DB-mutating event handlers

  • Exclusive subscribers (subscribe_exclusive()) for cache invalidation (per-instance fan-out)

  • Module naming — trust the parent context: name a module with the bare domain word, not a service-prefixed compound. services/craig-cases/src/matching/ is correct; person_matching/ is wrong even though craig-placement also has a matching/ module — the module path is the disambiguator, by design. Pre-emptively prefixing with the domain the path already carries is a forever-tax that compounds across every module. A slightly more complicated cross-service grep is acceptable; scope the grep to the service (grep -r matching services/craig-cases/) when context matters. This does not apply when the names are actually different concepts within a service (screening_decisions/ is correct when there is no bare-word decisions/ module) — the rule is about not stuttering the parent, not about avoiding descriptive names.

Batch-migration pattern

When a single mechanical change has to be applied across N partners, services, or crates (N ≥ 4), don’t hand-roll each one from scratch. Establish a PILOT in one MR with extra care, then execute the remaining N−1 as identical mechanical translations:

  1. PILOT MR — establish all the shared infrastructure: any restructure (lib/bin split, module extraction), shared helper extraction, the first concrete instance with the extensive doc comments that explain the pattern, and the full doc cascade (plan Status, CHANGELOG, nav) written for the first time. Budget real time here.

  2. Per-follow-up MR — pure mechanical translation against the proven shape: read the target’s typed shape, write the code, write the test, cascade the docs, ship. Once the PILOT is landed, each follow-up is short and predictable.

The PILOT investment only pays off when there are ≥ 4 follow-ups. A one-off migration (single partner / single service) gets no benefit from the PILOT split — just do the work inline. This is internal infrastructure improvement, not external API contract-locking, so it is safe to do pre-1.0.

CLI command args-struct pattern

When decomposing a fat-arm CLI dispatcher — a match cmd { Variant { fields } ⇒ { 20–30 LOC inline } } run fn — wrap each variant’s fields in a clap #[derive(Args)] struct referenced as a tuple variant, rather than passing the fields through to a helper individually:

pub enum CaseCmd {
    List(CaseListArgs),
    Get(CaseIdArgs),
}

#[derive(Args)]
pub struct CaseListArgs {
    #[arg(long)]
    pub worker: Option<String>,
    // …
}

#[derive(Args)]
pub struct CaseIdArgs {
    pub id: String,
}

// `run` becomes a thin dispatcher:
match cmd {
    CaseCmd::List(args) => run_list(args, format, client).await,
    CaseCmd::Get(args) => run_get(args, format, client).await,
}

Why not field-pass-through. Passing each field individually (run_variant(client, format, field1, field2, …, fieldN)) produces 7–15-parameter helper signatures that are hard to read and harder to refactor. The args-struct keeps every helper signature narrow (run_list(args: &CaseListArgs, format, client)).

The CLI surface is identical either way — clap derives the same --flag parsing from the struct fields that it would from the inline variant fields, so this is a pure internal refactor that users never see.

Tip: single-field variants that all take the same id can share one XxxIdArgs { pub id: String } struct rather than declaring a one-off struct per variant — e.g. Get(HealthIdArgs) and Delete(HealthIdArgs).

Documentation governance

Split every doc by time horizon and audience:

  • Durable engineering reference — design principles, architectural constraints and decisions, current-state snapshots — belongs in Antora at docs/modules/ROOT/pages/ (AsciiDoc), discoverable via nav.adoc. This is material carried forward across CRAIG versions.

  • Engagement-specific or one-time content — an RFP, a stakeholder hand-off bundle, a vendor evaluation, an engagement decision log — belongs in docs/handoffs/ (Markdown, universal for external partners), not in Antora. This is material that gets archived externally when the engagement concludes.

Cross-link, don’t duplicate. Antora-to-Antora references use xref: (e.g. …); handoff Markdown references Antora by relative path. Never copy Antora content into a handoff bundle, and never mix durable + engagement content in one doc — split them. Markdown does not belong under docs/modules/ROOT/pages/ (it breaks Antora’s AsciiDoc convention); engagement docs inside Antora dilute the engineering reference and go stale fast.

Edit this page · latest