Deployment Guide

Every service that touches the object store loads the same CRAIG_STORE* family. CRAIG_STOREBACKEND is explicit-required (#1362): a deployment that omits it refuses to boot with the remedy in the message (craig-rules only loads the family — and so only enforces the requirement — when its retention archiver is enabled) — the backend is never silently defaulted, because local writes blobs to a single container’s disk (dev/test only; they vanish with the container). A deployment that intends S3 can therefore never silently get local.

No service-specific environment variables beyond the shared CRAIG_PLACEMENT* settings listed under "Shared settings" (database, MQ, Keycloak, port, logging, CORS). Earlier versions carried a CRAIG_PLACEMENTRULES_ENGINE_URL variable; it was removed in 2026-04-19 when the unused Extension wrapper was deleted (see CHANGELOG under Step 12 #[allow] audit).

craig-intake is a stateless edge (ADR-017): it has no database and no message-queue connection — there are no DATABASE_URL/RABBITMQ_URL variables for this service. The full variable set (standalone mode, SHINES backend, signed-path limits, TLS) is in the Configuration Reference.

Each Track A unit (A2–A6) converts one service’s event consumers from the legacy at-least-once inbox path to the ADR-062 single-transaction attempt machine. The two implementations must never run concurrently against one event_inbox table: the legacy path’s autocommitted claim bypasses the row lock the transactional path serializes on, reopening the double-execution window mid-rollout. Deploy those releases with a stop-then-start barrier — compose recreates the container (the devstack’s single-replica shape already does this); on Kubernetes set the workload’s update strategy to Recreate for the conversion release (and back to rolling afterwards if desired). Pre-1.0 the platform ships single-replica, so this is a release-note discipline rather than an orchestration change.

All services expose three health endpoints (no auth required), aligned with the Kubernetes liveness/readiness convention (Step 6 of platform-stabilization):

Services report ready (/readyz → 200) when:

The migration gate is process-mode separation; the privilege boundary behind it is TWO Postgres principals per stateful service (the per-service least-privilege shape the RabbitMQ credentials already follow, #1202):

Per service (names are yours to choose; devstack uses craig_<svc>_owner / craig_<svc>_app with password == role name as a PUBLIC dev fixture):

Set the default privileges BEFORE the first gate run and every future migration-created table is DML-accessible to the runtime role with zero per-release grant churn (CRAIG has no sequences — UUIDv7 keys fleet-wide — so no sequence grants are needed). The gate’s CRAIG_<SVC>__DATABASE_URL carries the owner DSN; the serving container’s carries the runtime DSN — same variable NAME, two per-container values (the split is provisioning, not code).

Meta-table hardening (automatic since #1550): default privileges cannot exclude individual tables, so the runtime role would otherwise hold DML (not just SELECT) on the two schema meta-tables. The gate revokes INSERT, UPDATE, DELETE on _sqlx_migrations + schema_compat_floor from every non-owner grantee — PUBLIC included — on every successful verdict (idempotent; read from the table ACL itself, so it works whatever you named the runtime role and catches grants your session could not see through information_schema; SELECT stays — the verify-only boot reads both tables). No manual step remains; the statement it issues per grantee is the one the pre-#1550 guide asked operators to run by hand:

Rotation: the two principals rotate independently. The owner credential is needed only during deploys — scope it to the deploy pipeline’s secret store and rotate per release if desired (ALTER ROLE cases_owner PASSWORD '…' between deploys is invisible to serving). The runtime credential rotates like any service secret: ALTER ROLE, then roll the serving pods. Neither rotation touches grants — privileges attach to the role, not the password.

Cutover from a shared credential (the pre-#1310 posture), per service and rollback-safe: provision both roles + ALTER DATABASE … OWNER + default privileges as above, then REASSIGN OWNED BY <old_shared_role> TO <svc>_owner; in each database (existing tables become owner-owned; grant the runtime role on them once — GRANT SELECT, INSERT, UPDATE, DELETE ON ALL TABLES IN SCHEMA public TO <svc>_app; — since default privileges only cover FUTURE objects), flip the gate DSN, verify a deploy, flip the serving DSN, verify boot, then retire the shared credential. Rollback at any step = point the DSNs back. One sharp edge: REASSIGN OWNED also reassigns SHARED objects (databases, tablespaces) the old role owns, cluster-wide — run the per-database ALTER DATABASE … OWNER steps for ALL services FIRST so no database can transiently land on the wrong service’s owner.

Devstack: devstack/postgres/init.sql provisions all 16 roles (drift-pinned against the xtask service registry); the craig superuser remains the devstack ADMIN principal (seed, test-plane scratch/template databases, xtask tooling, pg_isready). Because init.sql runs only on a fresh data directory, upgrading an existing devstack across #1310 requires cargo xtask dev reseed. One operational upside: the serving roles are no longer superuser, so superuser_reserved_connections (3) now genuinely reserves admin access under connection-pool saturation.

Schema application is a DEPLOY phase, not a serving-startup side effect. Every service image doubles as its own migration gate: <binary> migrate (exact argv — nothing else) connects with ONE env var (CRAIG_<SVC>__DATABASE_URL — the MIGRATION-OWNER DSN since #1310; serving containers carry the runtime DSN under the same variable name), classifies the schema against its embedded migration set, and acts by verdict:

The reference compose graph carries the gates automatically: one craig-<svc>-migrate one-shot per stateful service, sharing the app’s image: + build: via a merged fragment (a gate can never run a stale :local image the app wasn’t built from), with the app gating on service_completed_successfully. Locally, cargo xtask migrate apply [--service <name>] builds + runs the gates and reports per service (partial success is retry-safe — applied migrations re-verdict as no-ops).

Kubernetes deployments run the gate as a parallelism-1 Job or deploy hook per service before rolling the serving image. A per-pod init container is NOT equivalent — per-pod means N concurrent appliers, which recreates the boot convoy the gate removes. Production uses immutable image tags/digests so gate and app provably run the same binary.

Since M3b, serving boot is VERIFY-ONLY (a pre-1.0 breaking change, CHANGELOG’d): the gate step is mandatory and serving binaries never issue DDL. Boot verifies BEFORE any JWKS/RabbitMQ init (craig_api::bootstrap_verified — data plane → schema verdict → control plane), so an identity-provider or broker outage can never mask a schema refusal. A Behind database refuses serving startup with the gate named as the remedy; Ahead (rollback) boots. Rollback support is explicit: back to the newest release whose embedded migrations reach the schema’s schema_compat_floor (destructive contract migrations bump it; the verifier refuses binaries older than the floor with "roll forward").

Postgres refuses sessions beyond its server-wide max_connections, and a CRAIG deployment holds more sessions than its pool arithmetic suggests. Budget every class:

Rule of thumb: max_connections ≥ Σ pools + 14 + (1 × services) + tooling + 3, then margin. The devstack pins 250, with its instance arithmetic (including the pre-push battery’s scratch-DB test pools) recorded next to the knob in docker-compose.yml and floor-pinned by crates/craig-db/tests/connection_budget.rs; production deployments substitute their own pool sizes and replica counts.

Per-service pool sizes were reviewed for #1160 and stay at the uniform default 10: differentiated devstack pools would re-introduce per-service acquire cliffs under bursty e2e load for roughly 20 sessions of headroom the right-sized ceiling already provides. Production deployments size pools per service from observed db_pool_* saturation, not from this default.

Two durable topic exchanges. Since #1202 they are operator-owned — services no longer declare them at runtime (granting a service configure on a shared exchange would include DELETE, letting a compromised service delete/redeclare craig.dlx and silently suppress forensic delivery). Devstack pre-declares both in devstack/rabbitmq/definitions.json; production operators MUST provision them before first boot (a missing exchange fails the first bind/publish loudly rather than being silently recreated).

Each service declares its own queues on startup (the exchanges are operator-owned — see above). The queue names here are what the #1202 per-service credentials are scoped to (User Configuration, below).

Competing consumers (shared queue): use for DB-mutating handlers — ensures each event is processed by exactly one instance.

Exclusive consumers (per-instance queue): use for cache invalidation — every instance must receive the event.

The "Type" column above is the consumer pattern (competing vs exclusive). This section is about the RabbitMQ queue type property (classic vs quorum vs stream) — a different axis.

Every CRAIG queue is declared x-queue-type=classic (ADR-003 §Amendment #1198): the fleet’s redelivery semantics assume classic-no-delivery-limit, and a quorum queue’s default delivery-limit = 20 would silently drop or re-route deliveries — on the DLX audit queue that means destroying forensic records. The explicit argument makes CRAIG immune to a default_queue_type at fresh declare and turns any conflict into a loud boot failure.

Operator MUST-NOTs:

Each DLQ subscription (production: craig-security.dlq) declares two durable classic companions at boot (ADR-059):

Automated drain (#1206): cargo xtask quarantine-drain <queue> --yes [--amqp-url <url>] [--out <dir>] [--limit <n>] implements the procedure below exactly — returnable captures republish bytes-unchanged and ack only on a clean confirm; malformed captures file to a timestamped NDJSON (fsynced before their ack); any unclean confirm stops the drain with the capture left in place. Devstack runs it as the craig-test identity by default; production needs an operator-scoped --amqp-url — the service identities deliberately cannot consume quarantine (#1202), and the recommended production shape is a purpose-scoped account holding exactly read on {queue}.quarantine + write on craig.dlx. The manual procedure below remains the fallback.

Replaying quarantined captures (manual-ack AMQP procedure — never a destructive management get; a management-UI peek must use requeue mode):

Verify the contract on a live broker (the first command shows the stored client arguments, which distinguish an explicit pin from a resolved default):

Failure signature and remedies:

Each service connects as its OWN broker account, scoped to exactly the queues it declares plus the shared exchanges it uses. A single shared account is a pre-1.0-retired anti-pattern: it let any service read every other service’s dead-letter records and (via configure on a shared exchange, which includes DELETE) delete craig.dlx and suppress forensic delivery. RabbitMQ permission patterns are a substring match, so every pattern below is anchored (^(…​)$).

An administrator-tagged operator account (here craig) is still needed for the management plane (vhost/policy administration). It is NOT a service identity — no service connects as it.

The per-op derivation: write carries craig.events (the outbox publish) and craig.dlx (every events-role queue is declared with an x-dead-letter-exchange argument, which RabbitMQ authorizes as a write on the DLX); read carries craig.events (bind reads) plus the service’s own queues; configure is the service’s own queues ONLY. craig-security additionally holds the dead-letter plane: amq.default write (the parking/quarantine capture publishes go via the default exchange), craig.dlx read (the dlq.# bind), and configure/read on its dlq/dlq.parking/dlq.quarantine (quarantine is configure-only — no consumer).

The canonical spec is devstack/rabbitmq/definitions.json (imported at first boot, / vhost). Provision the same via the management API or rabbitmqctl; the set_permissions argument order is configure write read. Example for two representative accounts (a plain service and the audit service):

The other services follow the same three-column shape scoped to their own queue names, with these differences from the craig-rules exemplar (which is the only account carrying a .cache family): craig-placement, craig-exchange, craig-financial, and craig-reporting have .events + .authz-cache only (no .cache); craig-cases carries .events + .authz-cache (its craig-cases.events queue arrived with the ADR-062 §G4 auto-link consumer, #1269); craig-composition also omits .events and carries .composition-cache + .authz-cache. Copy the exact triples from definitions.json — it is the executable source of truth.

Re-running the import or set_permissions is idempotent (a permission set replaces, never appends). Verify:

CRAIG requires a craig realm with:

In containerized deployments, Keycloak is typically accessible at different URLs from the host and from service containers:

Both must point to the same Keycloak realm. The iss claim in JWTs is validated against OIDC_ISSUER.

The boot-time JWKS/discovery fetch tolerates a saturated bring-up: connect/timeout-class failures retry up to 3 attempts with a 2 s pause (a starved GET on a cold host — 14 containers + a cold JVM — must not brick a service into exited (1) for everything downstream of its compose edge). Fail-loud is preserved for genuine misconfiguration: HTTP status errors, JSON-decode failures, and refused connects still surface within seconds (a refused connect fails each attempt instantly, so the whole window is ~4 s); only a fetch that is genuinely WAITING burns its per-attempt HTTP budget, giving real saturation 30-60 s of patience (one vs both GETs timing out per attempt) before the same loud failure.

The data-plane’s eager DB dial gets the same treatment: PoolTimedOut — the ONLY retried class — means the first connect starved past the serving pools' fixed 5 s acquire cliff (#1160; sqlx retries refused connects internally until that deadline, so a busy or not-yet-accepting server surfaces the same way). Three attempts with a 2 s pause bound the window at ~19 s before the loud failure. Every other error class (bad URL, DNS, TLS, auth) fails on the FIRST attempt, and the pool’s acquire policy is untouched — once serving, overload still cliffs visibly at 5 s (#1340’s migration gate remains the one acquire-bound exception).

Create a single bucket for the deployment:

Files are stored with the key pattern:

Example: cases/contact_attachments/abc123/def456/report.pdf

Boot REFUSES (D10 probe, hard startup failure) when the archiver is enabled and:

Required by policy (not machine-checkable — verify before enabling):

The craig-security release carrying the rules.evaluations_archived parser arms must be deployed FLEET-WIDE before CRAIG_RULESRETENTION_ARCHIVEENABLED=true — parser-first, so rules' fleet-bookkeeping events never land in a security instance that would audit-log them as unknown without ledger bookkeeping (a pre-parser instance acks the event, so its fleet copy is permanently absent — bookkeeping-only, never data loss: rules' own archive_ledger is the authority).

An evaluation_id (or any archived row id) stays resolvable forever — hot via the service, cold via the owning service’s ledger id-ranges:

The tool reads the ledger (devstack DSN by default; --database-url otherwise) and the store via the services' own CRAIG_STORE__* contract, and emits the archived NDJSON lines VERBATIM — the manifest’s ColumnSpec schema (stored beside every object) says how to interpret five-year-old lines without the live schema.

Purge refuses EVERYTHING until DFCS names a records-retention schedule: every destructive statement independently requires retention_until IS NOT NULL AND retention_until ⇐ CURRENT_DATE AND NOT legal_hold (D14), and the engine writes every ledger row with retention_until = NULL. When a schedule exists, an operator sets retention_until (and legal_hold where litigation requires), then drains in bounded slices: craig archive purge --limit 100 until more is false. Every run/purge is attributed — the manual operator’s claims.sub (or system:retention for scheduled passes) lands in archived_by/purged_by and in the staged security.archive_completed / security.archive_purged events.

A crash between the object puts and the prune commit leaves data+manifest objects with NO ledger row — harmless (the next pass re-archives under a fresh id). Garbage-collect ONLY against the owning service’s LOCAL ledger (D8): list retention-archives/{service}/{table}/, delete objects whose archive_id has no ledger row older than a day. Never GC from the security fleet-copies of rules' rows — archive_ledger in craig_rules is the authority for rules objects.

All services are stateless and can be scaled horizontally:

All services: GET /livez (k8s liveness, always 200), GET /readyz (k8s readiness, 200/503), GET /healthz (JSON detail). See Health Checks above for the full table.

Every service serves Prometheus text at GET /metrics. Instruments export only when OTEL_EXPORTER_OTLP_ENDPOINT is set (otherwise the endpoint returns an empty 200 — the devstack default).

Database session exhaustion is directly observable via the db_pool_* family (#1160), wired to every service’s pool by the shared bootstrap:

sqlx additionally logs any single acquire slower than 2 s at WARN on target sqlx::pool::acquire (its built-in acquire_slow default) — a free per-request signal that complements the sampled probe.

craig-cases also exports the person-matching latency pair (#1157): person_match_candidates_per_entry and person_match_entry_eval_duration_seconds — per ranked report entry, how many candidates were seeded and how long the (bounded, 8 in flight) rules-evaluation phase took.

Structured JSON logging via tracing + tracing-subscriber:

Each API service serves Swagger UI at /swagger-ui: