Plan: Operational Infrastructure Remediation

On this page

Status

Step Description Status

1

Add test, lint, and build jobs to GitLab CI pipeline

Not started

2

Implement column-level encryption for sensitive PII fields

Not started

3

Create backup/disaster recovery tooling and runbooks

Not started

4

Establish migration rollback strategy with pre-migration snapshots

Not started

5

Integrate secret management (Vault or sealed-secrets)

Not started

6

Add API versioning and deprecation headers

Not started

7

Implement retry with exponential backoff and jitter for external calls

Not started

8

Tune database connection pool per service and export pool metrics

Not started

9

Add SSE endpoint for real-time portal updates

Not started

10

Expand automated accessibility testing (axe-core) across all pages

Not started

11

Build general-purpose data export API for FOIA and audit portability

Not started

12

Replace in-memory idempotency/JWKS cache with PostgreSQL or Redis-backed store

Not started

13

Add documentation testing for API examples and CLI commands

Not started

14

Full validation pass

Not started

Epic: TBD
Branch: chore/operational-infrastructure
Labels: type::chore, priority::critical

Context

A cross-project audit of CRAIG and its sibling project Canopy identified 13 shared operational infrastructure gaps. Both projects invested heavily in application architecture (service isolation, rules engines, event buses, E2E testing) but underinvested in operational concerns (backups, encryption, CI enforcement, retry resilience, observability under failure).

The most critical findings:

  1. CI does not run tests. Both projects defer entirely to optional pre-push hooks. A developer pushing with --no-verify or from a machine without hooks configured can land broken code in main with zero automated test signal. For a CCWIS handling child welfare case records, this is an unacceptable risk.

  2. No encryption at rest. Child names, SSNs, case narratives, placement records, and court orders sit in PostgreSQL as plaintext. HHS requires encryption at rest for child welfare data. Application-level column encryption provides defense-in-depth beyond filesystem encryption.

  3. No backup or disaster recovery tooling. CRAIG is a system of record for child welfare cases. Data loss from ransomware, accidental deletion, or failed migrations has no recovery path today.

  4. Forward-only migrations with no rollback. Failed deployments cannot revert schema changes. Combined with no backups, a bad migration could leave the system in an unrecoverable state.

CRAIG is further along than Canopy in some areas (working portal, E2E tests, rate limiting, security headers, i18n, k6 load tests) but shares all 13 operational gaps. This plan addresses them for CRAIG specifically.

Scope

In scope:

  • CI pipeline: cargo fmt, cargo clippy, cargo nextest run as blocking merge jobs

  • JUnit XML artifact consumption and test reporting in CI

  • Docker build validation on feature branches

  • Column-level encryption for SSN, DOB, case narratives, and court order details using aes-gcm-siv

  • Encryption key management via environment variable (phase 1) with Vault integration (phase 2)

  • pg_basebackup wrapper script with WAL archiving configuration

  • Documented RTO/RPO targets and tested restore procedure

  • Pre-migration snapshot tooling in cargo xtask

  • Down migration templates for critical tables

  • HashiCorp Vault integration or sealed-secrets operator support

  • Accept-Version header support with Sunset and Deprecation headers (RFC 8594)

  • backoff crate integration for JWKS refresh, inter-service HTTP, and RabbitMQ reconnect

  • Per-service pool sizing with test_before_acquire, pool metrics exported to Prometheus

  • SSE endpoint in craig-web for real-time case/placement event updates

  • axe-core coverage expansion across all craig-web pages (building on existing ARIA work)

  • General-purpose /v1/export endpoints beyond AFCARS reporting

  • PostgreSQL-backed idempotency key store (replacing in-memory DashMap)

  • API example validation in CI; CLI example validation

Out of scope:

  • Mobile/offline client (separate plan exists)

  • Full Redis deployment (PostgreSQL-backed cache is sufficient for phase 1)

  • Multi-jurisdiction deployment orchestration (Kubernetes operator)

  • HSM-backed encryption keys (phase 3, post-production)

  • ICPC partner integration changes (separate plan)

Design

CI Pipeline Architecture

Add a test stage to .gitlab-ci.yml that runs before promote. Use the existing rust:1.94-alpine builder image. Three jobs run in parallel:

cargo-fmt:
  stage: test
  script: cargo fmt --check --all

cargo-clippy:
  stage: test
  script: cargo clippy --workspace -- -D warnings

cargo-test:
  stage: test
  script: cargo nextest run --workspace --profile ci
  artifacts:
    reports:
      junit: test-results/**/*.xml

The docker-promote job gains a needs: [cargo-fmt, cargo-clippy, cargo-test] dependency so broken code cannot be promoted.

Encryption at Rest

Use aes-gcm-siv (AEAD, nonce-misuse resistant) for column-level encryption. Create a craig-crypto shared crate:

// crates/craig-crypto/src/lib.rs
pub struct FieldEncryptor { key: aes_gcm_siv::Aes256GcmSiv }

impl FieldEncryptor {
    pub fn from_env(var: &str) -> Result<Self, Error>;
    pub fn encrypt(&self, plaintext: &[u8]) -> Vec<u8>;   // nonce || ciphertext || tag
    pub fn decrypt(&self, blob: &[u8]) -> Result<Vec<u8>, Error>;
}

Encryption key loaded from CRAIG_FIELD_ENCRYPTION_KEY (base64-encoded 256-bit key). Phase 2 replaces env var with Vault transit engine.

CRAIG-specific encrypted fields: * persons.ssn — Social Security Number * persons.date_of_birth — Date of birth * case_narratives.content — Free-text case narratives (may contain sensitive details) * court_orders.order_text — Court order content

Retry with Backoff

Add backoff crate to workspace dependencies. Wrap all external calls (JWKS refresh, rules client, inter-service HTTP, exchange partner APIs) with exponential backoff:

backoff::future::retry(
    backoff::ExponentialBackoffBuilder::new()
        .with_initial_interval(Duration::from_secs(1))
        .with_max_interval(Duration::from_secs(300))
        .with_randomization_factor(0.3)
        .with_max_elapsed_time(Some(Duration::from_secs(3600)))
        .build(),
    || async { /* call */ },
).await

Particularly important for craig-exchange which calls external partner APIs (ICPC, SSA) that may be intermittently unreachable.

SSE for Real-Time Portal Updates

craig-web already has a mature Askama + htmx portal. Add an SSE endpoint that subscribes to relevant RabbitMQ events and pushes to connected browser sessions:

// services/craig-web/src/api/sse.rs
async fn event_stream(
    session: AuthenticatedWorker,
) -> Sse<impl Stream<Item = Result<Event, Infallible>>> {
    // Subscribe to events for this worker's caseload
    // Map RabbitMQ EventEnvelope → SSE Event
}

htmx natively supports SSE via hx-sse="connect:/sse". Events to push: case.updated, referral.created, placement.changed, court_order.filed, investigation.completed.

Data Export Beyond AFCARS

CRAIG already has AFCARS export in craig-reporting. Add general-purpose export endpoints for non-federal use cases:

  • GET /v1/export/cases — Case records (CSV/JSON, admin/supervisor only)

  • GET /v1/export/persons — Person records with PII redaction option for FOIA

  • GET /v1/export/audit-events — Audit log export (admin only)

  • GET /v1/export/placements — Placement history (supervisor only)

All exports require admin or supervisor role, produce audit log entries, and support Accept: text/csv content negotiation.

Steps

Step 1: Add CI test and lint jobs

Files: .gitlab-ci.yml, .config/nextest.toml

  1. Add test stage to stages list (before promote)

  2. Add cargo-fmt job: cargo fmt --check --all

  3. Add cargo-clippy job: cargo clippy --workspace — -D warnings

  4. Add cargo-test job: cargo nextest run --workspace --profile ci

  5. Publish test-results/*/.xml as JUnit artifacts

  6. Add cargo-build-docker job on MR branches (build only, no push)

  7. Gate docker-promote on test jobs: needs: [cargo-fmt, cargo-clippy, cargo-test]

Step 2: Implement column-level encryption

Files: New crates/craig-crypto/, services/craig-cases/src/store/, Cargo.toml

  1. Create craig-crypto crate with FieldEncryptor (encrypt/decrypt using AES-256-GCM-SIV)

  2. Add craig-crypto to workspace members and dependencies

  3. Update craig-cases person store: encrypt SSN and DOB on write, decrypt on read

  4. Update case narrative store: encrypt narrative content

  5. Add migration to backfill existing plaintext data (encrypt in place)

  6. Add unit tests for encrypt/decrypt roundtrip, tamper detection, and key rotation

  7. Document key management in .claude/docs/security.md

Step 3: Backup and disaster recovery tooling

Files: New tools/craig-backup/, new docs/modules/ROOT/pages/disaster-recovery.adoc

  1. Create tools/craig-backup/backup.sh: wrapper around pg_basebackup for craig database

  2. Configure WAL archiving in devstack PostgreSQL container

  3. Create tools/craig-backup/restore.sh: tested point-in-time recovery

  4. Document RTO (4 hours) and RPO (1 hour) targets

  5. Add quarterly restore test procedure to operations documentation

  6. Add cargo xtask backup command that invokes the script

Step 4: Migration rollback strategy

Files: xtask/src/cmd/migrate.rs (new), .claude/docs/coding-conventions.md

  1. Add cargo xtask migrate snapshot command that takes a pg_dump before running pending migrations

  2. Add cargo xtask migrate rollback command that restores from the most recent snapshot

  3. Document the rollback strategy in coding conventions

  4. Create down migration templates for critical tables (cases, persons, placements, court_orders)

  5. Add snapshot step to cargo xtask validate before running migrations in integration tests

Step 5: Secret management integration

Files: New crates/craig-secrets/, crates/craig-common/src/settings.rs, .env.example

  1. Create craig-secrets crate with trait-based secret provider: EnvSecretProvider (phase 1), VaultSecretProvider (phase 2)

  2. Settings loader uses SecretProvider to resolve database_url, rabbitmq_url, encryption_key

  3. Phase 1: EnvSecretProvider reads from env vars (current behavior, wrapped in trait)

  4. Phase 2: VaultSecretProvider reads from HashiCorp Vault via HTTP API

  5. Add secret access audit logging

  6. Document secret rotation procedure

Step 6: API versioning and deprecation

Files: crates/craig-api/src/versioning.rs (new), crates/craig-api/src/lib.rs

  1. Add Accept-Version header extraction middleware

  2. Default to v1 when header is absent

  3. Add Sunset and Deprecation response headers (RFC 8594) for deprecated endpoints

  4. Add /v1/api-versions endpoint listing available versions with sunset dates

  5. Document versioning strategy in developer guide

Step 7: Retry with exponential backoff

Files: Cargo.toml, crates/craig-auth/src/jwks.rs, services/craig-exchange/src/, crates/craig-mq/src/subscriber.rs

  1. Add backoff = "0.4" to workspace dependencies

  2. Replace infinite loop in JWKS refresh with exponential backoff (1s → 300s max, 30% jitter)

  3. Wrap exchange partner API calls with retry (3 attempts, 1s → 4s)

  4. Add retry on RabbitMQ reconnect in subscriber

  5. Add tests for retry behavior

Step 8: Database connection pool tuning

Files: crates/craig-db/src/lib.rs, crates/craig-common/src/settings.rs

  1. Add per-service pool configuration: db_max_connections, db_min_connections, db_acquire_timeout_secs

  2. Enable test_before_acquire(true) for connection health checks

  3. Export pool metrics to Prometheus: db_pool_active, db_pool_idle, db_pool_waiting, db_pool_acquire_duration_seconds

  4. Set production-appropriate defaults: max 25 connections, min 5, 10s acquire timeout

  5. Add pool exhaustion alert threshold

Step 9: SSE for real-time portal updates

Files: services/craig-web/src/api/sse.rs (new), services/craig-web/src/api/mod.rs, services/craig-web/templates/base.html

  1. Add SSE route: GET /sse returning Sse<impl Stream>

  2. Subscribe to RabbitMQ events filtered by worker’s assigned caseload

  3. Map EventEnvelope to SSE Event with JSON data

  4. Add hx-sse="connect:/sse" to base template for auto-reconnect

  5. Add SSE event handlers for: case.updated, referral.created, placement.changed, investigation.completed

  6. Add connection keepalive (30s heartbeat)

  7. Add E2E test: trigger case update, verify SSE event received in Playwright

Step 10: Expand automated accessibility testing

Files: tests/e2e/package.json, tests/e2e/fixtures/a11y.ts (new), tests/e2e/specs/*.spec.ts

  1. Add @axe-core/playwright to E2E dev dependencies

  2. Create shared fixture that runs checkA11y() after each page load

  3. Assert zero WCAG 2.1 AA violations on every page render

  4. Expand existing ARIA patterns to all forms (not just tabs)

  5. Add color contrast validation for all theme colors

  6. Run a11y tests as part of cargo xtask e2e

Step 11: General-purpose data export API

Files: services/craig-cases/src/api/export.rs (new), services/craig-security/src/api/export.rs (new)

  1. Add GET /v1/export/cases to craig-cases (CSV and JSON, supervisor+ role)

  2. Add GET /v1/export/persons to craig-cases (with PII redaction for FOIA)

  3. Add GET /v1/export/audit-events to craig-security (admin role required)

  4. Add GET /v1/export/placements to craig-placement (supervisor+ role)

  5. Add Accept: text/csv content negotiation

  6. Add audit log entry for every export request

  7. Add integration tests for export endpoints

Step 12: Distributed idempotency and cache store

Files: crates/craig-api/src/idempotency.rs, new migration in craig-cases

  1. Create idempotency_keys table in service database: (cache_key TEXT PRIMARY KEY, response_status INT, response_headers JSONB, response_body BYTEA, created_at TIMESTAMPTZ)

  2. Replace DashMap in idempotency middleware with PostgreSQL queries

  3. Add TTL cleanup: DELETE FROM idempotency_keys WHERE created_at < now() - interval '24 hours'

  4. JWKS cache can remain in-memory (per-instance, refreshes hourly)

  5. Add tests for idempotency across simulated pod restarts

Step 13: Documentation testing

Files: .gitlab-ci.yml, new tests/doc-validation/

  1. Extract API request/response examples from AsciiDoc into testable fixtures

  2. Add CI job that validates fixture requests against running devstack

  3. Add cargo test --doc to CI for Rust doc examples

  4. Validate CLI examples (from cli.adoc) produce expected output

  5. Add cargo xtask check-docs --examples command

Step 14: Full validation

  1. cargo fmt --check --all

  2. cargo clippy --workspace — -D warnings

  3. cargo nextest run --workspace --profile ci — all tests pass (existing + new)

  4. cargo xtask e2e — E2E tests pass (including new a11y and SSE tests)

  5. cargo xtask validate — full pre-push validation

  6. CI pipeline successfully runs all new jobs on feature branch

  7. Verify encryption roundtrip: encrypt SSN, decrypt, compare

  8. Verify backup/restore: take backup, corrupt data, restore, verify integrity

  9. Verify retry: mock failing Keycloak, confirm backoff intervals in logs

  10. Verify SSE: open browser, trigger case update, confirm real-time notification

Files Touched

File Change

.gitlab-ci.yml

Add test stage with fmt, clippy, test, docker-build jobs

Cargo.toml

Add aes-gcm-siv, backoff, craig-crypto to workspace

crates/craig-crypto/

New crate: AES-256-GCM-SIV field encryption

crates/craig-db/src/lib.rs

Pool tuning, test_before_acquire, metrics export

crates/craig-common/src/settings.rs

Per-service pool config, secret provider integration

crates/craig-api/src/lib.rs

Versioning middleware, SSE wiring

crates/craig-api/src/idempotency.rs

Replace DashMap with PostgreSQL-backed store

crates/craig-api/src/versioning.rs

New: Accept-Version header, Sunset/Deprecation headers

crates/craig-auth/src/jwks.rs

Replace infinite loop with backoff crate

crates/craig-mq/src/subscriber.rs

Add connection retry with backoff

crates/craig-secrets/

New crate: trait-based secret provider

services/craig-cases/src/store/

Encrypt/decrypt SSN, DOB, narratives via craig-crypto

services/craig-web/src/api/sse.rs

New: SSE endpoint for real-time updates

services/craig-web/templates/base.html

Add hx-sse connection

services/craig-cases/src/api/export.rs

New: data export endpoints

services/craig-security/src/api/export.rs

New: audit event export

services/craig-exchange/src/

Add retry with backoff on partner API calls

tools/craig-backup/

New: pg_basebackup wrapper + restore script

xtask/src/cmd/migrate.rs

New: snapshot and rollback commands

tests/e2e/fixtures/a11y.ts

New: axe-core accessibility fixture

tests/doc-validation/

New: API and CLI example validation tests

Execution Priority

Priority Step Effort Reason

P0

Step 1 (CI tests)

Small

Highest-impact single change; blocks broken code from merging

P0

Step 2 (Encryption)

Medium

Regulatory requirement for child welfare PII

P0

Step 3 (Backups)

Medium

No recovery path today; existential risk for system of record

P1

Step 4 (Migration rollback)

Small

Prevents unrecoverable deployment failures

P1

Step 7 (Retry/backoff)

Small

Low effort; prevents cascading failures; critical for craig-exchange

P1

Step 8 (Pool tuning)

Small

Low effort; prevents connection exhaustion under load

P2

Step 5 (Secret management)

Medium

Env vars acceptable short-term; Vault needed for production

P2

Step 6 (API versioning)

Medium

Not urgent until v2 is needed, but foundation should exist

P2

Step 9 (SSE)

Medium

UX improvement; caseworkers will benefit immediately

P2

Step 10 (a11y expansion)

Small

Section 508 requirement; builds on existing ARIA work

P3

Step 11 (Data export)

Medium

Needed for FOIA and auditors; not blocking for initial deployment

P3

Step 12 (Distributed cache)

Medium

Only matters at multi-pod scale

P3

Step 13 (Doc testing)

Small

Quality-of-life; prevents doc drift

Verification

  1. cargo fmt --check --all — no formatting issues

  2. cargo clippy --workspace — -D warnings — zero warnings

  3. cargo nextest run --workspace --profile ci — all tests pass

  4. cargo xtask e2e — E2E tests pass (including a11y and SSE)

  5. cargo xtask validate — full pre-push validation passes

  6. CI pipeline runs fmt + clippy + test jobs and blocks promote on failure

  7. cargo xtask backup creates valid backup; cargo xtask migrate rollback restores from snapshot

  8. Encrypted SSN roundtrip: insert person with SSN, retrieve, verify match

  9. Retry test: stop Keycloak, verify JWKS refresh backs off (1s, 2s, 4s…​ in logs)

  10. SSE test: open portal, update case via API, verify browser receives update < 2s

  11. a11y test: cargo xtask e2e reports zero WCAG 2.1 AA violations across all pages

Documentation Updates

  • .claude/docs/services.md — export endpoint tables, SSE endpoint

  • .claude/docs/security.md — encryption at rest, secret management, key rotation

  • docs/modules/ROOT/pages/deployment-guide.adoc — backup/restore commands, pool tuning, secret rotation

  • docs/modules/ROOT/pages/developer-guide.adoc — migration rollback strategy, retry patterns

  • docs/modules/ROOT/pages/configuration-reference.adoc — new pool and encryption settings

  • CHANGELOG.adoc — entry under == Unreleased

  • Antora pages — new disaster-recovery.adoc

Edit this page · latest