Plan: Platform Stabilization (Pre-Offline-Worker)

On this page

Status

Step Description Status

1

Plan + 3 ADR drafts (ADR-020/021/022) + GitLab epic + 16 step issues + nav.adoc Active entry. No code changes.

Done (2026-04-28) — MR !166

2

ADR-020 acceptance + P0.1: encryption fail-closed (refactor encrypt_field/decrypt_field Result; propagate through 5 callsites in cases-side persons handler; integration tests).

Done (2026-04-30) — MR !170

3

ADR-021 acceptance + P0.2: JWT validation (extend Claims with aud/azp/scope/typ; per-service Validation::set_audience; Keycloak realm config; scope-string capability).

Done (2026-04-30) — MR !172

4

ADR-022 acceptance + P0.4: DLX/DLQ wiring (craig.dlx exchange; subscriber DLX args; DLQ consumer in craig-security with dead_letter_audit table).

Done (2026-04-29) — MR !168

5

P1.2: subscriber stubs — retire 6 stubs across cases/exchange/placement; file 3 domain-decision issues for the events that warrant real handlers (exchange case.created/placement.created, financial rate-cache refresh).

Done (2026-04-28) — MR !167

6

P2.1: split health endpoints into /livez + /readyz + /healthz with proper status codes; update Docker Compose healthcheck stanzas; document new k8s probe shape.

Done (2026-04-30) — MR !171

7

P2.2 + P2.3 + P3.14 + P3.12: CI gate hardening (validate-tests + cargo-deny jobs in scan stage; drop allow_failure: true on cargo-audit; pin floating image tags; add cargo deny to pre-push).

Done (2026-04-29) — MR !169

8

P0.3 cases-only: transactional outbox pattern (per-service event_outbox; Publisher::publish_in_tx; outbox worker; rewrite cases-side publish callsites). Defines the pattern; remaining services follow in Step 12.

Done (2026-04-30) — MR !173

9

P0.3 inbox: idempotent consumers (per-service event_inbox; handle_idempotently helper; subscriber pattern adoption across surviving subscribers from Step 5; 30-day cleanup matching outbox retention).

Done (2026-04-30) — MR !174

10

P1.1: persistent idempotency middleware (per-service idempotency_responses table; body fingerprinting; 1 MiB cached-response cap with body_too_large sentinel; 2 MiB request cap with 413; periodic cleanup).

Done (2026-04-30) — MR !175

11

P1.3: cargo xtask reconcile + cross-service orphan walker; markdown report.

Done (2026-04-30) — MR !176

12

P0.3 remaining services: outbox migration + publish-callsite rewrite for craig-{placement, exchange, financial, reporting, security, rules}.

Done (2026-04-30) — MR !177

13

P0.3 admin event replay endpoint (POST /v1/<service>/admin/events/replay with from/to/dry_run; tests).

Done (2026-05-01) — MR !178

14

P3 cleanup MR (delete dead helpers + stale #[allow]`s + obsolete comments; archive 4 complete plans; fix ADR-008 nav label; file 7 ADR open-question issues; dedupe `zen-engine 0.35.1+0.54.0; gitignore bun.lock; fix doc/stat drift; file the parameter-struct sibling plan).

Done (2026-05-01) — MR !179

15

P3.5: unskip 5 stale E2E specs; investigate borderline intake-embedded-ui.spec.ts:51.

Done (2026-05-02) — MR !180 (, partial — see errata)

16

Plan completion audit (subagent per delivery-protocol.md) + archive (move plan from nav Active to archive.adoc; update Phase Status final stats; close epic).

Done (pre-ADR-030)

Epic: &20 (closed)
Issues: #226–#241 (16 step-tracking issues)
Branch prefix: feat/platform-stab- (or chore/platform-stab- for cleanup steps)
Milestone: 2026 Q3 — Feature Initiatives

Wave sequencing:

  • W1: Step 1 (plan + ADRs + issues; solo, blocks all)

  • W2 — foundation, parallel: Steps 2, 3, 4, 5, 6, 7

  • W3 — build on foundation, parallel: Steps 8, 9, 10, 11

  • W4: Steps 12, 13 (12 depends on 8’s pattern; 13 depends on 8 outbox tables)

  • W5 — cleanup, parallel: Steps 14, 15

  • W6: Step 16 (audit + archive)

Context

External review (2026-04-28) plus four internal audit subagents converged on a single verdict: CRAIG is better than a throwaway prototype but not yet ready to support an offline worker application. Four platform-level invariants (P0) and three durability/correctness concerns (P1) are gaps an offline-sync layer would amplify into silent data loss, fail-open PII exposure, or cross-service token reuse.

This plan addresses every item in the merged audit punch-list:

  • 4 P0 platform-invariant gaps: encryption fail-open, JWT thinness, events not transactional, DLQ stub

  • 3 P1 durability/correctness gaps: idempotency middleware brittle, subscriber stubs, cross-service FK reconciliation

  • 4 P2 ops gaps: health checks, CI not a real gate, cargo-audit allow_failure, normalized debt

  • 14 P3 cleanup items: dead helpers, stale skips, doc/stat drift, plan archival, ADR open-question issue tree, dep dedupe, lockfile policy, image-tag pins, pre-push gap

The work is structured as 16 steps across 6 waves. After this plan ships, ADR-009 (mobile/offline-capable client) becomes safe to accept and execute.

The parameter-struct refactor (originally drafted as a step) is carved out as a sibling plan filed after Step 14 lands — it’s mechanical, risk-isolated, and shares no architectural decisions with this plan; bundling would dilute reviewer focus on the P0/P1 work.

Related ADRs (existing): ADR-014, ADR-013. New ADRs filed by this plan: ADR-020 (encryption fail-closed), ADR-021 (JWT validation policy), ADR-022 (transactional outbox + DLQ + inbox + idempotency middleware).

Scope

In scope:

  • Refactor encrypt_field / decrypt_field to fail closed; propagate Result through every callsite (5 sites, all in craig-cases)

  • Add aud / azp / scope / typ to Claims; enforce per-service audience via Validation::set_audience; configure Keycloak realm to issue audience claims

  • Implement transactional outbox (publisher side), idempotent inbox (consumer side), and admin-gated event replay endpoint per service

  • Wire x-dead-letter-exchange arg into craig-mq::Subscriber::queue_declare; declare craig.dlx exchange; add a DLQ consumer in craig-security

  • Replace in-memory idempotency cache with Postgres-backed store; fingerprint requests; fix the >2 MiB body-empty bug

  • Decide retire-or-defer per "not yet handled" subscriber stub (9 sites) before wrapping subscribers with the new inbox helper

  • Add cargo xtask reconcile for cross-service reference reconciliation

  • Split /healthz into /livez + /readyz + /healthz with proper status codes

  • Move test battery into CI; drop allow_failure: true on cargo-audit; add cargo deny to pre-push; pin floating image tags

  • Cleanup MR: dead code, stale #[allow], stale comments, obsolete E2E skips, doc drift, archive 4 complete plans, fix ADR-008 nav label, file ADR-017/019 OQ tracking issues, dedupe zen-engine 0.35.1+0.54.0, decide bun.lock-vs-package-lock

  • Plan completion audit + archive

Out of scope (filed separately):

  • Parameter-struct refactor of long-arg store fns (~40+ across cases/security/placement) → sibling plan, filed after Step 14 of this plan lands. Resurrects the data-integrity-hardening § Deferred (Store Parameter Structs) work.

  • ADR-009 mobile/offline-capable client implementation — depends on this plan landing first

  • New domain features (any of the standing-backlog tier-2/tier-3 items)

  • AWS deployment thread (#163, #172, #199 — owned by external team)

  • Texas screening-policy ruleset (#225 — deferred until real Texas deployment)

  • Local-outbox for air-gapped partners (#215 — wait for real partner case)

  • Outbox/inbox upgrade from polling to LISTEN/NOTIFY: deferred per the abandonment criterion in §D3.1

Design

D1. Encryption fail-closed semantics (ADR-020)

Today services/craig-cases/src/api/encryption.rs:15-43 does:

pub fn encrypt_field(encryptor: Option<&FieldEncryptor>, value: &str) -> String {
    match encryptor {
        Some(enc) => enc.encrypt_str(value).unwrap_or_else(|e| {
            tracing::warn!("field encryption failed, storing plaintext: {e}");
            value.to_string()
        }),
        None => value.to_string(),
    }
}

This silently stores plaintext on cipher failure. For child-welfare PII under 45 CFR Part 1355 / 42 USC §5106a, this must fail the write.

New signatures:

pub fn encrypt_field(
    encryptor: Option<&FieldEncryptor>,
    value: &str,
) -> Result<String, ApiError>;

pub fn decrypt_field(
    encryptor: Option<&FieldEncryptor>,
    value: &str,
) -> Result<String, ApiError>;

Semantics:

  • encrypt_field: encryptor present + cipher Err → ApiError::internal("field encryption failed: …​") (becomes 500)

  • encrypt_field: encryptor absent → return value as-is (encryption-disabled mode is a documented dev-loop path)

  • decrypt_field: encryptor present + is_encrypted true + cipher Err → ApiError::internal("field decryption failed: …​"). Tampered ciphertext or wrong key surface as 500 incidents, not silent passthrough.

  • decrypt_field: encryptor present + is_encrypted false → return value as-is (legacy plaintext from before encryption was enabled)

  • decrypt_field: encryptor absent + is_encrypted true → ApiError::internal("encryption disabled but stored value appears encrypted") (operational error: someone mis-deployed)

  • decrypt_field: encryptor absent + is_encrypted false → return value as-is

Affected callsites (all in services/craig-cases/src/api/persons.rs):

Line Context

172

create_person body construction

195

create_person response after read

226

get_person after fetch

263

update_person body construction

288

update_person response after read

The two helper fns in encryption.rs:

pub fn decrypt_person_pii(encryptor: Option<&FieldEncryptor>, person: &mut Person) -> Result<(), ApiError>;
pub fn decrypt_persons_pii(encryptor: Option<&FieldEncryptor>, persons: &mut [Person]) -> Result<(), ApiError>;

Both propagate Result.

Migration safety: Existing rows in cases.persons.ssn_last_four may contain plaintext (if encryption was disabled at write time) or ciphertext (if it was on). The is_encrypted() heuristic at crates/craig-crypto/src/lib.rs:135-139 (base64 + ≥28 bytes) is the existing guard. We retain it. Adding an explicit enc:v1: marker prefix is deferred — would require a one-shot migration to backfill all existing ciphertext.

D2. JWT validation policy (ADR-021)

Today crates/craig-auth/src/jwks.rs:117-119:

let mut validation = Validation::new(Algorithm::RS256);
validation.set_issuer(&[&self.issuer]);
validation.validate_exp = true;

Issuer + expiry only. No audience, scope, azp, or token-type. Any token correctly issued by the realm with broad roles can hit any service.

Claims struct extension (crates/craig-auth/src/claims.rs):

pub struct Claims {
    pub sub: String,
    pub preferred_username: String,
    pub email: Option<String>,
    pub realm_access: RealmAccess,
    // ── ADR-021 additions ──────────────────────────────────────────
    /// Audience claim per RFC 7519 §4.1.3. Keycloak emits as either a
    /// string or array; deserialized as Vec<String> either way.
    #[serde(default, deserialize_with = "claims::aud_or_vec")]
    pub aud: Vec<String>,
    /// Authorized party (client_id) per OIDC §2.
    #[serde(default)]
    pub azp: Option<String>,
    /// Space-separated scope string per RFC 6749.
    #[serde(default)]
    pub scope: Option<String>,
    /// Token type. Required-presence check; reject anything other than "Bearer".
    #[serde(default, rename = "typ")]
    pub typ: Option<String>,
}

Validator extension:

pub struct JwksProvider {
    issuer: String,
    fetch_url: String,
    expected_audience: String,  // ← NEW
    keys: Arc<RwLock<Option<JwkSet>>>,
}

impl JwksProvider {
    pub fn new(issuer: &str, expected_audience: &str) -> Result<Self> { ... }

    pub async fn validate_token(&self, token: &str) -> Result<Claims, anyhow::Error> {
        // ... existing kid lookup ...
        let mut validation = Validation::new(Algorithm::RS256);
        validation.set_issuer(&[&self.issuer]);
        validation.set_audience(&[&self.expected_audience]);
        validation.validate_exp = true;
        let token_data = decode::<Claims>(token, &decoding_key, &validation)?;

        if token_data.claims.typ.as_deref() != Some("Bearer") {
            return Err(anyhow::anyhow!("token typ is `{:?}`, expected `Bearer`", token_data.claims.typ));
        }
        Ok(token_data.claims)
    }
}

Per-service config (crates/craig-common/src/settings.rs): add jwt_audience: String loaded from CRAIG_<SERVICE>__JWT_AUDIENCE env var; default to service name.

Keycloak realm config: add a Hardcoded Audience client-mapper to craig-api and craig-ui clients that emits a multi-valued aud array containing all 8 service names (craig-rules, craig-cases, craig-placement, craig-exchange, craig-financial, craig-reporting, craig-security, craig-intake).

Multi-valued aud rationale (must be explicit in ADR-021): audience prevents cross-realm token reuse, not cross-service role escalation. RBAC remains the authorization boundary; aud ensures a token issued by the CRAIG realm is the one being presented (rejecting accidental reuse of tokens from a sibling Keycloak realm or a stolen access_token from a non-CRAIG context). Per-service tokens (the alternative) would add re-auth friction with no security gain because role checks already gate per-service authorization.

Scope strings (defined per service):

Service Scopes

craig-rules

rules.read, rules.write, rules.evaluate

craig-cases

cases.read, cases.write, cases.admin

craig-placement

placement.read, placement.write

craig-exchange

exchange.read, exchange.write, exchange.admin

craig-financial

financial.read, financial.write

craig-reporting

reporting.read, reporting.admin

craig-security

security.read, security.admin

craig-intake

(none — intake uses partner API key, not user JWT)

Scope enforcement is additive on top of existing role checks. Step 3 ships the capability (Claims fields + require_scope() method); per-handler adoption is incremental over Steps 5–12 as those handlers are touched anyway.

D3. Transactional outbox + idempotent inbox + replay (ADR-022)

D3.1 Outbox (publisher-side durability)

New table per service (7 services × 1 migration each — intake is stateless and excluded):

CREATE TABLE event_outbox (
    id              UUID PRIMARY KEY DEFAULT uuidv7(),
    aggregate_id    UUID,
    event_type      TEXT NOT NULL,
    envelope        JSONB NOT NULL,
    created_at      TIMESTAMPTZ NOT NULL DEFAULT now(),
    published_at    TIMESTAMPTZ,
    attempts        INT NOT NULL DEFAULT 0,
    last_error      TEXT,
    CONSTRAINT event_outbox_published_check
        CHECK (published_at IS NULL OR published_at >= created_at)
);
CREATE INDEX idx_event_outbox_pending
    ON event_outbox (created_at)
    WHERE published_at IS NULL;

Retention: 30 days for both published_at IS NOT NULL and NULL rows (replay window). Periodic cleanup task drops rows older than 30 days. Inbox retention must match (see §D3.2) so replay never bypasses dedup.

Publisher API change (crates/craig-mq/src/publisher.rs):

impl Publisher {
    /// Direct publish — used by the outbox worker and by services without a DB (intake).
    pub async fn publish(&self, envelope: &EventEnvelope) -> Result<(), lapin::Error> { ... }

    /// Transactional publish: stages the envelope to the outbox in the caller's tx.
    /// The outbox worker drains it asyncly. Every domain handler uses this.
    pub async fn publish_in_tx(
        tx: &mut sqlx::Transaction<'_, sqlx::Postgres>,
        envelope: &EventEnvelope,
    ) -> Result<(), sqlx::Error>;
}

Outbox worker (crates/craig-mq/src/outbox.rs — new): polls every 1s, SELECT * FROM event_outbox WHERE published_at IS NULL ORDER BY created_at LIMIT 100, publishes via Publisher::publish(), marks published_at = now(). Crash-safe.

Polling-vs-LISTEN/NOTIFY abandonment criterion (must be in ADR-022): v1 is polling. Upgrade to LISTEN/NOTIFY when outbox.depth p95 > 500 sustained over a 24h window, or when sustained outbox lag exceeds 5s p95. Either trigger files a follow-up plan.

Migration of existing publish call sites: every events::publish_* function in the 6 service-side events.rs files is rewritten to take &mut sqlx::Transaction instead of &Publisher. Type system forces every caller to update.

Step 8 ships cases-only, defining the pattern + worker + the migration; Step 12 ships the remaining 6 services.

D3.2 Inbox (consumer-side idempotency)

CREATE TABLE event_inbox (
    envelope_id     UUID PRIMARY KEY,
    source_service  TEXT NOT NULL,
    event_type      TEXT NOT NULL,
    received_at     TIMESTAMPTZ NOT NULL DEFAULT now(),
    processed_at    TIMESTAMPTZ
);
CREATE INDEX idx_event_inbox_received ON event_inbox (received_at);

Retention: 30 days, matching outbox. A replay older than 30 days bypasses dedup — that’s a deliberate operational signal ("we lost our recent history; reprocess"). If a consumer’s side-effects aren’t idempotent at the application layer, the >30d edge case is a real bug — fix it at the application layer, not the dedup layer.

Subscriber pattern (crates/craig-mq/src/inbox.rs — new):

pub async fn handle_idempotently<F, Fut>(
    db: &PgPool,
    envelope: EventEnvelope,
    handler: F,
) -> anyhow::Result<()>
where
    F: FnOnce(EventEnvelope) -> Fut,
    Fut: std::future::Future<Output = anyhow::Result<()>>,
{
    let claimed = sqlx::query!(
        r#"INSERT INTO event_inbox (envelope_id, source_service, event_type)
           VALUES ($1, $2, $3)
           ON CONFLICT (envelope_id) DO NOTHING
           RETURNING envelope_id"#,
        envelope.id, envelope.source_service, envelope.event_type,
    )
    .fetch_optional(db)
    .await?;

    if claimed.is_none() {
        tracing::debug!(envelope_id = %envelope.id, "duplicate event, skipping");
        return Ok(());
    }

    handler(envelope.clone()).await?;
    sqlx::query!(
        "UPDATE event_inbox SET processed_at = now() WHERE envelope_id = $1",
        envelope.id,
    ).execute(db).await?;
    Ok(())
}

Each surviving subscriber (post Step 5 retire decisions) wraps its handler in handle_idempotently. craig-security’s wildcard audit subscriber and craig-rules' rule- evaluation subscriber both adopt this.

D3.3 Replay endpoint (admin)

POST /v1/<service>/admin/events/replay?from=<rfc3339>&to=<rfc3339>&dry_run=<bool>
Authorization: Bearer <admin-token-with-{service}.admin scope>

Re-publishes outbox rows in the window. Implementation: walks SELECT * FROM event_outbox WHERE created_at BETWEEN $from AND $to, sets published_at = NULL, lets the outbox worker pick them up. Inbox dedup ensures no duplicate side-effects (within the 30-day window).

D4. DLX/DLQ wiring (within ADR-022)

Today at crates/craig-mq/src/subscriber.rs:88-90, queue declaration passes FieldTable::default() — no DLX args. Failed messages route nowhere on broker config that doesn’t have a wildcard DLX.

// New in craig-mq lib.rs:
pub const DLX_EXCHANGE: &str = "craig.dlx";

// craig-mq subscriber.rs:88-92 becomes:
let mut args = FieldTable::default();
args.insert("x-dead-letter-exchange".into(), DLX_EXCHANGE.into());
args.insert("x-dead-letter-routing-key".into(), format!("dlq.{}", queue_name).into());
self.channel
    .queue_declare(queue_name.into(), queue_options, args)
    .await?;

craig-mq bootstrap (called once on connect):

pub async fn declare_dlx(channel: &Channel) -> Result<(), lapin::Error> {
    channel.exchange_declare(
        DLX_EXCHANGE,
        ExchangeKind::Topic,
        ExchangeDeclareOptions { durable: true, ..Default::default() },
        FieldTable::default(),
    ).await
}

DLQ consumer in craig-security: declares queue craig-security.dlq bound to craig.dlx with routing key dlq.#. Handler writes to dead_letter_audit table. Threshold alert: when count > 10 within a 1-hour window for any single event_type, emit security.dlq.threshold_exceeded event.

D5. Persistent idempotency middleware (within ADR-022)

Today crates/craig-api/src/idempotency.rs:

  • In-memory Arc<DashMap> keyed by claims.sub:idempotency_key

  • >2 MiB body silently empties response (line 99-101)

  • Per-process; doesn’t survive restart; doesn’t span instances

New design: Postgres-backed table per service.

CREATE TABLE idempotency_responses (
    cache_key        TEXT PRIMARY KEY,
    method           TEXT NOT NULL,
    path             TEXT NOT NULL,
    user_sub         UUID NOT NULL,
    body_fingerprint TEXT NOT NULL,
    status           SMALLINT NOT NULL,
    headers          JSONB NOT NULL,
    body             BYTEA,
    body_too_large   BOOLEAN NOT NULL DEFAULT false,
    created_at       TIMESTAMPTZ NOT NULL DEFAULT now(),
    expires_at       TIMESTAMPTZ NOT NULL DEFAULT now() + INTERVAL '24 hours'
);
CREATE INDEX idx_idempotency_expires ON idempotency_responses (expires_at);

Cache key: sha256(method || ":" || path || ":" || user_sub || ":" || idempotency_key).

Body fingerprint: sha256(request_body_bytes). Different fingerprint, same key → 422.

Body cap policy (sized for actual workloads):

  • Request bodies stay capped at 2 MiB. Caseworker JSON bodies are <<1 MiB; attachments go through the object_store path. Above 2 MiB → 413.

  • Cached response bodies cap at 1 MiB. If the response is larger, store status + headers + body_too_large = true, leave body = NULL. Replay returns 200 with a replay-too-large body and the client re-executes. Asymmetric by design — caching multi-megabyte responses balloons the table.

Cleanup: DELETE FROM idempotency_responses WHERE expires_at < now() every 1 hour.

D6. Subscriber stubs decision (P1.2 — Step 5)

Audit each "not yet handled" log; decide retire-or-defer per-event. This must happen before Step 9 (inbox) because both touch subscriber call sites.

Service Line Event Decision

craig-cases

main.rs:132

rules.evaluated

Retire — rules engine writes evaluations to its own audit table

craig-cases

main.rs:138

placement.created

Retire — placement and cases don’t need bidirectional sync today

craig-cases

main.rs:144

eligibility.evaluated

Retire — financial owns eligibility outcomes

craig-exchange

main.rs:88

case.created

Defer (file domain issue) — would drive outbound CWCA notifications; partner-notification contract not specified

craig-exchange

main.rs:94

placement.created

Defer (file domain issue) — same shape

craig-exchange

main.rs:100

eligibility.evaluated

Retire — financial pushes eligibility to exchange explicitly when needed

craig-placement

main.rs:81

case.created

Retire — placement is read-pulled by the worker UI

craig-placement

main.rs:87

rules.evaluated

Retire — placement doesn’t run automated decisions on rules outcomes

craig-financial

main.rs:327

rules.evaluated (rate update)

Defer (file domain issue) — would refresh rate cache; invalidation strategy needs design

6 retires + 3 deferrals. Step 5 ships only the retires. The 3 "Defer" entries are filed as separate feat-labeled GitLab issues with domain-decision-required labels.

D7. Cross-service reconciliation (P1.3 — Step 11)

New cargo xtask reconcile subcommand that walks documented cross-service references and surfaces orphans:

From Field To

cases.reports.partner_id

UUID

security.partners.id

cases.report_persons.linked_by

UUID

(Keycloak sub)

placement.placements.case_id

UUID

cases.cases.id

placement.placements.child_id

UUID

cases.persons.id

financial.payments.case_id

UUID

cases.cases.id

financial.payments.placement_id

UUID

placement.placements.id

exchange.icpc_requests.case_id

UUID

cases.cases.id

READ-ONLY; writes a Markdown report to test-results/reconciliation/<UTC-date>.md. No --fix flag — orphans need human triage. Optional GitLab schedule wires the report into the existing weekly stale-work pipeline.

D8. Health-check semantics (P2.1 — Step 6)

Today only /healthz (always 200 with degraded-in-body).

Endpoint Status Purpose

GET /livez

200 always (process responding)

Kubernetes liveness probe

GET /readyz

200 if all required deps up; 503 otherwise (Retry-After: 5)

k8s readiness; load balancer signal

GET /healthz

200 with body (existing JSON shape)

Human/dashboard view

Required deps per service: DB + RabbitMQ. Optional: object store.

D9. CI gate hardening (P2.2 + P2.3 + P3.14 + P3.12 — Step 7 + revision)

The original §D9 design called for validate-tests to run cargo xtask validate --skip-docker directly. That doesn’t work in CI: validate’s earliest steps need git, gitleaks, cargo-deny, and a running devstack (Postgres + Rabbit + Keycloak + Garage), none of which rust:1.94-alpine provides. Step 2’s pipeline failed on that, exposing the design error. The shipped revision (under the same Step 7 epic) replaces validate-tests with ci-tests, which runs the deterministic no-devstack subset. See §Errata for full context.

What shipped (Step 7 + revision via Step 2):

  • New CI job ci-tests in scan stage: runs cargo fmt --check --all + cargo clippy --workspace --all-targets --locked — -D warnings + cargo build --workspace --locked --exclude xtask + cargo nextest run --workspace --locked --profile ci --lib --bins on MR pipelines touching Rust sources / Cargo manifests / xtask/src/*. Tag dhs-aws-autoscaler-docker.medium. *Blocks merge.

  • Drop allow_failure: true from cargo-audit job.

  • New CI job cargo-deny: runs cargo deny check on MR pipelines that touch dependency manifests or deny.toml.

  • Pin release-cli:latest → :v0.24.0 (current stable per gitlab-org/release-cli tags).

  • Pin jaegertracing/jaeger:2 → :2.17.0 (current stable per Docker Hub).

  • Pre-push: add cargo deny check step before cargo xtask validate.

Devstack-dependent integration tests stay local-only — pre-push runs them, CI gates everything else. Matches the CI philosophy already documented at the top of .gitlab-ci.yml.

D10. P3 cleanup MR contents (Step 14)

Single MR bundling:

  • Delete dead helpers in services/craig-cases/src/store/report_persons.rs (3 fns) + persons.rs (1 fn) + screening_decisions.rs (1 fn)

  • Remove module-level #![allow(dead_code)] from report_persons.rs:2-5

  • Remove #[allow(dead_code)] on models.rs:330-333 ReportPerson

  • Rewrite services/craig-cases/src/api/reports.rs:163-164 comment

  • Move 4 plans to archive: case-routing-ruleset, object-storage-adoption, seed-attachments, plain-language-docs

  • Update nav.adoc:74 ADR-008 label (Pending) → (Partially Resolved)

  • File issues for ADR-017 OQ#2/3/4/7, ADR-019 OQ#2/3/4 (7 issues, P3-low, follow-up)

  • Trace + dedupe zen-engine 0.35.1 (find consumer, bump or drop)

  • Add sdks/typescript/bun.lock to sdks/typescript/.gitignore (npm/package-lock.json is the canonical lockfile manager for the TS SDK)

  • Fix .claude/CLAUDE.md Phase Status: cases endpoints 49→61, security 34→48, placement tests 68→77, workspace tests row to current count

  • Fix .claude/docs/services.md: cases + security endpoint counts

  • File the parameter-struct refactor sibling plan (per §Out of scope)

D11. E2E skip rot (Step 15)

Unskip 5 stale test.describe.skip / test.skip blocks whose rationale cited "ADR-017 Step 8 BFF re-pointing" (Step 8 shipped weeks ago in MR !148):

  • tests/e2e/specs/intake-review.spec.ts:41 (whole describe block)

  • tests/e2e/specs/public-report.spec.ts:89, 171

  • tests/e2e/specs/public-report-form.spec.ts:141, 214

Validate against current BFF routes; fix selectors if they broke. Borderline: intake-embedded-ui.spec.ts:51 — investigate standalone-mode status proxy semantics.

Steps

Step 1: Plan + ADR drafts + GitLab issue tree

Files written:

  • docs/modules/ROOT/pages/plans/platform-stabilization.adoc (this plan)

  • docs/modules/ROOT/pages/adrs/adr-020-encryption-fail-closed.adoc (Proposed)

  • docs/modules/ROOT/pages/adrs/adr-021-jwt-validation-policy.adoc (Proposed)

  • docs/modules/ROOT/pages/adrs/adr-022-event-durability-and-idempotency.adoc (Proposed)

Files edited:

  • docs/modules/ROOT/nav.adoc — under * Plans / ** Active, add this plan; add 3 new ADRs

  • CHANGELOG.adoc — entry under == Unreleased describing plan + ADR drafts + epic + 16 issues filed

GitLab artifacts created:

  • 1 epic at group level: epic: platform stabilization (pre-offline-worker)

  • 16 step-tracking issues, linked to the epic, labeled per character (chore for 1/14/16; feat for 2-13/15), milestone 2026 Q3 — Feature Initiatives

Branch: chore/platform-stab-step1-plan-and-adrs
MR title: chore(plans): file platform-stabilization plan + ADR-020/021/022 drafts + GitLab issue tree [Step 1]

Verification:

  1. cargo xtask check-docs — Tier 1 docs untouched; new ADR + plan AsciiDoc-correct

  2. cargo xtask validate --skip-docker — pre-push gate green (no code changes)

  3. After push: glab mr view <id> shows pipeline pass; glab issue list --milestone "…​" shows new epic + 16 issues

Step 2: ADR-020 acceptance + P0.1 encryption fail-closed

Files: services/craig-cases/src/api/encryption.rs, services/craig-cases/src/api/persons.rs, services/craig-cases/tests/api/persons.rs, docs/modules/ROOT/pages/adrs/adr-020-encryption-fail-closed.adoc

Implementation:

  1. Flip ADR-020 status Proposed → Accepted

  2. Change encrypt_field, decrypt_field, decrypt_person_pii, decrypt_persons_pii signatures per §D1

  3. Propagate Result through 5 callsites at persons.rs lines 172, 195, 226, 263, 288 (use ? operator)

  4. Add 4 new integration tests:

    1. encryption_disabled_passthrough

    2. encryption_enabled_round_trip

    3. encryption_failure_fails_write (inject tampered ciphertext via direct DB UPDATE; subsequent GET returns 500)

    4. encryption_disabled_but_stored_appears_encrypted_500s

Verification:

  1. cargo nextest run -p craig-cases --test api persons — 4 new tests pass + existing pass

  2. cargo clippy --workspace --all-targets --all-features — -D warnings — clean

  3. cargo xtask validate --skip-docker — pre-push green

Step 3: ADR-021 acceptance + P0.2 JWT validation policy

Files: crates/craig-auth/src/claims.rs, crates/craig-auth/src/jwks.rs, crates/craig-common/src/settings.rs, crates/craig-api/src/bootstrap.rs, devstack/keycloak/craig-realm.json, crates/craig-auth/tests/jwks.rs, crates/craig-auth/tests/claims.rs, docs/modules/ROOT/pages/adrs/adr-021-jwt-validation-policy.adoc

Implementation:

  1. Flip ADR-021 status Proposed → Accepted

  2. Extend Claims struct (aud/azp/scope/typ); helper methods + custom deserializer

  3. JwksProvider::new(issuer, expected_audience); set_audience + typ check

  4. jwt_audience: String field in settings; load from CRAIG_<SERVICE>__JWT_AUDIENCE

  5. init_auth accepts expected_audience: &str parameter

  6. Keycloak realm config: Hardcoded Audience client mapper (multi-valued, all 8 services); scope mappers per service

  7. 4 new unit tests in tests/jwks.rs: wrong audience rejection, missing audience rejection, wrong typ rejection, aud-as-string-or-array

  8. has_scope / require_scope tests in tests/claims.rs

Out of scope for Step 3: per-handler scope enforcement. Step 3 ships the capability; handlers adopt incrementally over Steps 5/8/9/10/12 as touched.

Verification:

  1. cargo xtask dev reseed (Keycloak realm change requires reseed)

  2. cargo nextest run -p craig-auth — all new tests pass

  3. Smoke: log in as jane.doe via curl → token has aud array + scope claim

  4. cargo xtask validate --skip-docker — pre-push green

Step 4: ADR-022 acceptance + P0.4 DLX/DLQ wiring

Files: crates/craig-mq/src/lib.rs, crates/craig-mq/src/subscriber.rs, all 8 services' main.rs, services/craig-security/migrations/<TS>_dead_letter_audit.sql, services/craig-security/src/store/dead_letter_audit.rs, services/craig-security/tests/api/dead_letter.rs, docs/modules/ROOT/pages/adrs/adr-022-event-durability-and-idempotency.adoc

Implementation:

  1. Flip ADR-022 status Proposed → Accepted (rationale must include polling-vs-LISTEN/NOTIFY abandonment criterion + retention policy)

  2. pub const DLX_EXCHANGE: &str = "craig.dlx"; declare_dlx fn

  3. Populate FieldTable with DLX args at subscriber.rs:88-92

  4. Each service main.rs: call craig_mq::declare_dlx(&publisher_channel) once during bootstrap

  5. New migration + store helpers + DLQ subscriber per §D4

  6. failed_handler_routes_to_dlq integration test

Verification:

  1. cargo xtask dev reseed

  2. cargo nextest run --workspace

  3. Manual: force a publish error → dead_letter_audit row appears in security DB

Step 5: Subscriber stubs — retire 6, defer 3

Files (this MR — retires only):

  • services/craig-cases/src/main.rs:132,138,144 — delete 3 stub subscriptions

  • services/craig-exchange/src/main.rs:100 — delete the eligibility.evaluated stub

  • services/craig-placement/src/main.rs:81,87 — delete 2 stubs

  • Net: 6 stub deletes across 3 services

Files (file domain issues — separate from this MR):

File 3 feat-labeled GitLab issues with domain-decision-required label:

  1. feat(craig-exchange): handle case.created event for outbound partner notifications — references services/craig-exchange/src/main.rs:88

  2. feat(craig-exchange): handle placement.created event for outbound CWCA notifications — services/craig-exchange/src/main.rs:94

  3. feat(craig-financial): refresh rate cache on rules.evaluated event — services/craig-financial/src/main.rs:327

Verification:

  1. cargo nextest run --workspace — no test changes; existing tests unaffected

  2. cargo clippy --workspace --all-targets --all-features — -D warnings — clean

  3. glab issue list --label domain-decision-required shows the 3 new issues

Step 6: P2.1 health-check semantics

Files: crates/craig-api/src/lib.rs:277-300, docker-compose.yml, docs/modules/ROOT/pages/deployment-guide.adoc

Implementation:

  1. Refactor health endpoint registration; add /livez and /readyz; 503 when readyz checks fail

  2. Update Docker Compose healthcheck stanzas for all 8 services to /readyz

  3. Document three-endpoint shape with k8s probe snippets in deployment-guide.adoc

Verification:

  1. Smoke: curl -i http://localhost:8002/readyz → 200; bring DB down → 503

  2. cargo xtask validate --skip-docker — pre-push green

Step 7: CI gate hardening

Files: .gitlab-ci.yml, docker-compose.yml, .githooks/pre-push

Implementation:

  1. .gitlab-ci.yml insertion point: after line 82 (end of cargo-audit job), add validate-tests job:

    validate-tests:
  stage: scan
  image: ${RUST_IMAGE}
  tags:
    - dhs-aws-autoscaler-docker.medium
  before_script:
    - apk add --no-cache curl bash
    - cargo install cargo-nextest --locked 2>/dev/null || true
  script:
    - cargo xtask validate --skip-docker
  rules:
    - if: $CI_MERGE_REQUEST_IID
      changes:
        - "**/*.rs"
        - "**/Cargo.toml"
        - "Cargo.lock"
        - "xtask/src/**"

  2. Insertion point: after the new validate-tests job, add cargo-deny job:

    cargo-deny:
  stage: scan
  image: ${RUST_IMAGE}
  tags:
    - dhs-aws-autoscaler-docker.small
  before_script:
    - cargo install cargo-deny --locked 2>/dev/null || true
  script:
    - cargo deny check
  rules:
    - if: $CI_MERGE_REQUEST_IID
      changes:
        - "**/Cargo.toml"
        - "Cargo.lock"
        - "deny.toml"

  3. Drop allow_failure: true from line 72 of the existing cargo-audit job

  4. Replace line 156 image: registry.gitlab.com/gitlab-org/release-cli:latest with current stable tag

  5. docker-compose.yml line 515: replace image: jaegertracing/jaeger:2 with image: jaegertracing/jaeger:2.6.0

  6. .githooks/pre-push insertion point: line 11, before cargo xtask validate --skip-docker:

    echo "==> [pre-push] cargo deny check"
cargo deny check --all-features || { echo "cargo deny failed"; exit 1; }

Verification:

  1. Open a sample MR touching a .rs file; verify both new jobs run + gate merge

  2. Open a sample MR touching only Cargo.toml; verify cargo-deny runs but validate-tests does not

Step 8: P0.3 transactional outbox (cases-only)

Defines the pattern; remaining 6 services follow in Step 12.

Files: services/craig-cases/migrations/<TS>_event_outbox.sql, crates/craig-mq/src/publisher.rs, crates/craig-mq/src/outbox.rs (new), crates/craig-api/src/bootstrap.rs, services/craig-cases/src/events.rs, all cases-side callers of events::publish_*, crates/craig-mq/tests/outbox.rs

Implementation:

  1. Migration per §D3.1

  2. publish_in_tx API; Publisher::publish retained for the worker

  3. New outbox worker module

  4. bootstrap spawns the worker

  5. Rewrite events.rs publish fns to take &mut sqlx::Transaction

  6. Restructure ~14 cases-side callers to publish-in-tx-then-commit

  7. 4 new integration tests:

    1. domain_write_and_publish_atomic

    2. outbox_worker_drains

    3. rabbit_down_keeps_events_in_outbox

    4. outbox_replay_after_restart

Verification:

  1. cargo xtask dev reseed

  2. cargo nextest run --workspace

  3. Manual: create a case, watch RabbitMQ admin UI, see events flow; event_outbox.published_at populated

Step 9: P0.3 idempotent inbox

Files: All 7 service migrations: <TS>_event_inbox.sql, crates/craig-mq/src/inbox.rs (new), all subscriber call sites in service main.rs, services/craig-security/tests/api/inbox.rs

Implementation:

  1. Migrations per §D3.2

  2. handle_idempotently helper

  3. Wrap all surviving subscribers (post Step 5) with handle_idempotently

  4. 3 integration tests: duplicate_envelope_skipped, processed_at_marked_on_success, cleanup_drops_old_rows

Verification:

  1. cargo xtask dev reseed

  2. Replay the same event twice via direct AMQP publish; verify handler ran once

Step 10: P1.1 persistent idempotency middleware

Files: All 7 service migrations: <TS>_idempotency_responses.sql, crates/craig-api/src/idempotency.rs, crates/craig-api/src/lib.rs, crates/craig-api/tests/idempotency.rs

Implementation:

  1. Migrations per §D5

  2. Postgres-backed cache; raise request cap to 2 MiB hard with 413; cap cached response body at 1 MiB with body_too_large sentinel; fingerprint check; periodic cleanup

  3. Pass db: PgPool to IdempotencyCache::new

  4. 5 integration tests: cross_instance_dedup, mismatched_fingerprint_returns_422, request_above_2_mib_returns_413, cached_response_above_1_mib_re_executes_with_sentinel, expired_entry_re_executes

Verification:

  1. cargo xtask dev reseed

  2. Manual: hit POST /v1/cases/persons with Idempotency-Key: x from two instances → cached response

Step 11: P1.3 reconciliation xtask

Files: xtask/src/cmd/mod.rs, xtask/src/main.rs, xtask/src/cmd/reconcile.rs (new), docs/modules/ROOT/pages/cross-service-reconciliation.adoc

Implementation:

  1. pub mod reconcile; in cmd/mod.rs; add Reconcile(reconcile::ReconcileArgs) variant to the Command enum (mirror the position of CheckDocs(check_docs::CheckDocsArgs))

  2. Dispatch arm in main.rs: Command::Reconcile(args) ⇒ cmd::reconcile::run(args),

  3. Template to copy: xtask/src/cmd/check_docs.rs — same #[derive(Args)] pub struct ReconcileArgs, same pub fn run(args: ReconcileArgs) → Result<()> signature, same section headers, same OK/FAIL lines, same bail! on aggregate failure

  4. ReconcileArgs shape:

    #[derive(Args)]
pub struct ReconcileArgs {
    /// Subset of references to walk. `all` (default) walks everything;
    /// otherwise comma-separated keys from the §D7 table.
    #[arg(long, default_value = "all")]
    pub scope: String,
    /// Output file path. Defaults to test-results/reconciliation/<UTC-date>.md.
    #[arg(long)]
    pub output: Option<PathBuf>,
}

  5. For each row in §D7, implement a walk_<reference>(pool_from: &PgPool, pool_to: &PgPool) → Result<Vec<OrphanReport>> fn. Aggregator collects all `OrphanReport`s and writes the markdown report.

  6. Operator runbook in cross-service-reconciliation.adoc

  7. Optional CI hookup: .gitlab-ci.yml add triage:reconcile job mirroring triage:stale (cron 0 14 * * 1)

Verification:

  1. Seed an orphan via direct SQL

  2. Run cargo xtask reconcile --scope placement_to_case

  3. Verify the orphan appears in the markdown report with the bad UUID

Step 12: P0.3 remaining services outbox

Mirrors Step 8 across craig-{placement, exchange, financial, reporting, security, rules}.

Files: mirrors Step 8 multiplied across 6 services.

Verification: mirrors Step 8.

Step 13: P0.3 admin event replay endpoint

Files: All 7 services' src/api/admin.rs (new module), all 7 services' src/api/mod.rs, services/craig-cases/tests/api/admin_replay.rs

Implementation:

  1. New admin module per service with POST /v1/{service}/admin/events/replay route per §D3.3

  2. Register route in each service’s api/mod.rs

  3. 3 integration tests: replay_window_re_publishes, dry_run_returns_count_without_republish, replay_requires_admin_scope

Verification:

  1. cargo nextest run --workspace

Step 14: P3 cleanup MR

Per §D10. ~30 files across services + docs + Cargo.lock + .gitignore. Plus: file the parameter-struct refactor sibling plan as docs/modules/ROOT/pages/plans/store-parameter-structs.adoc (sibling-plan stub referencing data-integrity-hardening § Deferred); add nav entry under Planned.

Verification:

  1. cargo xtask check-docs clean

  2. cargo build --workspace --locked clean (after dedupe)

Step 15: P3.5 unskip stale E2E specs

Walk each of the 5 specs cited in §D11; rewrite selectors against current BFF if needed.

Verification:

  1. cargo xtask e2e — all unskipped specs green

Step 16: Plan completion audit + archive

Spawn plan-completion-audit subagent per delivery-protocol.md. Audit Steps 1–15. Update CLAUDE.md final stats; move plan to archive.adoc; close epic.

Files touched:

  • docs/modules/ROOT/pages/plans/platform-stabilization.adoc — Status table → all Complete

  • docs/modules/ROOT/nav.adoc — Active → next plan or "(none)"

  • docs/modules/ROOT/pages/plans/archive.adoc — new row under Infrastructure & DevOps

  • .claude/CLAUDE.md — Phase Status final counts

  • CHANGELOG.adoc — wrap-up entry

Verification:

  1. cargo xtask check-docs clean

  2. glab issue list --milestone "…​" shows zero platform-stabilization issues open

  3. glab epic view <N> state == closed

Critical Files

File Purpose

services/craig-cases/src/api/encryption.rs

Step 2 — fail-closed signature change

services/craig-cases/src/api/persons.rs

Step 2 — 5 callsite updates

crates/craig-auth/src/claims.rs

Step 3 — Claims extension (aud, azp, scope, typ)

crates/craig-auth/src/jwks.rs

Step 3 — set_audience wiring

crates/craig-common/src/settings.rs

Step 3 — jwt_audience field

devstack/keycloak/craig-realm.json

Step 3 — audience + scope mapper config

crates/craig-mq/src/lib.rs

Step 4 — DLX_EXCHANGE constant + declare_dlx

crates/craig-mq/src/subscriber.rs:88-92

Step 4 — DLX args insertion

crates/craig-api/src/lib.rs:277-300

Step 6 — health-endpoint split

.gitlab-ci.yml

Step 7 — CI gate jobs + pin tags

.githooks/pre-push

Step 7 — cargo deny step

crates/craig-mq/src/publisher.rs

Step 8 — publish_in_tx API

crates/craig-mq/src/outbox.rs

Step 8 — new outbox worker

crates/craig-mq/src/inbox.rs

Step 9 — handle_idempotently helper

crates/craig-api/src/idempotency.rs

Step 10 — persistent backing rewrite

xtask/src/cmd/reconcile.rs

Step 11 — new xtask subcommand

Verification (overall)

After every step:

  1. cargo xtask validate --skip-docker — pre-push gate green

  2. cargo xtask dev reseed — required after any schema change (Steps 4, 8, 9, 10, 12)

  3. cargo nextest run --workspace — workspace tests green

  4. cargo xtask check-docs — Tier 1 docs clean

  5. cargo deny check — after Step 7 lands, this is part of pre-push

After Step 16:

  1. cargo xtask e2e — all E2E specs green

  2. Spawn the plan-completion-audit subagent; expect zero residual findings

  3. Manually exercise: encryption fail-closed; JWT audience-mismatch; event replay

Documentation Updates

  • Step 1: this plan AsciiDoc + 3 ADR drafts + nav.adoc + CHANGELOG

  • Step 2: services.md (no count change), CHANGELOG, ADR-020 status

  • Step 3: services.md (per-handler scope additions where added), CHANGELOG, ADR-021 status

  • Step 4: services.md (new dead_letter_audit table on craig-security), CHANGELOG, ADR-022 status

  • Step 5: services.md (subscriber list updates per service), CHANGELOG

  • Step 6: deployment-guide.adoc (k8s probes), services.md (new endpoint registrations), CHANGELOG

  • Step 7: deployment-guide.adoc (new CI jobs), CHANGELOG

  • Step 8: services.md (new event_outbox table), CHANGELOG

  • Step 9: services.md (new event_inbox table), CHANGELOG

  • Step 10: services.md (new idempotency_responses table), CHANGELOG

  • Step 11: cross-service-reconciliation.adoc (new), CHANGELOG

  • Step 12: services.md (outbox tables for 6 more services), CHANGELOG

  • Step 13: services.md (new admin/replay endpoints), CHANGELOG

  • Step 14: services.md + CLAUDE.md drift fixes, CHANGELOG, sibling plan filed

  • Step 15: CHANGELOG only (E2E counts)

  • Step 16 (last MR): .claude/CLAUDE.md Phase Status final counts; plan moved to archive; CHANGELOG wrap-up

Risks

Risk Mitigation

Step 8 + Step 12 are the largest single touches — 14+28 publish sites need signature change. Easy to miss callers.

Type system enforcement (signature change forces compile errors); MR descriptions list every modified callsite.

Step 3 Keycloak realm changes break dev-loop login.

Step 3 includes a smoke test (curl auth → token has aud claim) before merging.

Step 10 idempotency cap raise (cached response 1 MiB) increases DB storage growth.

Body cleanup hourly; 24h TTL keeps table small. body_too_large sentinel for oversized.

Step 5 retiring 6 subscriber stubs could surface a downstream consumer no one realized depended on the (no-op) handler.

Pre-1.0 with no live deployment — handlers are confirmed no-ops. Each retire is a pure delete.

Step 14 sibling-plan filing for the parameter-struct refactor risks the work staying perpetually planned.

The cleanup MR includes the sibling plan adoc + nav entry; listed under Planned with a target deadline.

ADR-009 (mobile/offline) decisions may evolve in parallel and conflict with this plan’s assumptions.

Document in ADR-009 status that "implementation begins after platform-stabilization Step 9 (inbox)."

Open questions

  1. Encryption marker prefix (Step 2): retain the heuristic is_encrypted() indefinitely vs. backfill an explicit enc:v1: prefix. Recommendation: defer until a real false-positive shows up.

  2. JWT scope enforcement granularity (Step 3 + downstream): Step 3 ships the capability; per-handler adoption is incremental over Steps 5/8/9/10/12.

  3. Outbox poll cadence (Step 8): 1s default. Abandonment-to-LISTEN/NOTIFY threshold: outbox.depth p95 > 500 over 24h or sustained lag p95 > 5s.

  4. DLQ alerting threshold (Step 4): count > 10 / event_type / hour. Tune after first real broker traffic.

  5. Zen-engine duplicate (Step 14): cargo tree -i zen-engine:0.35.1 will name the consumer; bump or pin.

  6. Subscriber retire vs. defer (Step 5): the §D6 matrix is a starting position. Each event reviewed by a maintainer with domain context before retiring.

  7. Step 12 split (Step 12): ship as one MR if reviewer-tractable, otherwise per-service.

Potential improvements (deferred)

These are real opportunities surfaced during implementation but not in scope for this plan:

  • DLQ threshold query rate-limiting (Step 4 follow-up): count_recent_by_event_type runs on every dead-letter insert. Under sustained DLQ traffic this is O(N) per insert. Two reasonable evolutions: (a) move the threshold check into a periodic job (every 1m) rather than per-insert, or (b) compute breach-detection as a RETURNING clause on the INSERT itself. Defer until a real DLQ workload makes this measurable.

  • DLQ alert dedup (Step 4 follow-up, addressed by Step 9): the threshold handler emits one alert per breaching event_type per dead-letter insert. Once Step 9 wires the inbox, the consumer of security.dlq.threshold_exceeded will dedup naturally via inbox claim. Until then, a small number of duplicate alerts is acceptable.

  • DLQ retention (Step 4 follow-up): dead_letter_audit has no retention cap today. Step 8 introduces 30-day retention for outbox/inbox; this same window should apply to dead_letter_audit. File as a Step 9 or Step 12 sub-task once the retention plumbing is real.

  • gitleaks in ci-tests (Step 7 follow-up): the new ci-tests job covers fmt + clippy + build + unit tests but not gitleaks (validate’s step 7). The GitLab Secret-Detection template already runs in the scan stage and covers the same ground via its own engine, so the gap is small. If we ever drop the template, apk add gitleaks and add a step.

  • Dedicated outbox failure-mode tests (Step 8 follow-up): the workspace test battery exercises the happy path through event_outbox + worker for every cases event. The plan §D3.1 listed four explicit failure-mode tests (domain_write_and_publish_atomic, outbox_worker_drains with row count assertions, rabbit_down_keeps_events_in_outbox, outbox_replay_after_restart). Adding them requires test orchestration (broker pause via management API; service kill+restart in-test) that’s not in the current test infrastructure. File when the orchestration ships.

  • Dedicated handle_idempotently unit tests (Step 9 follow-up): the helper’s claim-once + duplicate-skip semantics are exercised by every event-emitting test that flows through a wrapped subscriber. The plan §D3.2 listed three explicit unit tests (duplicate_envelope_skipped, processed_at_marked_on_success, cleanup_drops_old_rows). Direct unit tests would require Postgres-in-unit-test scope (no fixtures available today). Same deferral rationale as Step 8.

  • Step 9 — cache-invalidation subscriber stays unwrapped: plan §D3.2 said "every surviving subscriber" wraps with handle_idempotently. Exception: craig-rules' cache-invalidation subscribe_exclusive consumer. Per-instance auto-delete queues mean cross-instance dedup doesn’t apply, and reload-from-DB is idempotent at the application layer. Wrapping would add a DB INSERT on every cache-invalidation message for no gain. Documented for §D3.2 readers.

  • Dedicated idempotency middleware integration tests deferred (Step 10 follow-up): plan §D5 listed 5 integration tests (cross_instance_dedup, mismatched_fingerprint_returns_422, request_above_2_mib_returns_413, cached_response_above_1_mib_re_executes_with_sentinel, expired_entry_re_executes). Shipped with 9 unit tests on the helper functions instead — same deferral rationale as Steps 8 and 9. The Postgres-fixture testing infra needed for the integration tests doesn’t exist yet; file when the orchestration ships.

  • GitLab CI schedule for reconcile (Step 11 follow-up): plan §D7 mentioned an optional triage:reconcile job mirroring triage:stale. Not shipped — the runner needs DB access (the existing triage:stale runs in alpine:3.20 with no DB), and orphan-triage cadence is currently better served by ops running on demand. File when there’s a real prod-shape DB to scan.

  • Step 12 — rules.cache_invalidated stays synchronous: plan §D3.1 implied every event would route through the outbox. Exception: the rules.cache_invalidated event in craig-rules/src/engine.rs::notify_cache_invalidated. It’s a cache-coherence signal that horizontally-scaled instances consume to reload their in-memory rule-set caches. Routing through the outbox would add ~1s of staleness to cache invalidation, which defeats the purpose. The event is best-effort — if it drops, the next stale read forces a reload, so durability isn’t required. Documented as the principled exception to "events go through outbox".

  • Step 13 — replay endpoint URL shape: plan §D3.3 specified POST /v1/<service>/admin/events/replay (per-service nesting). Shipped as POST /v1/admin/events/replay — a single shared mount auto-included by ApiServer::router. Same effect (every stateful service has the endpoint), cleaner without per-service wiring. Per-service-name URL would be marginally more discoverable in admin UIs; logged as a Potential improvement.

  • Step 13 — scope vs role enforcement: plan §D3.3 mentioned {service}.admin scope enforcement. Shipped with admin-role enforcement (claims.require_admin()) because the per-handler scope adoption is still incremental (Step 3 errata: scope capability shipped, adoption happens as handlers are touched). When the rest of the codebase adopts scope checks, the replay endpoint should be tightened to require <service>.admin scope.

  • Triage the 12 reports.partner_id orphans found on first run (Step 11 follow-up): the seed data points all 12 reports' partner_id at the all-zeros UUID, but the seeded partners table doesn’t include that row. Either the seed should include a placeholder partner with id 00000000-0000-0000-0000-000000000000, or the seed reports should reference a real partner. Fix in seed; not Step 11 territory.

Errata

Implementation deltas vs. the original plan, captured for reviewer clarity:

  • Step 4 — subscribe_dlq handler signature: shipped as Fn(EventEnvelope, String) → Fut rather than Fn(EventEnvelope) → Fut. The extra String is the original DLQ routing key (dlq.<original-queue>) so the audit row can record original_queue accurately. The plan §D4 didn’t specify the handler signature, so this is an implementation detail rather than a deviation, but worth flagging since it diverges from the regular subscribe() shape.

  • Step 7 — image tag pins: plan §D9 used release-cli:v0.18.0 and jaegertracing/jaeger:2.6.0 as illustrative pins; the implementation pinned current stable (:v0.24.0 and :2.17.0) verified against gitlab-org/release-cli tags and Docker Hub at the time of merge. The plan’s "current stable" intent stands.

  • Step 7 — cargo deny flag: plan §D9 specified cargo deny check --all-features; cargo-deny does not accept --all-features (errors with unexpected argument). Shipped as bare cargo deny check, which is the documented invocation in cargo-deny’s help text. Same coverage.

  • Step 2 — callsite count: plan §D1 listed 5 callsites in services/craig-cases/src/api/persons.rs; actual is 6 (the search_persons decrypt loop at line 132 was missed in the plan’s audit). All 6 are now Result-propagating. No semantic difference — the plan undercounted by one.

  • Step 2 — test placement: plan §D1 called for "integration tests"; shipped as unit tests in the same module because the §D1 semantics matrix is pure-function behavior on the helpers, which isolates better and runs without devstack. The full create→GET→tampered-DB→GET path would be valuable but adds an integration-test infrastructure dependency for negligible additional confidence; deferred.

  • Step 7 → Step 2 revision — CI gate scope: the validate-tests job shipped in Step 7 invoked cargo xtask validate --skip-docker directly, but validate needs git/gitleaks/cargo-deny/devstack which rust:1.94-alpine doesn’t provide and CI infrastructure shouldn’t try to. Step 2’s pipeline failed on it, exposing the design error. Replaced with ci-tests running the deterministic no-devstack subset (fmt + clippy + build + unit tests via --lib --bins). Pre-push remains the gate for integration tests + e2e + perf + security. This matches the CI philosophy already documented at the top of .gitlab-ci.yml ("Pre-push hook handles the full test battery locally; CI is security scans + Docker image promotion only.").

  • Step 6 — non-shared router services: plan §D8 implied a single edit to the shared craig-api router would cover all 9 services. Actual: craig-intake (per ADR-017 stateless edge) and craig-web (BFF) build their own routers and don’t go through ApiServer::router. Both got their own /livez + /readyz routes pointing at the existing health handler since neither has DB/MQ deps (probes reduce to "process is up").

  • Step 3 — JWT audience config plumbing: plan §D2 called for a new jwt_audience field on ServiceSettings loaded from CRAIG_<SERVICE>__JWT_AUDIENCE. Shipped using the service_name argument already passed to bootstrap("CRAIG_X", "craig-x") — same value, no new config surface. The audience is whatever each service already calls itself.

  • Step 3 — Keycloak audience mapper: plan §D2 specified a single oidc-hardcoded-claim-mapper with jsonType.label=JSON for the multi-valued aud. Keycloak (per the version pinned in our devstack) emitted that as a literal escaped-JSON string instead of parsing it. Shipped 8 oidc-audience-mapper entries instead — Keycloak natively aggregates same-name claims into an array, simpler debug path, no JSON-string-vs-array footgun.

  • Step 3 — fail-closed aud requirement: plan §D2 implied Validation::set_audience was sufficient for audience enforcement. It isn’t — jsonwebtoken’s set_audience only validates present aud values match; a token with no aud claim at all silently passes. Added explicit post-decode check that the token’s aud array contains the required value when audience is configured. Tested by validate_token_rejects_missing_audience_when_required.

  • Step 8 — dedicated outbox tests deferred: plan §D3.1 specified 4 dedicated integration tests in crates/craig-mq/tests/outbox.rs (domain_write_and_publish_atomic, outbox_worker_drains, rabbit_down_keeps_events_in_outbox, outbox_replay_after_restart). Shipped with: (a) the existing 1313-test workspace battery exercising the happy path through every cases handler (every event-emitting test now flows through the outbox + worker → broker → wildcard audit subscriber), and (b) the unit test on OutboxWorker poll-interval invariants. The 4 dedicated tests for explicit failure modes (broker shutdown mid-flight, etc.) require additional test orchestration (broker pause via management API) and are tracked as a Step 9 follow-up.

  • Step 8 — create_case retry tx-poisoning: plan §D3.1 implied a single tx for create_case would suffice. Reality: the handler retries on case-number collision (cases_case_number_key unique violation), and a poisoned tx can’t be reused. Inlined the retry loop into the handler so each attempt opens its own tx and rolls back on conflict before retrying. Outbox event still stages atomically with the successful insert.

  • Step 8 — Publisher::publish_in_tx shipped as free fn: plan §D3.1 named the staging API Publisher::publish_in_tx. Shipped as free fn craig_mq::stage_event(tx, envelope) instead — Publisher itself is no longer needed in the handler call chain (only the outbox worker holds one), so making stage_event a free function avoids constructing a Publisher just to stage. Same coverage.

  • Step 15 — partial unskip: plan §D11 listed 5 stale skip blocks across 3 specs (intake-review, public-report, public-report-form). Shipped: 4 of 4 unskipped tests in public-report.spec.ts and public-report-form.spec.ts passing cleanly post-Step-8 BFF re-pointing; the test.describe.skip wrapper on intake-review.spec.ts was removed. Of the 6 inner tests in that file, 4 pass against the post-Step-8 routes (review queue page loads, submitted report appears in review queue, record screening decision (screened_out), public status reflects screening decision); 2 are statically skipped with rationale (report detail shows concern and reporter info — .kv-val 'Sexual Abuse' selector no longer matches; record screened_in decision and convert to referral — button with hasText: 'Convert' matches two elements, tripping Playwright strict mode). Both are mechanical selector-rewrites tracked as #263 rather than blocking the plan.

After Step 16 lands

  • ADR-009 (mobile/offline-capable client) becomes safe to accept and execute. Spawn its implementation as a follow-on plan.

  • The parameter-struct sibling plan (filed in Step 14) is the next mechanical-cleanup MR.

  • Pre-1.0 hardening posture: most platform-invariant gates closed; CRAIG can begin evaluating against real users without multiplying data-loss risk.

GitLab issues to file (Step 1 creates these)

  1. feat(craig-cases): encryption fail-closed [Step 2 of platform-stabilization]

  2. feat(craig-auth): JWT audience/scope/azp validation [Step 3]

  3. feat(craig-mq): DLX/DLQ wiring + dead_letter_audit [Step 4]

  4. chore(events): retire 6 no-op subscriber stubs [Step 5]

  5. feat(craig-api): livez/readyz/healthz semantics [Step 6]

  6. chore(ci): validate-tests + cargo-deny + drop allow_failure [Step 7]

  7. feat(craig-mq): transactional outbox (cases) [Step 8]

  8. feat(craig-mq): idempotent inbox + handle_idempotently [Step 9]

  9. feat(craig-api): persistent idempotency middleware [Step 10]

  10. feat(xtask): cross-service reconciliation walker [Step 11]

  11. feat(craig-mq): outbox rollout to remaining 6 services [Step 12]

  12. feat: admin event replay endpoint [Step 13]

  13. chore: P3 cleanup bundle [Step 14]

  14. chore(e2e): unskip 5 stale specs [Step 15]

  15. chore(plans): platform-stabilization completion audit + archive [Step 16]

Plus 3 follow-on domain-decision-required issues filed by Step 5 (deferrals).

Edit this page · latest