Plan: DB-Degradation Posture — Server-Yields-First Typed Failure

On this page

Status

Step Description Status

U0

Plan page (this file, nav Active) + ADR-068 (deadline model, taxonomy, health, load-shed comparison, censuses)

Done (2026-08-23) — lands as the branch’s first commit, docs-first

U1

DbErrorClass + phase-aware ApiError constructors + manual From<sqlx::Error> + connection-family fn + variant-specific Retry-After

Done (2026-08-23) — classifier + GatewayTimeout + sub-typed ServiceUnavailable in craig-common; commit-phase refined (server-reported failure = known outcome, connection-death = db-outcome-unknown); 9 new pins + 2 proptests

U2

Request-ceiling middleware (reusable; intake wired explicitly) + (Method, MatchedPath) class tables + coverage test + funnel SET LOCAL after-claim + _unbudgeted variants + settings refusals + delete map_begin_error + OpenAPI Modify + caller-budget legs + S2S hop budgets

Done (2026-08-23) — craig_common::ceiling (module in craig-common so the craig-api-free intake edge installs it; craig-api re-exports + carries the pool-acquire assert); 8 service tables + intake table + per-service coverage tests; budget hook lives INSIDE claim_first (opt-out = claim_first_unbudgeted; no unclaimed variant — no user); settings floors (statement ≥10s; authz+rules eval ≤19s); map_begin_error deleted; DegradationResponsesAddon ×9 + api-docs regen; caller budgets at BFF post_multipart/get_bytes, intake attachment forward, reporting whole-FFY export (Batch pairing), CLI post_batch ×5, xtask finalize (mirrored const); hops 8s at cases→rules ×3

U3

Dedicated-connection health probes + saturation hysteresis + db_pool_degraded gauge + /healthz value + telemetry decoupling

Done (2026-08-23) — craig_api::db_health (2s-bounded fresh-connection probes; drop = socket close; hysteresis 2-BAD/4-OK; process-global per the #1236 one-pool doctrine); probe task claims independently of metric registration (no-otel builds keep functional health); /healthz db value ok|degraded|error; /readyz reachability-only; deployment-guide alert row; 4 unit pins incl. the wire-stall bound

U4

Edge pass-through: BFF 503/504 statuses + class copy, intake sink, cases rules_upstream + matching guard

Done (2026-08-23) — BFF http_status passes 503/504 (other 5xx stay 502; Retry-After not forwarded, recorded); class copy (504 flags the indeterminate outcome); intake sink typed 503/504 arms (body still redacted); cases upstream_rules_refusal + matching guard extend to 504; Transport.extended_budget test seam (per-request budgets override client bounds)

U5

Classifier fleet sweep (census) + the db-error-classifier coverage lint (allowlist, monotonic)

Done (2026-08-23) — 3 direct Db( wraps → .into(); 110 method-ref internal-flattens → the classifier (compiler-verified; 11 serde/config survivors sanctioned); enum Db arms (rules Engine, upload-attempts, security Archive, composition Write) split through the classifier; composition commit → from_db_commit; the db-error-construction lint BLOCKING in validate (zero baseline; serde lines sanctioned; DB-FLATTEN-OK: marker idiom; Self::Db in foreign enums recorded false-negative)

U6

Fault legs 1–10 + l2 extensions + classifier proptest + C21 promotion

Done (2026-08-23) — J-review remediated (guard self-satisfaction fixed via ceilings.rs siblings ×8 with a helper tripwire + ghost-template teeth proof; wire-stall leg re-routed through the REAL ceiling middleware; 504 detail honesty; lint scan no longer truncates at mid-file cfg(any(test; probes single-flight behind a 1s verdict cache). degradation.rs ×7 legs, ALL GREEN LIVE (budget/57014-rollback-reuse/5s-cliff-Retry-After/hysteresis-never-readyz/full-stack-504-with-correlation-id/wire-stall-hits-ceiling/commit-ack-cut-outcome-unknown + claimed-replay-no-dup); outbox_db_stall.rs at-least-once drain leg (craig-mq); l2 legs assert the wire-taxonomy classification (Protocol stays Other); C21 PROMOTED to load-robust latency shape (green live); router_with_ceiling_config seam; classifier proptests landed in U1

U7

Close-out: architecture.adoc RFC-9457 row, CHANGELOG, plan → Done/Archive

Done (2026-08-23) — architecture RFC-9457 row carries the ADR-068 taxonomy; CHANGELOG Added + Changed (pre-1.0 breaking) entries; plan archived

Epic: &83 (last child)
Issues: #1404
Branch: feature/1404-degradation-posture

Context

The 2026-08-11 U4a battery exposed the fleet’s degradation posture under Postgres I/O stall: POST /v1/cases/persons hung the full 30 seconds until the client gave up. The arithmetic guaranteed it — the S2S client budget (30s, build_shared_client) exactly equals the session statement_timeout (30s), and no inbound request deadline exists anywhere (bare axum::serve, no timeout layer). Error mapping compounds it: a statement kill (SQLSTATE 57014) surfaces as a blanket RFC-9457 500 via ApiError::Db, and sqlx::Error::PoolTimedOut is a typed 503 only on the tx-begin path.

The 2026-08-16 steer ratified mode B: server-yields-first typed failure + honest health. A degraded database must never become a caseworker-facing raw stall; the server gives up first with a typed RFC-9457 error; the event pipeline queues and catches up; health reporting is honest without inviting restart flapping.

ADR-068 carries the full design. The plan went through one internal adversarial review (11 findings) and the maintainer’s external stop-ship review (26 findings) before ratification; both censuses (route classes, classifier call sites) in the ADR are review outputs, not estimates.

Couplings: #1403 (merged) removed the test-infra aggressor that first exposed this, so pool-timeout collateral in future batteries is clean product signal. Epic &83’s C21 shipped the pool-contention harness report-only with its enforcement promotion gated on this design merging (contested-surfaces.toml, pool.craig-db.serving-acquire). #1404 is epic &83’s last open child.

Scope

In scope:

  • The two-inequality deadline model: route-class request ceilings (20s / 300s / 600s), funnel-scoped 10s statement budget (SET LOCAL after claim_first), per-hop S2S budgets (≤ 8s) at synchronous in-request call sites, settings-load refusals for knob legs.

  • The phase-aware DB error taxonomy: from_db_acquire (503 + Retry-After), from_db_statement (57014 → 504; connection family → 503), from_db_commit (503 db-outcome-unknown), plus the fleet sweep and the coverage lint that keeps it closed.

  • Caller-budget pairing for Extended/Batch classes (per-request reqwest timeouts at the first-party call sites).

  • Health: dedicated-connection reachability probes (2s bound), pool-saturation hysteresis (2-BAD enter / 4-OK clear), db_pool_degraded gauge, /healthz ok|degraded|error; /readyz semantics unchanged (saturation never gates readiness).

  • Edge pass-through: BFF 503→503 / 504→504 (other 5xx stay 502), intake public sink, cases rules-upstream + matching guards.

  • Fault legs 1–10 asserting the posture (including lost-COMMIT-ack and hysteresis transitions) + the C21 enforcement promotion.

Out of scope (recorded in ADR-068 §Consequences):

  • A BFF inbound ceiling (its handlers legitimately fan out multiple 30s upstream calls; its bound is per-upstream-call).

  • Absolute deadline propagation (revisit trigger: the S2S graph deepens past 2).

  • Changing the session statement_timeout default (worker-plane authority, ADR-062) or fleet-wide per-consumer MQ pools.

  • Backgrounding the Batch engines (202 + poll), streaming object-store downloads, and hysteresis-driven fast-fail — each filed as a follow-up issue at close-out.

  • Prometheus alert-rules infrastructure (the gauge + documented alert rule is the deliverable).

Design

The complete design — the two-inequality model with its margins, the route-class and caller-budget tables, the full sqlx variant decision table, the cancellation-safety census, the load-shed comparison, and the health-signal state machine — lives in ADR-068. This plan does not duplicate it; the ADR is the implementation spec. Unit boundaries and file lists follow.

Steps

U0: Governance docs

Files: this page, docs/modules/ROOT/pages/adrs/adr-068-db-degradation-posture.adoc, docs/modules/ROOT/nav.adoc

Docs land first (the &83 precedent): ADR-068 is the spec the code is reviewed against.

U1: Taxonomy + classifier (craig-common)

Files: crates/craig-common/src/error.rs, crates/craig-common/src/http.rs

DbErrorClass enum + classify_db_error(&sqlx::Error) → DbErrorClass; phase constructors ApiError::{from_db_acquire, from_db_statement, from_db_commit}; manual From<sqlx::Error> (conservative statement-phase fallback; field keeps #[source]); new problem types db-timeout, db-connection-lost, db-outcome-unknown, request-timeout; Retry-After: 5 ONLY on the acquire-phase 503 (variant field, never status-wide); DEFAULT_TIMEOUT becomes pub for the const-assert chain.

U2: Ceiling + budgets

Files: crates/craig-api/src/lib.rs (+ new ceiling.rs), crates/craig-common/src/settings.rs, per-service route-class tables, services/craig-intake/src/main.rs, S2S call sites per the ADR census, config/README/.env.example

Class-table middleware (reusable pub fn, innermost global layer, test-injectable seam); funnel SET LOCAL statement_timeout = '10s' AFTER claim_first (execute_within_tx: after begin); execute_within_tx_unbudgeted + execute_claimed_tx_unbudgeted (census: ncands generate, sweep preview/execute); settings refusals (db_statement_timeout_ms < 10_000, authz/rules eval budgets ≥ 19s); map_begin_error deleted; shared utoipa Modify adding the 503/504 problem responses per operation + cargo xtask api-docs regen; caller-budget legs (BFF upload/download proxies .timeout(310s), CLI/xtask batch calls .timeout(610s)); S2S hop budgets ≤ 8s (cases→rules ×3, reporting→cases).

U3: Health

Files: crates/craig-api/src/{lib,bootstrap}.rs, crates/craig-common/src/metrics.rs, docs/modules/ROOT/pages/deployment-guide.adoc

Dedicated-connection reachability probes (fresh PgConnection, 2s overall bound, drop = socket close) for /readyz + /healthz; DbSaturation hysteresis fed by the 15s probe loop; probe task decoupled from telemetry registration (always runs; metrics optional); db_pool_degraded gauge; alert-rule row in the deployment guide.

U4: Edge pass-through

Files: services/craig-web/src/routes/error.rs, services/craig-web/src/clients/transport.rs, error/flash templates, services/craig-intake/src/sink/upstream_error.rs, services/craig-cases/src/matching/error.rs, cases rules_upstream.rs

Statuses pass through; Retry-After is NOT forwarded by the BFF (its UpstreamError::Status carries no headers — recorded); mutations keep redirect + flash with class-appropriate copy; get_bytes 5xx-as-data recorded as-is.

U5: Fleet sweep + coverage lint

Files: the ADR census sites; xtask/src/cmd/lints.rs

Every sqlx→ApiError conversion in the census routes through the phase constructors; the new lint (allowlist + monotonic shrink) bans direct ApiError::Db(/Self::Db( construction and sqlx-typed map_err(ApiError::internal) outside craig-common.

U6: Fault legs + C21 promotion

Files: crates/craig-api/tests/fault.rs + tests/fault/degradation.rs, crates/craig-db/tests/l2_db_faults.rs, crates/craig-test-lib/tests/pool_contention.rs, contested-surfaces.toml, ADR-067

Legs 1–10 per the ADR (ceiling, statement kill, acquire cliff, wire stall, lost COMMIT ack, recovery drain, hysteresis, S2S hop, C21 shape promotion, edge); classifier proptest (family disjointness, redaction, source preservation); l2 legs assert the new mappings.

U7: Close-out

Files: docs/modules/ROOT/pages/architecture.adoc, CHANGELOG.adoc, this page, nav.adoc

RFC-9457 row gains the 504 classes; CHANGELOG Changed entries (pre-1.0 breaking: 57014→504, PoolTimedOut→503 fleet-wide, edge pass-through, ceiling classes, statement-timeout floor); Status → Done; nav Active → Archive.

Post-merge (not commits): close #1404 with the closing record; close epic &83 (last child); update the .claude/CLAUDE.md &83 line; file the follow-up issues (background Batch engines; streaming downloads; hysteresis fast-fail).

Verification

  1. cargo fmt --check --all, cargo clippy --all-targets — -D warnings, the cheap lint gates — bare, before every push.

  2. cargo nextest run --workspace — including the new unit pins (classifier table, const-asserts, settings refusals).

  3. cargo xtask validate with the fault profile REQUIRED — legs 1–10 execute and record; zero-fault green is a failure (ADR-067 D9).

  4. cargo xtask e2e — BFF error-page rendering under upstream 503/504.

  5. Never-easier check in the J-review: zero nextest.toml serialization or envelope hunks; every fault leg asserts a typed refusal at a bounded time.

Documentation Updates

  • Architecture — RFC 9457 row (new 503/504 problem classes)

  • CHANGELOG.adoc — == Unreleased Changed entries (all pre-1.0 breaking changes)

  • ADR-068 — the as-built design

  • Deployment Guide — db_pool_degraded alert row; probe semantics

  • ADR-067 — C21 promotion resolution pointer

Edit this page · latest