ADR-068: Live-Service Degradation Posture Under DB I/O Stall — Server-Yields-First Typed Failure, Route-Class Request Ceilings, a Phase-Aware DB Error Taxonomy, and Saturation Hysteresis

On this page

Status

Accepted (2026-08-23). Design unit of #1404 (epic &83’s last child, outside the &83 program by the ratified boundary — ADR-067 §Open questions). Drafted FIRST per the draft-first path; implemented on feature/1404-degradation-posture. Plan: DB-Degradation Posture (archived). The design survived one internal adversarial review (11 findings) and the maintainer’s external stop-ship review (26 findings); the censuses in the appendices are review outputs, not estimates. Ratified direction (steer 2026-08-16, mode B): server-yields-first typed failure + honest health. Merging this design promotes the C21 pool-contention characterization from report-only to enforcing (contested-surfaces.toml, pool.craig-db.serving-acquire).

Context

The 2026-08-11 U4a battery (forensics on #1403) showed the fleet’s actual posture under Postgres I/O stall: POST /v1/cases/persons hung the full 30 seconds until the client timed out; outbox single-row UPDATEs ran 25.6s/17.2s. The arithmetic guarantees the hang:

  • The shared S2S client budget (craig_common::build_shared_client) is 30s total / 5s connect. The session statement_timeout (set once per connection by the pool’s after_connect hook) defaults to 30s. Equal budgets mean the client always eats the whole stall.

  • No inbound request deadline exists anywhere: apply_global_layers has no timeout layer and axum::serve runs bare. The only server-side bounds are the fixed 5s pool-acquire cliff (craig_db::SERVING_ACQUIRE_TIMEOUT, #1160) and the session statement timeout.

  • Error mapping is dishonest about all of it: SQLSTATE 57014 (statement kill) arrives as sqlx::Error::Database → the blanket ApiError::Db → a redacted RFC-9457 500 after up to 30s; sqlx::Error::PoolTimedOut maps to a typed 503 ONLY on the tx-begin path (map_begin_error); everything else — connection loss, commit-phase failures — is a 500 regardless of retry safety or outcome certainty.

A degraded database must never convert into a caseworker-facing raw stall. The server gives up first, typed; already-durable event-pipeline work catches up; health is honest without inviting restart flapping. That is the ratified posture this ADR realizes.

Decision

D1 — Deadline model: the two-inequality rule, enforced by route-class ceilings

Requiring the sum of all inner worst cases to fit under one ceiling is neither achievable nor necessary — every inner budget (pool acquire, statement timeout, authz/rules eval, S2S hop) is itself a fail-fast path that terminates the request with its own typed error. Two inequalities are required, each with explicit margin, and only these:

  1. Specific-beats-generic: each single inner budget + processing margin < the route’s ceiling — when exactly one component is slow, the caller gets that component’s typed error, never the generic ceiling 504.

  2. Server-yields-first: ceiling + client connect + margin < the client’s total budget — the server always answers (typed) before any first-party caller gives up.

The Standard chain:

Bound Value Enforcement

Pool acquire

5s

craig_db::SERVING_ACQUIRE_TIMEOUT — unchanged, fixed (#1160: not a knob)

In-request S2S hop

≤ 8s

tokio::time::timeout at each synchronous in-request S2S call site (census in Appendix A — In-request synchronous S2S hop census (per-hop budget ≤ 8s))

Per-tx statement budget

10s

SET LOCAL statement_timeout = '10s' inside the craig-api tx funnels, placed AFTER claim_first (see below)

Authz / rules eval budgets

5s (existing defaults)

unchanged; settings load refuses any configured value ≥ 19s

Request ceiling (Standard)

20s

the new ceiling middleware (D1a)

S2S client

30s total / 5s connect

build_shared_client — unchanged; DEFAULT_TIMEOUT exported for the assert chain

Checks: 5+ε<20, 8+ε<20, 10+ε<20, 20+5+5<30. Where both sides are consts the order is pinned by const _: assertions; where one side is a knob it is pinned by settings-load refusal: db_statement_timeout_ms < 10_000 is refused (SET LOCAL would otherwise silently raise an operator-tightened bound — the floor makes the joint decision explicit), and authz_eval_timeout_ms / the rules eval budget are refused at ≥ 19s.

SET LOCAL placement — as-built mechanism: the wrapper cannot inject between the claim and the domain statements (both live inside the handler closure), so the post-claim budget hook lives INSIDE claim_first itself — it applies the SET LOCAL as its final act, after the claim resolves. ADR-062’s claim-first contract is untouched, and the claim INSERT keeps the session-default (30s) lock-wait allowance its duplicate-arbitration math requires. In execute_within_tx (unclaimed): immediately after begin. The explicit opt-out for the censused Batch routes is claim_first_unbudgeted (full ADR-062 claim semantics, only the SET LOCAL omitted — the route rides the session default under its Batch ceiling). Census: ncands generate (10k-row INSERT…UNNEST batches), subsidy sweep preview/execute. A new claim_first_unbudgeted call site requires a justifying comment naming the >10s statement. No unclaimed _unbudgeted variant exists — the census has no unclaimed batch funnel user (unused API is not built).

One recorded interaction: the 8s hop budget sits BELOW the 19s eval-budget caps. Under default configs (5s eval) the callee’s typed 503 always arrives first; a deployment that raises an eval budget past ~7s trades error specificity for the same bounded outcome — the caller’s typed hop 503 fires at 8s while the callee finishes uselessly. Bounded and typed either way; specificity degrades gracefully. The reporting→cases whole-FFY export hop is NOT an OLTP hop — it targets a Batch-class route and carries BATCH_CALLER_BUDGET (the caller-budget pairing), not the 8s hop budget.

Session statement_timeout stays 30s by default — it is the worker-plane authority (ADR-062 §H derives the inbox winner-hold math from it). Standard routes that begin() directly on the pool (census: financial adjustments-resolution, placement kinship, cases report-create, among others) ride ceiling-only until migrated to the funnels; the funnels are the new-code convention.

Depth-2 honesty (per-hop margins instead of propagation). Absolute deadline propagation is rejected while the synchronous S2S graph has depth ≤ 2: it is header + re-budgeting machinery on every hop, and it still cannot validate external budgets. Instead every synchronous in-request S2S call carries a per-call budget ≤ 8s (the financial CASES_S2S_DEADLINE precedent), asserted < ceiling. The accepted residue: a downstream service may finish work after its caller has already yielded — inherent to non-propagation; effects are claims-idempotent (ADR-062). Revisit trigger: the graph deepens past 2.

D1a — The ceiling middleware and route classes

The fleet has a real population of legitimately-long synchronous routes, so the ceiling is class-based — ONE global middleware selecting the class per request; never an inner route_layer (an inner layer cannot extend an outer timeout):

Class Ceiling Members Caller budget pairing

Standard

20s

everything not listed below

shared 30s client (inequality 2)

Extended

300s

upload nests (intake signed 30 MiB, cases attachments + contact attachments + court-order documents, placement home documents, exchange ICPC attachments, intake attachment proxies) and buffered download/export GETs (object_store.get returns the whole blob before the response head — reporting ncands/afcars exports, attachment downloads)

the specific first-party call sites get per-request .timeout(310s): BFF upload/download proxies, CLI export fetches, intake attachment forward

Batch

600s

the synchronous admin/batch/federal engines: subsidy generate, import finalize, sweep preview + execute, ncands + afcars generate/export, rules promote-preview canary, admin-replay, security archive + purge

CLI/xtask call sites get per-request .timeout(610s); external callers documented below

A class ceiling without its caller-budget leg is dead policy — the server would keep working for minutes after every first-party caller abandoned at 30s. The two land together.

Mechanics:

  • The table is data on ServerOptions: (Method, route template) → Class, matched at runtime via MatchedPath (the http_metrics_middleware precedent) — never raw path prefixes (dynamic IDs and shared paths across methods make prefixes ambiguous). A syn-based coverage test (the validate_authz_coverage idiom) fails on any table entry absent from the service’s route registration, so the table cannot go stale silently.

  • Position: innermost of the global layers — inside http_metrics_middleware (which has no drop guard; an outer timeout would leak the in-flight gauge and lose the 504 from the duration histogram) and inside request_id_layer (the problem document carries the correlation id).

  • Implementation: axum::middleware::from_fn + tokio::time::timeout(next.run()) — no new dependencies, no tower feature enables, full control of the RFC-9457 body. The bound is cooperative and covers request processing up to the response head: streamed response bodies are not covered (today’s download handlers buffer, hence their Extended class), and a non-yielding CPU section cannot be preempted (none exists on these paths; rules and authz evals carry their own budgets and the ADR-024 wedge deadman).

  • Prod values are consts; a test-injectable seam (the HTTP_DRAIN_DEADLINE / serve_with_listener precedent) lets fault legs run at ~1s.

  • craig-intake does not use ApiServer: the middleware ships as a reusable pub fn and intake installs it in its own stack explicitly. As-built: the whole ceiling module (craig_common::ceiling) lives in craig-common, not craig-api — intake is deliberately craig-api-free (no sqlx on the stateless edge), and craig-api re-exports it; only the pool-acquire leg of inequality 1 is asserted in craig-api (craig-common cannot depend on craig-db). The xtask import tool mirrors BATCH_CALLER_BUDGET with a named comment (xtask is deliberately craig-common-free; tool-time OUT-class per ADR-067).

  • craig-web (BFF) gets NO inbound ceiling: its handlers legitimately fan out multiple sequential 30s upstream calls; its bound is per-upstream-call. Recorded, not deferred.

  • External (non-first-party) API consumers cannot be validated at config load. The contract is documented instead: responses are bounded at 20s/300s/600s by class; set client budgets above your class.

D1b — Cancellation safety (census, not hope)

The ceiling’s cut is mechanically the same event as a client disconnect, which hyper already inflicts on every handler when the peer goes away — the ceiling makes an existing hazard deterministic; it does not create the class. The census:

Sequence class Outcome when cut

Funnel tx routes (all convert-class routes, ADR-062)

drop = rollback; no partial commit; replay-same-id answers from the claim

Direct-begin() routes

drop = rollback (same mechanics, no claim replay)

Single-statement autocommit writes

statement-atomic

Multi-system non-tx sequences (DB row THEN blob delete: contacts, contact/report attachments, court orders)

a cut between the two leaves an orphan blob — the standing convergence owner is the orphan-blob janitor issue #1274 (pre-existing exposure via client disconnect; unchanged by this ADR)

Commit-phase cut

outcome unknown — the D2 db-outcome-unknown class exists precisely for this

Batch routes cut by deploy (HTTP_DRAIN_DEADLINE 20s ≪ 600s)

abandoned mid-flight — acceptable because the money engines are crash-resumable BY DESIGN (claims, staged finalize, durable sweep records: ADR-055/056/057); their resumability is their own ADRs' contract, restated here per route in the implementation

One sharp edge stated honestly: sqlx 0.8 returns a dropped pooled connection through an unbounded async ping (the connection waits for Postgres ReadyForQuery). Under a wire blackhole a pool slot can therefore wedge past statement_timeout, bounded only by the TCP stack and the pool’s 30-minute max_lifetime. The acquire cliff sheds load meanwhile (typed 503s — the designed degraded capacity). TCP keepalives via PgConnectOptions are enabled if the API offers them (implementation-verified). Health probes avoid this trap entirely by never using the serving pool (D4).

D2 — Phase-aware DB error taxonomy

One context-free mapping cannot be honest: retry safety depends on the request phase, not just the sqlx variant. The classifier is a pure function in craig-common (it must not depend on craig-db — dependency direction), consumed through phase-specific ApiError constructors; the funnels supply the phase.

Phase (constructor) Condition Status / problem type Retry signal

acquire — from_db_acquire

PoolTimedOut, PoolClosed

503 service-unavailable

Retry-After: 5 — ONLY here. The request was refused before any work; retry is truthfully cheap.

statement — from_db_statement

SQLSTATE 57014

504 db-timeout; detail: "canceled — statement timeout or administrative" (57014 is query_canceled, not uniquely a timeout)

none on the wire; in-tx the work is rolled back

statement — from_db_statement

connection family: Io, Tls, SQLSTATE 08*, 57P01/57P02/57P03. Protocol is EXCLUDED — production code synthesizes sqlx::Error::Protocol for serialization failures (subsidy import finalize), which must never read as retryable connection loss

503 db-connection-lost

none

commit — from_db_commit

the connection died during COMMIT (connection family) — the server’s verdict never arrived

503 db-outcome-unknown — the commit may or may not have landed

none — no retry invitation on an unknown outcome. A server-REPORTED commit failure (serialization failure, deferred constraint, statement cancel) means the transaction aborted — outcome KNOWN — and rides the ordinary classifier mapping instead.

any

everything else

500 internal (redacted, unchanged)

—

ceiling middleware

elapsed

504 request-timeout

none

  • The blanket From<sqlx::Error> impl becomes manual and applies the conservative statement-phase mapping — the safety net for unswept sites. The Db field keeps #[source].

  • Full variant decision table (no variant undecided): PoolTimedOut/PoolClosed → acquire-class 503 · Database(code) per the SQLSTATE rows above, other codes → 500 · Io/Tls → db-connection-lost 503 · Protocol → 500 · RowNotFound, TypeNotFound, ColumnIndexOutOfBounds, ColumnNotFound, ColumnDecode, Decode, Encode, Configuration, Migrate, WorkerCrashed, others → 500 (programming/config errors, not degradation).

  • No retry-safety promises ride the wire details: the BFF re-render mints a fresh client_request_id today, and the classifier cannot know route idempotency. Per-class retry semantics are documented HERE; the SDKs (which retry same-id per ADR-062) cite this section. Retry-After is variant-specific — the rules eval-timeout design explicitly declined it (no truthful estimate) and that decision stands.

  • map_begin_error is deleted — subsumed by from_db_acquire; keeping it would be pre-1.0 compat residue. Funnel arms: begin → acquire-phase; domain statements → statement-phase; commit → commit-phase (a connection loss during COMMIT is exactly the outcome-unknown case, today mapped to a plain 500).

  • Coverage is enforced, not promised: a new xtask lint (allowlist + monotonic shrink, the quality-budgets idiom) bans direct ApiError::Db(/Self::Db( construction and sqlx-typed map_err(ApiError::internal) outside craig-common. The implementation sweep works the census in Appendix B — Classifier sweep census (U5; the lint keeps it closed); the lint keeps it closed.

  • Untouched, verified disjoint: craig-security’s dead-letter classify_sqlx (MQ plane) and craig-mq’s inbox savepoint 57014 handling — neither converts into ApiError.

  • OpenAPI: utoipa cannot see middleware. A shared Modify adds the 503/504 problem responses to every operation in every service’s ApiDoc; the committed API pages regenerate in the same commits (#1022 gate).

D2a — Edge pass-through

  • BFF (craig-web): upstream 503 → 503, 504 → 504; all other upstream 5xx stay 502 (correct gateway semantics — 502 means "the upstream failed", 503/504 alone carry retry meaning). UpstreamError::Status carries no headers, so the BFF does NOT forward Retry-After (it renders HTML to humans; recorded). Mutation handlers keep redirect + flash; the flash copy becomes class-appropriate ("busy — try again shortly" vs "timed out"). `get_bytes’ 5xx-as-data contract is unchanged (recorded).

  • craig-intake (public portal): the sink stops flattening upstream 503/504 into a redacted 500 — the public edge keeps the typed posture.

  • craig-cases: the rules-upstream guard and the matching-error guard (both have 503-only arms today) extend to 504.

All of this is pre-1.0 breaking and rides the CHANGELOG.

D3 — The honest durability claim

  • Work that was already staged (outbox rows) before or during the stall drains when the DB recovers: the OutboxWorker’s warn-and-retry loop plus lease-healed claims. Delivery is at-least-once with consumer dedup — the outbox’s own contract; this ADR does not upgrade it to exactly-once and no test may claim otherwise.

  • NEW work while the DB is down cannot become durable, and therefore fails typed (503 at the acquire cliff / 504 at the statement kill). There is no queueing fig leaf.

  • Bounds: outbox backlog is bounded by DB disk (the DB-up/broker-down case); the DLX tier (ADR-059) is a consumer-side mechanism, not the pending-outbox path; broker-down ≠ DB-down and keeps its existing posture (ADR-003/ADR-059).

D4 — Health: dedicated-connection probes + saturation hysteresis; readiness unchanged

  • Probes never touch the serving pool. Reachability = a fresh dedicated PgConnection per probe, 2s overall bound including connect (the compose healthcheck window is 3s — the probe must answer inside it). Dropping a non-pooled connection is a socket close — none of the pooled return-ping hazard. This also removes today’s amplification: the unbounded /healthz SELECT 1 through the pool wedges a connection per hit under a wire stall.

  • /readyz semantics unchanged: reachable ⇒ ready. Pool saturation NEVER gates readiness — flipping readiness on latency invites the restart/eviction flapping the ratified direction forbids (ADR-061 alignment). No contradiction with the 2s bound: pool wait cannot affect a dedicated connection.

  • DbSaturation hysteresis, fed by the existing 15s pool-probe loop: a probe is BAD when its acquire wait exceeds 2.5s (half the cliff) or times out; degraded after 2 consecutive BAD (~30s of signal); clears after 4 consecutive OK (~60s). Exposed as the new db_pool_degraded gauge (0/1) and the /healthz database value ok|degraded|error.

  • The probe task decouples from telemetry: today it spawns only when metric registration succeeds and non-otel builds skip it entirely. Functional health cannot depend on observability being compiled in — the probe always runs and feeds health state; metric emission is the optional part. Startup state is ok; the probe task is supervisor-watched (Observed).

  • /livez untouched. The deployment guide gains the db_pool_degraded alert row (the worker-staleness convention’s sibling).

D5 — Load-shed / circuit-break: the pool is the admission control, stated precisely

Judgment-protocols comparison (all in-stack options considered):

Option Verdict

Pool acquire cliff + request ceiling (CHOSEN)

Admission control lives at the constrained resource: the fixed 5s cliff sheds to a typed 503 exactly when DB capacity is exhausted, with no proxy metric to mis-tune; the pool also caps the reconnect herd against a recovering database at max_connections — the breaker’s classic protective role, provided structurally. Zero new dependencies or feature enables. Scale remedy stays capacity + replicas (#1160).

tower LoadShed + ConcurrencyLimit

A second admission bound ABOVE the pool that must be sized against max_connections forever (two knobs that fight); the features are compiled nowhere in the graph today; axum’s Infallible services need HandleErrorLayer plumbing. Rejected as overengineering.

Circuit-breaker crate (failsafe, recloser, …)

A new dependency and per-route trip state duplicating what the acquire cliff provides at the resource itself; its half-open probing is already served by the pool’s bounded reconnects. Rejected.

Stated precisely: the pool is a connection-concurrency cap with a bounded (5s) per-request wait — it is NOT a circuit breaker. It has no open state, no cooldown, no pending-waiter cap: during a full outage every DB-bound request parks one acquire future for ≤ 5s and then fails typed, so at arrival rate R roughly R×5 futures are parked (1,000 rps ⇒ 5,000 parked futures — well inside tokio’s envelope; the arithmetic is the recorded justification for not adding a waiter cap). A hysteresis-driven fast-fail (shed at 0s while db_pool_degraded is set) is explicitly DEFERRED, with its trigger recorded: operators needing sub-5s shed under sustained outage. The per-IP rate limiter (429) and the authz/rules eval budgets keep their existing roles.

Consequences

  • Caseworkers and S2S callers see typed, bounded failures — worst-case time-to-error on a Standard route drops from 30s (client transport error) to ≤ 20s (server-typed 504), and for the common single-slow-component cases to 5s/8s/10s with the specific diagnosis.

  • Pre-1.0 breaking changes (CHANGELOG’d): 57014 becomes 504 (was 500-after-30s); PoolTimedOut becomes 503 fleet-wide (was begin-path-only); commit-phase failures become 503 db-outcome-unknown (was 500); BFF/intake/cases edges pass 503/504 through; db_statement_timeout_ms gains a 10s floor.

  • Existing tests pinning 500-on-DB-error or BFF-502 are updated as part of the same change — a legitimate oracle change for a deliberate semantic change, called out per commit (testing-discipline).

  • The C21 pool-contention oracle promotes to load-robust shape assertions (wins < the acquire bound; timeouts ≥ ~the bound — never exact milliseconds, per the harness’s own "load shifts latencies, not counts" note).

  • Deferred, each as a filed issue: backgrounding the Batch engines (202 + poll — removes the 600s class’s reason to exist), streaming object-store downloads (removes most of the Extended-download class), hysteresis fast-fail (trigger recorded above).

Appendix A — In-request synchronous S2S hop census (per-hop budget ≤ 8s)

Caller → callee Site Budget

cases → rules (investigation safety assessment)

services/craig-cases/src/api/investigations/safety.rs

8s (new)

cases → rules (screening policy)

services/craig-cases/src/screening_policy.rs

8s (new)

cases → rules (matching ruleset metadata)

services/craig-cases/src/matching/ruleset_metadata.rs

8s (new)

reporting → cases (detail hydration)

services/craig-reporting/src/cases_client.rs

8s (new)

financial → cases (person DOB, MQ path)

services/craig-financial/src/main.rs (CASES_S2S_DEADLINE)

8s (exists)

exchange → cases (custody)

services/craig-exchange/src/cases_client.rs (CUSTODY_TIMEOUT_SECS)

5s (exists)

Appendix B — Classifier sweep census (U5; the lint keeps it closed)

Site Today

financial subsidy maps ×7 (subsidy_error_map.rs, subsidy_agreements.rs, subsidy_review_sweep.rs ×5)

direct ApiError::Db(…​) wrap

the funnels' own begin/commit arms (crates/craig-api/src/lib.rs)

commit errors → internal

crates/craig-api/src/admin_replay.rs, upload_attempts.rs, subsidy_mutations.rs acquire

→ internal

rules store map (services/craig-rules/src/api.rs + ~20 sites)

→ internal

reporting store maps (~30 sites incl. api/mod.rs rereads)

→ internal

composition src/write_support.rs + read paths (overrides.rs, user_delta.rs)

own begin(); bypasses even today’s PoolTimedOut 503

security archive (src/api/archive.rs, src/store/archive.rs)

→ internal

exchange store maps

→ internal

service error enums: GenerateError, EngineError, AttemptError::Sqlx, WriteError::Db, ArchiveError::Db

Db arms flattened

Alternatives considered

  • Absolute deadline propagation (header + per-statement re-budgeting): rejected at graph depth ≤ 2 — see D1. Revisit trigger recorded.

  • Lowering the session statement_timeout default so the typed 57014 always beats the ceiling: rejected — the session default is the worker-plane authority and ADR-062’s inbox lock-wait math derives from it; the funnel-scoped SET LOCAL delivers the same ordering on exactly the HTTP tx plane with zero worker blast radius.

  • tower timeout/load-shed layers and breaker crates: see D5.

  • One flat ceiling with per-route exemptions: rejected — the review census showed a real population of legitimate 25s+ routes; exemptions-by-default invert into the Extended/ Batch classes with caller-budget pairing instead.

Edit this page · latest