Idempotency Middleware Body Integrity (empty/synthetic body substitution)

On this page

Status

Unit Description Status

U0

Choreography: label, issue amendments (#1183 v2 / #1182 scope expansion), #1194 filed, this plan

Done (2026-07-28) — commit f2407513

U1

Fix (terminal rule, frame capture, drop-guard, typed finalize outcomes) + 17-scenario matrix + proptest + RED-proofs

Done (2026-07-28) — commit b00eacf2; RED-proof 13/17 integration failures against the pre-fix binary (the 4 passes are the designed pre-fix-green pins); all 17 green post-fix + capture proptest + cache/response unit tests; J-review PASS with 2 doc-honesty flags remediated pre-commit (terminal-rule overclaim reworded; two panic-before-teardown escapes collected)

U2

Docs: ADR-022 §D5 refresh + amendment, data-model-security, CHANGELOG

Done (2026-07-28) — stale pre-atomic-claim DDL refreshed; the unsafe "client re-executes" instruction removed at both cites; mixed-fleet + no-repair honesty in CHANGELOG

Ship

Battery, MR, merge, close-out

Done (2026-07-28) — MR !1096 merged (48353f49); battery attempt 2 green (attempt 1 fast-failed at the fn-name-and cheap gate — four renames, b9c7e843); #1183 closed with AC walk; epic &74 ticked

Epic: &74
Issue: #1183 (Plan::IDEM-BODY)
Follow-ups: #1194 (expiry race — filed by this plan’s U0); #1182 (scope EXPANDED by U0: generation fencing, fingerprint-checked recovery, sentinel fidelity, client recovery policy — still gated on the maintainer’s contract steer)
Review state: v1 internal (Explore + Plan subagent passes, version-pinned API verification); v2 folded the 2026-07-28 external review (~27 findings — terminal rule, cap-split, trailers, cancellation, typed outcomes, expiry-race correction, wire/metrics honesty, test-matrix + routing corrections). Steer 2026-07-28: standalone v2; fencing → expanded #1182. Approved 2026-07-28.

Context

run_handler_and_finalize (crates/craig-api/src/idempotency/middleware.rs:301-365), mounted once (crates/craig-api/src/lib.rs:373-376) and inherited by all 8 ApiServer services (authed POSTs carrying Idempotency-Key; no first-party client sends it — exposure is contract-following SDK/external clients):

  • (a) Oversized 2xx poisons the cache. to_bytes(resp_body, RESPONSE_BODY_LIMIT + 1) errs for any body > limit+1; Err(_) ⇒ Bytes::new() feeds EMPTY bytes to finalize_succeeded (cache.rs:270-307) which computes too_large = false and persists a canonical succeeded row with an empty body — 2xx/empty live AND on every replay until expiry+cleanup. The documented sentinel (ADR-022 §D5) only materializes at EXACTLY limit+1 bytes. RESPONSE_BODY_LIMIT = 1 MiB (cache.rs:30).

  • (b) Finalize DB error swaps in an empty body. finalize_succeeded consumes the bytes; unwrap_or_else(|_| Bytes::new()) (middleware.rs:351-354) discards them on Err while the log claims "returning live response".

  • (c) Non-2xx arm truncates for nothing. Same capped to_bytes + empty-on-err, but finalize_failed stores no body. Both arms rebuild via a builder loop that drops response extensions + version.

  • (d) The terminal rule (v2). Once next.run returns 2xx the handler’s side effects are COMMITTED: any later delivery failure (over-cap tail error, mid-stream body error, client cancellation) must finalize into a NON-reclaimable success state — failed would let Path-D re-claim and re-execute the POST. The existing succeeded + body_too_large=true sentinel IS that state ("completed, response unavailable on replay") — no new status, no migration; one rule resolves over-cap/mid-error/cancellation uniformly.

Design

# Decision Substance

D1

Post-2xx terminal rule

After a 2xx, the only legal cache outcomes are succeededfull body (≤ 1 MiB, wholly captured) or `succeeded`body_too_large=true sentinel (over-cap, mid-stream error, cancellation, defensive oversize). NEVER failed; never left processing by our control flow (D5 drop-guard). Non-2xx keeps finalize_failed. Consequence: a different-fingerprint reuse after a post-2xx body error hits the succeeded-row fingerprint check → 422 (the failed/expired-reclaim fingerprint bypass stays #1182’s).

D2

Frame-level capture, cap-split zero-copy

capture_up_to_cap(body, cap) consumes http_body_util::BodyStream (FRAMES — trailer frames survive): data frames into BytesMut; on the crossing frame split_to(cap - buf.len()) pins the prefix at EXACTLY the cap and the remainder rides zero-copy ahead of the un-consumed stream. Outcomes Within { bytes, trailers } / Overflow { prefix, remainder+rest } / Failed { prefix, error }. Middleware buffering bounded by cap (+ transient frame refs — the crossing frame itself may be arbitrarily large; its tail is never copied). Cap is a parameter (prod = RESPONSE_BODY_LIMIT; proptest = small). Deps: futures-util + http-body-util (promoted to a direct workspace dep) in craig-api; Cargo.lock regenerated + committed.

D3

Arm handling

Within → finalize_succeeded(CachedBody::Full) — cache stores data bytes only (trailers live-only, documented); live body = recorded frames incl. trailers. Overflow → finalize_succeeded(CachedBody::TooLarge) awaited BEFORE the response returns (honest under D1 whatever the tail does), then prefix + remainder + rest streamed. Failed → sentinel (NOT failed — D1), then the response STREAM terminates with the error after the prefix. Pool note: finalize-early awaits a connection while the tail is un-polled; no craig handler body lazily holds a pooled connection — recorded as an ADR constraint + a one-connection-pool test.

D4

finalize_succeeded borrows + typed transition result

CachedBody<'a> { Full(&'a [u8]), TooLarge }; Result<FinalizeOutcome, sqlx::Error> with FinalizeOutcome { Applied, Lost } from rows_affected — Lost (row deleted/expired mid-flight) and transport Err (outcome UNKNOWN — may or may not have committed) get distinct honest structured warns; the "returning live response" lie is gone. Defensive: Full > cap stores the sentinel + warn! (caller bug, fail-safe). Drop the dangling Bytes import in cache.rs. NOT added: status='processing' guards / generation fencing (#1182 — a stale finalizer can still overwrite a reclaimed slot; pre-existing, recorded).

D5

Drop-guarded post-2xx finalization

FinalizeGuard (pool + cache_key + status/headers snapshot) arms after a 2xx, disarms on inline finalize; on Drop (client cancellation mid-capture/mid-finalize) spawns a best-effort sentinel finalize (idempotency.cancel_guard_fired). Closes the post-2xx cancellation → stuck-processing → Path-D re-execution window (the pre-2xx window is #1182’s lease design).

D6

Response::from_parts everywhere; non-2xx pure passthrough

Non-2xx: finalize_failed (typed-outcome warn on Err) → Response::from_parts(resp_parts, resp_body) untouched. Success: resp_parts.headers.insert(HeaderName::from_static("x-idempotency-replay"), HeaderValue::from_static("false")) on the mut parts (insert replaces a handler-set value), then from_parts — version + extensions survive. Helper split keeps fns ≤40 lines; the fused-shape [expect(too_many_lines, cognitive_complexity, single_match_else)] MUST be removed (unfulfilled [expect] fails the build); new private items get intent doc comments; drop the unused finalize_response import.

D7

Observability

Structured events: idempotency.overflow, idempotency.body_stream_error, idempotency.cancel_guard_fired, idempotency.finalize_lost, idempotency.finalize_unknown. ADR records the metrics discontinuity: http_observability completes at response-object return, so over-cap tail streaming/errors/cancellation are invisible to today’s latency/inflight metrics.

D8

Wire honesty

No schema migration. Framing MAY change for over-cap winners: explicit handler Content-Length survives (byte count unchanged); absent one, HTTP/1.1 goes chunked; HTTP/2 has no chunking; HTTP/1.0 is close-delimited. Bodies previously EMPTIED are now delivered — a stated behavior change, and the fix. Rolling deploy is schema-safe only: old replicas keep writing poisoned rows until drained; the behavioral guarantee starts when the last old replica exits. Poisoned rows are unrepairable-by-design (indistinguishable from legitimate empty-body successes) — recorded in CHANGELOG with the corrected expiry rationale (#1194: rows are dead-but-blocking up to an hour past expiry, not "harmless at 24h").

Units

U0 — label Plan::IDEM-BODY; #1183 v2 amendment note; #1182 scope-expansion note (generation fencing, fingerprint-checked recovery, sentinel fidelity, client recovery policy, lease-crossing pin, contention tests); #1194 filed (expiry race) + /relate #1183/#1182; this plan + nav; cargo xtask plan-lint.

U1 — code per D1–D8; proptest on the capture invariant (compact descriptors biased around cap−1/cap/cap+1, small test cap, ≈64 cases, one shared runtime; recorded as outside cargo xtask reliability’s `::properties:: selector — default sweep + MR CI --lib run it); integration target crates/craig-api/tests/idempotency_middleware.rs (#[ignore = "requires devstack"]; per-test throwaway DBs craig_idem_1183_{uuidv7.simple()} from craig_test_lib::postgres_test_db_url("postgres"); schema via sqlx::raw_sql over the two byte-identical craig-cases idempotency migrations; collect-observations → unconditional teardown → assert); harness = Router + idempotency layer + claims-inject layer (prod order), oneshot on a Router clone per request, sole-row DB asserts, AtomicUsize invocation counts; finalize faults via a scoped BEFORE UPDATE …​ WHEN (NEW.status='succeeded') RAISE trigger (table stays usable). 17-scenario matrix (the exact-cap replay pin folded into the exact-cap test) — (oversized-streamed RED, exact-cap pin, one-byte-over, giant single frame, finalize-failure RED, overflow-finalize-failure, oversized-non-2xx RED, body-error-before-cap RED, body-error-after-cap, different-fingerprint-422, loser-replays-while-winner-streams, cancellation RED, marker/extensions/version preservation, trailers live-only, one-connection pool, lease-crossing #1182 pin, real-TCP wire check) + response.rs/cache.rs unit tests (sentinel replay shape; defensive oversize; finalize-on-missing-row = Lost). RED-proof: git stash push — crates/craig-api/src (tests + manifests + lockfile stay), run, record, pop.

U2 — ADR-022: §D5 schema block refreshed (pre-atomic-claim DDL is stale) + the unsafe "client re-executes against the live handler" sentence REMOVED + dated #1183 amendment (terminal rule, full-body guarantee, framing deltas, sentinel-before-tail visibility, trailers live-only, finalize outcome taxonomy, metrics discontinuity, the no-pooled-connections-in-bodies constraint, mixed-fleet honesty); data-model-security.adoc cache-key (4→5-tuple) + re-execute text corrected; CHANGELOG === Fixed (schema-compat vs mixed-version behavioral safety separated; no-repair + corrected expiry rationale; framing/metrics deltas); plan Status per unit.

Verification

  • Integration: cargo nextest run -p craig-api --test idempotency_middleware --run-ignored=all (devstack up). Joins cargo xtask validate’s integration battery. Pre-push-only coverage: MR CI runs `--lib --bins; cargo xtask reliability’s `::properties:: selector matches neither this target nor the unit proptest.

  • Unit/prop: cargo nextest run -p craig-api capture_up_to_cap + cargo nextest run -p craig-api replay_of_too_large (default sweep; MR CI runs them).

  • cargo clippy -p craig-api --all-targets --locked — -D warnings (unfulfilled-#[expect] trap; --locked proves the committed lockfile).

  • Full battery pre-push (no reseed — no migration).

Out of scope (routed)

  • Generation fencing, conditional finalizers, fingerprint-checked failed/expired reclaim, sentinel replay fidelity, client recovery policy, lease-crossing double-execution, stale-finalizer contention → #1182 (scope expanded by U0; maintainer-gated).

  • Expiry race → #1194.

  • Repeated-header (set-cookie) collapse → #968.

  • Poisoned-row repair → impossible to target safely; rationale in CHANGELOG.

Edit this page · latest