Plan: Retention Enforcement — Transport Sweeps (#965) + Archive-Then-Prune (#1129)

On this page

Status

Unit Description Status

A1

Step 0: D-A resolution recorded on #965 + #1129; labels/relations/epic; both plans updated + nav

Done (2026-07-26) — notes 3603012745/3603012780; #965 → epic &71

A2

#965 Steps 2–4: retention settings (30/31 split, load()-validated), OutboxWorker::prune_expired (pinned-connection advisory lock, batching, cycle cap, drain-before-prune, jitter, metric)

Done (2026-07-26) — craig-mq retention.rs + OutboxWorker integration; 4 scratch-DB tests green live

A3

Extensions: prune_table helper exposed; exchange exchange_send_jobs + financial subsidy_reconcile_queue sweeps; admin-replay boundary clamp

Done (2026-07-26) — spawn_local_sweep shared loop; exchange gained its minimal lib target; clamp threaded via ServerOptions

A4

Watchdog invariants (4 transport) + grace constants + catalog mapping + red-harness (scratch-DB seams)

Done (2026-07-26) — committed with A3 (the red-harness folds into A3’s scratch suites)

A5

ADR-058 + ADR-022 correction + docs + CHANGELOG; MR-A battery/merge; close #965

Done (2026-07-26) — MR !1062 merged (a3e8a459); #965 closed; + f7cfc0ed rustdoc module-doc link fix

B1

crates/craig-retention: lease, spec, manifest, engine (quarantine, budgets, scheduler) + Store::put_create / delete_if_exists

Done (2026-07-26) — 20 unit/property + 6 scratch-DB engine tests green live; J-reviewed (consent re-check blocker fixed in-commit)

B2

craig-security: engine wiring + archive_records migration (repair + legal_hold) + run/purge rework + attribution + parser arms + event validation

Done (2026-07-26) — 7-test direct-router suite (enabled end-to-end, D14 purge matrix, D16 forge/replay/divergence, D11 403) green live; live-API consent tests flip green at the battery’s container rebuild

B3

craig-rules: LOCAL archive_ledger + archive worker + event + boot probe + compose credentials

Done (2026-07-26) — production-spec scratch suite green live (ledger + contract-shaped event + atomicity); archiver OFF in compose per D20

B4

CLI + web/model consumers; archive invariants (3) + red-harness; the non-vacuous AC3 gate

Done (2026-07-26) — CLI destructive rework + 3 watchdogs red-harnessed in the B2/B3 suites; 2 CLI consent tests flip at the battery rebuild

B5

Runbook + xtask archive-fetch + remaining docs; close #1129; ⁂ #1073 note; plans → Archive

Done (2026-07-26) — this commit (runbook, archive-fetch, ADR-058 as-built amendments, ADR-003, config/data-model/shared-crates/CHANGELOG, plan → Archive); #1129 closes at MR-B merge

Issues: #1129 (program, epic &71, weight 5) + #965 (MR-A vehicle)
Branches: feature/1129-retention-sweeps (MR-A, Closes #965 + Relates to #1129); feature/1129-archive-engine (MR-B, Closes #1129)
Related: #1073 (⁂ money/policy readings — the DFCS retention schedule rides it); the #965 plan (MR-A’s detailed spec — settings, lock, batch loop, loop integration, its own test plan; archived at MR-A merge)
ADRs: ADR-058 (new, lands in MR-A), ADR-022 §Retention correction, ADR-003 additions

This program plan was reshaped by an external stop-ship review (2026-07-26; ~10 stop-ships
~15 high-severity findings). The Review-Disposition Index at the end maps every finding to its fix. The single largest correction: MR-A does not invent a new sweeper — it implements the pre-existing, adversarially-reviewed #965 plan, whose one BLOCKING open decision (D-A, compliance classification) was resolved by the user on 2026-07-26 and is recorded on both issues.

Context

The 2026-07-25 performance pass (epic &71) found every churn table unbounded; only idempotency_responses and jws_seen_jti have working cleanup. #965 already carries the corrected transport-sweep design — published-only outbox pruning, the corrected §D3.1 window invariant, the pinned-connection advisory lock, anomaly caps — and was blocked solely on the records-classification question the user resolved: transport tables are delivery/dedup artifacts, not records of the system of truth; the audit-class tables (audit_log, dead_letter_audit, rule_evaluations — the latter the SINGLE copy of every evaluation’s input/output since #1130) are records surfaces and are never hard-deleted while unarchived.

CRAIG is a child-welfare system of record: fair hearings, child-fatality reviews, and federal audits can demand the trail years later, and retention length is DFCS’s call, not a code constant. Deletes are irreversible; archives preserve every possible compliance answer. Hence the two classes.

The flow at a glance

  1. Transport sweep (inside the existing OutboxWorker loop — zero new tasks): hourly + jittered, after the drain; batched deletes under a pinned-connection advisory lock; outbox published-only at 30 d, inbox all-aged at 31 d (inbox > outbox + cadence + skew); cycle cap against clock jumps. Exchange/financial reuse the exposed batch helper for their two terminal-state tables.

  2. Archive pass (consent knob, default OFF everywhere including devstack): select rows older than the 90-day hot window → NDJSON + manifest to the object store with no-overwrite conditional puts → ONE transaction: id-keyed DELETE + the service’s LOCAL ledger row (+ staged event). Bounded per-pass budgets; poison rows quarantined, never wedging the pass.

  3. Ledgers: each archiving service owns a durable local ledger written in the prune transaction (security: archive_records; rules: new archive_ledger). The rules.evaluations_archived event to security is secondary fleet bookkeeping — validated, idempotent, divergence-detected. Orphan GC is defined ONLY against the local ledger.

  4. Purge (operator, bounded, attributed): every destructive statement independently requires retention_until IS NOT NULL AND retention_until ⇐ CURRENT_DATE AND NOT legal_hold — refusing everything until DFCS names a schedule.

Decisions

# Fork Choice Why

D1

Transport sweep home

OutboxWorker::prune_expired in the existing loop (the #965 design) + an exposed prune_table batch helper for the two service-local tables

Zero new tasks/wiring on 8 services; drain-before-prune ordering comes free.

D2

Outbox predicate

Published-only: published_at IS NOT NULL AND created_at < cutoff

#965 D-B: pending rows are the ONLY evidence of a lost cross-service notification (no outbox retry-cap/DLQ exists); published-only also never touches an active claim (claims exist only on unpublished rows). ADR-022’s "both" wording is corrected; the migration comments are sqlx-checksummed (immutable) and noted as superseded.

D3

Windows

Outbox 30 d / inbox 31 d, separate knobs; ServiceSettings::load() rejects inbox < outbox + 1 when both non-zero; 0 disables that table’s sweep

#965’s corrected §D3.1: equality fails at the boundary (two DBs prune asynchronously); inbox ≥ outbox + cadence + skew, and the 1-day margin dominates both.

D4

Replay boundary

admin_replay mechanically refuses ranges older than now() - outbox_window (typed 400 naming the window)

Replay currently has NO retention cutoff — the dedup row for an older replay may already be pruned → re-executed consumer side effects. Mechanical, not runbook.

D5

Single-pruner guard

pg_try_advisory_lock on a DETACHED connection whose drop closes the socket (releases the session lock on every exit path — the subsidy-import-finalize lease precedent; an async unlock cannot run in Drop)

The on-the-pool version leaks the lock onto an idle connection and retention silently self-disables.

D6

Anomaly guard

PRUNE_BATCH = 5_000, PRUNE_CYCLE_CAP = 1_000_000, yield_now + cancellation check between batches, stop-and-warn at cap

A clock jump / NTP glitch must not mass-delete; bounds WAL and shutdown latency.

D7

Archiver home

New crate crates/craig-retention (deps: craig-store, craig-validation, sqlx, sha2, serde_json, futures, uuid, chrono, thiserror, tokio, tokio-util, tracing — NO craig-mq: the event-staging closure lives in each service)

Two consumers; per-service copies bust the B8 budget; craig-store stays storage-only.

D8

Local ledgers

rules gets its OWN archive_ledger table written in the prune tx; the event to security is secondary bookkeeping

An async event can DLX or (post-MR-A) age out while pending — ledger absence must never imply orphan-hood. GC is local-ledger-only.

D9

Store hardening

Store::put_create (no overwrite) + Store::delete_if_exists (typed NotFound → Ok(false)); dedicated retention-archives/ prefix; bucket+endpoint identity recorded in manifest AND ledger; Local backend FORBIDDEN for the archiver (fail at boot, explicit or defaulted); versioning/object-lock/least-privilege/lifecycle + encrypted-bucket posture = documented deployment REQUIREMENTS (devstack’s shared credential noted as dev-only)

The archive must be immutable and cross-service durable; /tmp/craig-store inside one container is neither.

D10

Boot posture

When enabled: config load + backend check + a scoped write/read/delete probe (retention-archives/{service}/.probe) must pass or boot refuses, naming the env var + knob; rules compose gains depends_on: garage: service_healthy

Initialization only constructs a client today; consent means the operator asked for durable archives.

D11

Consent at every entry

The scheduled worker AND POST /archive/run both refuse when enabled=false (run → typed 403 naming the knob)

An endpoint bypassing the knob would prune into an unchecked store.

D12

Bounded everything

run_pass(budget) — endpoint budget 8 batches, scheduled budget 120/pass, round-robin across tables, cancellation between batches, more: bool (any table’s selection full OR byte-truncated); /purge bounded (limit ≤ 100/call, more), under the same lease, manifest deleted FIRST then data, purged_at after both

Unbounded HTTP drains die silently on client disconnect; purge needs the same discipline.

D13

Poison rows

A row alone exceeding max_batch_bytes is QUARANTINED: skipped with a loud WARN + retention_quarantined_total metric; the pass continues; per-table error isolation

A halt-on-oversize would wedge archival forever on one row.

D14

Purge predicate

Every destructive SELECT/UPDATE/DELETE independently requires retention_until IS NOT NULL AND retention_until ⇐ CURRENT_DATE AND NOT legal_hold; migration repairs stale purge_eligible and adds legal_hold BOOLEAN NOT NULL DEFAULT false; the e2e seed row (retention 2031) must NOT purge — test-pinned

The boolean alone can be stale (the seed proves it); legal holds model DFCS’s future per-record reality.

D15

Attribution

Manual /run + /purge record the operator (claims.sub) in the ledger (archived_by/purged_by) and stage a security event; scheduled passes record system:retention

Irreversible acts carry actors.

D16

Event trust

Security’s typed branch validates BEFORE any write: source_service == "craig-rules", table in a fixed vocabulary, key under the canonical prefix, sane counts/timestamps; ON CONFLICT (id) DO NOTHING + divergence detection (conflicting replay → loud WARN + metric, never overwrite); explicit parser arms for rules.evaluations_archived AND the redefined security.archive_completed

The wildcard subscriber consumes #; envelope fields are data, not transport identity.

D17

Addressability

Ledgers carry first_record_id/last_record_id (UUIDv7 = time-ordered) + first/last timestamps; cargo xtask archive-fetch <service> <table> (--id | --from --to) resolves ledger → object → row; runbook procedure; the rules.evaluated evaluation_id pointer doc + audit/rules list-API docs state the hot-window semantics

After 90 d an evaluation_id must remain resolvable — cold, not gone.

D18

Immutability precondition

All three tables verified insert-only (no UPDATE sites in-tree); ADR-058 records "archive-then-prune requires insert-only sources" as the onboarding precondition

Selection-then-delete archives a version; mutation in between would silently diverge.

D19

Watchdogs vs monitors

4 transport + 3 archive invariant specs, report-only, thresholds = default windows + grace, documented as devstack/CI drift tools; production observability = event_rows_pruned_total / archive_last_success_timestamp metrics

The invariant runner is a pre-push tool; hard-coded thresholds are config-blind by nature.

D20

Devstack posture

Archiver default OFF in compose; devstack tests construct engines directly (scratch DBs, per-test retention-archives/test-{uuid}/ prefixes); AC3 asserts non-vacuously — the harness drives run_pass itself, asserts objects + ledger rows, then runs the BLOCKING gate

Enabling in compose proves nothing (hourly tick) and would race shared-devstack tests destructively.

MR-A — transport sweeps

Branch feature/1129-retention-sweeps, Closes #965 + Relates to #1129. The detailed mechanics (settings shape, lock protocol, batch loop, loop integration, #965’s own test plan) live in the #965 plan; this section carries only the program-level deltas.

  • A1 — this commit: D-A recorded on both issues (notes 3603012745 / 3603012780), Plan::RETENTION labels, #965 ↔ #1129 ↔ #1073 relations, #965 onto epic &71; both plan files + nav.

  • A2 — #965 Steps 2–4 as specified there, plus: checked u32→i32 conversions for make_interval/LIMIT binds; window upper bound (≤ 3650) in load(); a defined PruneReport { outbox_deleted, inbox_deleted, capped }; BOTH explicit settings test initializers updated (crates/craig-common/src/settings.rs + crates/craig-api/src/bootstrap.rs).

  • A3 — prune_table(conn, &RetentionPolicy) exposed from craig-mq. Exchange gains a minimal lib target exposing retention_policies (binary-only today); policy status IN ('sent','failed') AND completed_at < cutoff. Financial’s policy (processed_at IS NOT NULL AND processed_at < cutoff; dead-letter rows untouched by construction) lives in its exposed lib modules. Both run tiny hourly sweeps with their own advisory keys. Admin-replay clamp per D4.

  • A4 — invariants outbox_retention_overrun (published-only, > 32 d), inbox_retention_overrun (> 33 d), send_jobs_retention_overrun, reconcile_settled_retention_overrun; grace constants
    grace_literals_match_constants mapping rows; red-harness on scratch DBs (the scratch schemas deliberately OMIT the outbound FKs — both point out of the swept tables and cannot affect the DELETE path; recorded here as the as-built deviation from the earlier FK-valid-parents wording).

  • A5 — ADR-058 (both classes; the archive protocol marked "Accepted — MR-B implements"); ADR-022 §Retention correction; outbox/inbox module docs; configuration-reference; data-model rows across all 8 service pages; CHANGELOG. Battery → merge → close #965 → its plan → Archive.

MR-A ships ZERO migrations (#965’s index deferral stands: post-retention the tables are bounded and the batched hourly scan is acceptable; a follow-up files if prune latency shows).

MR-B — archive engine

Branch feature/1129-archive-engine, Closes #1129.

  • B1 — crates/craig-retention (workspace member): lease.rs (pinned detached-connection AdvisoryLease + drop-guard, key retention-archive); spec.rs (ArchiveTableSpec; select returns (id, ts, to_jsonb(t)) so first/last timestamps are typed, not JSON re-parsed); manifest.rs (ArchiveManifest: format ndjson/v1, archive_id, service, table, record_count, data_key
    manifest_key + data_sha256, store endpoint + bucket, first/last record id + timestamp, hot_window_days, archived_at, full ColumnSpec schema {name, data_type, is_nullable, char_max_len, numeric_precision, default} + fingerprint over the full tuples); engine.rs (run_pass(budget), PassReport { per_table, more }, errors AlreadyRunning | Store | Db | Bookkeep(BoxError) | Config; batch = SELECT → NDJSON with byte-cap/quarantine → sha256 → put_create data then manifest → ONE tx: id-keyed DELETE with rows_affected mismatch rollback + the service’s bookkeep closure (BoxFuture<Result<(), BoxError>>) → commit); the scheduled worker (biased-select, jittered, budget 120, per-table isolation, archive_last_success_timestamp gauge); RetentionArchiveSettings { enabled: false, hot_window_days: 90 (1..=3650), interval_seconds: 3600, batch_size: 1000 (1..=10_000), max_batch_bytes: 32 MiB (validated ≤ the Store cap at boot) }. craig-store gains put_create + delete_if_exists. Unit + proptest suite.

    As-built deltas (2026-07-26): PassReport.more is deliberately FALSE for error-stopped tables (re-running into the same failure would loop an operator’s /run client; TableReport.error carries that signal) — a refinement of D12’s wording; max_batch_bytes gains a 4096-byte floor in load(); the D9 Local-backend refusal lives in boot_probe (NOT engine construction) so the engine stays hermetically testable against a tempdir store; spawn_archive_worker takes enabled explicitly and refuses an unconsented spawn (D11 defense-in-depth); craig-store additionally gains Store::is_local_backend + StoreError::AlreadyExists and re-exports StoreBackend; deps add bytes/serde/opentelemetry (otel twin) beyond D7’s list; the workspace serde_json gains float_roundtrip (archived NDJSON must parse back to the exact f64s written — a round-trip proptest caught the default parser’s ULP drift; regression seed committed).

  • B2 — craig-security: migration (SET statement_timeout = 0): archive_records DROP NOT NULL on retention_until, ADD data_key (no object-key column exists today)/manifest_key/data_sha256/store_bucket/ first_record_id/last_record_id/first_record_at/last_record_at/hot_window_days/legal_hold/purged_by, REPAIR stale purge_eligible, + dead_letter_audit (dlq_received_at) index. Model retention_until → Option<NaiveDate>; craig-web ArchiveView Option + template placeholder; utoipa. Engine wiring (specs audit_log + dead_letter_audit; bookkeep = ledger insert
    publish_archive_completed extended payload); D10 boot probe; detection-window boot WARN guard; api/archive.rs rework per D11/D12/D14/D15; existing archive tests + the direct-router fixture rewritten; parser arms + validate-then-write inbound branch per D16.

    As-built deltas (2026-07-26): the ledger ALSO carries store_endpoint (D9 names bucket+endpoint for the ledger; the unit list above omitted it); /purge is consent-gated too (D11 — an unconsented deployment cannot verify its store); the D14 predicate is re-checked per candidate IMMEDIATELY before the object deletes (the store deletes are destructive statements too), and a stamp refused after deletes escalates to error! naming bucket versioning as recovery; purge more = selection-full heuristic (an empty slice is the drain proof); D15 rides per-invocation engines (manual = claims.sub, scheduled = system:retention); a security.archive_purged event carries the purge actor; refused D16 envelopes WARN-and-skip (never an error — a forged event must not poison-loop the queue; the generic audit row still records it); handle_inbound_event is pub as the D16 test seam; the seeder’s placements archive row drops purge_eligible=true (the S4 stale state) with the SECURITY byte-identity hash re-blessed.

    J-review remediations (2026-07-26, in-commit): purge is scoped source_service = 'craig-security' at all THREE statements — the craig-rules rows in archive_records are fleet BOOKKEEPING copies whose objects live under rules' store identity and whose lifecycle rules' own archive_ledger drives (a fleet row’s survival is pinned in the purge matrix); the D16 divergence comparison is a pure fn (unit-killable) and includes manifest_key; the rules.evaluations_archived contract REQUIRES non-empty store_endpoint/store_bucket (D9 ledger identity — B3’s emitter carries them; security’s own event extended to match); archived_by_name stays NULL at ledger inserts (it is the display-name column). Accepted untested residues (reviewed, recorded): the migration’s one-shot REPAIR UPDATE (a no-op in every fixture DB that runs it), the stamp-refused-after-object-deletes branch (raceless injection impossible; 6 reviewed lines escalating to error!), and `boot_archiver’s probe→warn→spawn sequence beyond B1’s spawn-refusal tests.

  • B3 — craig-rules: archive_ledger migration (PK archive_id, the manifest ledger facts); archive_worker.rs (spec rule_evaluations; bookkeep = local ledger insert + stage rules.evaluations_archived); D10 boot probe; compose garage credentials + healthy dependency (archiver OFF in compose per D20); the evaluation_id pointer doc updated.

    As-built deltas (2026-07-26): the module is retention_archive.rs behind a NEW minimal lib target (the craig-exchange precedent — tests drive the production spec/bookkeep, never a copy); rules has NO manual /run surface (operators drive security’s; rules archives on its own schedule under system:retention); the store is constructed lazily in main ONLY when the archiver is enabled (nothing else in rules touches one); the staged event carries the D9 store identity per the B2-tightened contract (its required-field list is pinned from the emit side by the scratch suite, matching B2’s validator-side forgery matrix); the D17 id-range index is (first_record_id, last_record_id).

    J-review remediation (2026-07-26, in-commit; touches the B1 crate): the manifest’s id range is now MIN/MAX, not positional — batches page by the AGE column, and under write concurrency id order can diverge from age order within a batch, so a positional range could INVERT and silently match nothing in a D17 lookup (the review’s counterexample came from the suite’s own seeding). D17 lookups scan CANDIDATE ranges: normally exactly one; ranges can overlap at batch boundaries, never invert. The B3 atomicity test gained fault injection (outbox renamed away → staging fails inside the prune tx → rollback proves no-ledger-row-without-its-event structurally, not by counting).

  • B4 — CLI cmd/archive.rs rework (destructive semantics, confirmation, new shapes, disabled-403) + cli.adoc; archive invariants (97 d) + red-harness; the non-vacuous AC3 gate test.

    As-built deltas (2026-07-26): the CLI confirmation is a type-yes stdin gate skippable with --yes (the tests pass --yes and assert the D11 refusal surfaces as an error naming the knob); purge gains --limit (1..=100). The three 97-day watchdogs share ONE constant (AUDIT_HOT_WINDOW_OVERRUN_GRACE) and are red-harnessed against the LIVE spec files inside the existing B2/B3 suites rather than new files. AC3’s "gate test" collapses honestly into standing machinery: the artifact assertions (objects + manifests verify + ledger rows + hot rows gone) already run in the B2/B3 suites that drive run_pass directly, and the pre-push battery’s devstack phase runs cargo xtask invariants --gate with the new (report-only) specs in the catalog — no redundant subprocess test. Seed honesty fix (reshaped by the B4 J-review’s concrete-date arithmetic): seeded audit_log timestamps move from the 2024 range to within 35 days of the as-of month’s 1st. Frozen seed data ages against wall-clock, so NO deterministic seed stays green under a 97-day watchdog forever — the 35-day lookback gives a fresh devstack a ≥62-day green shelf life from the seed month’s 1st, after which red IS the staleness signal (reseed); the spec’s remediation text says exactly that. SECURITY byte-identity re-blessed (twice more today), other five hashes verified unchanged each time. Accepted residue: the CLI’s interactive type-yes path is untested (hardcoded stdin; fails CLOSED on EOF/garbage — an injectable reader is a nicety, not a gate).

  • B5 — deployment-guide == Retention & archive runbook (knobs, parser-first enable order, D9 storage requirements, archive-pull + xtask archive-fetch, purge posture, local-ledger-only orphan GC); ADR-058 as-built amendment; ADR-003; api-docs/implementation-guide; shared-crates (craig-retention + Store); data-model-security/-rules; CHANGELOG; ⁂ #1073 note; close #1129; epic &71 tick; this plan → Archive.

Test matrix

All tests @axis-tagged; devstack tests #[ignore = "requires devstack"]; scratch DBs + scratch store prefixes per D20.

# Test Kind axis

1

Outbox prune: aged published deleted; aged PENDING kept (D2); aged claimed-but-unpublished kept; fresh kept

scratch-DB

happy

2

Inbox 31-d window distinct from outbox 30-d (32-d row deleted, 30-d row kept)

scratch-DB

happy

3

Batching: > PRUNE_BATCH rows cleared across batches, exact counts; cycle cap stops + warns

scratch-DB

fault

4

Two concurrent pruners: one yields (advisory lock); lock released on early return (drop-guard)

scratch-DB

conc

5

Window 0 skips that table only; inbox < outbox+1 rejected in load(); > 3650 rejected

unit

sad

6

Replay clamp: in-window replays work; older range → typed 400

devstack

sad

7

Send-jobs/reconcile sweeps: terminal-aged deleted; pending/in-flight/dead-letter rows survive

scratch-DB

sad

8

NDJSON + manifest round-trip; sha256 vector; fingerprint flips on ANY column-attribute change

unit + proptest

happy

9

Byte-cap truncation → more: true; oversize single row → quarantined, pass continues, other tables unaffected

unit

evil

10

Archive round-trip: aged rows → put_create objects + manifest verifies + rows gone + LOCAL ledger row + event staged

devstack

happy

11

Crash-resume: puts ok, prune tx aborted → second pass re-archives (new id) + prunes

devstack

fault

12

Concurrent run_pass → AlreadyRunning; /run 409 mid-pass; /run 403 when disabled

devstack

conc

13

Rules event → security ledger row; identical replay → one row; CONFLICTING replay → row unchanged + divergence WARN; forged source/table/prefix → refused

devstack

evil

14

Purge: NULL retention → 0 purged (incl. the repaired stale-eligible seed row); dated+held → refused; dated+unheld → manifest-then-data deleted + purged_at/purged_by; bounded limit + more; re-run idempotent (split into sad + happy tests)

devstack

sad, happy

15

Boot: enabled + Local/probe-fail → refuses naming knob + env var; disabled → no worker, no probe

devstack

sad

16

Parser arms resolve both events (never the unknown fallback); cadence consts pinned

unit

happy

17

Red-harness: each of the 7 watchdogs fires on seeded aged rows; sweep/pass clears them

scratch-DB

sad

18

AC3: harness-driven pass, asserted artifacts, then cargo xtask invariants --gate green

gate step

—

Risks

# Risk Mitigation

1

First backlog drain on grown tables

PRUNE_BATCH + cycle cap + hourly cadence + yield/cancel between batches; the no-index deferral is #965’s recorded call — follow-up files if latency shows

2

Replica stampede

Pinned-connection advisory lock per service DB (D5); the helper sweeps use their own keys

3

Cross-service replay-past-dedup

D3 asymmetric windows + load() validation + D4 mechanical replay clamp

4

Archive PII in a shared dev bucket

Dedicated prefix + D9 documented production requirements; devstack caveat recorded; archiver OFF in compose

5

Poison row wedges archival

D13 quarantine + per-table isolation + metric; the hot-window watchdog names the leftover

6

Ledger/object divergence

Local ledger in the prune tx (D8); manifest-first purge deletes; divergence WARN on conflicting event replays (D16)

7

Migration timeout on grown tables

SET statement_timeout = 0 in MR-B migrations; pre-1.0 plain-DDL posture + the post-1.0 CONCURRENTLY note (the #1132 precedent)

Review-Disposition Index

Stop-ships S1–S10, high-severity H1–H15, completeness C1–C8 from the 2026-07-26 external review.

Finding Disposition

S1 second contradictory plan (#965)

MR-A implements #965 (Closes #965); D-A resolved by the user’s classification; plans reconciled in A1

S2 equal windows unsafe

D3 30/31 split + load() validation; D4 mechanical replay clamp

S3 deleting never-published events

D2 published-only (also excludes active claims by construction)

S4 purge ignores the posture at the destructive step

D14 predicate at every statement + repair migration + legal_hold + seed-row pin (test 14)

S5 /run bypasses consent

D11 typed 403 when disabled

S6 orphan GC vs unguaranteed rules ledger

D8 local archive_ledger in the prune tx; GC local-ledger-only

S7 Local storage insufficient

D9 Local forbidden for the archiver; bucket/endpoint recorded

S8 store not immutable

D9 put_create + dedicated prefix + documented versioning/lock/least-privilege/encryption requirements

S9 event = unchecked deletion capability

D16 validate-before-write + divergence detection; purge keys only from the LOCAL ledger

S10 addressability broken

D17 id ranges + xtask archive-fetch + runbook + pointer-doc updates

H1 BookkeepFn error type

BoxError accommodating StageError AND sqlx::Error; deps corrected (futures in, craig-mq out)

H2 workspace/test-literal edits unnamed

Named in A2/B1/B2 (both settings initializers, the authz_fallback router fixture, workspace members)

H3 purge Store wiring / NotFound contract

Engine purge method + typed delete_if_exists

H4 unbounded replica-hostile drain

D5 lock + D6 caps/yield/cancel

H5 interval-0 tight loop

Window-0 skips per table; archive interval-0 = worker not spawned (tests 5/15)

H6 scheduled worker undesigned

B1 scheduler spec (jitter, budget 120, isolation, gauge)

H7 unbounded purge

D12 bounded + lease + manifest-first + idempotent re-run

H8 oversize row wedges

D13 quarantine

H9 SQL types / validation

Checked i32 conversions + bounds + max_batch_bytes ≤ Store cap at boot

H10 index rollout blocks writes

MR-A ships no index (#965 deferral); MR-B’s two migrations carry timeout-0 + the CONCURRENTLY note

H11 dead_letter_audit scan index

B2 migration adds (dlq_received_at)

H12 boot doesn’t prove storage

D10 probe + compose healthy dependency

H13 detection windows vs hot window

B2 boot WARN guard

H14 checksum/key/marker semantics

data_key + manifest_key + data_sha256; manifest = object-completeness marker only; DB commit = prune truth (stated in ADR-058)

H15 schema contract + timestamp source

Select returns (id, ts, jsonb); full ColumnSpec attributes; error enum extended

C1 watchdogs ≠ monitors

D19 split: invariants = CI drift tools; metrics/gauges = production observability

C2 vacuous AC3

D20 harness-driven pass + asserted artifacts before the gate

C3 devstack races

D20 scratch DBs + scratch prefixes + compose OFF

C4 missing negative paths

Tests 1, 3, 4, 5, 9, 11, 12, 13, 14, 15

C5 red-harness placement

Scratch-DB seams; exchange minimal lib target (A3)

C6 CLI + docs consumers

B4 CLI rework + api-docs/implementation-guide + the composition data-model page

C7 MR-A cites unlanded ADR

ADR-058 lands in MR-A (A5)

C8 sweeper contract + pins

PruneReport/PassReport defined; string pins + scratch-DB behavior tests
Edit this page · latest