SSA Screening Operations Runbook
On this page
E3 (#1566, plan Plan::SSA): the operator surface for the SSA screening
pipeline’s health, retention, and UD10 legal holds. Companion pages:
SSA Digest Rotation (the fork-4 sweep) and the
deployment guide’s worker-heartbeat table (staleness thresholds). All signals
are PII-free: ids, counts, categories.
Signals overview
| Signal | Kind | Meaning |
|---|---|---|
|
gauge |
Age of the oldest |
|
gauges |
The H16 pause depths by categorical token ( |
|
counter |
Per-attempt failures by categorical class ( |
|
counter |
Jobs terminally failed at the retry cap (member + job + run finalized in one transaction; counted post-commit). |
|
histogram |
SOLQ partner round-trip wall time, |
|
gauge |
Rows past their retention window still present: |
|
gauge |
Runs under a UD10 legal hold (deliberate stops — excluded from the lag classes above). Same recorder as the lag gauge: absent while the sweep is disabled. |
|
gauge |
1 = the gate knob is ON and every live dispatch precondition holds; 0 =
enabled-but-unready (the probe logs the held token on transitions). ABSENT
while the gate is off. This is deliberately a metrics-plane signal, not
|
Threshold-alert interpretation
ssa_screening_oldest_pending_seconds > 360(3 × the 120 s claim lease)-
The typed worker is not draining. Check the
ssa-workerheartbeat (worker_last_success_timestamp_seconds{worker="ssa-worker"}) and the craig-exchange log forssa drain failed. ADisabledworker with the gate ON meansWORKER_POLL_SECONDS=0— the devstack posture leaking into a real deployment. ssa_screening_ready == 0(sustained)-
The gate is on but every dispatch will pause. The transition warn names the held token; remedy per token —
contract-pendingis Phase-P-remediable ONLY,partner-unavailable/dsa-not-activeare partner-row/DSA fixes,egress-refusedisEGRESS_ALLOWED_HOSTSconfig. After remedying, re-admit paused jobs via the #1469 resume endpoint (never automatic). ssa_retry_exhaustion_totalmoving-
Members are terminally failing at the cap. Inspect
ssa_screening_job_errors_total{class}for the dominant class; the affected members readfailedand their runs complete (a re-screen is a NEW run). ssa_retention_lag_rows{class="transport"}not draining-
Check the
ssa-transport-sweepheartbeat FIRST: a mismatch-aborted pass deliberately withholds its beat, so a persistent count-mismatch wedge (the log shows repeatedcount mismatch) alarms as heartbeat staleness — a transient hold race retries clean next cycle; persistence is a bug, file it. The report-only invariantssa_transport_retention_overrunis the devstack/CI twin, and it is WIDER than the sweep (it also catches window-0 postures and unsweepable family shapes the gauge cannot report when the sweep is off). ssa_retention_lag_rows{class="provenance"}growing with archive consent ON-
Check the
retention-archiveheartbeat andarchive_last_success_timestamp; quarantines surface onretention_quarantined_total. With consent OFF this gauge is the expected accumulation curve (the D11 posture).
UD10 legal holds
A hold stops EVERY retention path for one run’s whole family (members, jobs, transaction rows, handoffs) — the flag joins every destructive statement’s predicate AND its delete re-assert, so a hold landing mid-sweep rolls that batch back whole.
-
Set:
POST /v1/exchange/ssa-screening-runs/{run_id}/holdwith{"reason": "…"}. -
Clear:
DELETE /v1/exchange/ssa-screening-runs/{run_id}/holdwith{"reason": "…"}— the reason rides a JSON body on BOTH verbs (a deliberate deviation from the D2?reason=precedent: hold rationales can carry case narrative, and query strings land in access logs and proxies). -
Both are HUMAN-ADMIN-only (service identities and non-admin humans are refused), reason-required, idempotent (
changed: false= the no-op replay), and writes-gate-EXEMPT — a hold is settable even after the screening feature is disabled. -
Each genuine transition stages
exchange.screening_hold_set/_cleared(actor + reason) — the audit log is the hold’s narrative; the run row carries only the boolean.ssa_screening_held_runscarries the live count. -
Batch-granularity holds (SDX/BENDEX imports) land with the C1 import tables.
Retention knobs
| Knob | Default | Semantics |
|---|---|---|
|
90 |
The UD4 transport window — a CEILING, |
|
false |
The D11 consent knob for the provenance archiver (handoffs → members → runs,
FK-ordered). Enabling requires an S3-class store (the D10 boot probe refuses
Local) and the exchange floor |
Purge |
— |
No purge tooling exists — refused until DFCS names a records-retention schedule (#1480). Archived objects are held indefinitely. |
Archived rows stay addressable: cargo xtask archive-fetch craig-exchange
ssa_screening_runs --id <run-id> resolves the batch via the exchange
archive_ledger and emits the exact NDJSON bytes (D17). Cases-side pointers
(ssa_screening_outcomes.run_id, ive_determinations.screening_run_id)
resolve through the same path once their run is cold.