SSA Screening Operations Runbook

On this page

E3 (#1566, plan Plan::SSA): the operator surface for the SSA screening pipeline’s health, retention, and UD10 legal holds. Companion pages: SSA Digest Rotation (the fork-4 sweep) and the deployment guide’s worker-heartbeat table (staleness thresholds). All signals are PII-free: ids, counts, categories.

Signals overview

Signal Kind Meaning

ssa_screening_oldest_pending_seconds

gauge

Age of the oldest pending/in_flight job — the pipeline-stall watchdog. Recorded every worker drain pass, explicit 0 when drained. paused rows are NOT counted here (they have their own pair below).

ssa_screening_paused_jobs{reason}
ssa_screening_oldest_paused_age_seconds

gauges

The H16 pause depths by categorical token (contract-pending, gate-disabled, partner-unavailable, dsa-not-active, egress-refused) and the oldest pause’s age. Re-admission is ONLY the #1469 resume endpoint.

ssa_screening_job_errors_total{class}

counter

Per-attempt failures by categorical class (custody-unavailable, outcome-unavailable, the partner-* classes, request-encoding, partner-malformed).

ssa_retry_exhaustion_total

counter

Jobs terminally failed at the retry cap (member + job + run finalized in one transaction; counted post-commit).

ssa_partner_request_duration_seconds{outcome}

histogram

SOLQ partner round-trip wall time, outcome ∈ {ok, error} (the error CLASS is the counter above — join on time, not labels).

ssa_retention_lag_rows{class}

gauge

Rows past their retention window still present: transport (should drain to ~0 within one sweep cycle; the detector is deliberately WIDER than the sweep’s eligibility, so unsweepable family shapes surface here too) and provenance (nonzero is EXPECTED while the consent-gated archiver is off). Recorded hourly by the transport sweep’s executed pass — with the sweep disabled (TRANSPORT_RETENTION_DAYS=0) this gauge AND ssa_screening_held_runs are ABSENT, not zero (the report-only overrun invariant is the remaining backstop; window 0 is a devstack/test posture, UD4-noncompliant in production, and refused outright while the archiver is consented).

ssa_screening_held_runs

gauge

Runs under a UD10 legal hold (deliberate stops — excluded from the lag classes above). Same recorder as the lag gauge: absent while the sweep is disabled.

ssa_screening_ready

gauge

1 = the gate knob is ON and every live dispatch precondition holds; 0 = enabled-but-unready (the probe logs the held token on transitions). ABSENT while the gate is off. This is deliberately a metrics-plane signal, not /readyz — a partner-side condition must never unroute the pod.

Threshold-alert interpretation

ssa_screening_oldest_pending_seconds > 360 (3 × the 120 s claim lease)

The typed worker is not draining. Check the ssa-worker heartbeat (worker_last_success_timestamp_seconds{worker="ssa-worker"}) and the craig-exchange log for ssa drain failed. A Disabled worker with the gate ON means WORKER_POLL_SECONDS=0 — the devstack posture leaking into a real deployment.

ssa_screening_ready == 0 (sustained)

The gate is on but every dispatch will pause. The transition warn names the held token; remedy per token — contract-pending is Phase-P-remediable ONLY, partner-unavailable/dsa-not-active are partner-row/DSA fixes, egress-refused is EGRESS_ALLOWED_HOSTS config. After remedying, re-admit paused jobs via the #1469 resume endpoint (never automatic).

ssa_retry_exhaustion_total moving

Members are terminally failing at the cap. Inspect ssa_screening_job_errors_total{class} for the dominant class; the affected members read failed and their runs complete (a re-screen is a NEW run).

ssa_retention_lag_rows{class="transport"} not draining

Check the ssa-transport-sweep heartbeat FIRST: a mismatch-aborted pass deliberately withholds its beat, so a persistent count-mismatch wedge (the log shows repeated count mismatch) alarms as heartbeat staleness — a transient hold race retries clean next cycle; persistence is a bug, file it. The report-only invariant ssa_transport_retention_overrun is the devstack/CI twin, and it is WIDER than the sweep (it also catches window-0 postures and unsweepable family shapes the gauge cannot report when the sweep is off).

ssa_retention_lag_rows{class="provenance"} growing with archive consent ON

Check the retention-archive heartbeat and archive_last_success_timestamp; quarantines surface on retention_quarantined_total. With consent OFF this gauge is the expected accumulation curve (the D11 posture).

A hold stops EVERY retention path for one run’s whole family (members, jobs, transaction rows, handoffs) — the flag joins every destructive statement’s predicate AND its delete re-assert, so a hold landing mid-sweep rolls that batch back whole.

  • Set: POST /v1/exchange/ssa-screening-runs/{run_id}/hold with {"reason": "…​"}.

  • Clear: DELETE /v1/exchange/ssa-screening-runs/{run_id}/hold with {"reason": "…​"} — the reason rides a JSON body on BOTH verbs (a deliberate deviation from the D2 ?reason= precedent: hold rationales can carry case narrative, and query strings land in access logs and proxies).

  • Both are HUMAN-ADMIN-only (service identities and non-admin humans are refused), reason-required, idempotent (changed: false = the no-op replay), and writes-gate-EXEMPT — a hold is settable even after the screening feature is disabled.

  • Each genuine transition stages exchange.screening_hold_set / _cleared (actor + reason) — the audit log is the hold’s narrative; the run row carries only the boolean. ssa_screening_held_runs carries the live count.

  • Batch-granularity holds (SDX/BENDEX imports) land with the C1 import tables.

Retention knobs

Knob Default Semantics

CRAIG_EXCHANGESSA_SCREENINGTRANSPORT_RETENTION_DAYS

90

The UD4 transport window — a CEILING, load()-validated 0..=90 (0 = the sweep registers Disabled; a devstack/test posture that is UD4-noncompliant in production, silences the lag/held gauges, and is REFUSED at boot while the archiver is consented). Terminal jobs + their SSA transaction rows are snapshotted into ssa_transport_handoffs and pruned in one transaction.

CRAIG_EXCHANGERETENTION_ARCHIVEENABLED

false

The D11 consent knob for the provenance archiver (handoffs → members → runs, FK-ordered). Enabling requires an S3-class store (the D10 boot probe refuses Local) and the exchange floor HOT_WINDOW_DAYS ≥ 365.

Purge

No purge tooling exists — refused until DFCS names a records-retention schedule (#1480). Archived objects are held indefinitely.

Archived rows stay addressable: cargo xtask archive-fetch craig-exchange ssa_screening_runs --id <run-id> resolves the batch via the exchange archive_ledger and emits the exact NDJSON bytes (D17). Cases-side pointers (ssa_screening_outcomes.run_id, ive_determinations.screening_run_id) resolve through the same path once their run is cold.

Edit this page · latest