ADR-040: Federal-Export Data Collection and Emit Architecture

On this page

Status

Accepted 2026-06-21. Sets the architecture for the NCANDS Child File export program (epic &55, plan NCANDS Child File Federal Export). The AFCARS export is expected to follow the same shape when built. Distinct from ADR-032 (partner→federal-category routing), which is a different axis and is not used by case-level row emit.

Amended 2026-06-21 (D1, issue #651). During D1 implementation the original "app-layer decrypt of reports.children`" collection premise (Context fact 2 + Decision 2 below) was found to be unnecessary: the curated child demographics are plaintext structured columns on `persons (only ssn_last_four is encrypted), reachable via the non-null allegation.victim_id FK, and the report-level fields (reporter type, admin unit, receipt date) are plaintext on the referral. The collection is therefore a plaintext cross-entity assembly rooted on screened-in referrals — no decryption. Context fact 2 and Decision 2 are rewritten accordingly; the broader asymmetry (subject PII plaintext in the persons master index vs. encrypted at the intake edge) is captured as a separate posture review in issue #671, and the report-date-precision refinement (carrying the originating report’s received_at onto the referral) is issue #672.

Context

craig-reporting’s NCANDS/AFCARS generation is placeholder MVP: generate_ncands/generate_afcars write a submission row with record_count = 0; there is no per-row table and no cross-service data collection. A real federal case-level export must (a) obtain the source data, (b) map each domain value to the literal federal codebook code, and (c) emit a multi-row file.

Investigation of the real craig-cases model surfaced two structural facts that shape the architecture:

  1. The case-level source data is partly uncaptured. Child date-of-birth is not collected at intake (ChildEntry has no DOB → NCANDS age unsourceable); per-allegation disposition is never written (allegations.disposition always NULL → NCANDS maltreatment disposition unsourceable). Investigation-level disposition is captured.

  2. The curated child demographics are plaintext; only the raw intake snapshot is encrypted. The as-reported intake payload (reports.children, Plan B §D2) is application-layer-encrypted JSONB. But the curated person record — persons.gender/race/ethnicity/date_of_birth — is plaintext structured columns (only ssn_last_four is encrypted), and every allegation links to its victim via a non-null allegation.victim_id FK. So the authoritative demographics are reachable as a plaintext cross-entity assembly; decrypting the intake JSONB is unnecessary. (Original premise — corrected during D1; see the Status amendment + issue #671.)

Amended 2026-07-04 (#782, capture-complete reportability). ADR-040’s original "include every screened-in referral and leave incomplete rows flagged/unpopulated" rule (Decision 5, "leave-unpopulated fidelity policy"
the Consequences' "honestly partial pre-capture-completeness") was correct while per-allegation disposition was uncaptured (Context fact 1): every row was then a capture gap, so emitting them flagged-but-present was the honest partial file. Now that allegation- and investigation-level dispositions are captured, a report with no disposition is a pending case, not a capture gap — it is not yet reportable and must not appear in the annual Child File (ACF edit-checks reject a record with no disposition; a pending report is reported in the year it is disposed). The NCANDS mapper therefore now excludes (report, child) groups that carry no disposition, and validation_errors counts only genuine data-quality defects among reportable rows (e.g. >4 distinct maltreatment types). Consequently finalize_ncands_submission stamps validated only when the file has reportable records and zero validation errors, else validated_with_errors, and transmit_ncands blocks a zero-record or errored file. This supersedes the include-and-flag behavior for the "no disposition" case only; the leave-unpopulated policy for genuinely uncaptured coded fields (Decision 5) is unchanged.

CRAIG is pre-1.0 with no live federal submission, so correctness and not locking premature contracts matter more than speed.

Decision

  1. Capture-first sequencing. Add the missing data capture (intake child DOB; per-allegation disposition) before building the export. The export is the last phase, on real data — not the first. An export built on absent data would emit a mostly-empty file in exactly the fields that carry NCANDS value.

  2. Plaintext cross-entity assembly for collection (no decrypt). A purpose-built cases endpoint (GET /v1/cases/ncands-export?ffy=YYYY) assembles the export rows from the plaintext domain entities, rooted on screened-in referrals received in the FFY → their allegations + investigations → the victim persons record (via allegation.victim_id), returning a typed DTO. No decryption is required: the NCANDS fields are plaintext (demographics on persons; reporter type / admin unit / receipt date on the referral). Rooting on referrals (rather than reports) also captures referrals created directly, not only those converted from an intake report. Rejected: decrypting reports.children (unnecessary, and it reads the as-reported snapshot rather than the curated person record); a store-level SQL join across the encrypted JSONB; an event-sourced read model (premature for a yearly/periodic export). Report date is referrals.received_at, which carries the originating report’s receipt timestamp onto referrals converted from an intake report (#672) — so FFY bucketing uses the true report date, not the conversion instant. Directly-created referrals (no originating report) keep the column DEFAULT now().

  3. Per-row materialization with snapshot. generate writes one ncands_child_rows row per emitted child, materializing the jurisdiction/FFY-derived fields (#1 submission year, #2 state) into the row, so a later re-export is byte-stable even if the active bundle/jurisdiction changes. record_count and child_fatalities derive from the rows, never hardcoded.

  4. Literal federal codes as pure functions in craig-reference, returning Option. The enum→code mapping lives in craig-reference (mirroring the afcars codecs) as pure functions that return Option<u8> (or total u8). This keeps the verified, codebook-cited mapping in one tested place. The existing internal-mnemonic *_to_ncands functions (ADR-precedent #640) remain; the new functions are named for the federal variable (*_to_rptsrc, *_to_chmal) to avoid confusion.

  5. Leave-unpopulated fidelity policy. Where a CRAIG domain value has no faithful federal code (e.g. ReporterType::Mandated is a legal status, not an NCANDS source category; Disposition::Inconclusive has no RptDisp equivalent), the mapping returns None → a NULL column → an empty cell. CRAIG never emits a guessed/wrong federal code. The faithful fix (capturing a finer source/disposition at intake) is a tracked follow-up, not an inline guess. No domain enum is enriched as a side effect of the export.

    Update (#649): #34 (Maltreatment Death) is now sourced from a real per-allegation child_fatality field captured in craig-cases — a true fatality emits 1, while a non-fatal (default-false) allegation is still left unpopulated rather than asserted as "No" (false is a default, not a recorded determination). The remaining leave-unpopulated coded fields are #10 RptSrc / #11 RptDisp, tracked in #648.

  6. FFY semantics + stable surrogate export IDs. reporting_year is the Federal Fiscal Year (Oct 1 – Sep 30). Federal record/child identifiers in the emitted file are stable surrogate keys, never raw internal UUIDs — internal join keys are not leaked into an external file.

Consequences

  • The export program is correctly ordered: data-capture MRs (intake DOB, allegation disposition) precede the export pipeline. Investigation disposition needs no capture work (already live).

  • The collection path is an HTTP call from craig-reporting’s service identity to a cases endpoint that assembles the plaintext entities. D1 gates it with require_service_caller() + an explicit service_id == "craig-reporting" allowlist (mirroring the jws_replay service-to-service pattern), not the per-row ruleset engine. The explicit CasesClient wiring into craig-reporting’s router state remains a D2 task (the authz-boot http_client is not in router state today).

  • The emitted Child File is honestly partial pre-capture-completeness: unfaithful/uncaptured coded fields are empty, documented in the CHANGELOG, and tracked as follow-ups — not presented as full ACF compliance.

  • The exact ACF fixed-width submission layout, field-level edit-checks, ACF transmission, and the AFCARS file are out of scope here; this ADR’s collection/emit shape is intended to carry them later.

Edit this page · latest