ADR-040: Federal-Export Data Collection and Emit Architecture
On this page
Status
Accepted 2026-06-21. Sets the architecture for the NCANDS Child File export program (epic &55, plan NCANDS Child File Federal Export). The AFCARS export is expected to follow the same shape when built. Distinct from ADR-032 (partner→federal-category routing), which is a different axis and is not used by case-level row emit.
Amended 2026-06-21 (D1, issue #651). During D1 implementation the original "app-layer decrypt of
reports.children`" collection premise (Context fact 2 + Decision 2 below) was found to be unnecessary: the
curated child demographics are plaintext structured columns on `persons (only ssn_last_four is encrypted),
reachable via the non-null allegation.victim_id FK, and the report-level fields (reporter type, admin unit,
receipt date) are plaintext on the referral. The collection is therefore a plaintext cross-entity assembly
rooted on screened-in referrals — no decryption. Context fact 2 and Decision 2 are rewritten accordingly; the
broader asymmetry (subject PII plaintext in the persons master index vs. encrypted at the intake edge) is
captured as a separate posture review in issue #671, and the report-date-precision refinement (carrying the
originating report’s received_at onto the referral) is issue #672.
Context
craig-reporting’s NCANDS/AFCARS generation is placeholder MVP: generate_ncands/generate_afcars write a
submission row with record_count = 0; there is no per-row table and no cross-service data collection. A
real federal case-level export must (a) obtain the source data, (b) map each domain value to the literal
federal codebook code, and (c) emit a multi-row file.
Investigation of the real craig-cases model surfaced two structural facts that shape the architecture:
-
The case-level source data is partly uncaptured. Child date-of-birth is not collected at intake (
ChildEntryhas no DOB → NCANDS age unsourceable); per-allegation disposition is never written (allegations.dispositionalways NULL → NCANDS maltreatment disposition unsourceable). Investigation-level disposition is captured. -
The curated child demographics are plaintext; only the raw intake snapshot is encrypted. The as-reported intake payload (
reports.children, Plan B §D2) is application-layer-encrypted JSONB. But the curated person record —persons.gender/race/ethnicity/date_of_birth— is plaintext structured columns (onlyssn_last_fouris encrypted), and every allegation links to its victim via a non-nullallegation.victim_idFK. So the authoritative demographics are reachable as a plaintext cross-entity assembly; decrypting the intake JSONB is unnecessary. (Original premise — corrected during D1; see the Status amendment + issue #671.)
Amended 2026-07-04 (#782, capture-complete reportability). ADR-040’s original "include every screened-in
referral and leave incomplete rows flagged/unpopulated" rule (Decision 5, "leave-unpopulated fidelity policy"
the Consequences' "honestly partial pre-capture-completeness") was correct while per-allegation disposition was
uncaptured (Context fact 1): every row was then a capture gap, so emitting them flagged-but-present was the
honest partial file. Now that allegation- and investigation-level dispositions are captured, a report with
no disposition is a pending case, not a capture gap — it is not yet reportable and must not appear in the
annual Child File (ACF edit-checks reject a record with no disposition; a pending report is reported in the year
it is disposed). The NCANDS mapper therefore now excludes (report, child) groups that carry no disposition,
and validation_errors counts only genuine data-quality defects among reportable rows (e.g. >4 distinct
maltreatment types). Consequently finalize_ncands_submission stamps validated only when the file has
reportable records and zero validation errors, else validated_with_errors, and transmit_ncands blocks a
zero-record or errored file. This supersedes the include-and-flag behavior for the "no disposition" case only;
the leave-unpopulated policy for genuinely uncaptured coded fields (Decision 5) is unchanged.
CRAIG is pre-1.0 with no live federal submission, so correctness and not locking premature contracts matter more than speed.
Decision
-
Capture-first sequencing. Add the missing data capture (intake child DOB; per-allegation disposition) before building the export. The export is the last phase, on real data — not the first. An export built on absent data would emit a mostly-empty file in exactly the fields that carry NCANDS value.
-
Plaintext cross-entity assembly for collection (no decrypt). A purpose-built cases endpoint (
GET /v1/cases/ncands-export?ffy=YYYY) assembles the export rows from the plaintext domain entities, rooted on screened-in referrals received in the FFY → their allegations + investigations → the victimpersonsrecord (viaallegation.victim_id), returning a typed DTO. No decryption is required: the NCANDS fields are plaintext (demographics onpersons; reporter type / admin unit / receipt date on the referral). Rooting on referrals (rather than reports) also captures referrals created directly, not only those converted from an intake report. Rejected: decryptingreports.children(unnecessary, and it reads the as-reported snapshot rather than the curated person record); a store-level SQL join across the encrypted JSONB; an event-sourced read model (premature for a yearly/periodic export). Report date isreferrals.received_at, which carries the originating report’s receipt timestamp onto referrals converted from an intake report (#672) — so FFY bucketing uses the true report date, not the conversion instant. Directly-created referrals (no originating report) keep the columnDEFAULT now(). -
Per-row materialization with snapshot.
generatewrites onencands_child_rowsrow per emitted child, materializing the jurisdiction/FFY-derived fields (#1 submission year, #2 state) into the row, so a later re-export is byte-stable even if the active bundle/jurisdiction changes.record_countandchild_fatalitiesderive from the rows, never hardcoded. -
Literal federal codes as pure functions in craig-reference, returning
Option. The enum→code mapping lives incraig-reference(mirroring theafcarscodecs) as pure functions that returnOption<u8>(or totalu8). This keeps the verified, codebook-cited mapping in one tested place. The existing internal-mnemonic*_to_ncandsfunctions (ADR-precedent #640) remain; the new functions are named for the federal variable (*_to_rptsrc,*_to_chmal) to avoid confusion. -
Leave-unpopulated fidelity policy. Where a CRAIG domain value has no faithful federal code (e.g.
ReporterType::Mandatedis a legal status, not an NCANDS source category;Disposition::Inconclusivehas noRptDispequivalent), the mapping returnsNone→ a NULL column → an empty cell. CRAIG never emits a guessed/wrong federal code. The faithful fix (capturing a finer source/disposition at intake) is a tracked follow-up, not an inline guess. No domain enum is enriched as a side effect of the export.Update (#649): #34 (Maltreatment Death) is now sourced from a real per-allegation
child_fatalityfield captured in craig-cases — a true fatality emits1, while a non-fatal (default-false) allegation is still left unpopulated rather than asserted as "No" (false is a default, not a recorded determination). The remaining leave-unpopulated coded fields are #10 RptSrc / #11 RptDisp, tracked in #648. -
FFY semantics + stable surrogate export IDs.
reporting_yearis the Federal Fiscal Year (Oct 1 – Sep 30). Federal record/child identifiers in the emitted file are stable surrogate keys, never raw internal UUIDs — internal join keys are not leaked into an external file.
Consequences
-
The export program is correctly ordered: data-capture MRs (intake DOB, allegation disposition) precede the export pipeline. Investigation disposition needs no capture work (already live).
-
The collection path is an HTTP call from craig-reporting’s service identity to a cases endpoint that assembles the plaintext entities. D1 gates it with
require_service_caller()+ an explicitservice_id == "craig-reporting"allowlist (mirroring thejws_replayservice-to-service pattern), not the per-row ruleset engine. The explicitCasesClientwiring into craig-reporting’s router state remains a D2 task (the authz-boothttp_clientis not in router state today). -
The emitted Child File is honestly partial pre-capture-completeness: unfaithful/uncaptured coded fields are empty, documented in the CHANGELOG, and tracked as follow-ups — not presented as full ACF compliance.
-
The exact ACF fixed-width submission layout, field-level edit-checks, ACF transmission, and the AFCARS file are out of scope here; this ADR’s collection/emit shape is intended to carry them later.