Object Storage Adoption for Placement & Reporting

On this page

Status

Step Description Status

1

Database migration for placement documents table

Done (pre-ADR-030)

2

craig-placement document upload/download/list/delete endpoints

Done (pre-ADR-030)

3

craig-reporting AFCARS export generation and download endpoints

Done (pre-ADR-030)

4

craig-store bucket initialization for new buckets

Done (Store init added to both services; local backend auto-creates dirs)

5

Seed sample files for testing

Not started

6

Integration tests for all new endpoints

Done (pre-ADR-030)

Epic: &TBD
Issues: #TBD
Branch: feature/object-storage-adoption

Context

craig-placement needs photo and document upload capabilities for foster home management — inspection photos, license documents, and training certificates. craig-reporting needs to generate and store AFCARS/NCANDS flat files for federal submission. The craig-store crate already provides a thin wrapper around the object_store crate (v0.13) with put/get/delete/list operations, upload validation, and filename sanitization. craig-cases already uses this pattern for contact attachments (services/craig-cases/src/api/contact_attachments.rs), so this plan extends the same approach to placement and reporting.

The craig-store::Store supports two backends: Local (filesystem, used in tests) and S3 (Garage in devstack, any S3-compatible in production). Configuration is via CRAIG_<SERVICE>STORE* env vars.

Scope

In scope:

  • Document upload/download/list/delete endpoints on craig-placement for foster home documents

  • AFCARS flat file export generation and download on craig-reporting

  • Database migration for a home_documents table on craig-placement

  • craig-store bucket initialization for placement-documents and reporting-exports paths

  • Seed sample files (test PDF, test image) for E2E testing

  • Integration tests for all new endpoints

Out of scope:

  • Virus scanning (future hardening task)

  • Thumbnail generation for images

  • Document versioning

  • Bulk download (ZIP archives)

Design

New API Endpoints

craig-placement

Method Path Description

POST

/v1/placement/homes/{home_id}/documents

Upload a document (multipart/form-data)

GET

/v1/placement/homes/{home_id}/documents

List documents for a foster home

GET

/v1/placement/homes/{home_id}/documents/{doc_id}/download

Download a specific document

DELETE

/v1/placement/homes/{home_id}/documents/{doc_id}

Soft-delete a document

craig-reporting

Method Path Description

POST

/v1/reporting/afcars/{submission_id}/export

Generate AFCARS flat file and store in object store

GET

/v1/reporting/afcars/{submission_id}/export/download

Download the generated flat file

Database Schema

New table on craig_placement database:

-- Migration: YYYYMMDDHHMMSS_add_home_documents.sql
CREATE TABLE home_documents (
    id          UUID PRIMARY KEY DEFAULT uuidv7(),
    home_id     UUID NOT NULL REFERENCES foster_homes(id),
    filename    VARCHAR(255) NOT NULL,
    content_type VARCHAR(127) NOT NULL,
    size_bytes  BIGINT NOT NULL,
    object_key  VARCHAR(512) NOT NULL,
    document_type VARCHAR(50) NOT NULL,  -- 'inspection_photo', 'license', 'training_certificate', 'other'
    uploaded_by UUID NOT NULL,
    description TEXT,
    active      BOOLEAN NOT NULL DEFAULT TRUE,
    created_at  TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    updated_at  TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

CREATE INDEX idx_home_documents_home_id ON home_documents(home_id) WHERE active = TRUE;

No new table needed for craig-reporting — the existing afcars_submissions table gains an export_object_key column:

-- Migration: YYYYMMDDHHMMSS_add_afcars_export_key.sql
ALTER TABLE afcars_submissions
    ADD COLUMN export_object_key VARCHAR(512),
    ADD COLUMN export_generated_at TIMESTAMPTZ;

Object Store Path Convention

Follow the pattern from craig-cases contact attachments:

  • Placement documents: placement-documents/{home_id}/{doc_id}/{sanitized_filename}

  • Reporting exports: reporting-exports/afcars/{submission_id}/{filename}

Upload Handling Pattern

Follow the existing pattern from services/craig-cases/src/api/contact_attachments.rs:

// Pattern: multipart extraction → validation → store → DB record
pub async fn upload_document(
    State(app): State<AppState>,
    Extension(claims): Extension<Claims>,
    Extension(object_store): Extension<Store>,
    Path(home_id): Path<Uuid>,
    mut multipart: axum::extract::Multipart,
) -> Result<Json<HomeDocument>, ApiError> {
    claims.require_caseworker_or_above()?;

    // Verify home exists
    // Extract multipart fields (file, document_type, description)
    // Validate upload (size, MIME type) using craig_store::validate_upload()
    // Sanitize filename using craig_store::sanitize_filename()
    // Build object_key: "placement-documents/{home_id}/{doc_id}/{filename}"
    // Put to object store
    // Insert DB record
    // Return HomeDocument
}

Download Pattern

pub async fn download_document(
    State(app): State<AppState>,
    Extension(claims): Extension<Claims>,
    Extension(object_store): Extension<Store>,
    Path((home_id, doc_id)): Path<(Uuid, Uuid)>,
) -> Result<impl IntoResponse, ApiError> {
    claims.require_caseworker_or_above()?;

    // Look up document record from DB
    // Verify it belongs to home_id
    // Get bytes from object store
    // Return with Content-Type and Content-Disposition headers
    let headers = [
        (header::CONTENT_TYPE, doc.content_type.clone()),
        (header::CONTENT_DISPOSITION, format!("attachment; filename=\"{}\"", doc.filename)),
    ];
    Ok((headers, Body::from(bytes)))
}

Steps

Step 1: Database Migration for Placement Documents

Files: services/craig-placement/migrations/YYYYMMDDHHMMSS_add_home_documents.sql

Create the home_documents table as specified in the Design section. Check existing migration timestamps to avoid collisions. Follow the pattern of other CRAIG migrations (UUID v7 default, active column, timestamps).

Step 2: craig-placement Document Endpoints

Files: services/craig-placement/src/api/home_documents.rs (NEW), services/craig-placement/src/api/mod.rs, services/craig-placement/src/store/home_documents.rs (NEW), services/craig-placement/src/store/models.rs, services/craig-placement/src/store/mod.rs, services/craig-placement/src/main.rs

Create the store layer:

services/craig-placement/src/store/models.rs — add HomeDocument struct:

#[derive(Debug, Clone, Serialize, Deserialize, sqlx::FromRow, utoipa::ToSchema)]
pub struct HomeDocument {
    pub id: Uuid,
    pub home_id: Uuid,
    pub filename: String,
    pub content_type: String,
    pub size_bytes: i64,
    pub object_key: String,
    pub document_type: String,
    pub uploaded_by: Uuid,
    pub description: Option<String>,
    pub active: bool,
    pub created_at: DateTime<Utc>,
    pub updated_at: DateTime<Utc>,
}

services/craig-placement/src/store/home_documents.rs — CRUD functions:

  • insert_document(pool, doc) → Result<HomeDocument>

  • list_documents(pool, home_id) → Result<Vec<HomeDocument>> (WHERE active = TRUE)

  • get_document(pool, doc_id) → Result<Option<HomeDocument>>

  • delete_document(pool, doc_id) → Result<()> (SET active = FALSE)

Create the API layer following the pattern in services/craig-cases/src/api/contact_attachments.rs:

  • upload_document — POST multipart handler with craig_store::validate_upload() and craig_store::sanitize_filename()

  • list_documents — GET returning Json<Vec<HomeDocument>>

  • download_document — GET returning binary stream with Content-Type/Content-Disposition

  • delete_document — DELETE soft-deleting DB record and removing from object store

Wire routes in services/craig-placement/src/api/mod.rs:

.route("/homes/:home_id/documents", post(home_documents::upload_document).get(home_documents::list_documents))
.route("/homes/:home_id/documents/:doc_id/download", get(home_documents::download_document))
.route("/homes/:home_id/documents/:doc_id", delete(home_documents::delete_document))

Add craig-store dependency to services/craig-placement/Cargo.toml. Initialize Store in main.rs and add as Extension.

Step 3: craig-reporting AFCARS Export Endpoints

Files: services/craig-reporting/migrations/YYYYMMDDHHMMSS_add_afcars_export_key.sql, services/craig-reporting/src/api/afcars.rs, services/craig-reporting/src/store/models.rs

Add the migration for export_object_key and export_generated_at columns.

Add two handlers to the existing AFCARS API module:

generate_export — POST /v1/reporting/afcars/{submission_id}/export:

  1. Verify submission exists and is in submitted state

  2. Generate AFCARS flat file content (tab-delimited text per federal spec)

  3. Store in object store at reporting-exports/afcars/{submission_id}/afcars-{date}.txt

  4. Update export_object_key and export_generated_at on the submission record

  5. Return updated submission

download_export — GET /v1/reporting/afcars/{submission_id}/export/download:

  1. Look up submission, verify export_object_key is set

  2. Get bytes from object store

  3. Return with Content-Type: text/plain and Content-Disposition: attachment

Add craig-store dependency to services/craig-reporting/Cargo.toml if not already present. Initialize Store in main.rs.

Step 4: craig-store Bucket Initialization

Files: crates/craig-store/src/config.rs, docker-compose.yml, devstack/garage/ (if bucket creation scripts exist)

Ensure the S3 buckets (or path prefixes for local backend) are initialized:

  • For local backend: Store::from_config() already calls create_dir_all — no changes needed

  • For S3 (Garage in devstack): add bucket creation for placement-documents and reporting-exports to the Garage init script (or use path-style access within an existing bucket)

Add store configuration to docker-compose.yml environment blocks for craig-placement and craig-reporting:

# craig-placement
CRAIG_PLACEMENT__STORE__BACKEND: s3
CRAIG_PLACEMENT__STORE__BUCKET: craig-placement
CRAIG_PLACEMENT__STORE__S3_ENDPOINT: http://garage:3900
CRAIG_PLACEMENT__STORE__S3_REGION: garage
CRAIG_PLACEMENT__STORE__S3_ACCESS_KEY: ${GARAGE_ACCESS_KEY}
CRAIG_PLACEMENT__STORE__S3_SECRET_KEY: ${GARAGE_SECRET_KEY}
CRAIG_PLACEMENT__STORE__MAX_UPLOAD_BYTES: 10485760

# craig-reporting (same pattern, different bucket)
CRAIG_REPORTING__STORE__BUCKET: craig-reporting

Step 5: Seed Sample Files

Files: tests/fixtures/sample.pdf, tests/fixtures/sample.png, xtask/src/seed.rs (or seed command)

Create minimal test fixtures:

  • tests/fixtures/sample.pdf — 1-page PDF with "Test Document" text (generate with a simple PDF library or commit a pre-built minimal file)

  • tests/fixtures/sample.png — 100x100 solid blue PNG (generate programmatically or commit a pre-built file)

Update the seed process (cargo xtask seed) to upload these fixtures:

  1. After SQL seeding completes, upload sample.pdf to the placement-documents path for a seeded foster home

  2. Upload sample.pdf to the reporting-exports path as a sample AFCARS export

  3. Insert corresponding DB records in home_documents and update export_object_key on a seeded AFCARS submission

Step 6: Integration Tests

Files: services/craig-placement/tests/home_documents.rs (NEW), services/craig-reporting/tests/afcars_export.rs (NEW)

Follow existing integration test patterns using TestHarness:

services/craig-placement/tests/home_documents.rs:

  1. Upload a document to a foster home (multipart POST)

  2. List documents — verify uploaded document appears

  3. Download document — verify content matches uploaded data

  4. Delete document — verify soft-deleted (GET still returns, LIST excludes)

  5. Upload with invalid MIME type — verify 400 error

  6. Upload exceeding size limit — verify 400 error

services/craig-reporting/tests/afcars_export.rs:

  1. Create AFCARS submission, generate export — verify 200 and export_object_key set

  2. Download export — verify file content is valid flat file format

  3. Download before generation — verify 404 or appropriate error

Each test file should have 4-6 test functions minimum. Use reqwest::multipart::Form and reqwest::multipart::Part for multipart uploads in tests (same pattern as services/craig-cases/tests/contact_attachments.rs).

Files Touched

File Change

services/craig-placement/migrations/YYYYMMDDHHMMSS_add_home_documents.sql

NEW: home_documents table

services/craig-placement/src/api/home_documents.rs

NEW: upload, list, download, delete handlers

services/craig-placement/src/api/mod.rs

Wire new routes

services/craig-placement/src/store/home_documents.rs

NEW: CRUD queries

services/craig-placement/src/store/models.rs

Add HomeDocument struct

services/craig-placement/src/store/mod.rs

Add home_documents module

services/craig-placement/src/main.rs

Initialize Store, add Extension

services/craig-placement/Cargo.toml

Add craig-store dependency

services/craig-reporting/migrations/YYYYMMDDHHMMSS_add_afcars_export_key.sql

NEW: add export columns

services/craig-reporting/src/api/afcars.rs

Add generate_export, download_export handlers

services/craig-reporting/src/store/models.rs

Update AfcarsSubmission model

services/craig-reporting/src/main.rs

Initialize Store, add Extension

services/craig-reporting/Cargo.toml

Add craig-store dependency (if needed)

docker-compose.yml

Add store config env vars for placement and reporting

tests/fixtures/sample.pdf

NEW: test PDF fixture

tests/fixtures/sample.png

NEW: test PNG fixture

xtask/src/seed.rs

Upload seed files to object store

services/craig-placement/tests/home_documents.rs

NEW: integration tests

services/craig-reporting/tests/afcars_export.rs

NEW: integration tests

Verification

  1. cargo nextest run --workspace --lib — unit tests pass

  2. cargo xtask dev reload — services rebuild with new migrations

  3. cargo nextest run -p craig-placement — placement integration tests pass (including new document tests)

  4. cargo nextest run -p craig-reporting — reporting integration tests pass (including new export tests)

  5. cargo xtask seed — seed files uploaded without errors

  6. Manual verification: upload a file via curl, download it, verify content matches

Documentation Updates

  • .claude/docs/services.md — add new endpoints to craig-placement and craig-reporting tables

  • CHANGELOG.adoc — entry under == Unreleased

  • docs/modules/ROOT/pages/data-model-placement.adoc — add home_documents table

  • docs/modules/ROOT/pages/data-model-reporting.adoc — note export columns on afcars_submissions

Edit this page · latest