API Contract Tests

On this page

Status

Step Description Status

1

OpenAPI spec generation and snapshot storage

Not started

2

Response schema validation test harness

Not started

3

Integration with existing tests

Not started

4

CI breaking-change detection

Not started

Epic: &TBD
Issues: #TBD
Branch: feature/api-contract-tests

Context

CRAIG has 8 backend services exposing REST APIs with ~180 endpoints total.

utoipa Status (Verified)

utoipa v5 is a workspace dependency in the root Cargo.toml (with features chrono, uuid, decimal). utoipa-swagger-ui v9 is also a workspace dependency. Every service:

  • Derives #[derive(utoipa::OpenApi)] on an ApiDoc struct (e.g., services/craig-rules/src/api.rs line 21)

  • Annotates every handler with #[utoipa::path(…​)] including responses(…​) declarations

  • Derives utoipa::ToSchema on all request/response model types

  • Derives utoipa::IntoParams on query parameter structs

  • Serves the spec at /api-doc/openapi.json via the shared craig_api::build_app() which merges SwaggerUi::new("/swagger-ui").url("/api-doc/openapi.json", doc) (see crates/craig-api/src/lib.rs lines 80-103)

All 8 backend services (rules, cases, placement, exchange, financial, reporting, security, intake) follow this pattern. The OpenAPI specs are fully machine-generated and available from running services.

ServiceClient Response Type (Verified)

The test infrastructure in crates/craig-test-lib/src/client.rs exposes:

/// API response carrying HTTP status, optional deserialized body, and raw text.
#[derive(Debug)]
pub struct ApiResponse<T> {
    pub status: StatusCode,   // reqwest::StatusCode
    pub body: Option<T>,      // deserialized via serde_json::from_str (None on error/empty)
    pub raw: String,          // raw response body text — always available
}

The raw field contains the unparsed JSON string for every response. Tests access it like:

let resp = client.create_case(&body).await.unwrap();
assert_eq!(resp.status, StatusCode::OK);
let data = resp.body.unwrap();          // Option<serde_json::Value> unwrapped
assert!(data["id"].is_string());

ServiceClient methods: get<T>, get_with_query<T,Q>, post<T,B>, put<T,B>, delete (status only), delete_with_body<T>, post_multipart<T>, get_bytes, get_raw. All generic methods return Result<ApiResponse<T>>. In practice, typed clients (e.g., CasesClient) wrap ServiceClient and hardcode T = serde_json::Value.

The raw field is the key integration point: serde_json::from_str::<Value>(&resp.raw) gives the JSON value for schema validation without changing any existing API.

Typed Client Architecture (Verified)

Each service has a typed client wrapper (e.g., CasesClient in crates/craig-test-lib/src/clients/cases.rs):

pub struct CasesClient {
    inner: ServiceClient,
}

The TestHarness (in crates/craig-test-lib/src/harness.rs) creates these clients via factories like admin_cases_client(), caseworker_cases_client(), etc. Each factory gets a Keycloak token and constructs the typed client with a shared reqwest::Client.

There is no harness.cases field — tests call harness.caseworker_cases_client().await? each time.

Scope

In scope:

  • Export OpenAPI JSON spec per service as committed snapshot files

  • Validate integration test responses against the OpenAPI schema at runtime

  • Detect breaking changes by comparing generated specs against committed snapshots

  • CI job to enforce schema consistency

Out of scope:

  • Request validation (server-side input validation is handled by serde + manual checks)

  • Client SDK generation from OpenAPI specs

  • API versioning strategy (v2 endpoints)

  • Documentation formatting of OpenAPI specs

Design

OpenAPI Snapshot Files

Each service’s OpenAPI spec is committed as a JSON file:

api-specs/
  craig-rules.json
  craig-cases.json
  craig-placement.json
  craig-exchange.json
  craig-financial.json
  craig-reporting.json
  craig-security.json
  craig-intake.json

These files serve as the contract baseline. When the API changes, the developer regenerates the spec and commits the updated file. CI detects uncommitted changes as a failing check.

Spec Generation

Add a cargo xtask api-specs command that fetches each service’s /api-doc/openapi.json from a running devstack and saves the result. This validates the actual running service rather than building OpenAPI structs in a separate binary.

// xtask: cargo xtask api-specs
async fn generate_api_specs() -> Result<()> {
    let services = [
        ("craig-rules", "http://localhost:8001"),
        ("craig-cases", "http://localhost:8002"),
        ("craig-placement", "http://localhost:8003"),
        ("craig-exchange", "http://localhost:8004"),
        ("craig-financial", "http://localhost:8005"),
        ("craig-reporting", "http://localhost:8006"),
        ("craig-security", "http://localhost:8007"),
        ("craig-intake", "http://localhost:8008"),
    ];
    let client = reqwest::Client::new();
    for (name, base_url) in services {
        let url = format!("{base_url}/api-doc/openapi.json");
        let spec = client.get(&url).send().await?.text().await?;
        // Pretty-print the JSON for readable diffs
        let value: serde_json::Value = serde_json::from_str(&spec)?;
        let pretty = serde_json::to_string_pretty(&value)?;
        std::fs::write(format!("api-specs/{name}.json"), pretty)?;
        println!("  wrote api-specs/{name}.json");
    }
    Ok(())
}
The xtask does NOT need utoipa as a dependency. It just fetches JSON over HTTP from already-running services. The utoipa dependency stays only in the service crates where it generates the specs.

Response Schema Validation

Add a SchemaValidator to craig-test-lib that validates JSON response bodies against committed OpenAPI spec files. The implementation uses the jsonschema crate (v0.33, already a transitive dependency via zen-engine).

Complete SchemaValidator Implementation

// crates/craig-test-lib/src/schema.rs

use anyhow::{Context, Result, bail};
use serde_json::Value;
use std::path::PathBuf;
use std::sync::OnceLock;

/// Validates API response bodies against committed OpenAPI spec files.
pub struct SchemaValidator {
    spec: Value,
    service_name: String,
}

/// Workspace root detection: walks up from CARGO_MANIFEST_DIR to find
/// the directory containing `api-specs/`.
fn workspace_root() -> &'static PathBuf {
    static ROOT: OnceLock<PathBuf> = OnceLock::new();
    ROOT.get_or_init(|| {
        let mut dir = PathBuf::from(env!("CARGO_MANIFEST_DIR"));
        loop {
            if dir.join("api-specs").is_dir() {
                return dir;
            }
            if !dir.pop() {
                // Fallback: assume cwd
                return PathBuf::from(".");
            }
        }
    })
}

impl SchemaValidator {
    /// Load from committed spec file at `<workspace>/api-specs/<service_name>.json`.
    pub fn load(service_name: &str) -> Result<Self> {
        let path = workspace_root().join(format!("api-specs/{service_name}.json"));
        let content = std::fs::read_to_string(&path)
            .with_context(|| format!("failed to read spec file: {}", path.display()))?;
        let spec: Value = serde_json::from_str(&content)
            .with_context(|| format!("failed to parse spec file: {}", path.display()))?;
        Ok(Self {
            spec,
            service_name: service_name.to_string(),
        })
    }

    /// Validate a response body against the schema declared for the given
    /// operation (method + path template) and HTTP status code.
    ///
    /// `path_template` uses OpenAPI path syntax, e.g., "/v1/cases/cases/{id}".
    /// `method` is lowercase, e.g., "get", "post".
    pub fn validate(
        &self,
        method: &str,
        path_template: &str,
        status: u16,
        body: &Value,
    ) -> Result<()> {
        let schema = self.extract_response_schema(method, path_template, status)?;

        // Resolve $ref pointers to produce a self-contained schema.
        let resolved = self.resolve_refs(&schema)?;

        // Validate using jsonschema crate.
        let validator = jsonschema::validator_for(&resolved)
            .map_err(|e| anyhow::anyhow!("failed to compile schema: {e}"))?;

        let errors: Vec<String> = validator
            .iter_errors(body)
            .map(|e| format!("  - {e} (at {})", e.instance_path))
            .collect();

        if errors.is_empty() {
            Ok(())
        } else {
            bail!(
                "schema validation failed for {} {} {} on {}:\n{}",
                method.to_uppercase(),
                path_template,
                status,
                self.service_name,
                errors.join("\n")
            )
        }
    }

    /// Extract the JSON Schema for a specific response from the OpenAPI spec.
    fn extract_response_schema(
        &self,
        method: &str,
        path_template: &str,
        status: u16,
    ) -> Result<Value> {
        let paths = self.spec.get("paths")
            .context("spec missing 'paths'")?;

        let path_item = paths.get(path_template)
            .with_context(|| format!(
                "path '{}' not found in spec (available: {})",
                path_template,
                paths.as_object().map(|o| o.keys().cloned().collect::<Vec<_>>().join(", "))
                    .unwrap_or_default()
            ))?;

        let operation = path_item.get(&method.to_lowercase())
            .with_context(|| format!(
                "method '{}' not found for path '{}'",
                method, path_template
            ))?;

        let responses = operation.get("responses")
            .context("operation missing 'responses'")?;

        let status_str = status.to_string();
        let response = responses.get(&status_str)
            .with_context(|| format!(
                "status {} not found in responses for {} {}",
                status, method.to_uppercase(), path_template
            ))?;

        // Follow $ref on the response object itself (utoipa sometimes uses
        // response references).
        let response = if let Some(ref_path) = response.get("$ref") {
            self.resolve_ref(ref_path.as_str().context("$ref is not a string")?)?
        } else {
            response.clone()
        };

        let content = response.get("content")
            .with_context(|| format!(
                "response {} has no 'content' (may be empty body like 204)",
                status
            ))?;

        let media_type = content.get("application/json")
            .context("response content missing 'application/json'")?;

        media_type.get("schema")
            .cloned()
            .context("media type missing 'schema'")
    }

    /// Recursively resolve all `$ref` pointers in a schema against
    /// the spec's `components/schemas` section.
    fn resolve_refs(&self, schema: &Value) -> Result<Value> {
        match schema {
            Value::Object(map) => {
                // If this object is a $ref, resolve it and then resolve its contents.
                if let Some(ref_val) = map.get("$ref") {
                    let ref_str = ref_val.as_str().context("$ref is not a string")?;
                    let resolved = self.resolve_ref(ref_str)?;
                    return self.resolve_refs(&resolved);
                }

                // Otherwise, recursively resolve all values.
                let mut new_map = serde_json::Map::new();
                for (key, value) in map {
                    new_map.insert(key.clone(), self.resolve_refs(value)?);
                }
                Ok(Value::Object(new_map))
            }
            Value::Array(arr) => {
                let new_arr: Result<Vec<Value>> =
                    arr.iter().map(|v| self.resolve_refs(v)).collect();
                Ok(Value::Array(new_arr?))
            }
            other => Ok(other.clone()),
        }
    }

    /// Resolve a single `$ref` string like `#/components/schemas/RuleSet`
    /// against the spec.
    fn resolve_ref(&self, ref_path: &str) -> Result<Value> {
        let pointer = ref_path
            .strip_prefix('#')
            .with_context(|| format!("only local $ref supported, got: {ref_path}"))?;

        // Convert OpenAPI JSON Reference to JSON Pointer (/ separated).
        // The ref_path is already in the right format after stripping #.
        self.spec.pointer(pointer)
            .cloned()
            .with_context(|| format!("$ref '{}' not found in spec", ref_path))
    }
}

Cargo.toml Change for craig-test-lib

# In crates/craig-test-lib/Cargo.toml, add to [dependencies]:
jsonschema = "0.33"
jsonschema 0.33 is already in the workspace lockfile (transitive via zen-engine). Adding it as a direct dependency does not pull in new code, just pins the existing version.

Integration with Existing Tests

Schema validation is opt-in per test via a standalone SchemaValidator instance. The existing ServiceClient and typed clients (CasesClient, etc.) are NOT modified. This avoids changing the response type, method signatures, or adding state to ServiceClient.

Approach: Side-Channel Validation

Tests create a SchemaValidator alongside their client and validate the raw field from ApiResponse:

// In services/craig-cases/tests/api/cases.rs

use craig_test_lib::schema::SchemaValidator;

#[tokio::test]
async fn create_case() {
    if !devstack_available().await {
        return;
    }
    let harness = TestHarness::new().await.unwrap();
    let client = harness.caseworker_cases_client().await.unwrap();
    let validator = SchemaValidator::load("craig-cases").unwrap();

    let body = json!({
        "admin_unit": "Fulton",
        "assigned_worker": "bob.smith",
        "icwa_flag": false
    });
    let resp = client.create_case(&body).await.unwrap();
    assert_eq!(resp.status, StatusCode::OK);

    // Schema validation — parse raw response and validate against OpenAPI spec
    let raw_json: serde_json::Value = serde_json::from_str(&resp.raw).unwrap();
    validator.validate("post", "/v1/cases/cases", 200, &raw_json).unwrap();

    let data = resp.body.unwrap();
    assert!(data["id"].is_string());
    assert_eq!(data["status"], "open");
}

#[tokio::test]
async fn get_case() {
    if !devstack_available().await {
        return;
    }
    let harness = TestHarness::new().await.unwrap();
    let case_id = create_case_id(&harness).await;
    let client = harness.caseworker_cases_client().await.unwrap();
    let validator = SchemaValidator::load("craig-cases").unwrap();

    let resp = client.get_case(&case_id).await.unwrap();
    assert_eq!(resp.status, StatusCode::OK);

    // Schema validation
    let raw_json: serde_json::Value = serde_json::from_str(&resp.raw).unwrap();
    validator.validate("get", "/v1/cases/cases/{id}", 200, &raw_json).unwrap();
}

#[tokio::test]
async fn list_cases() {
    if !devstack_available().await {
        return;
    }
    let harness = TestHarness::new().await.unwrap();
    let client = harness.caseworker_cases_client().await.unwrap();
    let validator = SchemaValidator::load("craig-cases").unwrap();

    let resp = client
        .list_cases(&[("page", "1"), ("per_page", "10")])
        .await
        .unwrap();
    assert_eq!(resp.status, StatusCode::OK);

    // Schema validation — list endpoints return PageResponse<T>
    let raw_json: serde_json::Value = serde_json::from_str(&resp.raw).unwrap();
    validator.validate("get", "/v1/cases/cases", 200, &raw_json).unwrap();
}

Why NOT Modify ServiceClient

The plan originally proposed adding a SchemaValidator field to ServiceClient and auto-validating in get(), post(), etc. This is problematic because:

  1. ServiceClient methods are generic over T: DeserializeOwned — they don’t know the OpenAPI path template (only the concrete URL with IDs filled in)

  2. Auto-validation would need the path template (with {id} placeholders), which is only known at the call site

  3. Adding state to ServiceClient would require changing all typed client constructors and TestHarness factories

  4. Side-channel validation is simpler, explicit, and does not break any existing tests

Convenience Macro (Optional Follow-Up)

For less boilerplate, a macro can be added later:

/// Validate an ApiResponse against the OpenAPI schema.
/// Usage: assert_schema!(validator, "post", "/v1/cases/cases", 200, resp);
macro_rules! assert_schema {
    ($validator:expr, $method:expr, $path:expr, $status:expr, $resp:expr) => {
        let __raw: serde_json::Value = serde_json::from_str(&$resp.raw)
            .expect("response body is not valid JSON");
        $validator
            .validate($method, $path, $status, &__raw)
            .expect("schema validation failed");
    };
}

Steps

Step 1: OpenAPI Spec Generation and Snapshot Storage

Files: xtask/src/main.rs, api-specs/ directory (8 JSON files)

  1. Create api-specs/ directory at workspace root

  2. Add api-specs subcommand to xtask:

ApiSpecs {
    /// Regenerate all OpenAPI spec snapshots from running services
    #[arg(long)]
    check: bool,  // If true, fail if specs differ from committed versions
},
  1. Implement the generation logic (see Design section)

  2. Add --check mode that compares generated specs against committed files and exits non-zero on differences

  3. Run cargo xtask api-specs against the devstack to generate initial snapshots

  4. Commit the generated api-specs/*.json files

Step 2: Response Schema Validation Harness

Files: crates/craig-test-lib/src/schema.rs (NEW), crates/craig-test-lib/src/lib.rs, crates/craig-test-lib/Cargo.toml

  1. Add jsonschema = "0.33" to crates/craig-test-lib/Cargo.toml [dependencies]

  2. Create crates/craig-test-lib/src/schema.rs with the complete SchemaValidator implementation shown in the Design section (includes load(), validate(), extract_response_schema(), resolve_refs(), resolve_ref())

  3. Add pub mod schema; to crates/craig-test-lib/src/lib.rs

  4. Write unit tests for the validator using a minimal inline OpenAPI spec:

#[cfg(test)]
mod tests {
    use super::*;
    use serde_json::json;

    fn sample_spec() -> Value {
        json!({
            "openapi": "3.1.0",
            "info": {"title": "Test", "version": "0.1.0"},
            "paths": {
                "/v1/things": {
                    "get": {
                        "responses": {
                            "200": {
                                "description": "OK",
                                "content": {
                                    "application/json": {
                                        "schema": {
                                            "$ref": "#/components/schemas/Thing"
                                        }
                                    }
                                }
                            }
                        }
                    }
                }
            },
            "components": {
                "schemas": {
                    "Thing": {
                        "type": "object",
                        "required": ["id", "name"],
                        "properties": {
                            "id": {"type": "string", "format": "uuid"},
                            "name": {"type": "string"},
                            "count": {"type": "integer"}
                        }
                    }
                }
            }
        })
    }

    #[test]
    fn valid_response_passes() {
        let v = SchemaValidator {
            spec: sample_spec(),
            service_name: "test".into(),
        };
        let body = json!({"id": "550e8400-e29b-41d4-a716-446655440000", "name": "foo"});
        v.validate("get", "/v1/things", 200, &body).unwrap();
    }

    #[test]
    fn missing_required_field_fails() {
        let v = SchemaValidator {
            spec: sample_spec(),
            service_name: "test".into(),
        };
        let body = json!({"id": "550e8400-e29b-41d4-a716-446655440000"});
        assert!(v.validate("get", "/v1/things", 200, &body).is_err());
    }

    #[test]
    fn wrong_type_fails() {
        let v = SchemaValidator {
            spec: sample_spec(),
            service_name: "test".into(),
        };
        let body = json!({"id": "550e8400-e29b-41d4-a716-446655440000", "name": "foo", "count": "not_a_number"});
        assert!(v.validate("get", "/v1/things", 200, &body).is_err());
    }

    #[test]
    fn unknown_path_returns_error() {
        let v = SchemaValidator {
            spec: sample_spec(),
            service_name: "test".into(),
        };
        let body = json!({});
        assert!(v.validate("get", "/v1/nonexistent", 200, &body).is_err());
    }
}

Step 3: Integration with Existing Tests

Files: Selected test files in services/craig-cases/tests/, services/craig-placement/tests/

  1. Add use craig_test_lib::schema::SchemaValidator; import and let validator = SchemaValidator::load("craig-cases").unwrap(); to 2-3 existing test files as proof of concept

  2. Add validator.validate(…​) calls after existing status assertions (see Integration with Existing Tests in Design section for exact code)

  3. Validate both single-resource responses (GET by ID, POST create) and list responses (GET list with pagination)

  4. Verify existing tests still pass with validation added

  5. Do NOT modify ServiceClient or CasesClient — use side-channel validation only

Step 4: CI Breaking-Change Detection

Files: .gitlab-ci.yml, xtask/src/main.rs

  1. Add CI job that runs cargo xtask api-specs --check:

api-contract-check:
  stage: test
  script:
    - cargo xtask api-specs --check
  rules:
    - if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
  1. The --check mode:

    • Starts the devstack (or assumes it is running in CI)

    • Fetches each service’s /api-doc/openapi.json

    • Compares against committed api-specs/*.json

    • Fails with a diff if they don’t match

    • Developer workflow: run cargo xtask api-specs to update snapshots, commit the changes

  2. Add a note in the output when specs differ, explaining how to update:

ERROR: API spec for craig-cases has changed.
Run `cargo xtask api-specs` to regenerate and commit the updated specs.

Files Touched

File Change

api-specs/*.json (8 files)

NEW: OpenAPI spec snapshots

xtask/src/main.rs

Add api-specs subcommand

crates/craig-test-lib/src/schema.rs

NEW: Schema validation with $ref resolution (~130 lines)

crates/craig-test-lib/src/lib.rs

Add pub mod schema;

crates/craig-test-lib/Cargo.toml

Add jsonschema = "0.33"

services/craig-cases/tests/api/cases.rs

Add schema validation to 3 tests (proof of concept)

.gitlab-ci.yml

Add api-contract-check job

Verification

  1. cargo xtask dev start — devstack running

  2. cargo xtask api-specs — generates all 8 spec files without errors

  3. cargo xtask api-specs --check — passes (specs match committed versions)

  4. Modify a response type (e.g., add a field), run cargo xtask api-specs --check — fails with diff

  5. cargo nextest run --workspace — existing tests pass with schema validation enabled

  6. Verify schema validation catches an intentional type mismatch (change a field type, run tests)

Documentation Updates

  • .claude/docs/testing.md — add section on API contract testing

  • CHANGELOG.adoc — entry under == Unreleased

  • docs/modules/ROOT/pages/developer-guide.adoc — document cargo xtask api-specs command

Edit this page · latest