API Contract Tests
On this page
Status
| Step | Description | Status |
|---|---|---|
1 |
OpenAPI spec generation and snapshot storage |
Not started |
2 |
Response schema validation test harness |
Not started |
3 |
Integration with existing tests |
Not started |
4 |
CI breaking-change detection |
Not started |
Epic: &TBD
Issues: #TBD
Branch: feature/api-contract-tests
Context
CRAIG has 8 backend services exposing REST APIs with ~180 endpoints total.
utoipa Status (Verified)
utoipa v5 is a workspace dependency in the root Cargo.toml (with features chrono, uuid, decimal). utoipa-swagger-ui v9 is also a workspace dependency. Every service:
-
Derives
#[derive(utoipa::OpenApi)]on anApiDocstruct (e.g.,services/craig-rules/src/api.rsline 21) -
Annotates every handler with
#[utoipa::path(…)]includingresponses(…)declarations -
Derives
utoipa::ToSchemaon all request/response model types -
Derives
utoipa::IntoParamson query parameter structs -
Serves the spec at
/api-doc/openapi.jsonvia the sharedcraig_api::build_app()which mergesSwaggerUi::new("/swagger-ui").url("/api-doc/openapi.json", doc)(seecrates/craig-api/src/lib.rslines 80-103)
All 8 backend services (rules, cases, placement, exchange, financial, reporting, security, intake) follow this pattern. The OpenAPI specs are fully machine-generated and available from running services.
ServiceClient Response Type (Verified)
The test infrastructure in crates/craig-test-lib/src/client.rs exposes:
/// API response carrying HTTP status, optional deserialized body, and raw text.
#[derive(Debug)]
pub struct ApiResponse<T> {
pub status: StatusCode, // reqwest::StatusCode
pub body: Option<T>, // deserialized via serde_json::from_str (None on error/empty)
pub raw: String, // raw response body text — always available
}
The raw field contains the unparsed JSON string for every response. Tests access it like:
let resp = client.create_case(&body).await.unwrap();
assert_eq!(resp.status, StatusCode::OK);
let data = resp.body.unwrap(); // Option<serde_json::Value> unwrapped
assert!(data["id"].is_string());
ServiceClient methods: get<T>, get_with_query<T,Q>, post<T,B>, put<T,B>, delete (status only), delete_with_body<T>, post_multipart<T>, get_bytes, get_raw. All generic methods return Result<ApiResponse<T>>. In practice, typed clients (e.g., CasesClient) wrap ServiceClient and hardcode T = serde_json::Value.
The raw field is the key integration point: serde_json::from_str::<Value>(&resp.raw) gives the JSON value for schema validation without changing any existing API.
Typed Client Architecture (Verified)
Each service has a typed client wrapper (e.g., CasesClient in crates/craig-test-lib/src/clients/cases.rs):
pub struct CasesClient {
inner: ServiceClient,
}
The TestHarness (in crates/craig-test-lib/src/harness.rs) creates these clients via factories like admin_cases_client(), caseworker_cases_client(), etc. Each factory gets a Keycloak token and constructs the typed client with a shared reqwest::Client.
There is no harness.cases field — tests call harness.caseworker_cases_client().await? each time.
Scope
In scope:
-
Export OpenAPI JSON spec per service as committed snapshot files
-
Validate integration test responses against the OpenAPI schema at runtime
-
Detect breaking changes by comparing generated specs against committed snapshots
-
CI job to enforce schema consistency
Out of scope:
-
Request validation (server-side input validation is handled by serde + manual checks)
-
Client SDK generation from OpenAPI specs
-
API versioning strategy (v2 endpoints)
-
Documentation formatting of OpenAPI specs
Design
OpenAPI Snapshot Files
Each service’s OpenAPI spec is committed as a JSON file:
api-specs/
craig-rules.json
craig-cases.json
craig-placement.json
craig-exchange.json
craig-financial.json
craig-reporting.json
craig-security.json
craig-intake.json
These files serve as the contract baseline. When the API changes, the developer regenerates the spec and commits the updated file. CI detects uncommitted changes as a failing check.
Spec Generation
Add a cargo xtask api-specs command that fetches each service’s /api-doc/openapi.json from a running devstack and saves the result. This validates the actual running service rather than building OpenAPI structs in a separate binary.
// xtask: cargo xtask api-specs
async fn generate_api_specs() -> Result<()> {
let services = [
("craig-rules", "http://localhost:8001"),
("craig-cases", "http://localhost:8002"),
("craig-placement", "http://localhost:8003"),
("craig-exchange", "http://localhost:8004"),
("craig-financial", "http://localhost:8005"),
("craig-reporting", "http://localhost:8006"),
("craig-security", "http://localhost:8007"),
("craig-intake", "http://localhost:8008"),
];
let client = reqwest::Client::new();
for (name, base_url) in services {
let url = format!("{base_url}/api-doc/openapi.json");
let spec = client.get(&url).send().await?.text().await?;
// Pretty-print the JSON for readable diffs
let value: serde_json::Value = serde_json::from_str(&spec)?;
let pretty = serde_json::to_string_pretty(&value)?;
std::fs::write(format!("api-specs/{name}.json"), pretty)?;
println!(" wrote api-specs/{name}.json");
}
Ok(())
}
The xtask does NOT need utoipa as a dependency. It just fetches JSON over HTTP from already-running services. The utoipa dependency stays only in the service crates where it generates the specs.
|
Response Schema Validation
Add a SchemaValidator to craig-test-lib that validates JSON response bodies against committed OpenAPI spec files. The implementation uses the jsonschema crate (v0.33, already a transitive dependency via zen-engine).
Complete SchemaValidator Implementation
// crates/craig-test-lib/src/schema.rs
use anyhow::{Context, Result, bail};
use serde_json::Value;
use std::path::PathBuf;
use std::sync::OnceLock;
/// Validates API response bodies against committed OpenAPI spec files.
pub struct SchemaValidator {
spec: Value,
service_name: String,
}
/// Workspace root detection: walks up from CARGO_MANIFEST_DIR to find
/// the directory containing `api-specs/`.
fn workspace_root() -> &'static PathBuf {
static ROOT: OnceLock<PathBuf> = OnceLock::new();
ROOT.get_or_init(|| {
let mut dir = PathBuf::from(env!("CARGO_MANIFEST_DIR"));
loop {
if dir.join("api-specs").is_dir() {
return dir;
}
if !dir.pop() {
// Fallback: assume cwd
return PathBuf::from(".");
}
}
})
}
impl SchemaValidator {
/// Load from committed spec file at `<workspace>/api-specs/<service_name>.json`.
pub fn load(service_name: &str) -> Result<Self> {
let path = workspace_root().join(format!("api-specs/{service_name}.json"));
let content = std::fs::read_to_string(&path)
.with_context(|| format!("failed to read spec file: {}", path.display()))?;
let spec: Value = serde_json::from_str(&content)
.with_context(|| format!("failed to parse spec file: {}", path.display()))?;
Ok(Self {
spec,
service_name: service_name.to_string(),
})
}
/// Validate a response body against the schema declared for the given
/// operation (method + path template) and HTTP status code.
///
/// `path_template` uses OpenAPI path syntax, e.g., "/v1/cases/cases/{id}".
/// `method` is lowercase, e.g., "get", "post".
pub fn validate(
&self,
method: &str,
path_template: &str,
status: u16,
body: &Value,
) -> Result<()> {
let schema = self.extract_response_schema(method, path_template, status)?;
// Resolve $ref pointers to produce a self-contained schema.
let resolved = self.resolve_refs(&schema)?;
// Validate using jsonschema crate.
let validator = jsonschema::validator_for(&resolved)
.map_err(|e| anyhow::anyhow!("failed to compile schema: {e}"))?;
let errors: Vec<String> = validator
.iter_errors(body)
.map(|e| format!(" - {e} (at {})", e.instance_path))
.collect();
if errors.is_empty() {
Ok(())
} else {
bail!(
"schema validation failed for {} {} {} on {}:\n{}",
method.to_uppercase(),
path_template,
status,
self.service_name,
errors.join("\n")
)
}
}
/// Extract the JSON Schema for a specific response from the OpenAPI spec.
fn extract_response_schema(
&self,
method: &str,
path_template: &str,
status: u16,
) -> Result<Value> {
let paths = self.spec.get("paths")
.context("spec missing 'paths'")?;
let path_item = paths.get(path_template)
.with_context(|| format!(
"path '{}' not found in spec (available: {})",
path_template,
paths.as_object().map(|o| o.keys().cloned().collect::<Vec<_>>().join(", "))
.unwrap_or_default()
))?;
let operation = path_item.get(&method.to_lowercase())
.with_context(|| format!(
"method '{}' not found for path '{}'",
method, path_template
))?;
let responses = operation.get("responses")
.context("operation missing 'responses'")?;
let status_str = status.to_string();
let response = responses.get(&status_str)
.with_context(|| format!(
"status {} not found in responses for {} {}",
status, method.to_uppercase(), path_template
))?;
// Follow $ref on the response object itself (utoipa sometimes uses
// response references).
let response = if let Some(ref_path) = response.get("$ref") {
self.resolve_ref(ref_path.as_str().context("$ref is not a string")?)?
} else {
response.clone()
};
let content = response.get("content")
.with_context(|| format!(
"response {} has no 'content' (may be empty body like 204)",
status
))?;
let media_type = content.get("application/json")
.context("response content missing 'application/json'")?;
media_type.get("schema")
.cloned()
.context("media type missing 'schema'")
}
/// Recursively resolve all `$ref` pointers in a schema against
/// the spec's `components/schemas` section.
fn resolve_refs(&self, schema: &Value) -> Result<Value> {
match schema {
Value::Object(map) => {
// If this object is a $ref, resolve it and then resolve its contents.
if let Some(ref_val) = map.get("$ref") {
let ref_str = ref_val.as_str().context("$ref is not a string")?;
let resolved = self.resolve_ref(ref_str)?;
return self.resolve_refs(&resolved);
}
// Otherwise, recursively resolve all values.
let mut new_map = serde_json::Map::new();
for (key, value) in map {
new_map.insert(key.clone(), self.resolve_refs(value)?);
}
Ok(Value::Object(new_map))
}
Value::Array(arr) => {
let new_arr: Result<Vec<Value>> =
arr.iter().map(|v| self.resolve_refs(v)).collect();
Ok(Value::Array(new_arr?))
}
other => Ok(other.clone()),
}
}
/// Resolve a single `$ref` string like `#/components/schemas/RuleSet`
/// against the spec.
fn resolve_ref(&self, ref_path: &str) -> Result<Value> {
let pointer = ref_path
.strip_prefix('#')
.with_context(|| format!("only local $ref supported, got: {ref_path}"))?;
// Convert OpenAPI JSON Reference to JSON Pointer (/ separated).
// The ref_path is already in the right format after stripping #.
self.spec.pointer(pointer)
.cloned()
.with_context(|| format!("$ref '{}' not found in spec", ref_path))
}
}
Integration with Existing Tests
Schema validation is opt-in per test via a standalone SchemaValidator instance. The existing ServiceClient and typed clients (CasesClient, etc.) are NOT modified. This avoids changing the response type, method signatures, or adding state to ServiceClient.
Approach: Side-Channel Validation
Tests create a SchemaValidator alongside their client and validate the raw field from ApiResponse:
// In services/craig-cases/tests/api/cases.rs
use craig_test_lib::schema::SchemaValidator;
#[tokio::test]
async fn create_case() {
if !devstack_available().await {
return;
}
let harness = TestHarness::new().await.unwrap();
let client = harness.caseworker_cases_client().await.unwrap();
let validator = SchemaValidator::load("craig-cases").unwrap();
let body = json!({
"admin_unit": "Fulton",
"assigned_worker": "bob.smith",
"icwa_flag": false
});
let resp = client.create_case(&body).await.unwrap();
assert_eq!(resp.status, StatusCode::OK);
// Schema validation — parse raw response and validate against OpenAPI spec
let raw_json: serde_json::Value = serde_json::from_str(&resp.raw).unwrap();
validator.validate("post", "/v1/cases/cases", 200, &raw_json).unwrap();
let data = resp.body.unwrap();
assert!(data["id"].is_string());
assert_eq!(data["status"], "open");
}
#[tokio::test]
async fn get_case() {
if !devstack_available().await {
return;
}
let harness = TestHarness::new().await.unwrap();
let case_id = create_case_id(&harness).await;
let client = harness.caseworker_cases_client().await.unwrap();
let validator = SchemaValidator::load("craig-cases").unwrap();
let resp = client.get_case(&case_id).await.unwrap();
assert_eq!(resp.status, StatusCode::OK);
// Schema validation
let raw_json: serde_json::Value = serde_json::from_str(&resp.raw).unwrap();
validator.validate("get", "/v1/cases/cases/{id}", 200, &raw_json).unwrap();
}
#[tokio::test]
async fn list_cases() {
if !devstack_available().await {
return;
}
let harness = TestHarness::new().await.unwrap();
let client = harness.caseworker_cases_client().await.unwrap();
let validator = SchemaValidator::load("craig-cases").unwrap();
let resp = client
.list_cases(&[("page", "1"), ("per_page", "10")])
.await
.unwrap();
assert_eq!(resp.status, StatusCode::OK);
// Schema validation — list endpoints return PageResponse<T>
let raw_json: serde_json::Value = serde_json::from_str(&resp.raw).unwrap();
validator.validate("get", "/v1/cases/cases", 200, &raw_json).unwrap();
}
Why NOT Modify ServiceClient
The plan originally proposed adding a SchemaValidator field to ServiceClient and auto-validating in get(), post(), etc. This is problematic because:
-
ServiceClientmethods are generic overT: DeserializeOwned— they don’t know the OpenAPI path template (only the concrete URL with IDs filled in) -
Auto-validation would need the path template (with
{id}placeholders), which is only known at the call site -
Adding state to
ServiceClientwould require changing all typed client constructors andTestHarnessfactories -
Side-channel validation is simpler, explicit, and does not break any existing tests
Convenience Macro (Optional Follow-Up)
For less boilerplate, a macro can be added later:
/// Validate an ApiResponse against the OpenAPI schema.
/// Usage: assert_schema!(validator, "post", "/v1/cases/cases", 200, resp);
macro_rules! assert_schema {
($validator:expr, $method:expr, $path:expr, $status:expr, $resp:expr) => {
let __raw: serde_json::Value = serde_json::from_str(&$resp.raw)
.expect("response body is not valid JSON");
$validator
.validate($method, $path, $status, &__raw)
.expect("schema validation failed");
};
}
Steps
Step 1: OpenAPI Spec Generation and Snapshot Storage
Files: xtask/src/main.rs, api-specs/ directory (8 JSON files)
-
Create
api-specs/directory at workspace root -
Add
api-specssubcommand to xtask:
ApiSpecs {
/// Regenerate all OpenAPI spec snapshots from running services
#[arg(long)]
check: bool, // If true, fail if specs differ from committed versions
},
-
Implement the generation logic (see Design section)
-
Add
--checkmode that compares generated specs against committed files and exits non-zero on differences -
Run
cargo xtask api-specsagainst the devstack to generate initial snapshots -
Commit the generated
api-specs/*.jsonfiles
Step 2: Response Schema Validation Harness
Files: crates/craig-test-lib/src/schema.rs (NEW), crates/craig-test-lib/src/lib.rs, crates/craig-test-lib/Cargo.toml
-
Add
jsonschema = "0.33"tocrates/craig-test-lib/Cargo.toml[dependencies] -
Create
crates/craig-test-lib/src/schema.rswith the completeSchemaValidatorimplementation shown in the Design section (includesload(),validate(),extract_response_schema(),resolve_refs(),resolve_ref()) -
Add
pub mod schema;tocrates/craig-test-lib/src/lib.rs -
Write unit tests for the validator using a minimal inline OpenAPI spec:
#[cfg(test)]
mod tests {
use super::*;
use serde_json::json;
fn sample_spec() -> Value {
json!({
"openapi": "3.1.0",
"info": {"title": "Test", "version": "0.1.0"},
"paths": {
"/v1/things": {
"get": {
"responses": {
"200": {
"description": "OK",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/Thing"
}
}
}
}
}
}
}
},
"components": {
"schemas": {
"Thing": {
"type": "object",
"required": ["id", "name"],
"properties": {
"id": {"type": "string", "format": "uuid"},
"name": {"type": "string"},
"count": {"type": "integer"}
}
}
}
}
})
}
#[test]
fn valid_response_passes() {
let v = SchemaValidator {
spec: sample_spec(),
service_name: "test".into(),
};
let body = json!({"id": "550e8400-e29b-41d4-a716-446655440000", "name": "foo"});
v.validate("get", "/v1/things", 200, &body).unwrap();
}
#[test]
fn missing_required_field_fails() {
let v = SchemaValidator {
spec: sample_spec(),
service_name: "test".into(),
};
let body = json!({"id": "550e8400-e29b-41d4-a716-446655440000"});
assert!(v.validate("get", "/v1/things", 200, &body).is_err());
}
#[test]
fn wrong_type_fails() {
let v = SchemaValidator {
spec: sample_spec(),
service_name: "test".into(),
};
let body = json!({"id": "550e8400-e29b-41d4-a716-446655440000", "name": "foo", "count": "not_a_number"});
assert!(v.validate("get", "/v1/things", 200, &body).is_err());
}
#[test]
fn unknown_path_returns_error() {
let v = SchemaValidator {
spec: sample_spec(),
service_name: "test".into(),
};
let body = json!({});
assert!(v.validate("get", "/v1/nonexistent", 200, &body).is_err());
}
}
Step 3: Integration with Existing Tests
Files: Selected test files in services/craig-cases/tests/, services/craig-placement/tests/
-
Add
use craig_test_lib::schema::SchemaValidator;import andlet validator = SchemaValidator::load("craig-cases").unwrap();to 2-3 existing test files as proof of concept -
Add
validator.validate(…)calls after existing status assertions (see Integration with Existing Tests in Design section for exact code) -
Validate both single-resource responses (GET by ID, POST create) and list responses (GET list with pagination)
-
Verify existing tests still pass with validation added
-
Do NOT modify
ServiceClientorCasesClient— use side-channel validation only
Step 4: CI Breaking-Change Detection
Files: .gitlab-ci.yml, xtask/src/main.rs
-
Add CI job that runs
cargo xtask api-specs --check:
api-contract-check:
stage: test
script:
- cargo xtask api-specs --check
rules:
- if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
-
The
--checkmode:-
Starts the devstack (or assumes it is running in CI)
-
Fetches each service’s
/api-doc/openapi.json -
Compares against committed
api-specs/*.json -
Fails with a diff if they don’t match
-
Developer workflow: run
cargo xtask api-specsto update snapshots, commit the changes
-
-
Add a note in the output when specs differ, explaining how to update:
ERROR: API spec for craig-cases has changed.
Run `cargo xtask api-specs` to regenerate and commit the updated specs.
Files Touched
| File | Change |
|---|---|
|
NEW: OpenAPI spec snapshots |
|
Add |
|
NEW: Schema validation with $ref resolution (~130 lines) |
|
Add |
|
Add |
|
Add schema validation to 3 tests (proof of concept) |
|
Add api-contract-check job |
Verification
-
cargo xtask dev start— devstack running -
cargo xtask api-specs— generates all 8 spec files without errors -
cargo xtask api-specs --check— passes (specs match committed versions) -
Modify a response type (e.g., add a field), run
cargo xtask api-specs --check— fails with diff -
cargo nextest run --workspace— existing tests pass with schema validation enabled -
Verify schema validation catches an intentional type mismatch (change a field type, run tests)
Documentation Updates
-
.claude/docs/testing.md— add section on API contract testing -
CHANGELOG.adoc— entry under== Unreleased -
docs/modules/ROOT/pages/developer-guide.adoc— documentcargo xtask api-specscommand