Ruleset Auto-Discovery Integration Test

On this page

Status

  • Create .adoc plan file and link in nav.adoc

  • Create GitLab issue (#30)

  • Create feature branch feature/ruleset-auto-discovery

  • Implement services/craig-rules/tests/ruleset_discovery.rs

  • Update documentation (CHANGELOG.adoc, .claude/docs/testing.md)

  • Full test battery

  • Commit with Closes #30, push, open MR

Problem

Ruleset evaluation tests are per-jurisdiction (services/craig-rules/tests/georgia_ruleset_evaluation.rs, services/craig-rules/tests/texas_ruleset_evaluation.rs) with hardcoded test functions. Adding a new jurisdiction directory under rulesets/ does not automatically test it — a developer must manually create a corresponding test file.

Solution

Add a single auto-discovery integration test (services/craig-rules/tests/ruleset_discovery.rs) that scans rulesets/ at runtime, discovers all .json files, and validates each one can be parsed, compiled, structurally validated, and evaluated with a synthetically generated default input.

This is additive — it does not replace the existing per-jurisdiction tests, which test specific business logic with representative inputs.

Existing Pattern to Follow

Reference file: services/craig-rules/tests/georgia_ruleset_evaluation.rs (lines 1–28).

use serde_json::{Value, json};
use zen_engine::Decision;
use zen_engine::model::DecisionContent;

async fn evaluate_ruleset(name: &str, input: Value) -> Value {
    let path = format!(
        "{}/../../rulesets/georgia/{}.json",
        env!("CARGO_MANIFEST_DIR"),
        name,
    );
    let content =
        std::fs::read_to_string(&path).unwrap_or_else(|e| panic!("failed to read {path}: {e}"));
    let dc: DecisionContent =
        serde_json::from_str(&content).unwrap_or_else(|e| panic!("failed to parse {name}: {e}"));
    let decision: Decision = Decision::from(dc);
    let result = decision
        .evaluate((&input).into())
        .await
        .unwrap_or_else(|e| panic!("evaluation of {name} failed: {e}"));
    serde_json::to_value(&result.result).unwrap_or(Value::Null)
}

Key constraints from this pattern:

  • env!("CARGO_MANIFEST_DIR") resolves to services/craig-rules/ at compile time

  • Path to rulesets: {CARGO_MANIFEST_DIR}/../../rulesets/ (workspace root + rulesets/)

  • Decision::from(dc) compiles the ruleset (infallible, panics on bad input)

  • .evaluate&input).into( is async and returns !Send future — requires #[tokio::test(flavor = "current_thread")]

  • .result on the eval response is the output serde_json::Value

Ruleset JSON Structure

All rulesets follow the JDM (JSON Decision Model) format. Reference: rulesets/georgia/georgia-safety-assessment.json.

Top-level fields (all required)

{
  "name": "georgia-safety-assessment",
  "version": "v2.0",
  "description": "...",
  "nodes": [...],
  "edges": [...]
}

Node types (3)

  1. inputNode — single entry point, always "id": "input"

    { "id": "input", "name": "Input", "type": "inputNode", "position": { "x": 100, "y": 300 } }

  2. decisionTableNode — contains decision logic

    {
  "id": "present-danger",
  "name": "Present Danger Assessment",
  "type": "decisionTableNode",
  "content": {
    "hitPolicy": "first",
    "inputs": [
      { "id": "i_pd", "name": "Present Danger Identified", "field": "present_danger_identified" }
    ],
    "outputs": [
      { "id": "o_pd_active", "name": "Present Danger Active", "field": "present_danger_active" }
    ],
    "rules": [
      { "_description": "...", "i_pd": "== true", "o_pd_active": "true" },
      { "_description": "...", "i_pd": "", "o_pd_active": "false" }
    ]
  }
}

    • content.inputs[].field — the input field name the rule reads from

    • content.inputs[].id — the key used in content.rules[] to reference this input

    • content.outputs[].field — the output field name the rule writes to

    • content.rules[] — each rule maps input IDs to expressions and output IDs to values

    • Expression "" is a wildcard (matches any value)

    • hitPolicy: "first" — first matching rule wins, order matters

  3. outputNode — single exit point, always "id": "output"

    { "id": "output", "name": "Output", "type": "outputNode", "position": { "x": 1100, "y": 300 } }

Edges

Connect nodes into a DAG. Format:

{ "id": "e1", "sourceId": "input", "targetId": "present-danger" }

Intermediate fields

Some decision tables consume fields that are outputs of upstream tables. Example: safety-decision table has input field present_danger_active which is an output of the present-danger table. zen-engine evaluates upstream tables first (following the edge DAG) and overwrites intermediate fields before downstream tables read them. Providing default values for these in the synthetic input is harmless.

Implementation Detail

File: services/craig-rules/tests/ruleset_discovery.rs

Imports

use serde_json::Value;
use std::collections::HashMap;
use std::path::PathBuf;
use zen_engine::Decision;
use zen_engine::model::DecisionContent;

No additional dependencies needed — serde_json and zen_engine are already in services/craig-rules/Cargo.toml.

Function: discover_rulesets() → Vec<PathBuf>

  1. Build base path: PathBuf::from(env!("CARGO_MANIFEST_DIR")).join("../../rulesets")

  2. Canonicalize the path (resolves ..); panic if rulesets/ directory doesn’t exist

  3. Iterate subdirectories (each is a jurisdiction: georgia/, texas/, etc.)

  4. Within each subdirectory, collect all files with .json extension

  5. Sort the collected paths for deterministic test order

  6. Return the Vec<PathBuf>

  7. Error case: panic with descriptive message if rulesets/ dir is missing or unreadable

Function: validate_structure(parsed: &Value, name: &str) → Vec<String>

Accepts the parsed JSON Value and the ruleset filename (for error messages). Returns a Vec<String> of error descriptions (empty = valid).

Checks:

  1. parsed["nodes"] is an array — if not, return early with error

  2. Iterate nodes, collecting a HashMap<String, String> of node_id → node_type

  3. Track booleans: has_input_node, has_output_node, has_decision_table_node

  4. After iteration, error if any of the three booleans is false

  5. Iterate parsed["edges"] array — for each edge, check that sourceId and targetId exist in the node ID map; if not, record error with edge ID and missing node ID

  6. Error case: edges array missing or not an array — skip edge validation (not an error, zen-engine may accept it)

Function: infer_default_value(expressions: &[String]) → Value

Accepts all non-wildcard rule expressions observed for a single input field across all decision tables. Returns a serde_json::Value to use as default.

Algorithm (check expressions in order, return on first match):

  1. Skip empty strings (wildcards — no type info)

  2. If expression is "== true" or "== false" → return Value::Bool(false)

  3. If expression starts with " (literal double-quote char after serde deserialization, e.g. the JSON "\"SCREEN_OUT\"" deserializes to Rust string "SCREEN_OUT") → extract the inner string (trim surrounding " chars), return Value::String(inner)

  4. Strip leading operator chars (<, >, =, !) and whitespace. If remainder starts with an ASCII digit → return Value::Number(0)

  5. If remainder starts with an ASCII letter or _ (cross-field reference like > deadline_days) → return Value::Number(0)

  6. Fallback (all expressions were wildcards): return Value::Bool(false) — safe because wildcard rules match any value

Function: build_default_input(parsed: &Value) → Value

  1. Get parsed["nodes"] as array

  2. For each node where node["type"] == "decisionTableNode":

    1. Get node["content"]["inputs"] as array

    2. Get node["content"]["rules"] as array

    3. For each input definition in inputs:

      • Extract field (string — the actual input field name, e.g. "present_danger_identified")

      • Extract id (string — the rule key, e.g. "i_pd")

      • Skip if field is empty

      • For each rule in rules, look up rule[id] as string — collect into Vec<String>

    4. Store in HashMap<String, Vec<String>> mapping field → expressions

  3. Build a serde_json::Map by calling infer_default_value(&expressions) for each field

  4. Return Value::Object(map)

Test function: all_rulesets_parse_compile_and_evaluate

#[tokio::test(flavor = "current_thread")]
async fn all_rulesets_parse_compile_and_evaluate() {

Algorithm:

  1. Call discover_rulesets() — assert returned vec is non-empty

  2. Create failures: Vec<String>

  3. For each path in discovered rulesets:

    1. Extract name from filename stem (for error messages)

    2. Stage 1 — Read: std::fs::read_to_string(&path) — on error, push to failures, continue

    3. Stage 2 — JSON parse: serde_json::from_str::<Value>(&content) — on error, push to failures, continue

    4. Stage 3 — Structure: call validate_structure(&parsed, &name) — if errors returned, push all to failures, continue

    5. Stage 4 — DecisionContent parse: serde_json::from_str::<DecisionContent>(&content) — on error, push to failures, continue

    6. Stage 5 — Compile: Decision::from(dc) — this is infallible (panics internally on bad input, caught by test harness)

    7. Stage 6 — Build default input: call build_default_input(&parsed)

    8. Stage 7 — Evaluate: decision.evaluate&default_input).into(.await — on error, push to failures with the input JSON included for debugging, continue

    9. Stage 8 — Output check: convert result.result to Value; if null, push to failures with the input JSON included, continue

  4. Assert failures.is_empty() with a message listing failure count, total ruleset count, and all failure descriptions joined by newlines

Error message format

Each failure string follows the pattern:

[{ruleset-name}] Stage N ({stage-description}): {error-detail}

Example:

[georgia-safety-assessment] Stage 7 (evaluate): expression error in node 'present-danger'. Input was: {"present_danger_identified":false,...}

Including the default input in evaluation/output errors makes debugging straightforward — you can see exactly what was fed in.

Type Inference Reference

All expression patterns observed across the 10 current rulesets (5 Georgia + 5 Texas):

Pattern Regex Example (after serde deser) Default

Boolean

^== (true|false)$

== true

false

String literal

^".*"$

"SCREEN_OUT"

first literal (stripped of quotes)

Numeric compare

^[<>=!]+\s*\d

> 0, ⇐ 5, < 18

0

Cross-field ref

^[<>=!]+\s*[a-zA-Z_]

> deadline_days

0

Wildcard

^$

`` (empty)

skip (no type info)

Fallback when all expressions for a field are wildcards: false.

Documentation Updates

CHANGELOG.adoc

Add under == Unreleased → === Added:

* **Testing**: Auto-discovery integration test for rulesets — scans `rulesets/` at runtime, validates all jurisdictions compile and evaluate without manual per-jurisdiction test registration

.claude/docs/testing.md

Add a new subsection after the "nextest Profiles" section (after line 114):

## Ruleset Discovery Test

- File: `services/craig-rules/tests/ruleset_discovery.rs`
- Runs as part of `cargo nextest run --workspace` (no devstack needed — loads JSON from disk)
- Scans `rulesets/{jurisdiction}/*.json` at runtime
- Validates: JSON parse, structure (nodes/edges), zen-engine compilation, evaluation with synthetic default input
- Adding a new jurisdiction: just create `rulesets/{jurisdiction}/*.json` files — the discovery test picks them up automatically

.claude/docs/services.md

No change needed — this test is a standalone integration test file, not an API endpoint test. The test count in services.md under craig-rules tracks API integration tests, which remain unchanged.

Files Modified

File Action

docs/modules/ROOT/pages/plans/ruleset-auto-discovery-test.adoc

Created (this file)

docs/modules/ROOT/nav.adoc

Updated (plan link added)

services/craig-rules/tests/ruleset_discovery.rs

Create — the auto-discovery test

CHANGELOG.adoc

Add entry under == Unreleased → === Added

.claude/docs/testing.md

Add "Ruleset Discovery Test" subsection

GitLab

  • Issue: #30 (test: auto-discovery integration test for rulesets)

  • Branch: feature/ruleset-auto-discovery

  • Commit message must include Closes #30

Verification

Full test battery per CLAUDE.md:

  1. cargo fmt --all

  2. cargo clippy --workspace --locked

  3. cargo nextest run --workspace --lib (unit tests)

  4. cargo xtask dev reload (rebuild services)

  5. cargo nextest run --workspace (all tests including the new discovery test)

  6. cargo xtask e2e (E2E tests)

Edit this page · latest