Performance Testing Infrastructure
On this page
- Status
- Context
- Scope
- Design
- Steps
- Step 1: k6 Project Structure and Shared Helpers
- Step 2: Per-Service Scripts (craig-rules, craig-cases, craig-placement)
- Step 3: Per-Service Scripts (craig-exchange, craig-financial, craig-reporting)
- Step 4: Per-Service Scripts (craig-security, craig-intake, craig-web)
- Step 5: VU Ramp Profiles
- Step 6: Threshold Definitions and Assertions
- Step 7: cargo xtask perf Command
- Step 8: CI Integration
- Baseline Documentation Format
- Files Touched
- Verification
- Documentation Updates
Status
| Step | Description | Status |
|---|---|---|
1 |
k6 project structure and shared helpers |
Not started |
2 |
Per-service k6 test scripts (craig-rules, craig-cases, craig-placement) |
Not started |
3 |
Per-service k6 test scripts (craig-exchange, craig-financial, craig-reporting) |
Not started |
4 |
Per-service k6 test scripts (craig-security, craig-intake, craig-web) |
Not started |
5 |
VU ramp profiles (smoke, load, stress, soak) |
Not started |
6 |
Threshold definitions and assertions |
Not started |
7 |
cargo xtask perf command |
Not started |
8 |
CI integration (manual job on main) |
Not started |
Epic: &TBD
Issues: #TBD
Branch: feature/performance-testing
Context
CRAIG has 8 backend services and a BFF (craig-web) — all need performance characterization before production deployment. No load testing infrastructure currently exists. Without baseline metrics, regressions go undetected and capacity planning is guesswork.
k6 is chosen over alternatives (Locust, Gatling, JMeter) because it uses JavaScript for test scripts, runs as a single Go binary with no runtime dependencies, produces structured JSON output suitable for CI assertions, and has first-class support for threshold-based pass/fail.
Scope
In scope:
-
k6 test runner setup with shared utilities (auth token acquisition, base URL resolution)
-
Per-service k6 load test scripts targeting at least 3 critical endpoints each
-
Four VU ramp profiles: smoke (1 VU), load (50 VU), stress (100 VU), soak (10 VU for 30 minutes)
-
Threshold definitions with p95 latency targets per endpoint type
-
cargo xtask perfcommand to run tests locally -
CI integration as a manual pipeline job on
mainbranch -
Baseline documentation format for recording results
Out of scope:
-
Production monitoring and alerting (covered by OpenTelemetry plan)
-
Capacity planning and autoscaling configuration
-
Micro-benchmarks with
criterion(separate effort) -
tokio-consoleprofiling (separate tooling concern)
Design
Directory Structure
tests/k6/
helpers/
auth.js # Keycloak token acquisition
config.js # Base URL resolution, environment detection
thresholds.js # Shared threshold definitions
services/
rules.js # craig-rules load tests
cases.js # craig-cases load tests
placement.js # craig-placement load tests
exchange.js # craig-exchange load tests
financial.js # craig-financial load tests
reporting.js # craig-reporting load tests
security.js # craig-security load tests
intake.js # craig-intake load tests
web.js # craig-web (BFF) load tests
scenarios/
smoke.js # 1 VU, 1 iteration per endpoint
load.js # 50 VUs, 5 minute ramp-up, 10 minute sustained
stress.js # 100 VUs, 2 minute ramp-up, 5 minute sustained
soak.js # 10 VUs, 30 minute sustained
run-all.js # Orchestrator that imports all service scripts
baselines/
YYYY-MM-DD.json # Recorded baseline results
README.md # Baseline format documentation
Threshold Definitions
| Endpoint Type | p95 Target | Examples |
|---|---|---|
Read (GET single) |
<50ms |
GET /v1/cases/cases/{id}, GET /v1/rules/rulesets/{id} |
List (GET collection) |
<100ms |
GET /v1/cases/cases, GET /v1/placement/homes |
Write (POST/PUT) |
<200ms |
POST /v1/cases/referrals, PUT /v1/placement/homes/{id} |
Cross-service |
<500ms |
POST /v1/cases/investigations/{id}/safety-assessment (calls craig-rules) |
BFF page load |
<300ms |
GET /dashboard, GET /cases |
VU Ramp Profiles
| Profile | VUs | Duration | Purpose |
|---|---|---|---|
Smoke |
1 |
1 iteration per endpoint |
Sanity check — endpoints respond correctly under zero load |
Load |
50 |
5min ramp + 10min sustained + 2min ramp-down |
Expected production traffic simulation |
Stress |
100 |
2min ramp + 5min sustained + 1min ramp-down |
Peak traffic / breaking point discovery |
Soak |
10 |
30min sustained |
Memory leak and connection pool exhaustion detection |
Authentication Strategy
k6 tests acquire a Keycloak token once per VU init phase using ROPC grant (same as integration tests):
// tests/k6/helpers/auth.js
import http from 'k6/http';
export function getToken(username, password) {
const url = `${__ENV.OIDC_INTERNAL_URL}/realms/craig/protocol/openid-connect/token`;
const payload = {
grant_type: 'password',
client_id: 'craig-api',
username: username,
password: password,
};
const res = http.post(url, payload);
return JSON.parse(res.body).access_token;
}
Steps
Step 1: k6 Project Structure and Shared Helpers
Files: tests/k6/helpers/auth.js, tests/k6/helpers/config.js, tests/k6/helpers/thresholds.js
Create the directory structure. Implement shared helpers:
tests/k6/helpers/config.js — base URL resolution from environment variables with devstack defaults:
export const BASE_URLS = {
rules: __ENV.CRAIG_RULES_URL || 'http://localhost:8001',
cases: __ENV.CRAIG_CASES_URL || 'http://localhost:8002',
placement: __ENV.CRAIG_PLACEMENT_URL || 'http://localhost:8003',
exchange: __ENV.CRAIG_EXCHANGE_URL || 'http://localhost:8004',
financial: __ENV.CRAIG_FINANCIAL_URL || 'http://localhost:8005',
reporting: __ENV.CRAIG_REPORTING_URL || 'http://localhost:8006',
security: __ENV.CRAIG_SECURITY_URL || 'http://localhost:8007',
intake: __ENV.CRAIG_INTAKE_URL || 'http://localhost:8008',
web: __ENV.CRAIG_WEB_URL || 'http://localhost:8080',
};
export const OIDC_INTERNAL_URL = __ENV.OIDC_INTERNAL_URL || 'http://localhost:9090';
tests/k6/helpers/auth.js — Keycloak ROPC token acquisition (see Design section). Cache token per VU to avoid repeated auth calls during iterations.
tests/k6/helpers/thresholds.js — shared threshold definitions:
export const READ_THRESHOLDS = { http_req_duration: ['p(95)<50'] };
export const LIST_THRESHOLDS = { http_req_duration: ['p(95)<100'] };
export const WRITE_THRESHOLDS = { http_req_duration: ['p(95)<200'] };
export const CROSS_SERVICE_THRESHOLDS = { http_req_duration: ['p(95)<500'] };
export const BFF_THRESHOLDS = { http_req_duration: ['p(95)<300'] };
Step 2: Per-Service Scripts (craig-rules, craig-cases, craig-placement)
Files: tests/k6/services/rules.js, tests/k6/services/cases.js, tests/k6/services/placement.js
Each script exports a default function and named scenario functions. Minimum 3 critical endpoints per service:
tests/k6/services/rules.js:
-
GET /v1/rules/rulesets— list rulesets -
GET /v1/rules/rulesets/{id}— get specific ruleset -
POST /v1/rules/evaluate— evaluate a rule (the hot path)
tests/k6/services/cases.js:
-
GET /v1/cases/cases— list cases -
POST /v1/cases/referrals— create referral -
GET /v1/cases/cases/{id}— get case detail -
POST /v1/cases/investigations/{id}/safety-assessment— cross-service call to rules
tests/k6/services/placement.js:
-
GET /v1/placement/homes— list foster homes -
POST /v1/placement/homes— create foster home -
GET /v1/placement/placements— list placements
Each test function follows this pattern:
import http from 'k6/http';
import { check } from 'k6';
import { BASE_URLS } from '../helpers/config.js';
import { getToken } from '../helpers/auth.js';
let token;
export function setup() {
token = getToken('jane.doe', 'password');
return { token };
}
export default function(data) {
const headers = { Authorization: `Bearer ${data.token}`, 'Content-Type': 'application/json' };
// List
const listRes = http.get(`${BASE_URLS.cases}/v1/cases/cases`, { headers, tags: { endpoint: 'list_cases' } });
check(listRes, { 'list cases 200': (r) => r.status === 200 });
// Create
const createRes = http.post(`${BASE_URLS.cases}/v1/cases/referrals`, JSON.stringify({
reporter_name: 'k6 Test',
reporter_phone: '555-0100',
child_name: 'Test Child',
child_age: 5,
allegation_type: 'neglect',
narrative: 'k6 performance test referral',
source: 'hotline',
}), { headers, tags: { endpoint: 'create_referral' } });
check(createRes, { 'create referral 200': (r) => r.status === 200 });
}
Step 3: Per-Service Scripts (craig-exchange, craig-financial, craig-reporting)
Files: tests/k6/services/exchange.js, tests/k6/services/financial.js, tests/k6/services/reporting.js
tests/k6/services/exchange.js:
-
GET /v1/exchange/partners— list partners -
POST /v1/exchange/partners— create partner -
GET /v1/exchange/agreements— list agreements
tests/k6/services/financial.js:
-
GET /v1/financial/payment-requests— list payment requests -
POST /v1/financial/payment-requests— create payment request -
GET /v1/financial/claims— list claims
tests/k6/services/reporting.js:
-
GET /v1/reporting/afcars— list AFCARS submissions -
POST /v1/reporting/afcars— create AFCARS submission -
GET /v1/reporting/data-quality/checks— list data quality checks
Step 4: Per-Service Scripts (craig-security, craig-intake, craig-web)
Files: tests/k6/services/security.js, tests/k6/services/intake.js, tests/k6/services/web.js
tests/k6/services/security.js:
-
GET /v1/security/audit-logs— list audit logs -
GET /v1/security/admin-units— list admin units -
GET /v1/security/consent-records— list consent records
tests/k6/services/intake.js:
-
GET /v1/intake/reports— list public reports -
POST /v1/intake/reports— submit public report (unauthenticated) -
GET /v1/intake/reports/{id}— get report detail
tests/k6/services/web.js (BFF — HTML responses):
-
GET /dashboard— dashboard page load -
GET /cases— cases list page -
GET /placement— placement list page
| BFF tests require an authenticated session. Use the Keycloak code+PKCE flow or set a session cookie. If PKCE is complex for k6, test the underlying API calls instead and note the BFF overhead separately. |
Step 5: VU Ramp Profiles
Files: tests/k6/scenarios/smoke.js, tests/k6/scenarios/load.js, tests/k6/scenarios/stress.js, tests/k6/scenarios/soak.js
Each scenario file imports service functions and configures k6 options.scenarios:
// tests/k6/scenarios/load.js
import { cases } from '../services/cases.js';
import { rules } from '../services/rules.js';
export const options = {
scenarios: {
cases_load: {
executor: 'ramping-vus',
exec: 'cases',
startVUs: 0,
stages: [
{ duration: '5m', target: 50 }, // ramp up
{ duration: '10m', target: 50 }, // sustained
{ duration: '2m', target: 0 }, // ramp down
],
},
rules_load: {
executor: 'ramping-vus',
exec: 'rules',
startVUs: 0,
stages: [
{ duration: '5m', target: 50 },
{ duration: '10m', target: 50 },
{ duration: '2m', target: 0 },
],
},
},
thresholds: {
'http_req_duration{endpoint:list_cases}': ['p(95)<100'],
'http_req_duration{endpoint:create_referral}': ['p(95)<200'],
'http_req_duration{endpoint:evaluate_rule}': ['p(95)<500'],
},
};
Smoke profile uses shared-iterations executor with 1 VU and 1 iteration per endpoint.
Step 6: Threshold Definitions and Assertions
Files: tests/k6/helpers/thresholds.js, all scenario files
Define per-endpoint thresholds using k6 tags. Each service script tags requests with { tags: { endpoint: 'name' } }. Scenario files reference these tags in thresholds:
thresholds: {
// Global
http_req_failed: ['rate<0.01'], // <1% error rate
// Per-endpoint type
'http_req_duration{endpoint:list_cases}': ['p(95)<100'],
'http_req_duration{endpoint:get_case}': ['p(95)<50'],
'http_req_duration{endpoint:create_referral}': ['p(95)<200'],
'http_req_duration{endpoint:safety_assessment}': ['p(95)<500'],
},
k6 exits with non-zero status when thresholds are violated, which CI uses as a gate.
Step 7: cargo xtask perf Command
Files: xtask/src/main.rs
Add a perf subcommand to xtask:
// xtask/src/main.rs — add to command enum
Perf {
/// Test profile: smoke, load, stress, soak (default: smoke)
#[arg(long, default_value = "smoke")]
profile: String,
/// Specific service to test (default: all)
#[arg(long)]
service: Option<String>,
/// Save results to baselines directory
#[arg(long)]
save_baseline: bool,
},
Implementation:
-
Check that
k6is installed (runk6 version, error with install instructions if missing) -
Check that devstack is running (hit
http://localhost:8001/healthz) -
Build the k6 command:
k6 run tests/k6/scenarios/{profile}.js --out json=tests/k6/baselines/{date}.json -
If
--serviceis specified, run only that service script:k6 run tests/k6/services/{service}.js -
If
--save-baseline, copy JSON output totests/k6/baselines/with date-stamped filename -
Print summary table with pass/fail status per threshold
Step 8: CI Integration
Files: .gitlab-ci.yml
Add a manual CI job on main:
performance-smoke:
stage: test
image: grafana/k6:latest
when: manual
only:
- main
needs: ["build"]
services:
- name: docker:dind
variables:
OIDC_INTERNAL_URL: http://keycloak:9090
CRAIG_RULES_URL: http://craig-rules:8001
CRAIG_CASES_URL: http://craig-cases:8002
CRAIG_PLACEMENT_URL: http://craig-placement:8003
CRAIG_EXCHANGE_URL: http://craig-exchange:8004
CRAIG_FINANCIAL_URL: http://craig-financial:8005
CRAIG_REPORTING_URL: http://craig-reporting:8006
CRAIG_SECURITY_URL: http://craig-security:8007
CRAIG_INTAKE_URL: http://craig-intake:8008
CRAIG_WEB_URL: http://craig-web:8080
script:
- k6 run tests/k6/scenarios/smoke.js --summary-export=summary.json
artifacts:
paths:
- summary.json
when: always
expire_in: 30 days
| Full load/stress/soak profiles require the complete devstack running. The CI smoke test validates endpoint availability and basic response times. Full performance characterization should run against a dedicated environment. |
Baseline Documentation Format
Each baseline file (tests/k6/baselines/YYYY-MM-DD.json) is the raw k6 --summary-export JSON output. A companion tests/k6/baselines/README.md documents:
-
Date, commit hash, hardware specs
-
Profile used (smoke/load/stress/soak)
-
Summary table: endpoint, p50, p95, p99, error rate
-
Notable observations or regressions from previous baseline
Files Touched
| File | Change |
|---|---|
|
NEW: Keycloak ROPC token helper |
|
NEW: Base URL resolution |
|
NEW: Shared threshold constants |
|
NEW: craig-rules load tests |
|
NEW: craig-cases load tests |
|
NEW: craig-placement load tests |
|
NEW: craig-exchange load tests |
|
NEW: craig-financial load tests |
|
NEW: craig-reporting load tests |
|
NEW: craig-security load tests |
|
NEW: craig-intake load tests |
|
NEW: craig-web BFF load tests |
|
NEW: 1 VU smoke profile |
|
NEW: 50 VU load profile |
|
NEW: 100 VU stress profile |
|
NEW: 10 VU / 30min soak profile |
|
NEW: Orchestrator script |
|
Add |
|
Add manual |
Verification
-
k6 version— k6 CLI installed -
cargo xtask dev start— devstack running -
cargo xtask perf --profile smoke— smoke tests pass (all endpoints respond, thresholds met) -
cargo xtask perf --profile load— load tests complete, p95 latencies within thresholds -
cargo xtask perf --profile smoke --save-baseline— baseline JSON saved totests/k6/baselines/ -
Verify k6 exits non-zero when a threshold is violated (temporarily lower a threshold to test)
Documentation Updates
-
.claude/docs/testing.md— add performance testing section with xtask commands -
CHANGELOG.adoc— entry under== Unreleased -
docs/modules/ROOT/pages/developer-guide.adoc— reference k6 setup andcargo xtask perf