Validation

Source file: src/validation/results.ts

Purpose

Validation is a quality gate that catches systemic issues individual test results can’t reveal. A test result might parse correctly and have a valid status, but still indicate a deeper problem: duplicate IDs across files suggest copy-paste issues in test specifications, missing IDs mean the LLM couldn’t identify distinct test scenarios (often because the spec wasn’t structured with clear headings), and invalid statuses mean the LLM decided a specification isn’t actually testable. These are suite-level concerns, not per-test concerns — they only become visible when you look at results across all test files together.

After all tests execute, validateResults() inspects the aggregated results for these quality issues.

Three checks

1. Duplicate IDs

Scans all TestRunResult entries for duplicate id values across different source files.

Duplicate test ID "auth-check" found in: auth-middleware.md, auth-session-flow.md

2. Missing IDs

Flags tests where the LLM didn’t extract a meaningful ID. Detected when:

• The id field is empty or missing
• The id matches the fallback pattern filename#N (e.g. auth-middleware.md#0)

Missing LLM-extracted ID for test in auth-middleware.md (using fallback "auth-middleware.md#0")

3. Invalid results

Flags tests where the LLM returned "status": "invalid", meaning it couldn’t interpret the test file as a testable specification.

Invalid test scenario "auth-middleware.md#0" in auth-middleware.md

ValidationResult

interface ValidationResult {
  valid: boolean;
  issues: ValidationIssue[];
}

interface ValidationIssue {
  type: "duplicate-id" | "missing-id" | "invalid";
  id?: string;
  files: string[];
  message: string;
}

CLI flag effects

Flag	Effect
--strict	Validation issues cause exit code 2 (error) instead of being informational
--skip-validation	Validation is skipped entirely — no checks run, no issues reported
(neither)	Validation runs, issues are printed as warnings, but don’t affect exit code

The strict and skipValidation fields can also be set in the config file. CLI flags take precedence over config values.