Reading Reports

Markdown report

The primary human-readable output. Written to {output}/latest.md.

Summary table

Shows aggregate counts at a glance:

Metric	Count
Total Tests	12
Passed	10
Failed	2

Errored, skipped, and invalid rows only appear if their count is greater than zero.

Test results

Each failing test gets a detailed section:

### FAIL auth-check

Source: auth-middleware.md
Status: FAIL

Expectation:
The auth middleware should validate JWT tokens on protected routes.

Observed:
The middleware only checks for cookie-based sessions, no JWT validation exists.

Location:
src/middleware/auth.ts

Resolution:
Add JWT token validation logic to the auth middleware.

Passing tests are hidden by default. Use --include-passing to show them.

Validation issues

If validation finds problems, they appear at the bottom:

## Validation Issues

- **duplicate-id**: Duplicate test ID "auth-check" found in: auth.md, session.md
- **missing-id**: Missing LLM-extracted ID for test in config.md

JSON report

Machine-readable output for CI. Written to {output}/ci-results.json.

{
  "status": "fail",
  "summary": {
    "total": 12,
    "passed": 10,
    "failed": 2
  },
  "tests": [
    {
      "id": "auth-check",
      "sourceFile": "auth-middleware.md",
      "status": "fail",
      "location": "src/middleware/auth.ts"
    },
    {
      "id": "project-structure",
      "sourceFile": "project-structure.md",
      "status": "pass"
    }
  ]
}

Key fields:

Field	Type	Description
status	"pass" | "fail" | "error"	Overall run status
summary.total	number	Total test scenario count
tests[].id	string	LLM-extracted test ID
tests[].status	TestStatus	Per-test result
tests[].location	string?	Relevant file path (fail only)
tests[].error	string?	Error message (error only)

Terminal output

During execution

Live progress with spinner animation (TTY) or static lines (non-TTY):

Icon	Colour	Meaning
✔	Green	All scenarios in the file passed
✗	Red	At least one scenario failed
⚠	Yellow	At least one scenario errored
⟳	Yellow	Retrying after empty response

Final summary

Semantic tests completed

Report:    semantic-test-results/latest.md
CI Output: semantic-test-results/ci-results.json

Passed: 10
Failed: 2

FAILED

The final verdict is PASSED (green bold) or FAILED (red bold).