Config

Source files: src/config/schema.ts, src/config/loader.ts

Configuration is how the tool knows what to do. It answers the fundamental questions: where are the test files? Which LLM should evaluate them? What capability level? Should validation issues be treated as errors?

The config system has two layers. At compile time, the defineConfig() helper gives users TypeScript autocompletion as they write their semtest.config.ts file. At runtime, Zod validates the loaded config object — because TypeScript types are erased when code runs, the config file could contain anything (wrong types, invalid runner names, missing fields), and Zod catches these problems immediately with clear error messages rather than letting them cause cryptic failures later in the pipeline.

Loading the config involves a second tool: jiti. Node.js can’t natively import .ts files, but users want to write config in TypeScript for the autocompletion. jiti is a runtime TypeScript loader that transpiles on-the-fly when the config is loaded — no separate build step needed.

Schema

The config schema is defined with Zod in schema.ts. All fields have defaults, so a minimal config only needs tests and llm:

Field	Type	Default	Description
tests	string	"semantic-tests/"	Directory containing test files
output	string	"semantic-test-results/"	Directory for generated reports
llm.runner	"claude" | "codex" | "gemini" | "opencode"	"claude"	Which LLM CLI to invoke
llm.capability	"high" | "balanced" | "fast"	"balanced"	Maps to a specific model per runner
extensions	string[]	undefined	Filter test files by extension (e.g. [".md"])
strict	boolean	false	Exit code 2 if validation issues found
skipValidation	boolean	false	Skip post-run validation entirely
debug	boolean	false	Log raw LLM output to debug files
timestamp	boolean	false	Generate timestamped report copies
includePassing	boolean	false	Include passing tests in MD report

Types

Two types are exported:

• SemtestConfig — The fully resolved config (z.infer). All fields are present and typed.
• SemtestUserConfig — The input type (z.input). Fields are optional where defaults exist.

defineConfig()

An identity function that provides type safety when writing config files:

import { defineConfig } from "@thulanek/semtest-runner";

export default defineConfig({
  tests: "semtests/",
  output: "semantic-test-results/",
  llm: {
    runner: "claude",
    capability: "fast",
  },
});

The function simply returns its argument — it exists purely for TypeScript autocompletion and validation.

Config loader

loadConfig() in loader.ts handles finding and parsing the config file:

1. Search — walks up from cwd looking for one of:
   • semtest.config.ts
   • semtest.config.js
   • semtest.config.mjs
2. Load — uses jiti to import the file (handles TypeScript natively, no pre-compilation needed)
3. Resolve — reads the default export (or the module itself if no default)
4. Validate — runs semtestConfigSchema.safeParse() and throws a formatted error on failure

If no config file is found, loadConfig() throws with a clear message.