.test.yml files. Each file points at one agent
(its target), lists a set of cases — a prompt plus the answer you
expect — and says how many times to run each one. An LLM judge scores every
run against the expected answer.
Only
.agentic.yml agents can be tested. Inline tests: blocks in
.agent.yml or .workflow.yml files are no longer supported — .test.yml is
the only way to define tests.Test file format
sales.agentic.test.yml
Top-level fields
settings
cases
How evaluation works
For each case, Oxygen:- Runs the target agent
runstimes with the case’sprompt - Asks the judge model whether each answer is factually consistent with
expected— formatting and wording differences are ignored - Records a pass rate per case, and whether all runs reached the same verdict
Running tests
Run tests from the CLI, locally or in your repository’s CI:Accuracy thresholds
Use--min-accuracy to fail the command (exit code 1) when the score is below
a threshold — useful as a CI quality gate:
--threshold-mode controls how the threshold is applied: average (default)
checks the average score across files; all requires every file’s score to
meet it.
Machine-readable output
--format json prints the full results as JSON on stdout. --output-json
also writes the JSON to a file next to the test file
(sales.agentic.test.results.json, or test-results.json when running every
file).
CLI reference
FILE— path to a.test.ymlfile. Omit it to run every.test.ymlfile in the project.
CI example
Error handling
- Wrong file type —
oxy testrejects any file that doesn’t end in.test.yml, and any test whose target isn’t an.agentic.ymlagent. - No tests found — running
oxy testwith no file fails if the project has no.test.ymlfiles. - Threshold failures — the command exits with code 1 only when
--min-accuracyis set and not met. If no score was produced, it prints a warning and exits 0.