.test.yml file next to the agent. Name it after the agent —
sleep.agentic.test.yml tests sleep.agentic.yml — and list the questions
you care about along with the answer you expect:
sleep.agentic.test.yml
runs times, and an LLM
judge checks whether each answer is factually consistent with expected.
Only
.agentic.yml agents can be tested, and only from .test.yml files.
Inline tests: blocks in .agent.yml or .workflow.yml files are no longer
supported.Run your tests
Tests run from the CLI, locally or in your repository’s CI. Oxygen executes each case the configured number of times and reports the results for each case:What the judge ignores
Oxygen is built for data analysis, so the judge focuses on factual correctness and ignores differences that don’t change the answer:- Rounding and precision —
$1,081,396vs$1,081,395.67 - Wording — “Sales fell by 10%” vs “Sales decreased by 10%”
- Formatting — date formats, number formatting, whitespace
expected — a different
number, the opposite trend, or a different conclusion.
Quality gates in CI
Fail the build when accuracy drops below a threshold:--format json for machine-readable output, or --tag critical to gate on a
subset of cases.
For every field and flag, see the Testing reference.
At this point, you have a working agent as well as the ability to modify and
test this agent. Congratulations!