Skip to main content
Now that you have an agent working, you need to write tests to ensure that the quality of your answers doesn’t degrade as you add context or change the system instructions. Tests live in a .test.yml file next to the agent. Name it after the agent — sleep.agentic.test.yml tests sleep.agentic.yml — and list the questions you care about along with the answer you expect:
sleep.agentic.test.yml
You can add as many cases as you like. Each case runs runs times, and an LLM judge checks whether each answer is factually consistent with expected.
Only .agentic.yml agents can be tested, and only from .test.yml files. Inline tests: blocks in .agent.yml or .workflow.yml files are no longer supported.

​
Run your tests

Open the Tests page in the Oxygen IDE to see every test file in the project, run them, and review past runs. Oxygen executes each case the configured number of times and reports the results for each case. To run the same tests from a terminal or CI, use the CLI:

​
What the judge ignores

Oxygen is built for data analysis, so the judge focuses on factual correctness and ignores differences that don’t change the answer:
  • Rounding and precision — $1,081,396 vs $1,081,395.67
  • Wording — “Sales fell by 10%” vs “Sales decreased by 10%”
  • Formatting — date formats, number formatting, whitespace
A run fails when its answer materially disagrees with expected — a different number, the opposite trend, or a different conclusion.

​
Quality gates in CI

Fail the build when accuracy drops below a threshold:
The command exits with code 1 if the score is below the threshold. Add --format json for machine-readable output, or --tag critical to gate on a subset of cases. For every field and flag, see the Testing reference. At this point, you have a working agent as well as the ability to modify and test this agent. Congratulations!