Testing reference

Tests live in their own .test.yml files. Each file points at one agent (its target), lists a set of cases — a prompt plus the answer you expect — and says how many times to run each one. An LLM judge scores every run against the expected answer.

Only .agentic.yml agents can be tested. Inline tests: blocks in .agent.yml or .workflow.yml files are no longer supported — .test.yml is the only way to define tests.

Test file format

sales.agentic.test.yml

target: sales.agentic.yml   # optional — inferred from the filename

settings:
  concurrency: 5            # runs executed in parallel
  runs: 3                   # times to run each case
  judge_model: gpt-5-mini   # model that scores each run

cases:
  - name: signups-last-month
    prompt: "How many users signed up last month?"
    expected: "1,247 new user signups"
    tags: [users, critical]

  - prompt: "Show me top 5 customers by lifetime value"
    expected: |
      A ranked list of 5 customers ordered by total spend,
      including customer names and their LTV amounts.
    tags: [customers]

Top-level fields

Field	Type	Default	Description
`target`	string	inferred from filename	Path to the `.agentic.yml` agent under test. If omitted, `sales.agentic.test.yml` targets `sales.agentic.yml` in the same directory.
`name`	string	—	Optional display name for the test file.
`settings`	object	see below	Run settings.
`cases`	list	required	The test cases.

`settings`

Field	Type	Default	Description
`runs`	integer	`3`	Number of times each case is run. Consistency is measured as agreement across these runs.
`concurrency`	integer	`5`	How many runs execute in parallel.
`judge_model`	string	—	Model used as the LLM judge. Prefer a different model family than the agent’s to avoid self-preference bias.

`cases`

Field	Type	Default	Description
`prompt`	string	required	The question sent to the agent.
`expected`	string	required	The expected answer. The judge checks each run’s answer for factual consistency with it.
`name`	string	—	Stable label for the case, used by `--case` and shown in progress output instead of the prompt.
`tags`	list of strings	`[]`	Tags for filtering with `--tag`.
`tool`	string	—	Accepted by the schema, but tool-use assertions are not enforced yet — setting it has no effect on pass/fail.

How evaluation works

For each case, Oxygen:

Runs the target agent runs times with the case’s prompt
Asks the judge model whether each answer is factually consistent with expected — formatting and wording differences are ignored
Records a pass rate per case, and whether all runs reached the same verdict

Across the file, the score is the overall pass rate over every case and run, and consistency is the fraction of cases whose runs all agreed.

Running tests

Run tests from the Tests page in the Oxygen IDE, which lists every test file in the project along with past runs, or from the CLI:

# Run every .test.yml file in the project
oxy test

# Run one file
oxy test sales.agentic.test.yml

# Only cases tagged "critical"
oxy test --tag critical

# One case, by 0-based index, name, or exact prompt
oxy test sales.agentic.test.yml --case signups-last-month

Accuracy thresholds

Use --min-accuracy to fail the command (exit code 1) when the score is below a threshold — useful as a CI quality gate:

oxy test --min-accuracy 0.8

Each test file produces one score. When several files run, --threshold-mode controls how the threshold is applied: average (default) checks the average score across files; all requires every file’s score to meet it.

oxy test --min-accuracy 0.8 --threshold-mode all

Machine-readable output

--format json prints the full results as JSON on stdout. --output-json also writes the JSON to a file next to the test file (sales.agentic.test.results.json, or test-results.json when running every file).

oxy test sales.agentic.test.yml --format json --quiet

CLI reference

oxy test [FILE] [OPTIONS]

FILE — path to a .test.yml file. Omit it to run every .test.yml file in the project.

Flag	Short	Description	Default
`--tag <tag>`		Only run cases with this tag	—
`--case <case>`		Only run one case, by 0-based index, `name`, or exact prompt. Requires `FILE`	—
`--format <format>`		Output format: `pretty` or `json`	`pretty`
`--output-json`		Also write full JSON results to a file	`false`
`--min-accuracy <threshold>`		Minimum score (0.0–1.0). Exit code 1 if below	—
`--threshold-mode <mode>`		`average` or `all`	`average`
`--quiet`	`-q`	Show only the results summary	`false`
`--verbose`	`-v`	Show agent steps, actual output, and judge reasoning	`false`

CI example

- name: Run agent tests
  run: oxy test --min-accuracy 0.8 --quiet

Error handling

Wrong file type — oxy test rejects any file that doesn’t end in .test.yml, and any test whose target isn’t an .agentic.yml agent.
No tests found — running oxy test with no file fails if the project has no .test.yml files.
Threshold failures — the command exits with code 1 only when --min-accuracy is set and not met. If no score was produced, it prints a warning and exits 0.

Get started

Data Infrastructure

World Model

Agentic Intelligence

Custom Apps

Automations

Integrations

Edge fleet

Manage

Workspaces

Authentication

Observability

Enterprise

Reference

REST API

Resources

Troubleshooting

Testing reference

Test file format

Top-level fields

`settings`

`cases`

How evaluation works

Running tests

Accuracy thresholds

Machine-readable output

CLI reference

CI example

Error handling

​Test file format

​Top-level fields

​settings

​cases

​How evaluation works

​Running tests

​Accuracy thresholds

​Machine-readable output

​CLI reference

​CI example

​Error handling

Test file format

Top-level fields

`settings`

`cases`

How evaluation works

Running tests

Accuracy thresholds

Machine-readable output

CLI reference

CI example

Error handling