Skip to main content
Oxygen can record every agent run, workflow execution, and tool call as a trace — a hierarchical view of what the AI did, how long each step took, which SQL was generated, how many tokens were used, and whether anything errored. Traces show up in the Observability section of the Developer Portal so you can debug slow runs, audit what agents produced, and spot patterns across many executions. Observability has two independent switches:
  • OXY_OBSERVABILITY_BACKEND — turns span capture on. Set it to clickhouse, the only supported backend, and Oxygen starts writing traces there. No capture happens if it’s unset.
  • --enterprise — mounts the observability UI and API routes (the Traces / Metrics / Execution Analytics pages and their /api/* endpoints). Without it, traces are still captured (if the backend is set) but are only reachable via direct DB access.
For the normal “I want to see traces in the Developer Portal” path you want both.

​
Enabling observability

Set OXY_OBSERVABILITY_BACKEND=clickhouse, point Oxygen at a ClickHouse instance, then start the server with --enterprise to expose the UI.
With the Docker-managed stack, oxy start boots an oxy-clickhouse container for you and points the connection at it — the backend variable is all you need to set:
Open http://localhost:3000/ide/observability/traces to see traces. Run an agent or workflow and the corresponding trace appears within a second or two.

​
The backend

ClickHouse is the sole observability backend, and there is no default in any mode — capture is opt-in everywhere, including --local.
Trace data is a columnar, append-heavy workload with fixed query shapes and TTL-based retention, and the store has to stay up precisely when the rest of the system is misbehaving — that is what it is there to debug. ClickHouse matches that shape and isolates a bad query to the query itself. Earlier releases also offered duckdb, postgres, and airhouse backends; those were removed. Setting one of their names no longer falls back to anything — the server starts with observability off and prints a message telling you to migrate. Trace data is not carried across from them.
If you start the server with --enterprise but don’t set OXY_OBSERVABILITY_BACKEND, the Traces page renders a “not configured” banner and nothing is recorded. The server also prints a warning on startup.

​
What’s recorded

Oxygen emits spans for the major execution boundaries: Each span carries attributes like the agent name, model, execution status, and error messages if any. Nested spans are stitched into a waterfall view in the Trace Detail page so you can see where time was spent.

​
Environment variables

​
Retention

Trace data is automatically pruned after 90 days. Retention is derived from the longest time-window the Traces UI exposes, so the UI and the retention policy always agree. The policy is applied at the engine level via ALTER TABLE ... MODIFY TTL on schema init, so expiration is handled by ClickHouse’s own background merges — there is no pruning job to schedule or monitor.

​
No traces showing up?

  1. Confirm the server was started with --enterprise.
  2. Confirm OXY_OBSERVABILITY_BACKEND=clickhouse is set — it has no default, in any mode. Check the startup output for a line like Observability: clickhouse (...).
  3. If the Traces page shows a “not configured” banner, the backend env var wasn’t set when the server started.
  4. OXY_LOG_LEVEL does not affect what is recorded. Traces are captured even at warn.
  5. If startup printed a message about a removed backend, you’re carrying a duckdb, postgres, or airhouse value from an older release — switch it to clickhouse.
  6. Check docker logs oxy-clickhouse (if using oxy start) or your external instance’s logs for connection or schema errors.
For deeper architecture details see the internal observability docs.