> ## Documentation Index
> Fetch the complete documentation index at: https://oxy.tech/docs/llms.txt
> Use this file to discover all available pages before exploring further.
# Observability
Oxygen can record every agent run, workflow execution, and tool call as a **trace** — a hierarchical view of what the AI did, how long each step took, which SQL was generated, how many tokens were used, and whether anything errored. Traces show up in the Observability section of the Developer Portal so you can debug slow runs, audit what agents produced, and spot patterns across many executions.
Observability has two independent switches:
* **`OXY_OBSERVABILITY_BACKEND`** — turns span **capture** on. Set it to `clickhouse`, the only supported backend, and Oxygen starts writing traces there. No capture happens if it's unset.
* **`--enterprise`** — mounts the observability **UI and API routes** (the Traces / Metrics / Execution Analytics pages and their `/api/*` endpoints). Without it, traces are still captured (if the backend is set) but are only reachable via direct DB access.
For the normal "I want to see traces in the Developer Portal" path you want both.
## Enabling observability
Set `OXY_OBSERVABILITY_BACKEND=clickhouse`, point Oxygen at a ClickHouse instance, then start the server with `--enterprise` to expose the UI.
```bash theme={null}
export OXY_OBSERVABILITY_BACKEND=clickhouse
export OXY_CLICKHOUSE_URL=http://localhost:8123
oxy serve --enterprise
```
With the Docker-managed stack, `oxy start` boots an `oxy-clickhouse` container for you and points the connection at it — the backend variable is all you need to set:
```bash theme={null}
export OXY_OBSERVABILITY_BACKEND=clickhouse
oxy start --enterprise
```
Open `http://localhost:3000/ide/observability/traces` to see traces. Run an agent or workflow and the corresponding trace appears within a second or two.
## The backend
ClickHouse is the sole observability backend, and there is no default in any mode — capture is opt-in everywhere, including `--local`.
| Variable | Purpose |
| -------------------------------------- | ------------------------------------------------------------------------ |
| `OXY_OBSERVABILITY_BACKEND=clickhouse` | Enables capture. Unset disables it. |
| `OXY_CLICKHOUSE_URL` | HTTP endpoint of your ClickHouse instance, e.g. `http://localhost:8123`. |
Trace data is a columnar, append-heavy workload with fixed query shapes and TTL-based retention, and the store has to stay up precisely when the rest of the system is misbehaving — that is what it is there to debug. ClickHouse matches that shape and isolates a bad query to the query itself. Earlier releases also offered `duckdb`, `postgres`, and `airhouse` backends; those were removed. Setting one of their names no longer falls back to anything — the server starts with observability off and prints a message telling you to migrate. Trace data is not carried across from them.
If you start the server with `--enterprise` but don't set `OXY_OBSERVABILITY_BACKEND`, the Traces page renders a "not configured" banner and nothing is recorded. The server also prints a warning on startup.
## What's recorded
Oxygen emits spans for the major execution boundaries:
| Span | Description |
| ----------------------- | -------------------------------------------------------------------------------- |
| `agent.run_agent` | Top-level agent run. Parent span for everything the agent does. |
| `workflow.run_workflow` | Top-level workflow execution. |
| `analytics.run` | Agentic analytics pipeline run. |
| `llm.call` | Individual LLM request (includes prompt/completion token counts). |
| `tool.call` | Tool invocation — SQL execution, semantic query compilation, Looker lookup, etc. |
Each span carries attributes like the agent name, model, execution status, and error messages if any. Nested spans are stitched into a waterfall view in the Trace Detail page so you can see where time was spent.
## Environment variables
| Variable | Default | Description |
| ----------------------------- | ----------------------- | ------------------------------------------------------------------------------------------------------------- |
| `OXY_OBSERVABILITY_BACKEND` | *(unset)* | `clickhouse` — the only supported value. No default in any mode; unset disables observability entirely. |
| `OXY_CLICKHOUSE_URL` | `http://localhost:8123` | ClickHouse HTTP endpoint. |
| `OXY_CLICKHOUSE_USER` | `default` | ClickHouse username. |
| `OXY_CLICKHOUSE_PASSWORD` | *(empty)* | ClickHouse password. |
| `OXY_CLICKHOUSE_DATABASE` | `observability` | ClickHouse database name. |
| `OXY_OBSERVABILITY_LOG_LEVEL` | `debug` | Filter for span capture. Independent of `OXY_LOG_LEVEL` — console verbosity does not affect what is recorded. |
| `OXY_SERVICE_NAME` | `oxy` | Service name attached to every span. |
## Retention
Trace data is automatically pruned after **90 days**. Retention is derived from the longest time-window the Traces UI exposes, so the UI and the retention policy always agree. The policy is applied at the engine level via `ALTER TABLE ... MODIFY TTL` on schema init, so expiration is handled by ClickHouse's own background merges — there is no pruning job to schedule or monitor.
## No traces showing up?
1. Confirm the server was started with `--enterprise`.
2. Confirm `OXY_OBSERVABILITY_BACKEND=clickhouse` is set — it has no default, in any mode. Check the startup output for a line like `Observability: clickhouse (...)`.
3. If the Traces page shows a "not configured" banner, the backend env var wasn't set when the server started.
4. `OXY_LOG_LEVEL` does **not** affect what is recorded. Traces are captured even at `warn`.
5. If startup printed a message about a removed backend, you're carrying a `duckdb`, `postgres`, or `airhouse` value from an older release — switch it to `clickhouse`.
6. Check `docker logs oxy-clickhouse` (if using `oxy start`) or your external instance's logs for connection or schema errors.
For deeper architecture details see the [internal observability docs](https://github.com/oxy-hq/oxygen/blob/main/internal-docs/observability-analytics.md).