New Features
Oxy Function Jobs
Scheduled Oxy Functions are now first-class, observable jobs. A function that lives inside a custom-app bundle can be run from its definition, triggered on a cron schedule or manually on demand, and watched from the Orchestrator dashboard alongside your automations, pipelines, and agents.- Trigger on demand - Kick off any custom-app function directly from an API call and get back a run id, so a function is no longer tied to its cron cadence — trigger it manually whenever you need it.
- Retry policy in the manifest - Declare a
retriesblock (maxAttempts,minTimeoutMs,maxTimeoutMs) on a function inoxy-app.json; each run executes on Oxy’s durable task queue with exponential backoff, so a transient failure retries automatically instead of failing the whole job. - Persisted logs -
ctx.logandconsole.logoutput from a background run is now captured and saved, streamed live while the run is in flight and kept afterward — so a scheduled run’s output is no longer lost the moment it finishes. - Monitor every run - Function runs appear in the Runs and Jobs lists with their own Function badge, and a dedicated run detail view shows status, duration, the return value or error, and the full log panel — a complete picture of what a job did and why.
Observability Traces Console & Execution Analytics
The Observability surface gains an engineer-grade console for exploring traces and analyzing execution performance, rebuilt to stay fast and readable as trace history grows.- Traces search and filtering - Search traces by span name, agent, or prompt (or jump straight to an exact trace id), filter by success or error, and narrow to a rolling preset window (1h–90d) or an absolute custom from/to range. Every filter drives both the trace list and the summary charts so they always stay in sync.
- Live-tail and view toggle - Turn on live-tail to auto-refresh the list as new traces arrive, and switch between rich cards and a dense, scannable table depending on how much you want to see at once.
- Compare two traces - Select any two traces to open a side-by-side comparison with a per-metric delta, making it easy to see how one run differs from another.
- Execution analytics - New p50/p95/p99 latency and error-rate tiles with sparklines, a latency histogram with percentile markers, percentiles over time, and a cost-and-tokens breakdown by model — so you can see spend and performance at a glance instead of reading raw spans.
- Deeper trace detail - The trace waterfall is colored by span category with critical-path highlighting, and a full span inspector shows self-vs-children timing, every attribute with a filter, the inline SQL a span ran, and its events, with a flamegraph toggle.
Admin Apps Cockpit
For Global Owner and Global Admin operators, the admin Apps surface has been reworked into a dense, keyboard-navigable cockpit for managing custom-app deployments across every workspace.- Master-detail layout - Selecting an app opens a persistent registry rail (LED status indicators, grouped and collapsible) beside a live detail view, replacing the full-page swap so you can move between apps without losing context.
- Fleet strip - A slim rollup at the top shows totals at a glance — apps live vs. draft, number of orgs, and the mix of deployment sources.
- Responsive dossier - The app detail panel is collapsible and folds into an overlay sheet on narrower windows, fixing a clipped-toolbar issue on resized screens.
- Access tab revamp - The Access tab now lists orgs both with and without Oxy access, adds a Granted / No access / All filter, and uses compact icon actions with tooltips; opening a workspace’s home now always launches in a new tab so it never clobbers your own last-workspace state.
Platform Improvements
Observability
- No more crash when viewing a trace - Opening a trace on the ClickHouse-backed Observability surface could render-crash the whole app with an “Invalid time value” error and, under a slow backend, take the instance offline. Serving timestamps are now emitted as ISO-8601 UTC so the browser parses them correctly, the time-formatting helpers degrade gracefully to a dash on any bad value, and a new route-level error boundary shows a branded, recoverable page instead of a raw error screen.
- Trace serving fails fast instead of hanging - Observability queries now run under hard execution-time and result-size caps with a bounded trace-detail query, so a slow or oversized query returns a clean error rather than pinning the request and eventually 503-ing the instance.
- Trace waterfalls render correctly on ClickHouse - On the ClickHouse-backed Observability surface, a trace’s waterfall spans collapsed into thin slivers pinned to the left edge instead of stretching across the timeline, making it impossible to read where time was actually spent. Timestamps are now served in a format the browser parses correctly, so each span renders at its true width and position across the trace’s duration — the ClickHouse waterfall now matches the Airhouse one.
Onboarding
- Correct LLM-key detection in cloud - The home readiness check now resolves whether a model’s API key is set from the workspace secret store in cloud (matching how the runtime actually reads it), rather than only the process environment. This clears a false “LLM key not set” for keys added under Settings → Secrets — including OpenAI-compatible providers — that were correctly stored but reported missing.
Reliability
- Seamless updates after a new release - When a new version of Oxy ships while you have a tab open, opening a page such as the IDE could white-screen with an “Unexpected Application Error”. Oxy now detects the new version — polling on tab focus and every few minutes — and surfaces a “new version available, reload” prompt before you hit a dead page. Route errors caused by an outdated tab recover automatically with a single guarded reload instead of crashing, while genuine errors on the current build still show a recoverable error card.
Workspace Health
- Lighter, configurable health-check cadence - The default workspace health-check interval moves from every 10 minutes to once an hour, reducing background load; workspaces that need more frequent checks can still opt in via
health_check.intervalinconfig.yml. Deleting a workspace now also removes its scheduled health checks, so removed workspaces no longer keep generating failing background jobs.