Peer Cohorts on the Semantic Model, Document Libraries, Custom-App Data Stores, and Fleet Health

September 11, 2026 · v0.5.145, v0.5.146, v0.5.147

​

New Features

​

Peer Cohorts on the Semantic Model

“Is this store doing well?” is only answerable against comparable stores, and until now every app that asked it built the comparison itself — one shipped roughly 1,800 lines of hand-written SQL to do it. The semantic model answers it directly now: name a subject and a measure, and a cohort of comparable peers comes back with the baseline to judge the subject against.

• A baseline, not a leaderboard - The response carries the subject’s value, the peer median it is being compared with, the peers that formed it, and the band window the comparison covers. Everything is computed from the measures already declared in the semantic model, so a cohort and a dashboard tile cannot disagree about what a number means.
• The peer population is scoped to what you can see - Scoping is applied to the population the baseline is computed over, not just to the rows that come back — so a regional manager’s cohort is built from their own stores, and a comparison can never be a median over locations the caller has no access to.
• A comparison it cannot scope is refused, not approximated - Asking for a cohort over an entity that is not bound to the locations registry returns an explicit refusal rather than a population-wide number that looks scoped and isn’t.
• A thin cohort is reported, never silently dropped - A subject with too few comparable peers comes back with its comparison computed and marked as under-evidenced, alongside the peer count. Hiding it produced the worst of both worlds in the hand-rolled version: a panel saying “not enough data” directly above a chart already drawing the comparison.
• Excluded peers say why they were excluded - Each subject left out of the cohort carries its reason, so a cohort that looks too small is diagnosable instead of mysterious.
• Typed from TypeScript - The SDK ships types and a client for the endpoint, so a custom app calls it rather than reimplementing the comparison.

​

Seeing What Background Work Is Doing

Traces covered requests well and covered everything else barely at all: a queued task, an agent run, a scheduled scan and a warehouse query all executed with nothing recording them, so “the automation was slow” or “the function failed” had no trace to open. Background work is now instrumented end to end.

• Every queued task, agent run and scheduled pass is traced - Durable tasks, the agent run coordinator, the scheduler’s own tick, monitor scans down to each individual series, retention and cleanup passes, and the phases of serving a custom app. Work that used to vanish the moment it left a request now has a span you can find it by.
• Every warehouse query is timed - Each connector records a span per query, so the time an automation or an app function spent waiting on the warehouse is visible as its own measurement instead of being buried inside the step that issued it.
• An agent run links back to what started it - Runs stay addressable as their own traces, as the Traces console needs, and additionally carry a link to the request or task that kicked them off, so you can navigate from cause to run.
• A failed response records why it failed - A 4xx or 5xx used to log only its status code, leaving a wall of “request failed, 502” with no cause attached. The error message is now recorded on the request’s span.
• Logs are machine-readable without exception - Output from dependencies and native libraries used to arrive as plain text carrying no level, no source and no trace id — on some deployments nearly half of all log lines. Every line is now emitted in the same structured shape, repeated noise collapses with its count preserved, and a crash is reported as one structured record with its location and backtrace rather than scattered across several untagged lines.
• A guide for debugging a custom app’s functions - Which ids to look for, which endpoints serve a function’s logs, and how to follow an invocation from the app to the platform.

​

Organizations Arrive Ready to Use

Signing in to a new organization used to mean an eleven-screen setup wizard, because onboarding was still built for people setting Oxygen up themselves — which is no longer how anyone starts. Organizations are now created for you, and arrive with a workspace already in place.

• Every new organization comes with a workspace - Creating an organization from the admin console, or as a partner, provisions a Default workspace in the same step. The first person to sign in lands on Home rather than in setup.
• Onboarding never redirects you - /onboarding is now a “you’re not in an organization yet” page listing pending invites and join-by-link. An organization whose workspace isn’t ready yet shows a “being set up” card instead, and Home no longer jumps into the wizard. The wizard still exists, reachable on purpose rather than by accident.
• Setup shortcuts are staff-only - Home’s setup rows now show only to Oxy staff, so an organization owner sees their apps instead of a configuration checklist.

​

Kiosk Rosters Scoped to Their Store, and an Idle Timeout

A shared tablet in a store is not the same thing as a browser on a desk, and crew sign-in was treating them alike. Two gaps found during the first store pilot are closed.

• A tablet shows only its own store’s crew - The crew name picker listed every PIN holder in the organization, so a manager from one store appeared on another store’s tablet. A kiosk bound to a location now lists only workers holding an assignment at that location; a kiosk with no location keeps the organization-wide list. An organization-wide position is deliberately absent from every store tablet — those people sign in on the web. If the assignment lookup fails the picker comes back empty rather than falling back to the whole organization.
• A kiosk signs itself out when idle - Enrolled devices now carry an idle timeout, set when the device is enrolled. It defaults to five minutes and accepts anything from 30 seconds up to the 12-hour shift ceiling. Existing kiosks pick up the default with nothing to change. Previously the only limit was the shift length, so a counter tablet stayed signed in as whoever last touched it.

​

Sessions Last 30 Days

• One month between sign-ins - Sessions now last 30 days rather than 7. This covers browser sign-in, the custom-app subdomain session and oxyc login, which all share one token — so the CLI no longer asks for a browser re-login every week. Existing sessions keep their original expiry and pick up the longer window at the next sign-in.
• A cookie can no longer outlive its token - The cookie’s lifetime and the token’s lifetime were two separate values kept in step by hand. They are now one, closing a case where a browser held a live cookie against a dead token and got a 401 on every call. Frontline’s 12-hour shift session is unchanged.

​

One CLI for Talking to a Deployment

Everything the oxy binary did as a plain HTTP client now lives in oxyc (npm i -g @oxy-hq/cli). oxy keeps the commands that need a database, a workspace on disk or the server process — serve, worker, compile, validate and the rest. Existing logins keep working: oxyc reads the same credentials file.

• The proxy was fixed, not just moved - oxyc proxy now forwards trace headers and prints each function call’s request and trace id, survives an upstream error arriving after headers have been sent instead of taking the whole proxy down with it, and closes the upstream stream when the browser disconnects.
• Publishing from CI works - oxy init-ci generated a workflow referencing a GitHub action that does not exist, and the workspace template could not publish an app with Oxy Functions from CI at all. oxyc init-ci now writes a build job plus an environment-gated publish job, and oxyc publish performs the GitHub OIDC exchange itself — so a repository needs no long-lived token. New --build-only and --prebuilt flags split the build from the credentialed upload, and --prebuilt refuses if a declared function’s bundle didn’t arrive.
• Server warnings are printed - Advisories the server returns on a publish were dropped by the old client and are now shown.

​

Document Libraries

A knowledge base and a compliance library are both a folder tree of documents with a rule about who may read them, and neither was reachable from what the platform offered: an app could put files in its own private storage silo, but a silo cannot express “headquarters only” or “this store only”. Documents are now a platform primitive any custom app can build on.

• Folders, documents and versions - A document is either a chapter authored as rich text or an uploaded file, never both, and every edit keeps its predecessor as a version — so what a policy said last quarter stays readable. Folders nest, and categories group documents across the tree.
• Visibility belongs to the document - Each document names who may read it, and a single rule decides every listing, search hit, download and favourite. A frontline worker’s library, an officer’s library, and search over either cannot disagree about what a reader is entitled to see.
• Readable without an organization membership - Reads are scoped by the document’s own visibility rather than by a role, because the people a knowledge base exists for — crew on a shared tablet — usually hold no membership row at all. Creating, editing, uploading and deleting are organization-admin actions.
• Full-text search on day one - Search ships with a real full-text engine behind it rather than a placeholder, so an empty result means there is nothing to find instead of a search box that was never wired up.
• Review is tracked separately from publication - A compliance document is routinely published and still awaiting sign-off, so review state is its own axis: “nobody is asking for review” stays distinct from “waiting for review”, and a document in review is still readable by whoever its visibility names. Review is a workflow for tracking sign-off, not an access control.
• Pins, favourites and a trash - An officer pins a document organization-wide; any reader favourites one for themselves. Deleted documents go to a trash only officers can open — to everybody else it reads as an ordinary library rather than confirming that a trash exists and has something in it.

​

Custom Apps Store Data by Its Shape

A custom app had no right home for the facts it produced. Writing to the warehouse minted the calling user’s role, so a scheduled function could never write while its reads worked fine, and nothing stopped an app writing into a customer’s own warehouse. There is now one store per shape of data, and the SDK enforces the choice.

• ctx.airhouse for facts - Append-only facts land in the app’s own Airhouse schema, where the analytics agent can query them alongside everything else. Writes run as the app rather than as whoever invoked it, so a scheduled run writes exactly as a button click does. Tables come from the app’s declared airhouseMigrations, applied at publish; a statement reaching outside the app’s own schema is refused.
• Transactions on the app’s records - ctx.oltp.tx() runs a block of statements against the app’s own transactional Postgres schema and commits or rolls back as a unit, sharing the same commit bracket as ctx.tx.
• Customer warehouses are read-only - A write to a connected customer warehouse is now refused unless the function names that database in customerWarehouseWrites with a reason — so a newly refused write is a missing declaration, not a broken connector.
• oxyc checks placement before you publish - oxyc validate checks the new manifest fields, Airhouse migration DDL, and secrets being used as state. oxyc oltp status reports whether an app’s transactional store is provisioned, and oxyc oltp provision sets one up.

​

Fleet Health for Published Apps

“Which of our apps are broken right now” had nowhere to look. The only status an app showed was Live or Draft — publish state, not health — and four separate layers turned an absent measurement into a green result: telemetry not configured, a workspace never evaluated, a failed query, and traffic too low to judge.

• A health table across every app - Admin → Apps gains a health view listing each published app’s current verdict, read from request and function telemetry.
• “No reading” is not “healthy” - Verdicts are down, degraded, not measured, quiet and operational. An app with no telemetry, an unevaluated workspace, a failed query, or traffic below the floor reports as such instead of collapsing into a green tick, and carries a dashed outline rather than a colour — “no reading” is not a point on a severity scale.
• A quiet app can now fail loudly - The alerting rules were written for high-volume services and could never fire for an app serving one shared tablet in a store. A new rule fires on absolute counts — at least five failures and at least half of all requests, over both windows — so an app failing everything it serves reaches unhealthy rather than scoring “no opinion”.
• Silence is detected - A six-hour window compared against the same hours a week earlier catches an app that has stopped being used at all, which no error-rate rule can see because the denominator is zero. It reports as degraded and never pages.
• Per-invocation function measurements - Each function call now records how many outbound calls it made and how much of its elapsed time was platform startup versus your own code.

---

​

Platform Improvements

​

Performance

• Semantic queries lose about 0.8 seconds of fixed cost - Every semantic query re-read and re-materialised the whole semantic model before doing any work, fetching each file from object storage one at a time — a fixed cost per request whatever the query, and one that grew with the number of semantic files in the workspace. On one measured app the route’s own overhead was larger than the warehouse work: materialising a 708-row table instead of running a 700-line view changed the warm request time barely at all. A compiled revision never changes, so it is now materialised once per process and reused across requests.

​

Airway ELT

• A chunked backfill can be scoped to specific resources - A chunked backfill replayed every resource in a pipeline, and a snapshot resource has no historical form to fetch — so each chunk spent a report request pulling today’s snapshot again, against a daily quota shared with the upstream vendor’s own UI, which then started refusing the windowed reports the backfill existed for. Backfills now take a resources list (and --resources on the CLI), stored with the range so a resumed run keeps the same scope instead of silently widening. Omitting it keeps the previous behaviour.
• A Postgres source can hold its whole connection string as a secret - A sql_database source could only take a literal connection string, so a per-org transactional Postgres — whose credential is a whole DSN minted per organization, with no literal to commit — could not be read from a pipeline at all. It failed with an unexplained invalid sql_database config rather than saying what was wrong. connection_string_var now resolves from the secret manager, for sql_database and postgres_cdc alike.

​

Custom Apps

• Writes to a ClickHouse warehouse work again - Every ctx.warehouse.insert, and every ctx.warehouse.exec of an INSERT … VALUES, had failed against a ClickHouse warehouse since the audit trailer added to queries two releases ago. ClickHouse stops parsing SQL at the VALUES keyword and reads the rest of the request as row data, so the trailer was read as a malformed extra row and the insert was rejected whole — while reads, schema changes and INSERT … SELECT kept working, which made a platform-wide break look like a problem with one app. The trailer now leads the statement on that warehouse, where it also survives into the query log: the audit line it exists to leave was previously being lost even on the statements that did not crash. Other warehouses are byte-for-byte unchanged.
• A warehouse write starting with a comment works too - The fix above left one case behind: ctx.warehouse.exec of a statement that opens with a -- comment still failed on ClickHouse. The audit tag is no longer attached by editing the statement at all — each warehouse decides where its own tag goes, and on ClickHouse the SQL is sent exactly as written with the tag carried alongside it. A comment-prefixed write also lands an audit record again; it previously read as an unrecognized verb and left none.
• ctx.warehouse.upsert is refused by name outside Postgres and DuckDB - Conflict handling is not ClickHouse syntax, so an upsert there has never worked — it came back as the same unrelated parse error as the insert bug above. It now says plainly that the warehouse doesn’t support it. Use an explicit insert.
• A failing function reaches us before a customer does - A failed invocation now ends its trace as an error recording what kind of failure it was, and a failure a function hasn’t seen in a week pages the on-call once it repeats. The ClickHouse insert break was first reported by a customer, with nothing on the platform side having noticed.
• Publishing can no longer move a live app to another workspace - A draft publish naming a different --project overwrote the live app’s workspace, taking its connectors, secrets and schedules with it; a stray environment variable in a local file was enough to trigger it. A publish whose project doesn’t match the app’s is now refused before the bundle is even unpacked, with a message naming both workspaces and the supported way to move an app. An app sitting on a workspace that was deleted or orphaned is still re-homed on its next publish, since refusing would leave it stranded.
• An app can’t be created on, or moved to, another organization’s workspace - The admin app registry took the workspace id straight from the request without checking that it exists or belongs to the app’s organization — and that id is what an app’s connectors, secrets and data plane resolve against. Both create and move now refuse a workspace outside the app’s organization, and refuse before anything is written.
• A bundle missing its function code is refused at publish - A bundle whose oxy-app.json declared functions but carried no code for them published happily, and the app went live with every function missing — the only signal a 404 at the first call. Publish now names each missing function and refuses, storing nothing.
• A broken oxy-app.json is an error instead of being ignored - An unparseable manifest, or an environments entry without a target, was silently treated as no manifest at all: the publish went out with no functions bundled and took its identity from the directory name. A malformed manifest now stops the command with the file path and the parse error. target is genuinely optional — leaving it out falls back to the default, as blank build fields already did.
• ctx.storage delete and copy require write access - Both went through with only storage.read, so a function declared read-only could destroy or overwrite objects in its app’s storage. delete now requires storage.write, and copy requires both read and write since it does both. A function using either with read alone must declare "storage": { "write": true }.

​

Home and the App Launcher

• App cards load fast and stay cached - Card images were re-downloaded on nearly every visit — five cards at 2 MB with a five-minute cache lifetime, each taking close to two seconds. Icon and art URLs now carry the build they belong to, so a card image can be cached for a year and is replaced the instant the app is republished. The first row of cards loads eagerly rather than lazily.
• Publish warns about oversized card art - Card art over 200 KB now comes back as an advisory warning on publish, never a refusal. The screenshot scaffold writes WebP at 1280×640 — around 40–70 KB for a dashboard screenshot, against roughly 290 KB for the PNG it used to produce.

​

Workspaces

• A workspace with a broken config stops retrying every five minutes forever - A workspace whose configuration fails to parse was recompiled on a fixed five-minute cycle indefinitely, producing a permanent stream of identical errors that buried real ones. Repeated failures now back off progressively, up to six hours. Pushing a commit still triggers a prompt retry, so a fix lands as quickly as it always did.

​

Observability

• A trace-store restart no longer silences capture until the instance is restarted - If the ClickHouse trace store was unreachable at boot, the instance gave up permanently: no span was recorded again for the life of the process, and the Traces, Metrics and Execution Analytics consoles stayed disabled even after the store came back. A node roll produced exactly that — span ingest fell by 95% and stayed there for 35 minutes until the pods were restarted by hand, with one line at boot as the only signal. An unreachable store is now retried in the background with backoff until it opens, and the consoles enable themselves the moment it does. A store that accepts a connection but never answers counts as unavailable rather than holding up boot.

​

Cameras

• A forwarded address can no longer be spoofed into the proxy path - Edge authentication stored the first forwarded address it saw, so a request arriving through a local hop recorded a loopback address, and a link-local address could be presented as a camera’s upstream. Loopback and link-local hops are now stepped past and refused outright, closing the path by which a cloud metadata address could have been named as the snapshot proxy’s destination.
• A failed write says why it failed - Compliance report submissions were failing roughly 1,270 times a day with nothing but airhouse error: INSERT failed: db error — the actual database error was discarded, so there was no way to tell a missing table from a bad value from a connection problem. The database’s code, message, detail and hint are now kept all the way through. That also makes the “nothing provisioned yet” case behave as intended: a workspace with no tables returns an empty result rather than an error.

​

Logs

• Routine events no longer read as failures - Unauthenticated service-worker polls, a missing local dataset directory and routine git maintenance were logged at warning or error level, and a failed background task was logged under a name that said nothing about which run or task kind failed. Those are demoted to their real severity, and a failed task now names its run and kind.
• Expected outcomes are warnings, not errors - A run that can never resume because it has no checkpoint, a task that stopped to wait for user input, and a query rejected as a bad request were all logged at error level beside genuine failures, which made the error stream unusable for finding real problems. Those three are now warnings; a server-side query failure stays an error.