> ## Documentation Index > Fetch the complete documentation index at: https://oxy.tech/docs/llms.txt > Use this file to discover all available pages before exploring further. # App Operator Role, Airway Admission Policy, Custom App Storage Lifecycle, and Sharper Anomaly Detection > Ship apps without reaching the tenant, govern Airway sources, and cap custom-app storage. **August 10, 2026** · v0.5.121 – v0.5.126 ### New Features #### App Operator: Ship Apps Without Reaching the Tenant Managing custom apps used to require Global Admin — which silently included the ability to delete any org. Platform standing is now a **capability grant with a scope**, not a single boolean, and **App Operator** is the role for people who publish and develop apps and nothing else: * **A role that stops at apps** - An App Operator ships and develops custom apps. Org deletion, ownership transfer, member management, org settings, workspace admin, billing, the partner registry, internal jobs, compiles, and the cross-tenant explorer are all out of reach. * **Optionally bounded to named orgs** - A grant can be scoped to specific tenants, and the bound holds everywhere: publish, rollback, builds, functions, access, teams, members, activity, and API keys all refuse an org the grant doesn't cover. Out-of-scope requests return "not found" rather than "not allowed", so scope can't be probed. * **Global Admins can now delegate** - Issuing, re-scoping, and revoking an App Operator grant no longer needs the Global Owner. A writable grant must be strictly weaker than your own — a lower role, within a scope your own scope wholly contains — so nobody can edit their own row, mint a peer, or launder a bounded grant into an unbounded one through a second account. * **Grant staff access from the person's record** - A new **Staff access** card on the admin user detail page grants, changes, and revokes standing in place, instead of copying an email address over to a separate table. The role picker offers only what you may issue, the scope picker only the tenants you reach, and rows you cannot write are disabled with the reason in the tooltip. * **Every grant change is recorded** - Grant creation, update, and revocation now land in the audit trail in the same transaction as the change itself. * **Publish tokens belong to their minter** - CI publish tokens are now owned by whoever minted them, with the cross-admin view reserved for platform operations, so an app-focused role can't revoke another engineer's token. The admin console is now one door with per-section escalation: opening it and acting inside it are separate questions, and the navigation renders per role rather than offering a section that bounces on click. #### One Airway Run per Pipeline, per Workspace Two concurrent runs of the same **Airway** pipeline are incorrect, not merely wasteful — they read-modify-write a single cursor row and fold overlapping snapshots into the served table, which can silently skip a window or leave duplicate rows behind. A pipeline now holds a lease for the duration of its run, and a run that arrives while that lease is held waits its turn rather than being turned away: * **On by default for every pipeline** - At most one active run per pipeline per workspace, enforced in the database, so it holds across every serve and worker replica rather than just within one process. * **Every path is gated, including automations** - The lease is taken when a run actually starts rather than when it is submitted, so an Airway step inside an automation is covered too. That path previously bypassed the guard entirely, and two automations could run one pipeline at the same time. * **Starting a run that's already queued joins it** - Submitting a pipeline that is already waiting returns the run already in flight instead of an error or a second copy, so ten clicks on **Run now** produce one run. Backfills never collapse this way, since different windows are genuinely different work. * **Contention waits instead of failing** - A run that finds the pipeline busy is put back in the queue and retried about every 30 seconds, holding no worker slot while it waits. A run starved for more than 12 hours is failed rather than retried forever. * **A schedule out-pacing its own load is not a failure** - A tick landing while a run is already queued joins that run, and schedule history shows the run it joined rather than a bare "skipped". * **A failed run doesn't block the next one** - The lease is released on success and failure alike, and a lease still recorded against a run that has already finished is reclaimed on the next attempt — the case that used to wedge a pipeline for six hours with nothing actually running. * **Keyed by workspace and pipeline name** - Two tenants can both ship a `daily_analytics` pipeline without gating each other. * **Opt out per pipeline** - Set `allow_concurrent_runs: true` in the `.airway.yml` when a pipeline genuinely tolerates overlap; the run logs a warning naming the pipeline so its trace shows the guard was off. This prevents new duplicate rows from being created. It does not repair duplicates that already landed — those need a one-time cleanup. #### Airway Source Contracts and Admission Policy Every Airway source resource declares a contract: whether its rows are immutable, versioned, or opaque, whether its cursor tracks modification time, and how far back it can be restated. Those declarations are now both visible and enforceable: * **A policy per source kind** - A new **Admin → Airway** page sets the contract policy and environment for each source kind — Toast, QuickBooks, Weather, and generic REST. Per-workspace overrides exist for the rare exception and inherit field by field from the global setting, so an override that sets only the environment keeps the global policy. * **See what a stricter policy would stop, before saving it** - Tightening a policy halts every pipeline whose resources don't satisfy it. An on-demand preview lists exactly which resources would be refused and separates a real, fixable gap from an upstream limitation nobody can act on from this page. Saving asks for confirmation unless the preview ran clean across every pipeline of that kind — an unknown impact is never read as a safe one. * **The same policy on every path** - Manual runs, schedules, `oxy airway run`, and Airway steps inside an automation all resolve the same policy. Automation steps previously fell back to the built-in default and silently ignored what had been configured. * **A run records the policy it ran under** - Admission is resolved when the run is queued, so changing the configuration later does not retroactively rewrite the history of runs that already executed. * **Contracts in the run view** - Run topology now labels each resource with its contract, so an operator can see why a resource behaves the way it does and whether late edits are recoverable. A resource that declares nothing reads as undeclared, distinct from a source that was never asked. * **Backfill windows suggested from the contract** - The backfill dialog pre-fills its range from the declared restatement window instead of leaving you to guess how far back to re-pull, naming the resource the suggestion came from and what it does not speak for. Where nothing is declared, or the source is immutable and has nothing to re-pull, no range is suggested and the reason is stated rather than a number invented. * **Deployment-wide connector settings** - Request timeout, retry count and backoff, user agent, and TLS trust are now configurable for the whole deployment instead of running on built-in defaults. The page shows configured beside installed, flags drift, and says plainly that a save applies on restart. #### Custom App Storage: Retention, Quotas, and Usage An app's storage silo had no expiry, no ceiling, and no way to answer "how much is this org holding?" short of walking the bucket. Assets now have a lifecycle: * **Declare how long objects live** - A `storage.retention` block in `oxy-app.json` sets a lifetime per key prefix, and the expiry is carried out by object storage itself rather than by a sweeper an app has to trust. A prefix pinned with `expireAfter: null` keeps its objects, and still counts toward usage. * **Org-level quotas** - Soft and hard quotas apply at every write path, including an overwrite of an existing key, so one app can't quietly consume a tenant's whole allowance. * **A storage view in admin** - **Admin → Apps → Storage** ranks the fleet by usage, charts it over time, and browses an individual app's objects live. * **GB-month metering** - Time-weighted GB-month usage per org is measured and exposed. It is a meter only — nothing is invoiced from it yet. * **Everything fails in the safe direction** - An object with no matching rule never expires, a malformed retention rule expires nothing and warns, a partial measurement records a floor with its status attached rather than a smaller number, and a quota lookup that fails allows the write. Adding an app's **first** retention rule changes how its uploads are signed, so an uploader that bypasses `@oxy-hq/sdk` starts failing with an opaque storage error — use the SDK's upload path. Apps with no retention rule behave exactly as they do today. #### Sharper Anomaly Detection and Grouping Anomaly monitoring now decides what fires, how it ranks, and how related anomalies group using the same signal: * **One bad bucket no longer disables a monitor** - The empirical band was built from raw per-phase minimums and maximums, so a single contaminated historical bucket widened that phase's envelope permanently and nothing could ever breach it again. The envelope is now robust to outliers, so a bad bucket is clamped instead of becoming the new normal. * **Severity ranks on what actually bound the decision** - The Insights Inbox used to order anomalies by a score different from the one that decided whether they fired at all. Severity now reflects how far past the envelope edge a value landed, so the ordering and the filter answer the same question. * **A chain-wide drop is one event, not fifty** - Anomalies firing across many segments in the same bucket are grouped into a cohort, with members ranked by how far each deviates from the group. A `calendar:` block in `.monitor.yml` can label a cohort with a named date — a label only, never a filter. * **Explain can decompose by store** - Attribution used to drop every dimension whose name looked like a key, which threw out the foreign keys that make the most useful splits. It now consults the semantic layer's declared entities and falls back to the name heuristic only when they aren't declared. One behavior change worth knowing: a spike on a short, flat series now files as **Medium** rather than **High**. Such a series has too few measured points in the spike's seasonal phase for the band to have an opinion, and "no opinion" now falls to an honest default. Existing anomalies keep their stored severity; only newly-scanned ones rank the new way. #### Read-Only QuickBooks Credentials for Airway Intuit expires the previous refresh token whenever it issues a new one, so a QuickBooks connection tolerates exactly one component refreshing it. An **Airway** pipeline could only refresh — which made it a second rotation writer alongside any scheduled function or external ETL already rotating the same connection. When the two chains diverge, the loser is locked out with an `invalid_grant` error that only a manual re-authorization recovers: * **A read-only custody mode** - Setting `access_token_var` instead of `refresh_token_var` on a QuickBooks source tells the pipeline to authenticate with an access token read from the secret manager and never contact Intuit's token endpoint. Whichever component owns rotation stays the only writer. * **Resolved per request, not frozen when the run is queued** - An access token lives about an hour and a backfill can outlast that, so the token is read from the secret store as the run needs it rather than baked in at dispatch. App-scoped secret names work as-is. * **No silent fallback** - A source that declares `access_token_var` with no token available refuses to start rather than quietly reverting to refreshing and forking the chain at 3am. Rotating mode is unchanged and still requires both the client secret and the refresh token. Nothing changes for an existing pipeline until its `.airway.yml` moves from `refresh_token_var` to `access_token_var`. *** ### Platform Improvements #### Airway * **A run that fails before it starts no longer blocks the pipeline** - A run that died at dispatch — an unreadable `.airway.yml`, a parse error, an unresolvable destination — reached its failed state within milliseconds but kept holding the pipeline's lease for the full six-hour timeout. Every later run was refused while the UI correctly showed nothing in progress. Dispatch failures now release the lease like any other ending. * **Clearing a stuck lease by hand actually clears it** - `oxy airway release-lease` honored `--workspace-id` when listing and confirming, then discarded it when deleting, so it could only ever clear a lease in the legacy local workspace. It also reported success whether or not anything was released — the worse half, since an operator who believes a wedged lease is cleared goes looking for the problem somewhere else. The command now targets the workspace you name and reports what it actually released. * **Newer rows can no longer be overwritten by stale ones** - Toast and QuickBooks resolved their pipeline contracts through a wrapper that dropped the record-version declaration, which silently disabled the guard keeping an older copy of a row from replacing a newer one in the served table. The guard is now in force for every connector. * **Pipelines run correctly on any replica** - The Airway worker resolved a pipeline's `.airway.yml` from the local working copy, which the stateless worker fleet doesn't have. It now reads the compiled workspace, so a run no longer fails with "workspace directory not found" or a spurious missing-pipeline error depending on which instance happened to pick it up. A pipeline that has never been compiled now fails naming that reason instead of depending on what was on a node's disk, and the IDE's edit-then-run loop on a draft branch is unchanged. * **One wider pull on the first Toast and QuickBooks run** - The upgrade to Airway 0.1.23 renames the keys these pipelines track their progress with, so the first run after upgrading reads as a fresh start: Toast re-pulls seven days of time entries and a full pass over employees, QuickBooks re-pulls thirty days. The window is bounded and deliberate — the rename exists to recover late edits the previous cursors could miss — and later runs resume incrementally as before. * **Captured Toast payments no longer read as unpaid** - Toast does not advance an order's modified timestamp when a card moves from authorized to captured, so a pipeline trusting that timestamp alone kept the authorized snapshot forever and anything reading payment status booked settled card revenue as unpaid. Toast orders now re-read a trailing three-day window on every run. This prevents recurrence; rows that already landed stale stay stale until a bounded backfill re-pulls them, and that backfill must widen both ends of its range because the modified axis is windowed too. * **A pipeline reference that can't be resolved is retried, not failed** - When a replica could not answer whether a pipeline exists — mid-deploy, or a workspace that hasn't compiled yet — the answer came back looking like a bad request, so the automation run failed on the first attempt instead of waiting for the transient condition to clear. "Not compiled yet" and "genuinely not found" are now different answers: the first is retried and the API responds with a retry-after rather than a caller error, while a real typo in a pipeline reference still fails immediately instead of burning the retry budget. * **REST-based sources can declare a contract** - The admission-policy preview used to report an undeclared endpoint on a generic REST source as an upstream limitation nobody could act on. Those endpoints can now carry a contract of their own, so for the couple dozen SaaS connectors built on REST the preview names a fix the operator can actually make instead of calling it impossible. * **A deployment-wide cursor-lag floor** - A new connector setting guards against a source whose cursor index lags further behind than its contract declares, so a pipeline doesn't trust a high-water mark the source hasn't caught up to. Leaving it unset keeps each contract's own declared lag — absence is not a floor of zero, and a floor of zero is refused with an explanation rather than silently read as "no floor". * **Admin → Airway is reachable by platform operators** - The page's API already served anyone with platform-operate standing, but the sidebar entry and the route guard still demanded Global Owner — so the tab was hidden and `/admin/airway` bounced to Feature flags for exactly the admins the endpoint would have answered. The navigation and the endpoint now agree on who gets in. * **Duplicate rows collapse instead of multiplying** - A merge updated every target row sharing a key, so once a key existed twice both copies lived forever and each later load rewrote both with identical values — permanent, self-propagating, and invisible to any query that groups by the key. It was observed across two independent connectors sharing one write path. A merge now replaces per row, so N copies collapse to one on the next load that stages that key. Duplicates that already landed on keys no later load re-reads need a one-time cleanup. * **Late edits to Toast time entries are no longer lost** - Time entries were pulled with no overlap between windows, so an entry that a run missed sat below the next window's floor forever with no second chance, and the likelihood of loss climbed with how long after clock-in the entry was edited. The resource now re-reads a seven-day trailing window on every run. Rows already missing need a bounded backfill on the clock-in axis; the re-read is idempotent and adds no extra requests in steady state. * **QuickBooks journal entries can now be loaded** - A new `journal_entries` resource mirrors the QuickBooks journal, manual entries included, so posted status can be reconciled against QuickBooks itself rather than inferred. The table is rebuilt in full on every run, which means an entry deleted in QuickBooks disappears from the mirror by absence — something the incremental resources cannot express. Selecting it pulls the company's whole ledger history; pinning a start date is not available yet, so expect the first load to be a long one on a company with a long ledger. * **Nested tables no longer accumulate a copy per load** - When a resource is loaded by full replace, its nested child tables were appended to instead of replaced, so every load added another complete copy while the parent table stayed correct, so a child table grew by a full extra copy on every load. Child tables now clear with their parent, and the accumulated copies are dropped by the first run after upgrading rather than needing a manual cleanup. #### Automations * **A long-running delegated step runs once, not three times** - An automation step that hands work off to a sub-run — an Airway pipeline load, most commonly — and takes longer than a minute could be reclaimed and re-executed by the task queue up to three times, roughly a minute apart, before being abandoned. Nothing about this was visible from the outside: the automation file had one step and the run view showed one run, while the delegated work ran repeatedly, so a single day of data could cost several passes over the source API. A step parked waiting on delegated work now keeps reporting itself alive, so it is only reclaimed when its process genuinely dies, and a step already waiting on a sub-run refuses to launch a second one. Steps that finish inside a minute were never affected, and a step whose process really does die is still recovered exactly as before. One consequence worth knowing: a delegated step is now held to the standard 30-minute ceiling instead of being abandoned after roughly three minutes. That is strictly more room than before, but a load that genuinely runs longer than 30 minutes as an automation step will now fail at that ceiling rather than quietly duplicating itself. Duplicate rows that already landed are not repaired by this fix — it stops new ones from being created. #### Home * **No more "Oxygen Factory is temporarily unavailable" during a rollout** - Home probed an onboarding endpoint only the Factory can answer, so every time that pod restarted, anyone sitting on the launcher saw an app-wide unavailability banner for something they had never asked for. Home now renders from reads any replica can serve and never touches that route on the common path. * **No more being pulled into the setup wizard** - A missing LLM key in the workspace secret store was enough on its own to force a full-page redirect into the setup wizard — including when the key came from a server environment variable, when a teammate had already finished setup, or when stale browser state from an abandoned attempt made the wizard and Home hand you back and forth. The wizard now opens only when this browser is genuinely mid-setup on a workspace that isn't usable yet, and a workspace that completes setup clears the flag that kept re-triggering the check. One trade-off: Home no longer verifies the LLM key on the common path, so a workspace that truly has none surfaces it on the first failed message rather than up front. #### Custom Apps * **Faster asset delivery** - Assets are now compressed once at publish time rather than on every request on every replica, and the org, app, and build lookups behind a page load are cached — a 100-asset page load went from roughly 200 database queries to none in steady state. Apps published before this release still serve exactly as they did; the improvement applies from the next publish. * **No more blank page on back-navigation** - Pressing Back from a custom app sometimes rendered an empty page. The app shell went out with no cache directives, so a stored copy could name script files a later deploy had already rotated away, and the page died before it could recover. The shell now always revalidates. Anyone holding a pre-fix cached copy will hit the blank page once more and needs a single hard reload. * **Oxy Functions are no longer reachable as static files** - A function's compiled handler could be fetched over HTTP by anyone past the app's auth gate. The serve plane now refuses those paths outright, and published bundles no longer embed the original TypeScript source. Invoking a function is unaffected, and stack traces still resolve to file, line, and column. * **A warning when a build can't be traced to its source** - `oxy publish` now warns when no source repository is recorded, or when the working tree was dirty — a case where the recorded commit links to code that looks right but isn't. Neither blocks a publish. The admin app list and build history flag apps already in the field whose serving build has no traceable source, instead of rendering nothing where the commit link would be. #### Cameras * **People behind a counter are counted again** - Zone occupancy was decided by where a person's feet landed, so anyone whose lower body was occluded by a prep counter read as outside the zone — drawn grey in the clip and, more importantly, left out of the head count that fires a congestion flag. Occupancy is now anchored on the center of the detection. * **Busy periods are captured, not just long backups** - Congestion capture now triggers at 3 people sustained for 90 seconds, down from 4 for 5 minutes. It still archives one representative clip per episode rather than a flood, and the thresholds remain overridable per camera box. * **Upsell detection starts on the boxes configured for it** - The camera list that switches upsell detection on never reached the edge container, so a box ran with the feature silently off — detections, clips, and uploads all healthy, and not one upsell event in a day. The installer and the container now carry it, along with the speech model, confidence threshold, and per-item cooldown, so those stay tunable per box without a code change. A box picks this up on its next install or over-the-air update. * **No more phantom boxes in evidence clips** - Annotated clips drew boxes drifting across frames with nobody in them, because the annotator kept predicting the position of people who had already left the frame. A box is now drawn only where a person is actually detected on that frame, with a confidence gate that keeps the weakest false positives out without dropping real, distant CCTV detections. * **Audio capture recovers instead of going quiet** - A half-open audio stream delivers no data, no end-of-stream, and no error, so the upsell listener could sit wedged for a day with zero errors logged and not one event captured. A watchdog now tears the capture down after 20 seconds of silent delivery and lets the existing reconnect loop take over. A box already stuck picks this up on its next update or a container restart. * **A box can be dedicated to upsell audio** - An edge box can now be installed for audio only, skipping video detection, streaming, previews, and clips entirely, so upsell detection can run on a low-compute box instead of competing for cycles with a chronically saturated video box. Boxes default to running both, as before. * **Every speech window records a verdict** - The upsell classifier now logs its decision for all speech it hears rather than only the upsells, so "is the audio pipeline hearing and classifying at all?" is answerable from the logs instead of by inference from an empty event list. Transcripts stay out of the logs unless explicitly turned on. #### Workspace Health * **A failing smoke test no longer restarts Oxygen Factory every 20 minutes** - The hourly health pass recorded a workspace's smoke run only after it finished, so a connection probe that took the process down left the workspace due again immediately and the same fatal probe ran on every restart — roughly three restarts an hour, indefinitely. The smoke run is now recorded when it is attempted rather than when it succeeds, so a probe that can't complete costs at most one interruption per smoke interval (six hours by default) instead of one per pass. * **MotherDuck connections are reused across queries** - A MotherDuck workspace opened a fresh connection for every query, paying a network handshake and an extension load each time and holding multiple independent handles on one database — the pattern behind the crashes above. Connections now go through the shared pool like every other warehouse, so queries start faster and a rotated token replaces the pooled connection rather than accumulating alongside it. Two related notes: recording the attempt bounds a crashing probe rather than eliminating it, since a probe running inside the process can still take it down; and a workspace whose evaluation never completes still displays its last known status rather than saying the pass never finished.