Frontline sign-in shipped as a model last release with no door in front of it: nothing could enrol a worker, grant them an app, or reset a forgotten PIN, and a location’s roster came back empty on every deployment — indistinguishable from a store that had simply not enrolled anybody. It works end to end now, and a manager sets a store up from Settings → Organization → Crew rather than through the API:
Enrol a worker from Settings - Name, identifier, PIN and the apps they should reach, in one step. The roster lists each worker’s standing, which of the organization’s apps they hold, and whether they are currently locked out.
A PIN only works on an enrolled kiosk - Enrolling a tablet mints a one-time link that expires in a day; opening it shows a confirmation page, and confirming binds that device. Anywhere else a PIN is refused exactly as a wrong PIN is — same response, same delay, counted against the same budget — and the roster comes back empty, so a PIN stops being a credential anyone can present from anywhere. This closes the network, not the tablet: a device someone is holding is still a device someone is holding.
“Who’s on shift?” - On a bound kiosk the login page opens with the location’s roster as large tap targets, instead of offering magic link and SSO to someone who has no mailbox. A wrong PIN clears the field with one sentence, repeated failures hold the button for a minute, and a successful sign-in lands on the app that kiosk was enrolled for.
Reset a PIN, suspend a worker, reinstate them - A reset issues a new PIN and clears any running lockout, because a forgotten PIN is usually discovered by guessing wrong and a new PIN that still refuses reads as wrong too. Suspending revokes the login, drops the name from the kiosk’s picker and takes app access with it — while keeping the row, so who completed which inspection, submission or training stays on the record. Reinstating restores the same PIN rather than making a worker learn a new one.
Granting a worker an app finally has a door - Apps can be granted at enrolment or later from the app’s own access settings, which until now refused any grantee who was not an organization member — which a frontline worker is never, by design. A suspended worker is refused like an outsider.
Kiosk rows flip to Bound while you watch - The kiosk list refreshes while an enrol link is live and unspent, so a row reads “Waiting for the tablet” and then Bound the moment the device binds. Revoked kiosks stay in the table as the record of which tablet a shift was signed in on.
Worth knowing: login attempts are budgeted per organization rather than per kiosk. Crew also no longer waits on workspace setup — a brand-new organization can enrol its first worker before it has a workspace at all (below).
An app that models work assigned to people had no way to name one, so every such app grew its own user ids and held no human name at all — a task dashboard broke down by role because a person was not available, and a site visit dropped its “who conducted it” column rather than guess. ctx.org.people() ends that:
A read-only directory, with no arguments - The organization is the host’s, so an app’s manifest cannot point it at another tenant’s roster. Capability-gated and refused by default, like secrets, email and storage.
Names, not contact details - A display name and a role; no email, no phone. Naming a colleague and contacting them off-platform are different needs, and only the first was blocking anything.
The crew is in it too, scoped to who can reach the app - One rule covers both halves: the directory names people who can reach this app. For office staff that is organization membership; for a frontline worker it is the explicit grant that is already the only way they reach anything. A worker granted the app next door is absent, and a suspended worker leaves the directory along with their login.
kind tells the two apart - Every person carries member or frontline, so an app that must not name a worker — a document’s approver, say — refuses on the field rather than by convention, and one that should does not have to union two lists by hand.
oxy publish shipped code; tables arrived by someone running SQL by hand against a script that re-applied every file on every pass, with nothing recording what had already run. Renaming a plan row and re-running the seed left the old row sitting beside the new one; re-running a migration overwrote text an author had since edited. Migrations are part of the bundle now:
Declared in the manifest - A migrations directory in oxy-app.json. Files run in lexical order, once each, in a transaction, as the app’s own database role — so containment is enforced by the database rather than by reviewing the file.
A file that already ran is never re-run - Refused two ways: the same name carrying different bytes, and different bytes under a new name. The second is the one that bites, because a seed with ON CONFLICT duplicates rows quietly instead of failing with “already exists”.
A mismatch refuses the whole plan before any file runs - So a promote never fails with the tenant half-changed.
Two concurrent promotes cannot both apply - The plan is re-checked under a per-app lock, rather than letting both read “one file unapplied” and finding out from a unique-key violation after the schema change already ran twice.
Worth designing around: making a build live from the admin console does not apply migrations, so publishing with --no-promote and then promoting from the console can put new code in front of tables that were never created.
A custom-app host answers 200 with the app shell for every path, so an uptime monitor pointed at an invented /health is green forever — including while the app it is watching renders a white screen. Every request for every app already terminates on the platform, so availability is now a query over what actually happened rather than a probe:
Every served request, function invocation and client beacon is recorded - With the outcome stored rather than inferred from a status code, because the failure class this exists to catch is a 200 carrying the wrong thing. A status >= 500 rule would report a white-screened app as perfectly available.
A 4xx does not dent the score - Otherwise an app raises its own availability by deleting its 404 handler. Rate-limited and timed-out requests are the platform declining to serve and do count; a success that took too long is a failure; an invocation cancelled because someone navigated away is not.
An idle app returns no opinion, never healthy - “Nobody is using it” and “it works” are different facts. A traffic floor keeps a two-request blip on a quiet app from paging anyone at 3am.
A verdict that stops when the incident does - Two windows: the long one decides whether something is wrong, the short one whether it is still wrong. A recovered incident stops alerting instead of firing for hours afterwards and getting muted, and a brand-new spike waits for the long window to agree, so a page means “sustained”.
Readable per app - An availability endpoint serves the windows and the verdict. It does not replace the existing health check, which is a deployment-integrity ladder — a green one is entirely compatible with every request to the app failing.
Alert delivery is deliberately not wired yet: the verdict is served, and nothing pages anyone on a schedule.
Function Logs Are Durable, and One Id Joins a Request
ctx.log() and console.* output from a function invoked over a route was dropped from observability entirely — it reached the server’s own output and nowhere else, while the scheduled path kept its own. The two halves agree now, and the lines survive the request:
Route-mode logs are stored - They previously existed only inside the HTTP response carrying them, so a browser that navigated away took them with it. Kept for 30 days.
A background run’s logs are kept too - A run with no event channel — an Airway step, a worker running outside a coordinator — dropped its entire log output, with nothing recording that it had.
One id names a request everywhere it shows up - Every response carries a request id, and a served request, the function invocations under it, and their log lines all name the same one. An inbound value is ignored and replaced: it is a join key, not an input.
Every invocation gets a span - Carrying app, organization, project, build, function and mode, closing with status and duration — including on the paths that fail before the function body runs.
analytics: false now silences product analytics only. Two platform-health signals — a
boolean saying the bundle mounted, and error names and counts — are sent regardless, so an
operator can answer “is this app up” without depending on how someone else configured their
bundle. The line is drawn where health costs no privacy: app-ready carries no path, and
pageviews stay opt-out-able because a list of screen paths genuinely is product analytics.
Existing builds published with analytics: false will begin emitting these two events.
The Metric Tree decomposes a top-line metric into its drivers. Scenario mode asks the other direction — change a lever, and see what else moves:
Levers you can drag and re-drag for free - One batched warehouse query values the levers and everything reachable forward from them. Editing a lever re-runs the propagation locally, so exploring a scenario does not cost another round trip.
A metric that cannot be valued says so instead of showing 0 - Unvalued nodes are classified and rendered as unknown, because a fake zero on a canvas is indistinguishable from a real one.
The forecast ships with the evidence it works - On real data a converged answer and a confidently wrong one look identical, so the forecast is measured against declared worlds whose true parameters the model is never told — deliberately including the ones where it loses. Simulated rows land in a real table the semantic model queries normally, so there is no simulation-only path through the thing being tested.
Amazon Selling Partner Backfills Over a Date Window
Amazon caps a shipments report at one month and does not reject a request for more — it accepts the job, hands back a report id, and never finishes it. A pipeline asking for a longer window sat at “in progress” for the better part of an hour, run after run, and never landed a row:
A date window is now a supported backfill - Amazon Selling Partner joins the other sources that accept one, rather than being refused outright.
A long window is split into report-sized pieces - So an over-range request can no longer be built at all, including the 31-day month that came out one day past the cap.
A retry resumes the report Amazon is already building - The report id is persisted, so an interrupted run polls the existing job instead of paying tens of minutes of build time for a new one.
The window comes from the backfill control, not the pipeline file - A start and end hand-written into the YAML are ignored on ordinary and scheduled runs, so a pipeline cannot be pinned to a window its live cursor can never advance past.
The physical world an organization operates in — its stores, its regions, who works at which one and in what position — lived nowhere in particular, so every app that needed it grew its own tables of locations and staff, and the semantic model knew a store only by whatever identifier the point-of-sale system used. It is one registry now, owned by the organization and readable by every app:
Locations, in whatever shape the organization actually has - Nest regions, districts, stores and floors to any depth, naming the levels yourself rather than fitting a fixed hierarchy. Managed from Settings → Organization → Locations; a parent that would create a loop is refused at the point of saving.
One place, however many systems call it something else - A location carries an identifier map — what the point-of-sale calls it, what payroll calls it, what a camera console calls it — so a store known by three different ids reconciles to a single row. One id per system per location, enforced in both directions.
Positions and assignments - Define the organization’s positions, then assign an org member or an active frontline worker to a position at a place, optionally under a supervisor. Crew’s enrolment, worker and kiosk dialogs pick position × place, so a tablet now belongs to a store rather than floating free.
Reach decides what a person sees, and the platform decides reach - An assignment is resolved into the set of places a caller can act on before any app code runs: an admin or an organization-wide position reaches everywhere, an assigned person reaches exactly their places, and a frontline worker with no assignment reaches nothing. App-side helpers can narrow that set, never widen it, so a screen cannot promise what the server will refuse.
Semantic queries can be pinned to the caller’s places - A view’s primary entity declares which registry system its key belongs to, and a query asked with scope: "reach" is filtered to the caller’s locations before any row limit applies. A query that names no bound view is refused outright rather than quietly answered whole — a half-mapped model shows up the day it is wrong, not the day someone scoped notices. Where a key is mapped, World Model instances and Toast order events carry the location’s name alongside its id.
Apps read the registry instead of rebuilding it - ctx.org.places() and ctx.org.assignments() sit behind the same capability gate as ctx.org.people() and are read-only. Places come back whole, since an app must know a location exists before it can decide who reaches it; assignments are scoped to the people who can reach the app, by the same rule that governs the directory.
A custom-app function that was slow or failing could be seen from the outside — a duration, a status — but not followed. Each ctx.* call it made was its own disconnected record, so “which of the four warehouse queries took the nine seconds” had no answer. An invocation is now a single trace, from the click in the browser to the query in the warehouse:
The trace starts in the browser - The client SDK names the trace before the request leaves the page, so a call that never returns still identifies the exact server-side trace, and a failed invoke hands back that id along with the request id. The error beacon carries it too, which means an uncaught rejection can be traced back to the call that caused it.
Every host call is timed individually - Warehouse queries, outbound HTTP, transactional Postgres, semantic queries and outbound email each appear as their own step with duration and, on failure, an error class — so a slow invocation shows which call was slow rather than only that it was.
Shape, never payload - A query step records the operation and the table but never the SQL text; an HTTP step records scheme, host and port but never the full URL, which is where API keys travel. Content stays in the tenant’s own store, behind the app-admin gate it already had.
Logs are addressable by request - The function logs endpoint returns a trace id and accepts a request id as a filter, so the lines for one click can be pulled out without reading around them.
A scheduled run links back rather than nesting - A job queued by a request is joined to it in both directions but keeps its own timeline, so a click that queues an hour of work does not read as an hour-long click. A timer-fired run is an honest root.
Neither data plane could say who wrote a row. The transactional Postgres provider offers no self-serve statement auditing and clears its query statistics whenever a database suspends; a warehouse commit records whatever author the writer claims. The one place every ctx.oltp, ctx.tx and ctx.warehouse write passes through is the function host, so the host is now the record — in three layers that each stand on their own:
A statement can be traced back to the invocation that sent it - Every transaction names itself to the database, so a provider’s live-query views and a slow-statement report identify the invocation behind a statement instead of showing an anonymous connection.
Every SQL statement carries its origin - App, function, invocation and trace id ride along as a trailing comment, on writes to transactional Postgres and to the warehouse alike. It survives into the database provider’s own logs, which is where you end up looking when the platform’s records are the thing in question.
A committed write leaves an audit row - The verified user on a route, or the app itself on a schedule, an Airway step or a manual run; the table written, the operation, the row count, and the invocation, request and trace ids. A ctx.tx() handle records one row for the whole transaction, and a rollback leaves nothing behind.
What the statement does decides, not which method was called - ctx.oltp.query("INSERT … RETURNING") is audited; ctx.oltp.exec("SELECT …") is not.
Table and operation, never SQL text or values - The same reduction the host-call steps already use, so an audit trail cannot become a second copy of the data it describes.
Audit events are hash-chained per organization, but a chain only shows that rows were not edited if the head of it is trusted — and anyone who can rewrite rows can rewrite the head. Two additions make the chain evidence rather than a promise:
Append-only at the database itself - Updates and truncation of the audit table always fail, regardless of who is asking. Deletion is refused too, except for the daily retention prune, which has to declare itself to proceed.
Chain heads anchored outside the database, hourly - Each organization’s current position and hash are written to object storage under a compliance-mode lock: undeletable and un-overwritable for the retention period, by the bucket’s own owner included. Anything up to the last anchor can be shown intact — or shown not to be — without trusting the database it came from.
A verifier an admin can call - An admin endpoint fetches the latest anchor and reports whether the anchored point still exists and still hashes the same. Both true means nothing up to that anchor was rewritten or removed.
Identifiers and hashes only - No audit row, and no data from one, is ever written to the bucket.
Off until a bucket is configured - The endpoint reports plainly that anchoring is unconfigured rather than implying verified silence, so an unset deployment does not look like a verified one.
Every model call now emits its span in the OpenTelemetry GenAI conventions, so what a monitoring tool sums and what the tenant-facing Traces console prices are the same numbers:
Token usage, broken out - Input, output, cache-read and cache-write tokens on every call, alongside the finish reason and time to first chunk — enough to tell a slow model from a slow first token, and a cache hit from a cold prompt.
Every call path is instrumented - One of the three ways the platform calls a model previously emitted no span at all, which quietly under-reported spend and latency for the agents that used it.
Provider and endpoint are on the span - Which vendor answered and which host and port served it, so a call routed to a regional or self-hosted endpoint is visible as such rather than lumped in with everything else.
Which agent, which workspace - Conversation, agent name, organization, workspace and project where known, so cost and latency break down by the thing that caused them instead of only in total.
Prompts and completions stay out by default - Capture is opt-in per deployment and size-capped, recording tool names rather than arguments. Off by default because these spans land in the tenant-visible store.
An app had no way to end the session it was being viewed in. On a shared store tablet a crew member stayed signed in for the twelve hours a PIN session lasts, so the next person on shift worked as them — and the app in front of them could offer nothing but “close the browser”:
signOut() in the SDK - Clears the session on the server and returns the viewer to the login page with the app named as where to come back to. On an enrolled kiosk that page is “Who’s on shift?”, so the next PIN signs straight back into the same app as that person.
The login page is named by the platform - links.login joins home, threads and settings on the shell context. On a custom-app subdomain the login page lives on the product host, which a bundle has no way to work out for itself.
A sign-out that did not work never navigates - If the session was not actually cleared the call fails rather than sending someone to a login page while they are still signed in as somebody else.
signOutUrl() for apps that route themselves - The address without the navigation, for a bundle that would rather hand the destination to its own router.
Available in SDK 2.13.0. An app built against an older server falls back to /login on its own origin.
The Amazon Selling Partner source read Seller Central only. Vendor Central is a separate Amazon account with its own authorization, and its reports were not merely unconfigured but unreachable — a pipeline that tried to name the account type failed to parse at all:
partner_type: vendor on the source - Opens the three Vendor Central reports: vendor sales, vendor inventory and vendor forecasting. Amazon serves the forecast as a JSON document rather than a flat file, which the connector now reads, so that report lands rather than arriving empty.
The pipeline wizard asks which account - An Amazon account selector on the Selling Partner step, and the generated file states the account either way rather than leaving it to a default. Switching the selector re-points the secret names — a Vendor Central refresh token is not the seller’s — and clears any credential already pasted in, so a vendor pipeline built after a seller one cannot quietly run on the seller’s token. A name you typed yourself is never overwritten.
One account per pipeline - The account type belongs to the credential, not the run, so Seller and Vendor Central need separate pipelines rather than one naming both sets of reports. A pipeline sees only the reports its configured account can reach and refuses any other by name, saying which account owns it — so a misconfiguration reads as a refusal instead of pulling the wrong data.
Existing pipelines are untouched - The setting defaults to seller, and every Selling Partner pipeline already running behaves exactly as before.
A service that only pushes had nowhere to deliver to — Uber Eats’ Reporting API is webhook-only, with no endpoint to poll — so receiving one meant running a flow outside Oxy and moving the result back in. A function can now be published as a webhook endpoint, with the platform proving the sender before any app code runs:
Declared in the manifest, which names the secret and never holds one - A webhook block on a function names the signing-key secret, the header the signature arrives in, and its encoding. The address is /api/webhooks/apps/{organization}/{app}/{function}.
The platform verifies, not the function - The signature is checked, in constant time, before the request reaches app code. A check written inside a function would work, but an unverified request would already have arrived — and a forgotten one would be silent.
A rotation does not drop deliveries - The named secret accepts several keys, comma-separated, because a provider that keeps two live signing keys signs with either one during a rotation. Accepting only one silently loses half the events for the length of the overlap.
Fails closed, and says as little as possible - An unknown function, a function with no webhook block, and an app with no published build all answer 404 rather than “wrong signature” — the route is anonymous, so a more specific answer would turn it into a directory of every function an app has and of which apps exist. A declared function whose signature is missing or wrong answers 401.
A verified delivery is acknowledged straight away - The sender gets a 202 the moment it is proven and the run is queued durably, so a provider’s retry policy is not tripped by work that takes a while. Only published builds are reachable; a draft is unreviewed by definition and this address is reachable by anyone.
A webhook run knows it is one - The function reads webhook as its invocation mode, distinct from a manual run and from a scheduled one.
Worth designing around: a webhook run takes the system path, so its managed Airhouse credential is read-only and cannot write to the warehouse. It can write app storage — so land the raw payload there and let an Automation, which does resolve a role, do the warehouse write.
Per-App Secrets, and a Panel That Says What Is Missing
ctx.env has always resolved secrets scoped to a single app, but nothing could create one: the Secrets settings page rejected the name shape those keys use, and the only writer was ctx.secrets.set from inside a function — which cannot create the key the function needs in order to run at all. A webhook block was the sharp end of it, answering 401 on every delivery with no surface anywhere naming the key to fill:
An app declares the keys it expects - An env block in oxy-app.json lists key names only, never values, and marks which are required. Required is advisory — it flags a key as missing and never blocks a publish, because the first publish is exactly when nobody could have set one yet.
Declared and stored, reconciled in one view - The app’s Secrets panel lists every key the manifest declares alongside every key actually set, with a missing count on the collapsed header, so a blank one is visible before it becomes a failed delivery. Each function’s webhook signing key is folded in as a required key without being declared twice.
Set, reveal, rotate and delete - Values are created from the panel and revealed on demand rather than held in the browser’s cache, where a decrypted secret would outlive the row it came from. A rotation reaches a function on its next invocation, within five minutes.
App-scoped keys are legible in workspace settings - The workspace Secrets table now shows an app’s keys under their bare name, attributed to the app that owns them, and its create dialog gained an “Available to” scope so a key can be aimed at a single app.
One app’s keys are invisible to another - And an admin reaching for an app that belongs to a different organization gets a not-found rather than a refusal, so the surface cannot be used to probe which apps exist.
A malformed env block degrades to “declares nothing” and reports the parse error in the panel — it never fails a publish, but it does not stay quiet either, since a typo would otherwise look exactly like an app that declares no keys.
A tracked event names the app it belongs to - Activity events were attributed by workspace, and a workspace can publish several apps, so a click in one app could be recorded against another’s Activity tab. Every event now carries the app’s own id, honoured only if that app is published from the workspace serving the request. Republish to pick it up; an older bundle falls back to a deterministic choice rather than an arbitrary one.
A frontline worker can invoke an app’s functions - The gate behind the app shell and every function invoke had no frontline term at all, so a worker enrolled specifically to use an app got the shell and a 403 from every function inside it. The data plane had the matching defect, comparing a grant against the wrong kind of id, which meant a worker’s grant was invisible on every healthy request. Suspending a worker now takes effect immediately rather than a cache expiry later.
ctx.user.email is typed as nullable - The runtime has sent null for a frontline worker since they existed; the SDK type said string.
An enrol link points at the host the request came from - Creating a kiosk from the command line returned a localhost link, because link building read only headers a browser sends. The response also carries the bind path now, for a client that would rather compose the URL against its own origin.
A kiosk’s sign-in page names the store it belongs to - Now that a kiosk is enrolled at a place, the sign-in screen says which one, rather than presenting an unlabelled roster.
A new organization can set up its crew before it has a workspace - Settings were reachable only from inside a workspace, so a store operator had to create an empty workspace before enrolling a single worker or tablet. The onboarding screen now offers “Running stores? Set up the crew first” beside the workspace card and opens settings directly at Crew. Both options are admin-only; a member who joined by invite still waits for an admin.
Any settings section can be linked to directly - A ?settings= parameter opens the dialog on the named section — ?settings=organization.members, for instance — and strips itself from the URL afterwards, so a link shared into chat lands where it says it does. A section the viewer cannot see falls back to the first one they can rather than failing.
Amazon ledger summary lands at the daily grain it documents - The report options Amazon needs were never sent, so ledger_summary came back as a single monthly period regardless of the window asked for. It now arrives a day at a time, as its roster entry has always said it would.
Amazon shipments rows merge on a measured key - Shipments were combined on a key derived from the schema rather than one confirmed against real documents, which could fold two distinct shipments into one row. A row arriving with no key is now refused outright instead of merging into whatever it collided with.
NetSuite loads no longer exhaust a worker’s memory - The NetSuite load path allocated roughly 215 MB per batch — enough to have workers killed at a 500 MiB memory limit, which took out three pipelines in a single morning on one deployment. Peak usage is now 1.03× the size of the data being loaded, down from 2.00×, and emitted batches are capped at 5,000 rows.
An execute_sql step can run its .sql file on any worker - An ingest-then-rollup chain — the standard gold-layer shape — failed with “failed to read SQL file … No such file or directory” whenever the step ran on a worker rather than the Developer Portal instance. Every child task succeeded and the parent died on a filesystem read: the pipeline’s own definition came from the promoted workspace, but the .sql file it delegated to was still read from disk, which a worker does not have. Those files are read from the promoted workspace now, and a node holding no workspace files says so — a state a retry elsewhere can recover from — instead of surfacing a missing-file error that looked permanent.
Objects under a nested build directory are no longer listed - target, node_modules, dist and build were skipped only at the root of a workspace, so a stray copy of an agent or a Data App under a nested build output was listed as a real object — one source of the “duplicate name” errors that had no duplicate in the source tree. The rule applies at any depth now, and the Developer Portal and the serving fleet apply the same one, so a file only one of them listed no longer resolves in the IDE and 404s when served.
A workspace is judged on what was promoted, not on whatever is on one node - The health evaluator was the last semantic reader still going to the filesystem, and it runs on a node that holds the workspace record but not its directory — so a workspace whose semantic model every product surface queried happily alerted unhealthy for weeks, with an error naming a path that node was never going to have. The alert was about the node, worded as an alert about the workspace. Health now reads the compiled model, and deliberately will not fall back: a check that graded whichever uncommitted draft happened to sit on the node it landed on would page the on-call about a promoted revision that is fine. A workspace with nothing compiled reports its semantic checks as unavailable — Degraded, meaning “we learned nothing” — enqueues a compile so the next pass succeeds unattended, and still evaluates every other dimension.
Anomalies are displayed but no longer decide the verdict - An open high-severity anomaly used to make a workspace Unhealthy, which pages ops on the transition. Anomaly severity measures how far a number sits past its seasonal band, so a marketing push, a holiday or a new location scores High while nothing is broken, and a busy Saturday was indistinguishable from a dead worker fleet. Recovery needed human triage as well — a workspace stayed Unhealthy until someone acknowledged the anomaly in the Insights Inbox, while the re-alert clock kept firing every six hours — so the pager rang on a queue only the customer could drain. Anomaly counts stay on the Health tab, marked informational.
Worth knowing: reconcile.yml, which compares an Oxy measure against the live external source, is now the only data-correctness dimension. The five remaining dimensions are all infrastructure liveness, so a workspace with no reconcile.yml carries no correctness signal at all — correctness coverage is explicitly opt-in.
A busy trace store no longer wedges itself - Observability event tables were filed a month at a time, so a month of spans grew into pieces of millions of rows. Compacting them read hundreds of megabytes at once, which exhausted the memory the store was given, which failed the compaction — which was then retried against the same oversized pieces. The failing work was itself the cause of the pressure that failed it, and the end state is a trace store that can no longer answer a row count over its own table. Events are now filed a day at a time, which is small enough that compaction stays well inside its budget. Measured against a healthy instance running the same image, daily filing showed no compaction failures across 2,646 attempts, against roughly half failing on the monthly one.
Retention drops traces on the day they expire - The 90-day rule only ever removed a stored chunk once every trace inside it had aged out, so with a month of traces in one chunk the effective retention drifted as high as ~120 days. Days now expire on their own, which is both the retention you configured and materially less storage on a busy workspace.
Health checks no longer fill the trace store - Readiness, liveness and health probes each produced a span and a log line, and they run every few seconds per instance: on a four-instance deployment that is roughly 80,000 requests a day arriving ahead of any user traffic, and the Traces console was showing hundreds of thousands of probe rows a day — very nearly all of what was in it. Probes now record neither, so what is left is real traffic. A probe that actually fails still logs its error, which is the only part of a probe anyone wanted to read.
Only the paths a probe actually reaches are silenced - The silenced list named bare /health, /ready and /live, but the API is mounted under /api, so those paths are 404s in the served shape — meaning a monitor pointed at one would have gone unrecorded in every log and trace, and a probe aimed at the wrong path is exactly the misconfiguration worth catching. The list is now the three paths the server serves, and silence is conditional on success: a probe that returns anything else says so, with its status and latency. That same list is what load shedding exempts, so probes are no longer turned away by an instance shedding traffic — the previous list never matched a real path, so the exemption had never once applied.
The Traces console keeps the spans it actually shows - The tenant-facing store was recording every HTTP request plus framework internals — retry attempts, ClickHouse inserts, the AWS credential chain — on the order of 660,000 rows a day on a busy deployment, none of which the console reads: it roots on agent and automation runs. Those targets are dropped now, so the store holds the traces the console is built to display.
Existing deployments keep the old layout until they are migrated. How events are filed is
fixed when a table is first created and cannot be altered in place, so this change takes effect
on new deployments only. An existing trace store keeps monthly partitions — and keeps the
behavior above — until its tables are rebuilt. If your observability backend is showing failing
merges or a growing backlog, contact us before migrating: on an already-struggling instance the
rebuild is itself the expensive step and is best done after the backlog has drained.
Diagnosing a running deployment meant reading log lines that carried no way to tell which request they belonged to, grouped under a single generic entry per HTTP call that no tracing backend could break down by route. For anyone running Oxy, that changes:
Every log line names its request - Trace and span ids are on every line and every span emitted by serve, start and worker, with the service name alongside them, and they propagate across internal hops. This holds whether or not a tracing backend is configured — the ids are there to group by.
Export to an OpenTelemetry collector - Setting OTEL_EXPORTER_OTLP_ENDPOINT (OTLP over HTTP, the collector’s 4318, not the gRPC port) exports traces. Log export is a separate opt-in, OTEL_LOGS_EXPORTER=otlp, deliberately off by default because a collector that already tails standard output would otherwise store every line twice. Export verbosity is set independently of console verbosity.
HTTP spans you can group by route - One span per request, named by method and route and following the OpenTelemetry HTTP conventions. Query-string values and secret-shaped path parameters are redacted before anything leaves the process.
Startup failures are reported, not just printed - A failure to bind a port or load a TLS certificate now reaches error reporting and the exporters instead of ending the process silently. The exit code and console message are unchanged.
Debug logging shows Oxy, not its dependencies - Turning verbosity up buried the application under its own libraries: the OpenTelemetry exporter narrating each of its uploads and the AWS SDK walking its credential chain together accounted for around 12,000 lines an hour, before Oxy wrote anything. Those targets are held at warn in every filter the log configuration is built from, so OXY_LOG_LEVEL=debug now shows the application. The exporter’s own chatter is still reachable deliberately, through RUST_LOG or OXY_OTEL_FILTER.
Heartbeat lines are no longer written at info - Recovery sweeps, coordinator and worker lifecycle, camera ingest flushes and GitHub token refreshes each logged on every tick, so an idle deployment still produced thousands of lines an hour reporting that nothing had changed. They log at debug now; the lines that mark an actual change of state — a task assigned, a run recovered, a task re-launched — stay at info.
The router health probe is a metric now, not two thirds of the log volume - One line, logged on every probe receipt, was 67% of everything the fleet wrote in production and 56% in dev. The cost is quadratic: every instance announces itself on a minute tick and every listener on every instance logged the receipt, so fourteen routers meant fourteen lines per tick each. It was deliberately kept at info so ops could alert on its absence — but the store those lines reach is an investigation surface that carries no alerting, so the two thirds bought an alert that could never be written. The signal is exported from the worker’s metrics endpoint instead, as a receipt counter and a last-seen timestamp, which is where the alert actually goes; the per-receipt line drops to debug. The timestamp is absent rather than zero before the first probe arrives, so a pod that has just started does not read as decades stale and page on every rollout.
A repeating failure warns once an hour instead of every tick - The camera alerter re-logged the same summarize failure for the same workspaces on every pass. It now warns once per hour per workspace and repeats at debug in between, carrying the throttle window as a field so a reader can see the suppression rather than infer it from a gap.
Console JSON logs are flat now. Event fields sit at the top level alongside message,
trace_id, span_id and service, instead of nested under a fields object. Anything parsing
the previous fields.message shape — a log pipeline, a dashboard query, an alert rule — needs
updating. Note also that the tenant-facing Traces console is a separate store with its own ids:
a product trace and the operator log lines emitted inside it do not yet correlate.
