Skip to main content
An Airway pipeline is a .airway.yml file. Unknown top-level keys are a parse error rather than being silently ignored, so a typo surfaces as soon as the file is compiled. The .airway.yml extension is required, because the workspace only picks up files with that exact suffix; a pipeline saved as plain .yml is silently ignored.

​
Example

​
Keys

Note that omitting concurrency means one resource at a time rather than unlimited parallelism, so a pipeline with many resources that leaves it unset runs correctly but slowly.

​
Source kinds

Oxygen supports the following kind values. The field is free text, so an unsupported kind parses cleanly and fails only when the pipeline runs, with an unsupported source kind error. rest_api is the general-purpose connector: most vendor APIs without a dedicated kind can be described directly as a rest_api config, so a missing kind is rarely a blocker.

​
Credentials

Credentials live in the workspace’s secret manager and never in the YAML. A config key ending in _var names a secret key rather than holding a value, and Oxygen substitutes the secret’s value at run time:
Each kind manages its own set of _var keys, and those sets do not overlap, so copy them from an example for the same kind rather than borrowing from another. rest_api nests its credentials under auth instead:

​
How a run works

  • Queue. Only one run of a given pipeline is active per workspace. A run submitted while another is busy waits for it, and repeated submissions join the run already queued, so ten clicks of Run now produce one run.
  • Extract. The run reads only what changed since the cursor, extracting the source’s resources in parallel up to the pipeline’s concurrency.
  • Load. Rows are loaded into the destination, and the cursor advances once the load has committed. New tables and columns are added automatically, but a column is never dropped or narrowed in place, so a pipeline sitting on a wrong schema needs an explicit Reset schema, which drops its destination tables and cursor so that the next run starts fresh.
  • Finish. A run ends as completed, completed_with_errors, or failed. A run that loads some resources but fails on others finishes as completed_with_errors rather than as a failure, so alerting should watch for both.

​
Triggers

Pipelines can run on their own with Run now, from a schedule, or as a step in an automation, where the step finishes only once the load has committed, so that the tasks after it read fully loaded tables.

​
Destinations

Pipelines load into a database declared in config.yml. In practice this is almost always an Airhouse database declared as airhouse_managed, which mints credentials at run time so the pipeline never holds a warehouse password; Postgres destinations are also supported.

​
Next steps

Airway overview

How pipelines run, retry, and evolve their schema

Automation task types

Run a pipeline as a step in an automation