> ## Documentation Index
> Fetch the complete documentation index at: https://oxy.tech/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Pipeline reference

> The keys in an .airway.yml file, the supported source kinds, how credentials resolve, and how runs work

An Airway pipeline is a `.airway.yml` file. Unknown top-level keys are a
parse error rather than being silently ignored, so a typo surfaces as soon as
the file is compiled.

The `.airway.yml` extension is required, because the workspace only picks up
files with that exact suffix; a pipeline saved as plain `.yml` is silently
ignored.

## Example

```yaml theme={null}
# pipelines/quickbooks_financials.airway.yml
name: quickbooks_financials
source:
  kind: quickbooks
  config:
    client_id: <client-id>
    client_secret_var: QB_CLIENT_SECRET   # a secret key, never a value
    refresh_token_var: QB_REFRESH_TOKEN
    realm_id: "<realm-id>"
destination:
  database: warehouse                      # a database from config.yml
  dataset_name: quickbooks_financials
concurrency: 2
```

## Keys

| Key | Required | Default | Description |
| - | - | - | - |
| `name` | yes | | The pipeline's name, which must be non-empty |
| `source` | yes | | `{ kind, config }`, where `kind` picks the connector and `config` holds its settings |
| `destination` | yes | | `{ database, dataset_name }`, naming a database from `config.yml` and the dataset (schema) to load into |
| `description` | no | | Free text |
| `resources` | no | all | A subset of the source's resources to load |
| `concurrency` | no | `1` | How many resources extract in parallel, up to `16` |
| `allow_concurrent_runs` | no | `false` | Lifts the one-run-at-a-time guard; set it only on a pipeline with no cursor and no replacing tables |
| `streaming` | no | `true` | Streams rows to the destination, falling back to bulk loading automatically |

Note that omitting `concurrency` means one resource at a time rather than
unlimited parallelism, so a pipeline with many resources that leaves it unset
runs correctly but slowly.

## Source kinds

Oxygen supports the following `kind` values. The field is free text, so an
unsupported kind parses cleanly and fails only when the pipeline runs, with
an `unsupported source kind` error.

| Category | Kinds |
| - | - |
| Restaurant and commerce | `toast`, `ubereats`, `sp_api` |
| Accounting and ERP | `quickbooks`, `netsuite` |
| Databases | `sql_database`, `postgres_cdc`, `clickhouse` |
| Files and sheets | `http_file`, `filesystem`, `google_sheets` |
| Public and geographic data | `weather`, `besttime`, `overture`, `overpass` |
| Anything with an HTTP API | `rest_api` |

`rest_api` is the general-purpose connector: most vendor APIs without a
dedicated kind can be described directly as a `rest_api` config, so a
missing kind is rarely a blocker.

## Credentials

Credentials live in the workspace's secret manager and never in the YAML. A
config key ending in `_var` names a secret **key** rather than holding a
value, and Oxygen substitutes the secret's value at run time:

```yaml theme={null}
source:
  kind: toast
  config:
    client_id_var: TOAST_CLIENT_ID
    client_secret_var: TOAST_CLIENT_SECRET
```

Each kind manages its own set of `_var` keys, and those sets do not overlap,
so copy them from an example for the same kind rather than borrowing from
another. `rest_api` nests its credentials under `auth` instead:

```yaml theme={null}
source:
  kind: rest_api
  config:
    auth:
      type: api_key_query
      key_var: CENSUS_API_KEY
      param: key
```

## How a run works

* **Queue.** Only one run of a given pipeline is active per workspace. A run
  submitted while another is busy waits for it, and repeated submissions join
  the run already queued, so ten clicks of **Run now** produce one run.
* **Extract.** The run reads only what changed since the cursor, extracting
  the source's resources in parallel up to the pipeline's `concurrency`.
* **Load.** Rows are loaded into the destination, and the cursor advances once
  the load has committed. New tables and columns are added automatically, but
  a column is never dropped or narrowed in place, so a pipeline sitting on a
  wrong schema needs an explicit **Reset schema**, which drops its destination
  tables and cursor so that the next run starts fresh.
* **Finish.** A run ends as `completed`, `completed_with_errors`, or `failed`.
  A run that loads some resources but fails on others finishes as
  `completed_with_errors` rather than as a failure, so alerting should watch
  for both.

## Triggers

Pipelines can run on their own with **Run now**, from a schedule, or as a step
in an [automation](/docs/guide/build/automations/task-types#type-airway), where the
step finishes only once the load has committed, so that the tasks after it
read fully loaded tables.

## Destinations

Pipelines load into a database declared in `config.yml`. In practice this is
almost always an [Airhouse](/docs/guide/build/data-infrastructure/airhouse)
database declared as `airhouse_managed`, which mints credentials at run time
so the pipeline never holds a warehouse password; Postgres destinations are
also supported.

## Next steps

<CardGroup cols={2}>
  <Card title="Airway overview" icon="arrow-right-to-bracket" href="/docs/guide/build/data-infrastructure/airway">
    How pipelines run, retry, and evolve their schema
  </Card>

  <Card title="Automation task types" icon="diagram-subtask" href="/docs/guide/build/automations/task-types#type-airway">
    Run a pipeline as a step in an automation
  </Card>
</CardGroup>


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.