<region>, <account-id>, and names like airhouse-data-<env> with your own. For the general installation steps and all deployment options, see Installation.
Architecture
Terraform: AWS resources
The snippets assume you already have a VPC and an EKS cluster, for example from theterraform-aws-modules/vpc and terraform-aws-modules/eks modules, referenced here as module.vpc and module.eks.
KMS key and S3 bucket
s3.tf
Airhouse’s maintenance jobs delete Parquet files that are no longer referenced. If you enable S3 versioning for extra protection, add a lifecycle rule that expires noncurrent versions. Otherwise deleted files keep accruing storage cost.
RDS for PostgreSQL
rds.tf
rds_iam.
bootstrap.sql
IAM role for EKS Pod Identity
One role serves both planes. It grants read and write on the bucket, use of the KMS key, and IAM database login for the two control-plane users.iam.tf
serviceAccount.controlPlane.annotations and serviceAccount.dataPlane.annotations, and use an IRSA trust policy.
Admin API token in Secrets Manager
secrets.tf
Kubernetes: secrets and Helm values
Sync the admin token with External Secrets
admin-token-external-secret.yaml
airhouse_cp and airhouse_catalog_admin passwords the same way, and reference them with controlPlaneDatabase.passwordSecretRef and catalogDatabase.adminPasswordSecretRef.
Helm values
values-aws.yaml
Install with Argo CD
airhouse-application.yaml
helm install airhouse oci://ghcr.io/oxy-hq/helm-charts/airhouse --version <chart-version> -n airhouse -f values-aws.yaml.
Client access and TLS
-
Clients inside the cluster connect to the HAProxy Service on port
5445, for examplehost=airhouse-haproxy.airhouse.svc port=5445for a release namedairhousein theairhousenamespace. -
Restricting port 5445: the chart ships NetworkPolicies for the control plane and data plane only. HAProxy has none, so any pod in the cluster can reach port
5445(clients still need valid credentials).networkPolicy.allowedNamespacesdoesn’t change this: it only opens port8080on the two planes. To limit which namespaces can connect, add a policy like this one. It’s not part of the chart. Its pod selector matches the labels the chart puts on HAProxy pods:Add a rule for porthaproxy-networkpolicy.yaml5446if you enablehaproxy.analyticsBackend. If you expose HAProxy through a load balancer, allow the load balancer’s source ranges too. -
Clients outside the cluster: set
haproxy.service.type: LoadBalancerand add AWS Load Balancer Controller annotations for a Network Load Balancer. Prefer aninternalscheme, reachable over VPN or peering, to an internet-facing one: -
TLS for clients: HAProxy passes TCP through, so TLS terminates on the data plane. Enable it with a
kubernetes.io/tlsSecret and point the server at the mounted files:Enable TLS for any client traffic that leaves the cluster. Connections from Airhouse to RDS use TLS (sslMode: require), and connections to S3 use HTTPS (useSsl: true). -
Admin API: keep it private. Reach it from your backend inside the cluster (control-plane Service, port
8080), or withkubectl port-forwardfor manual administration. Don’t expose it through an ingress.
Operations
Metrics and alerts
Withmetrics.serviceMonitor.enabled, Prometheus or a compatible scraper (for example VictoriaMetrics’ agent) collects /metrics from both planes. The scraper’s namespace must be in networkPolicy.allowedNamespaces, or the chart’s NetworkPolicies block the scrape and none of the alerts below can fire. metrics.prometheusRule.enabled installs the chart’s alert rules, including:
AirhouseQueryAvailabilityBurningandAirhouseQueryAvailabilitySlowBurn: wire-protocol query error budget.AirhouseQueryLatencyP99HighAirhouseAdminAvailabilityBurningandAirhouseMintErrorRateHigh: Admin API and credential minting.AirhouseCircuitBreakerOpen: a pod is shedding all queries after repeated backend failures.AirhouseDataPlaneRestartandAirhouseControlPlaneLivenessRestartAirhouseQueryQueueSaturated,AirhouseWorkerPoolExhausted, andAirhouseQueriesSpillingToDisk: capacity pressure.AirhouseCPDBPoolSaturatedandAirhouseSweeperLag
airhouse:
airhouse-availability-rules.yaml
kubectl rollout restart of the data-plane Deployment clears it.
For RDS, CloudWatch alarms on DBLoad (above 50% and 100% of vCPUs), CPUUtilization (above 80%), DatabaseConnections, ReadLatency and WriteLatency (above 20 ms), FreeableMemory, and FreeStorageSpace are a good baseline.
Logs and traces
- Logs are JSON (
logging.format: json). Collect them with your cluster’s log agent, such as an OpenTelemetry Collector or Fluent Bit. Keeplogging.logStatementText: falseif SQL text may contain sensitive values. - Traces: set
tracing.enabled: trueandtracing.endpointto an OTLP/HTTP collector, for examplehttp://<otel-collector>.<namespace>.svc:4318.
Backups
- Catalogs and registry: RDS automated backups (14 days here) and snapshots cover every Airhouse database’s catalog and the registry. Keep
deletion_protectionon, and test restores periodically. - Data: Parquet files in S3 are immutable once written. S3 versioning, or replication to a second bucket, protects against accidental deletes. Restore the catalog and data to the same point in time, because a catalog references specific data files.
- Time travel window: snapshot expiry (below) sets how far back time travel can reach.
Maintenance jobs
DuckLake needs periodic maintenance: compacting small files, expiring old snapshots, and deleting files that are no longer referenced. By default the control plane runs this in process every 6 hours and keeps 30 days of snapshots (controlPlane.vacuum). The reference setup moves it into the chart’s vacuum CronJob, so a large merge can’t compete with the control plane for memory. The CronJob is the one job that expires snapshots and deletes files:
retentionDays to match how far back you need time travel and point-in-time restores.
The chart also ships a native compaction drain (compactionDrain) for working through a large backlog of small files. It’s a manual Job: keep it suspended, and trigger it on demand instead of scheduling it. Leave its expiry, cleanup, and reap switches off so it doesn’t duplicate the vacuum CronJob’s work:
dryRun: false makes the drain rewrite Parquet files, so make that a separate, deliberate change: read the dry-run report, pause the vacuum CronJob (vacuum.cronjob.suspend: true) so the two don’t work on the same catalogs at once, run one drain, then resume the CronJob.
Sizing starting point
To resize, keep
maxConcurrentQueries × memoryPerSessionMB + ~1 GB and maxTenantInstances × memoryPerSessionMB (which must stay under podMemoryBudgetMB) below resources.limits.memory, and keep maxConcurrentQueries × threadsPerSession near the CPU limit. For example, a 10 GiB pod fits 8 queries at 1 GiB. Watch AirhouseQueriesSpillingToDisk and pod memory. Add replicas for more concurrency, and raise pod memory together with memoryPerSessionMB for larger queries.