Deployment view · where each component runs

Deployment on Kubernetes (EKS)

A production setup on AWS. Airflow runs on Kubernetes (EKS) in a private VPC, and every task runs in a pod built from one Docker image that contains Airflow, dbt and Cosmos. Snowflake is SaaS in the same region and is reached over PrivateLink. GitHub Actions tests and ships every change.

Airflow runtime Build and release (dbt code, CI/CD) Snowflake Network connection

Figure 1 — Physical deployment on AWS. Numbered markers match the release and run steps below. On a phone, scroll the diagram sideways or use full screen.

Release and run, step by step

Develop. An engineer builds models locally against their own Snowflake schema (DBT_DEV_<USER>), then pushes a branch and opens a pull request.
CI checks the change. GitHub Actions runs SQLFluff, then dbt build --select state:modified+ --defer in a CI_PR_<N> database cloned from prod, so only changed models and their children are built and tested.
Build the image. After merge, the pipeline builds one Docker image with Airflow, dbt-snowflake, Cosmos and the dbt project (including a parsed manifest.json) and pushes it to ECR with a version tag.
Deploy. helm upgrade rolls the new image tag out to the scheduler, API server and triggerer on EKS. DAG files arrive with the image or through git-sync.
Scheduled run. The scheduler starts the DAG. Each Cosmos task runs in its own worker pod, reads the Snowflake private key from Secrets Manager, and runs dbt on TRANSFORM_WH over PrivateLink on port 443.
Observe. Task logs and dbt artifacts go to S3, metrics go to CloudWatch or Datadog, and failures page the team in Slack. Snowflake resource monitors cap credit use.

Where each component runs

Component	Runs on	Notes
AFScheduler	EKS deployment, 2 replicas	High availability; both replicas share the metadata DB
AFAPI server / UI	EKS deployment behind an ALB	SSO (Okta / Entra ID); never exposed without auth
AFTriggerer	EKS deployment	Runs deferrable sensors cheaply instead of holding worker slots
AFWorkers	Pods created per task (KubernetesExecutor)	Isolated; CPU and memory set per task; scale to zero when idle
AWSMetadata DB	RDS PostgreSQL, Multi-AZ	Stores DAG runs, task state and connections; back it up
AWSSecrets	Secrets Manager as the Airflow secrets backend	Snowflake key-pair, Fivetran API key; access granted via IAM roles (IRSA)
AWSLogs and artifacts	S3 ops bucket	Remote task logging; `manifest.json` kept for Slim CI; dbt docs hosted as a static site
dbtdbt Core	Inside the worker image, not a separate server	Only compiles SQL; Snowflake does the compute
SFSnowflake	SaaS, same cloud region as the VPC	PrivateLink plus a network policy that allows only the VPC endpoint
CIGitHub Actions	GitHub-hosted runners	Uses a CI service user with access only to `CI_PR_*` databases

Environments

Environment	Snowflake target	Who runs dbt	Warehouse
Dev	`DBT_DEV_<USER>`	Engineer, from a laptop	TRANSFORM_WH (XS in dev)
CI	`CI_PR_<N>`, cloned from prod and dropped after merge	GitHub Actions	CI_WH
Prod	`ANALYTICS`	Airflow on EKS, on schedule	TRANSFORM_WH (M)

Key deployment files

Dockerfile

FROM apache/airflow:3.0.6-python3.11

# dbt in its own virtualenv to avoid dependency clashes
RUN python -m venv /opt/dbt-venv && \
    /opt/dbt-venv/bin/pip install --no-cache-dir dbt-snowflake==1.9.*

RUN pip install --no-cache-dir astronomer-cosmos \
    apache-airflow-providers-snowflake

COPY dags/ /opt/airflow/dags/
COPY dbt/analytics/ /opt/airflow/dbt/analytics/

# pre-parse so Cosmos loads the manifest fast
RUN cd /opt/airflow/dbt/analytics && \
    /opt/dbt-venv/bin/dbt deps && \
    /opt/dbt-venv/bin/dbt parse --no-partial-parse

Pin exact versions in your own build; these are illustrative.

GitHub Actions (simplified)

on:
  pull_request:
  push: {branches: [main]}

jobs:
  ci:
    if: github.event_name == 'pull_request'
    steps:
      - run: sqlfluff lint dbt/analytics/models
      - run: aws s3 cp s3://ops/dbt/manifest.json prod-state/
      - run: |
          dbt build --target ci \
            --select state:modified+ \
            --defer --state prod-state

  deploy:
    if: github.ref == 'refs/heads/main'
    steps:
      - run: docker build -t $ECR/airflow-dbt:$SHA .
      - run: docker push $ECR/airflow-dbt:$SHA
      - run: |
          helm upgrade airflow apache-airflow/airflow \
            -n airflow -f values.yaml \
            --set images.airflow.tag=$SHA

Other ways to host it

Option	What you manage	Good fit
Docker Compose on one VM	Everything: the VM, upgrades, backups	Labs, small teams, proofs of concept (like the lab stack used in the dbt track)
Self-managed Kubernetes	Helm chart, cluster, scaling	Shown above; full control, needs platform skills
Linux VMs, no containers	VMs, virtualenvs, systemd, Ansible	Data centres and teams without Kubernetes — see the next chapter
Amazon MWAA	DAGs and requirements only	AWS shops that want managed Airflow
Astronomer (Astro)	DAGs and image	Managed Airflow from the Cosmos maintainers, any cloud
Google Cloud Composer	DAGs and packages	GCP shops
dbt Cloud + Airflow	Airflow triggers dbt Cloud jobs via API	Teams that want the dbt Cloud IDE and hosted docs

Explaining it in an interview

The two-minute answer

“Airflow runs on EKS in a private VPC using the official Helm chart and the KubernetesExecutor, so every task gets its own pod from one versioned image that bundles Airflow, dbt and Cosmos. Metadata lives in RDS, secrets in Secrets Manager, and logs and dbt artifacts in S3.

Snowflake is reached over PrivateLink with key-pair service users. Every pull request runs Slim CI against a zero-copy clone, and merging to main builds the image and rolls it out with Helm. Deployments are repeatable and easy to roll back by changing the image tag.”

Why bake dbt into the image? Every task runs the exact same tested code and dependencies; rolling back means deploying the previous tag.
Why KubernetesExecutor? Tasks are isolated, resources are set per task, and nothing runs when no tasks are queued. CeleryExecutor with Redis is the alternative for many short tasks.
How are credentials handled? No passwords in code: key-pair auth for Snowflake, keys in Secrets Manager, and pods get AWS access through IAM roles for service accounts.
How do you roll back? helm rollback or redeploy the previous image tag; for data, Snowflake Time Travel or clones.