Deployment view · Linux virtual machines, no containers

Deployment on Linux VMs

Airflow and dbt are installed directly on Linux VMs in Python virtual environments and run as systemd services. Airflow uses the CeleryExecutor, so the dbt work is spread across worker VMs. Ansible deploys every release over SSH, and Snowflake is reached over outbound HTTPS through the firewall.

Airflow VMs Build and release (dbt code, CI, Ansible) Snowflake Network connection

Figure 1 — Deployment on Linux VMs. Numbered markers match the steps below. On a phone, scroll the diagram sideways or use full screen.

Release and run, step by step

Develop. An engineer builds models locally in a Python venv against their own Snowflake schema (DBT_DEV_<USER>), then pushes a branch and opens a pull request.
CI checks the change. A self-hosted runner VM runs SQLFluff, then dbt build --select state:modified+ --defer in a CI_PR_<N> database cloned from prod. The prod manifest.json comes from the NFS artifacts folder.
Release. After merge, CI creates a version tag (for example v1.4.2) and calls the Ansible control node.
Deploy over SSH. Ansible checks out the tag into /opt/airflow/dags and /opt/dbt/analytics on every Airflow VM, installs pinned requirements into the venvs, runs dbt deps and dbt parse, then restarts services one VM at a time. All VMs must hold the same DAG and dbt code.
Scheduled run. The scheduler queues tasks in Redis, Celery workers pick them up, and Cosmos runs /opt/dbt/venv/bin/dbt for each model. dbt connects to Snowflake on port 443 through the firewall, with key-pair authentication.
Observe. StatsD metrics go to Prometheus and Grafana, logs go to NFS and ELK or Loki, and Grafana or Airflow callbacks alert Slack. Snowflake resource monitors cap credit use.

VM inventory

Host	Role	systemd services	Size (starting point)
AFairflow-web-01	UI and REST API	`nginx`, `airflow-api-server`	2 vCPU, 4 GB
AFairflow-sched-01	Scheduling and DAG parsing	`airflow-scheduler`, `airflow-dag-processor`, `airflow-triggerer`	4 vCPU, 8 GB
AFairflow-worker-01/02	Run tasks, including dbt	`airflow-worker`	4 vCPU, 8 GB each
DBairflow-db-01	Metadata DB and Celery broker	`postgresql`, `redis`	4 vCPU, 16 GB, SSD
Opsmonitor-01	Metrics, dashboards, logs, alerts	`prometheus`, `grafana`, `statsd-exporter`	2 vCPU, 8 GB
Relansible-01	Deployments, SSH jump host	none (runs on demand)	2 vCPU, 4 GB
CIci-runner-01	CI jobs for pull requests	`actions-runner` or `gitlab-runner`	2 vCPU, 4 GB

Why the workers are small

Worker VMs stay small because dbt only compiles SQL and waits. The heavy compute happens in Snowflake warehouses.

Ports and firewall rules

From	To	Port	Purpose
Users	airflow-web-01	443	Airflow UI through Nginx (TLS)
Nginx	api-server (local)	8080	Reverse proxy
All Airflow VMs	airflow-db-01	5432, 6379	Metadata DB and Celery broker
airflow-web-01	Workers	8793	Fetch task logs from the worker log server
monitor-01	All VMs	9100, 9102	Prometheus scrapes node and StatsD exporters
ansible-01	All VMs	22	SSH deployments
Workers, CI runner	*.snowflakecomputing.com	443 out	dbt and Airflow to Snowflake, via NAT with a static IP

Key configuration

Directory layout on each Airflow VM

/opt/airflow/
├─ venv/                 # apache-airflow[celery,postgres]
├─ dags/                 # from the release tag
└─ airflow.cfg
/opt/dbt/
├─ venv/                 # dbt-snowflake, separate venv
└─ analytics/            # dbt project, same tag
/etc/airflow/airflow.env # AIRFLOW__* settings, no secrets
/home/airflow/.ssh/      # deploy key (read-only)
/secure/snowflake/rsa_key.p8   # chmod 600, owner airflow
/data/airflow/logs       # NFS mount
/data/dbt/artifacts      # NFS mount

systemd unit for a worker

[Unit]
Description=Airflow Celery worker
After=network-online.target

[Service]
User=airflow
EnvironmentFile=/etc/airflow/airflow.env
ExecStart=/opt/airflow/venv/bin/airflow celery worker --queues default,dbt
Restart=on-failure
RestartSec=10

[Install]
WantedBy=multi-user.target

Cosmos pointing at the dbt venv

from cosmos import DbtTaskGroup, ProjectConfig, \
    ProfileConfig, ExecutionConfig

transform = DbtTaskGroup(
    group_id="dbt",
    project_config=ProjectConfig("/opt/dbt/analytics"),
    profile_config=profile,          # key-pair mapping
    execution_config=ExecutionConfig(
        dbt_executable_path="/opt/dbt/venv/bin/dbt"),
    operator_args={"queue": "dbt"},  # only dbt workers
)

Ansible deploy playbook (simplified)

- hosts: airflow
  become: true
  serial: 1                     # one VM at a time
  vars: {release: "v1.4.2"}
  tasks:
    - git:
        repo: git@git.local:data/analytics.git
        dest: /opt/release
        version: "{{ release }}"
    - synchronize: {src: /opt/release/dags/, dest: /opt/airflow/dags/}
      delegate_to: "{{ inventory_hostname }}"
    - synchronize: {src: /opt/release/dbt/, dest: /opt/dbt/analytics/}
      delegate_to: "{{ inventory_hostname }}"
    - pip:
        requirements: /opt/release/requirements.txt
        virtualenv: /opt/airflow/venv
    - command: /opt/dbt/venv/bin/dbt deps
      args: {chdir: /opt/dbt/analytics}
    - systemd:
        name: "{{ item }}"
        state: restarted
      loop: "{{ airflow_services }}"

Sizing options

Setup	VMs	Executor	Good fit
Single VM	1: everything plus local PostgreSQL	LocalExecutor	Labs, proofs of concept, under ~50 dbt models
Small production	3: web + scheduler, worker, database	LocalExecutor or Celery	One team, nightly batch
Standard (shown above)	5–7, with separate workers and monitoring	CeleryExecutor	Several teams, hourly runs, growing DAG count
High availability	2 schedulers, 2 web behind a load balancer, PostgreSQL primary + standby, Redis Sentinel	CeleryExecutor	Business-critical pipelines with SLAs

Explaining it in an interview

The two-minute answer

“We run Airflow on Linux VMs without containers. Airflow and dbt each have their own Python virtualenv and run as systemd services. A scheduler VM queues tasks in Redis, and Celery worker VMs run them. Cosmos turns each dbt model into an Airflow task that calls the dbt binary in its venv, and dbt runs the SQL in Snowflake over HTTPS with key-pair authentication.

Releases are Git tags. CI runs Slim CI against a zero-copy clone, then Ansible checks out the tag on every VM, installs pinned packages and restarts services one VM at a time. To roll back, we redeploy the previous tag.”

Why separate venvs for Airflow and dbt? Their dependencies often conflict. Separate venvs let you upgrade one without breaking the other.
Why CeleryExecutor on VMs? It spreads tasks across worker VMs and lets you add capacity by adding a VM. LocalExecutor runs everything on one machine.
How do you keep VMs consistent? Ansible is the only way changes reach them, with the same tag and pinned requirements.txt everywhere. No manual edits on servers.
Trade-off vs containers: VMs are simpler to understand and fit existing ops teams, but you patch the OS and manage Python versions yourself, and scaling is slower than on Kubernetes — compare with the EKS deployment.