Data Architecture Design track
dbt + Airflow + Snowflake Architecture
One design in three views. First the logical architecture: what each tool does and how a daily run flows. Then two physical deployments of the same design: a container platform on AWS, and plain Linux virtual machines.
The core idea
Load raw: Fivetran, Snowpipe→
Store + compute: Snowflake→
Transform as code: dbt→
Orchestrate: Airflow→
Serve: BI, ML, sharing
Each tool does one job. Snowflake stores the data and supplies compute. dbt only compiles SQL and sends it to Snowflake, so it needs very little CPU wherever it runs. Airflow decides when things run and what happens when they fail. Because of this split, the same logical design can be deployed in very different ways.
Chapters
Kubernetes or VMs?
| Kubernetes (EKS) | Linux VMs | |
|---|---|---|
| Executor | KubernetesExecutor: one pod per task | CeleryExecutor: Redis broker plus worker VMs |
| How dbt is installed | Baked into one versioned Docker image | Separate Python venv at /opt/dbt/venv |
| Release unit | Image tag, rolled out with helm upgrade | Git tag, rolled out with Ansible over SSH, serial: 1 |
| Rollback | helm rollback or the previous image tag | Redeploy the previous Git tag |
| Secrets | Secrets Manager plus IAM roles for service accounts | Vault or Fernet-encrypted connections, key file at chmod 600 |
| Path to Snowflake | PrivateLink VPC endpoint, port 443 | NAT egress with a static, allowlisted IP, port 443 |
| Scaling | Automatic; scales to zero when idle | Add a worker VM |
| Good fit | Cloud-native teams with platform skills | Data centres and ops teams without Kubernetes |