Chapter 01

The Big Picture

Before looking at individual modules, you need a mental model of the whole system. dbt Core is three things running in one Python process: a compiler, a scheduler and an executor. Almost everything else in this tutorial is detail about one of those three.

What dbt Core actually is

dbt Core is an open-source (Apache 2.0) Python command-line application. You give it a folder of SQL select statements, YAML files and Jinja macros. It works out the order they depend on each other, turns each one into the DDL and DML your warehouse understands, and sends that SQL to the warehouse over a database connection.

The important point is in that last step: dbt never processes your data itself. Rows don't pass through the Python process. All of the computation runs inside PostgreSQL (or Snowflake, BigQuery and so on). dbt's job is to generate and order SQL, so the "T" in ELT happens where the data already lives.

What dbt Core is not

The three-part architecture

Inside the process, the work splits into three areas of responsibility. Keep this split in mind: when something goes wrong, the first question to ask is which of the three it happened in.

1 · The compiler

The compiler turns files into a data structure. It reads dbt_project.yml and profiles.yml, then visits every model, test, seed, snapshot, macro and YAML file and creates a Python object (a node) for each one. Together these objects form the manifest, an in-memory catalogue of the whole project. Later, just before a node runs, the compiler renders that node's Jinja into the final SQL.

2 · The scheduler

The scheduler decides what runs and in what order. Every ref() and source() recorded during parsing becomes an edge in a directed acyclic graph (DAG). Your --select / --exclude arguments cut a sub-graph out of it. A dependency-aware queue then hands nodes to worker threads as soon as their parents finish. The number of threads comes from threads: in your profile.

3 · The executor

The executor turns a compiled node into changes in the database. It doesn't have a fixed "create table" routine. Instead it finds the right materialization (view, table, incremental, snapshot or a custom one) and calls it. A materialization is a Jinja macro that calls the adapter, the database-specific plugin (here dbt-postgres), to run SQL inside transactions, inspect existing relations and rename or drop objects.

dbt Core, dbt Fusion and the dbt platform

The name "dbt" now covers several products, so it helps to be precise about which one this tutorial describes.

This tutorial is about dbt Core. Fusion aims to stay compatible with the same project format (models, YAML, Jinja), so most of the Jinja tutorial still applies. The internals described here (the Python modules, threads and partial parsing) are specific to Core.

The life of a dbt run

Now let's slow one command down and follow it. This is what your Airflow worker runs in stage 4 of the DAG:

$ dbt --no-use-colors run --select dimemployee --vars '{"run_id": 42 }'

Between pressing Enter and the process exiting, dbt goes through eight phases:

1. Parse the command line. The dbt executable is a Click application. Click matches the run subcommand and collects every flag (--select, --vars, --no-use-colors, …) into a single flags object. Flags can also come from environment variables (DBT_*) and from flags: in dbt_project.yml. The same entry point is available to Python as dbtRunner, which is how tools such as Astronomer Cosmos call dbt without starting a separate process. dbt/cli/main.py · dbt/cli/flags.py · dbt/cli/requires.py
2. Load the project and the profile. A chain of decorators runs before the task starts: preflight → profile → project → runtime_config → manifest. Between them they read dbt_project.yml, choose a target from profiles.yml, render the Jinja in those YAML files (this is where env_var() is resolved), apply --vars, and combine everything into one RuntimeConfig object. If dbt debug fails, it fails here. dbt/config/project.py · dbt/config/profile.py · dbt/config/runtime.py
3. Register the adapter. The profile's type: postgres tells dbt which plugin to import. The adapter factory loads dbt-postgres, registers its PostgresAdapter class and its connection manager, and adds the adapter's own macro package (with things like postgres__create_table_as) to the list of macros the project can see. No database connection opens yet; connections are created lazily, the first time a thread needs one. dbt/adapters/factory.py (dbt-adapters) · dbt/adapters/postgres/ (dbt-postgres)
4. Parse the project into a manifest (Jinja pass 1). ManifestLoader visits every file in the project, not just the ones you selected. It renders each model's Jinja in a special parse context. Here ref() and source() don't resolve to real tables; they only record dependencies. config() records settings, and execute is False. The result is the Manifest object, written to target/manifest.json. To save time on the next run, dbt also saves target/partial_parse.msgpack and re-parses only the files that changed. dbt/parser/manifest.py · dbt/parser/models.py · dbt/contracts/graph/manifest.py
5. Build the graph. The Linker turns each node's depends_on list into edges of a networkx.DiGraph. dbt then checks it for cycles, which is where "Found a cycle" errors come from. In your project the chain is stg.* → int_* → int_dimemployee_prepared → dimemployee → rpt_load_summary. dbt/compilation.py (Linker) · dbt/graph/graph.py
6. Select nodes and build a queue. The selector engine evaluates --select dimemployee against the graph (methods such as path:, tag:, source: and state:, and operators such as + and @). The result is a set of unique IDs, for example model.dwlab.dimemployee. dbt wraps the selected sub-graph in a GraphQueue, which only releases a node once all of its selected parents have finished. dbt/graph/selector.py · dbt/graph/selector_methods.py · dbt/graph/queue.py
7. Compile and execute each node (Jinja pass 2). The task (RunTask for dbt run) starts a thread pool with threads workers. For each node it takes from the queue, a ModelRunner renders the node's Jinja again, this time in the runtime context: ref() returns real relations, execute is True and is_incremental() can query the database. The output is written to target/compiled/. The runner then calls the materialization macro, which runs pre-hooks, the main statement and post-hooks through the adapter. The final SQL that actually ran is written to target/run/. dbt/task/runnable.py · dbt/task/run.py · dbt/context/providers.py
8. Write artifacts and exit. dbt runs any on-run-end hooks, writes run_results.json with a status, timing and message for each node, prints the summary line, closes connections and exits with code 0 (success), 1 (a node failed) or 2 (dbt itself crashed). Airflow only looks at that exit code. Your record_dbt_step callback goes further and counts results to write them to audit.etl_step. dbt/artifacts/ · dbt/task/runnable.py

Two Jinja passes: why execute exists

Phases 4 and 7 both render your model's Jinja, but in different contexts. Many confusing Jinja errors come from code that is written for pass 2 also being run during pass 1.

Here is the problem in practice. Suppose you want dimemployee to log how many rows its input has:

{% set q %}
  select count(*) from {{ ref('int_dimemployee_prepared') }}
{% endset %}

{% set n = run_query(q).columns[0].values()[0] %}
{{ log("rows in: " ~ n, info=True) }}

{% set q %}
  select count(*) from {{ ref('int_dimemployee_prepared') }}
{% endset %}

{% if execute %}
  {% set n = run_query(q).columns[0].values()[0] %}
  {{ log("rows in: " ~ n, info=True) }}
{% endif %}

The version on the left fails in phase 4 with an error like 'None' has no attribute 'columns', because run_query returned None. It fails even when you aren't selecting dimemployee, since every node is parsed. In the version on the right, pass 1 skips the block but still records the ref() dependency. Pass 2 runs the query for real.

Mapping the architecture onto your lab

Your Airflow DAG calls dbt nine times per DAG run, once from each BashOperator, and each call starts a new process. Knowing the eight phases tells you what each call costs:

DBT_DIR  = "/opt/airflow/dbt/dwlab"
DBT_VARS = """--vars '{"run_id": {{ ti.xcom_pull(task_ids="start_run") }} }'"""

# stage 3
bash_command="dbt --no-use-colors run  --select path:models/int " + DBT_VARS
bash_command="dbt --no-use-colors test --select path:models/int " + DBT_VARS
# stage 4
bash_command="dbt --no-use-colors run  --select dimsalesterritory " + DBT_VARS
bash_command="dbt --no-use-colors run  --select dimemployee "      + DBT_VARS

• Phases 1–3 happen in every call. Starting Python, importing dbt and loading config typically take a second or more, whatever you select.
• Phase 4 depends on partial parsing. dbt does a full re-parse whenever certain inputs change between runs, and the values passed in --vars are one of those inputs. Your run_id changes on every DAG run, so expect the first dbt call of each DAG run to fully re-parse. Later calls in the same DAG run pass the same run_id and can reuse partial_parse.msgpack, as long as they share the same target/ folder. Changes to profiles.yml, dbt_project.yml, installed packages or the dbt version also force a full re-parse, and so do edits to override macros such as your generate_schema_name.
• Phases 5–8 are where the actual work happens, and that work is limited to what you selected.
• Ordering happens at two levels. Airflow orders the stages, and inside one call, dbt's DAG and GraphQueue order the nodes. In stage 3, --select path:models/int lets dbt order all six int_* views itself.

$ docker exec -w /opt/airflow/dbt/dwlab airflow_airflow-worker_1 dbt --version
$ docker exec -w /opt/airflow/dbt/dwlab airflow_airflow-worker_1 dbt --log-level debug parse
$ docker exec -w /opt/airflow/dbt/dwlab airflow_airflow-worker_1 dbt ls --select +dimemployee
$ docker exec -w /opt/airflow/dbt/dwlab airflow_airflow-worker_1 dbt ls --select +dimemployee --output json --output-keys unique_id depends_on

Where the chapters go from here

Exercises