From 7ff7d3fb1538f4d467738a56d434ce6c4f6ed0df Mon Sep 17 00:00:00 2001
From: Cortland Goffena <30168413+cmgoffena13@users.noreply.github.com>
Date: Wed, 29 Jul 2026 18:12:54 -0600
Subject: [PATCH 1/2] chore(docs): adding in nomissingunittest documentation
Signed-off-by: Cortland Goffena <30168413+cmgoffena13@users.noreply.github.com>
---
docs/examples/sqlmesh_cli_crash_course.md | 1898 +++++++++++----------
docs/guides/linter.md | 1 +
2 files changed, 1001 insertions(+), 898 deletions(-)
diff --git a/docs/examples/sqlmesh_cli_crash_course.md b/docs/examples/sqlmesh_cli_crash_course.md
index 0bf5780f12..c3f891c68a 100644
--- a/docs/examples/sqlmesh_cli_crash_course.md
+++ b/docs/examples/sqlmesh_cli_crash_course.md
@@ -1,6 +1,6 @@
# SQLMesh CLI Crash Course
-
+
This doc is designed to get you intimate with a **majority** of the SQLMesh workflows you’ll use to build *and* maintain transformation data pipelines. The goal is to get SQLMesh into muscle memory in 30 minutes or less.
@@ -12,7 +12,11 @@ If you're new to how SQLMesh uses virtual data environments, [watch this quick e
!!! tip
- Put this page on your second monitor or in a side by side window to swiftly copy/paste into your terminal.
+```
+Put this page on your second monitor or in a side by side window to swiftly copy/paste into your terminal.
+```
+
+
## Development Workflow
@@ -25,6 +29,8 @@ You’ll use these commands 80% of the time because this is how you apply the ch
5. Run data diff against prod
6. Apply the changes to prod
+
+
### Preview, Apply, and Audit Changes in `dev`
You can make changes quickly and confidently through one simple command: `sqlmesh plan dev`
@@ -37,131 +43,139 @@ Note: If you run this without making any changes, SQLMesh will prompt you to mak
=== "SQLMesh"
- ```bash
- sqlmesh plan dev
- ```
+```
+```bash
+sqlmesh plan dev
+```
- ```bash
- sqlmesh plan
- ```
+```bash
+sqlmesh plan
+```
- If you want to move faster, you can add the `--auto-apply` flag to skip the manual prompt and apply the plan. You should do this when you're familiar with the plan output, and don't need to see tiny changes in the diff output before applying the plan.
+If you want to move faster, you can add the `--auto-apply` flag to skip the manual prompt and apply the plan. You should do this when you're familiar with the plan output, and don't need to see tiny changes in the diff output before applying the plan.
- ```bash
- sqlmesh plan --auto-apply
- ```
+```bash
+sqlmesh plan --auto-apply
+```
+```
=== "Tobiko Cloud"
- ```bash
- tcloud sqlmesh plan dev
- ```
+```
+```bash
+tcloud sqlmesh plan dev
+```
- ```bash
- tcloud sqlmesh plan
- ```
+```bash
+tcloud sqlmesh plan
+```
- If you want to move faster, you can add the `--auto-apply` flag to skip the manual prompt and apply the plan. You should do this when you're familiar with the plan output, and don't need to see tiny changes in the diff output before applying the plan.
+If you want to move faster, you can add the `--auto-apply` flag to skip the manual prompt and apply the plan. You should do this when you're familiar with the plan output, and don't need to see tiny changes in the diff output before applying the plan.
- ```bash
- tcloud sqlmesh plan --auto-apply
- ```
+```bash
+tcloud sqlmesh plan --auto-apply
+```
+```
??? "Example Output"
I made a breaking change to `incremental_model` and `full_model`.
- SQLMesh:
-
- - Showed me the models impacted by the changes.
- - Showed me the changes that will be made to the models.
- - Showed me the models that need to be backfilled.
- - Prompted me to apply the changes to `dev`.
- - Showed me the audit failures that raise as warnings.
- - Updated the physical layer to validate the SQL.
- - Executed the model batches by inserting the data into the physical layer.
- - Updated the virtual layer's view pointers to reflect the changes.
-
- ```bash
- > sqlmesh plan dev
- Differences from the `dev` environment:
-
- Models:
- ├── Directly Modified:
- │ ├── sqlmesh_example__dev.incremental_model
- │ └── sqlmesh_example__dev.full_model
- └── Indirectly Modified:
- └── sqlmesh_example__dev.view_model
-
- ---
-
- +++
-
- @@ -9,7 +9,8 @@
-
- SELECT
- item_id,
- COUNT(DISTINCT id) AS num_orders,
- - 6 AS new_column
- + new_column
- FROM sqlmesh_example.incremental_model
- GROUP BY
- - item_id
- + item_id,
- + new_column
-
- Directly Modified: sqlmesh_example__dev.full_model (Breaking)
-
- ---
-
- +++
-
- @@ -15,7 +15,7 @@
-
- id,
- item_id,
- event_date,
- - 5 AS new_column
- + 7 AS new_column
- FROM sqlmesh_example.seed_model
- WHERE
- event_date BETWEEN @start_date AND @end_date
-
- Directly Modified: sqlmesh_example__dev.incremental_model (Breaking)
- └── Indirectly Modified Children:
- └── sqlmesh_example__dev.view_model (Indirect Breaking)
- Models needing backfill:
- ├── sqlmesh_example__dev.full_model: [full refresh]
- ├── sqlmesh_example__dev.incremental_model: [2020-01-01 - 2025-04-16]
- └── sqlmesh_example__dev.view_model: [recreate view]
- Apply - Backfill Tables [y/n]: y
-
- Updating physical layer ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100.0% • 2/2 • 0:00:00
-
- ✔ Physical layer updated
-
- [1/1] sqlmesh_example__dev.incremental_model [insert 2020-01-01 - 2025-04-16] 0.03s
- Executing model batches ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0% • pending • 0:00:00
- sqlmesh_example__dev.incremental_model .
- [WARNING] sqlmesh_example__dev.full_model: 'assert_positive_order_ids' audit error: 2 rows failed. Learn more in logs:
- /Users/sung/Desktop/git_repos/sqlmesh-cli-revamp/logs/sqlmesh_2025_04_18_10_33_43.log
- [1/1] sqlmesh_example__dev.full_model [full refresh, audits ❌1] 0.01s
- Executing model batches ━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━ 33.3% • 1/3 • 0:00:00
- sqlmesh_example__dev.full_model .
- [WARNING] sqlmesh_example__dev.view_model: 'assert_positive_order_ids' audit error: 2 rows failed. Learn more in logs:
- /Users/sung/Desktop/git_repos/sqlmesh-cli-revamp/logs/sqlmesh_2025_04_18_10_33_43.log
- [1/1] sqlmesh_example__dev.view_model [recreate view, audits ✔2 ❌1] 0.01s
- Executing model batches ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100.0% • 3/3 • 0:00:00
-
- ✔ Model batches executed
-
- Updating virtual layer ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100.0% • 3/3 • 0:00:00
-
- ✔ Virtual layer updated
- ```
+```
+SQLMesh:
+
+ - Showed me the models impacted by the changes.
+ - Showed me the changes that will be made to the models.
+ - Showed me the models that need to be backfilled.
+ - Prompted me to apply the changes to `dev`.
+ - Showed me the audit failures that raise as warnings.
+ - Updated the physical layer to validate the SQL.
+ - Executed the model batches by inserting the data into the physical layer.
+ - Updated the virtual layer's view pointers to reflect the changes.
+
+```bash
+> sqlmesh plan dev
+Differences from the `dev` environment:
+
+Models:
+├── Directly Modified:
+│ ├── sqlmesh_example__dev.incremental_model
+│ └── sqlmesh_example__dev.full_model
+└── Indirectly Modified:
+ └── sqlmesh_example__dev.view_model
+
+---
+
++++
+
+@@ -9,7 +9,8 @@
+
+ SELECT
+ item_id,
+ COUNT(DISTINCT id) AS num_orders,
+- 6 AS new_column
++ new_column
+ FROM sqlmesh_example.incremental_model
+ GROUP BY
+- item_id
++ item_id,
++ new_column
+
+Directly Modified: sqlmesh_example__dev.full_model (Breaking)
+
+---
+
++++
+
+@@ -15,7 +15,7 @@
+
+ id,
+ item_id,
+ event_date,
+- 5 AS new_column
++ 7 AS new_column
+ FROM sqlmesh_example.seed_model
+ WHERE
+ event_date BETWEEN @start_date AND @end_date
+
+Directly Modified: sqlmesh_example__dev.incremental_model (Breaking)
+└── Indirectly Modified Children:
+ └── sqlmesh_example__dev.view_model (Indirect Breaking)
+Models needing backfill:
+├── sqlmesh_example__dev.full_model: [full refresh]
+├── sqlmesh_example__dev.incremental_model: [2020-01-01 - 2025-04-16]
+└── sqlmesh_example__dev.view_model: [recreate view]
+Apply - Backfill Tables [y/n]: y
+
+Updating physical layer ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100.0% • 2/2 • 0:00:00
+
+✔ Physical layer updated
+
+[1/1] sqlmesh_example__dev.incremental_model [insert 2020-01-01 - 2025-04-16] 0.03s
+Executing model batches ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0% • pending • 0:00:00
+sqlmesh_example__dev.incremental_model .
+[WARNING] sqlmesh_example__dev.full_model: 'assert_positive_order_ids' audit error: 2 rows failed. Learn more in logs:
+/Users/sung/Desktop/git_repos/sqlmesh-cli-revamp/logs/sqlmesh_2025_04_18_10_33_43.log
+[1/1] sqlmesh_example__dev.full_model [full refresh, audits ❌1] 0.01s
+Executing model batches ━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━ 33.3% • 1/3 • 0:00:00
+sqlmesh_example__dev.full_model .
+[WARNING] sqlmesh_example__dev.view_model: 'assert_positive_order_ids' audit error: 2 rows failed. Learn more in logs:
+/Users/sung/Desktop/git_repos/sqlmesh-cli-revamp/logs/sqlmesh_2025_04_18_10_33_43.log
+[1/1] sqlmesh_example__dev.view_model [recreate view, audits ✔2 ❌1] 0.01s
+Executing model batches ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100.0% • 3/3 • 0:00:00
+
+✔ Model batches executed
+
+Updating virtual layer ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100.0% • 3/3 • 0:00:00
+
+✔ Virtual layer updated
+```
+```
+
+
### Run Data Diff Against Prod
-
+
Run data diff against prod. This is a good way to verify the changes are behaving as expected **after** applying them to `dev`.
@@ -169,196 +183,212 @@ To make this easier and faster, you can run data diff against all models in the
=== "SQLMesh"
- ```bash
- sqlmesh table_diff prod:dev sqlmesh_example.full_model --show-sample
- ```
+```
+```bash
+sqlmesh table_diff prod:dev sqlmesh_example.full_model --show-sample
+```
- ```bash
- sqlmesh table_diff : --show-sample
- ```
+```bash
+sqlmesh table_diff : --show-sample
+```
- ```bash
- sqlmesh table_diff prod:dev -m '*' --show-sample
- ```
+```bash
+sqlmesh table_diff prod:dev -m '*' --show-sample
+```
+```
=== "Tobiko Cloud"
- ```bash
- tcloud sqlmesh table_diff prod:dev sqlmesh_example.full_model --show-sample
- ```
+```
+```bash
+tcloud sqlmesh table_diff prod:dev sqlmesh_example.full_model --show-sample
+```
- ```bash
- tcloud sqlmesh table_diff : --show-sample
- ```
+```bash
+tcloud sqlmesh table_diff : --show-sample
+```
- ```bash
- tcloud sqlmesh table_diff prod:dev -m '*' --show-sample
- ```
+```bash
+tcloud sqlmesh table_diff prod:dev -m '*' --show-sample
+```
+```
??? "Example Output"
I compare the `prod` and `dev` environments for `sqlmesh_example.full_model`.
- - Verified environments and models to diff along with the join on grain configured.
- - Showed me schema diffs between the environments.
- - Showed me row count diffs between the environments.
- - Showed me common rows stats between the environments.
- - Showed me sample data differences between the environments.
- - This is where your human judgement comes in to verify the changes are behaving as expected.
-
- Model definition:
- ```sql linenums="1" hl_lines="6"
- -- models/full_model.sql
- MODEL (
- name sqlmesh_example.full_model,
- kind FULL,
- cron '@daily',
- grain item_id, -- grain is optional BUT necessary for table diffs to work correctly. It's your primary key that is unique and not null.
- audits (assert_positive_order_ids),
- );
-
- SELECT
- item_id,
- COUNT(DISTINCT id) AS num_orders,
- new_column
- FROM
- sqlmesh_example.incremental_model
- GROUP BY item_id, new_column
- ```
-
- Table diff:
- ```bash
- > sqlmesh table_diff prod:dev sqlmesh_example.full_model --show-sample
- Table Diff
- ├── Model:
- │ └── sqlmesh_example.full_model
- ├── Environment:
- │ ├── Source: prod
- │ └── Target: dev
- ├── Tables:
- │ ├── Source: db.sqlmesh_example.full_model
- │ └── Target: db.sqlmesh_example__dev.full_model
- └── Join On:
- └── item_id
-
- Schema Diff Between 'PROD' and 'DEV' environments for model 'sqlmesh_example.full_model':
- └── Schemas match
-
-
- Row Counts:
- └── PARTIAL MATCH: 5 rows (100.0%)
-
- COMMON ROWS column comparison stats:
- pct_match
- num_orders 100.0
- new_column 0.0
-
-
- COMMON ROWS sample data differences:
- Column: new_column
- ┏━━━━━━━━━┳━━━━━━┳━━━━━┓
- ┃ item_id ┃ PROD ┃ DEV ┃
- ┡━━━━━━━━━╇━━━━━━╇━━━━━┩
- │ -11 │ 5 │ 7 │
- │ -3 │ 5 │ 7 │
- │ 1 │ 5 │ 7 │
- │ 3 │ 5 │ 7 │
- │ 9 │ 5 │ 7 │
- └─────────┴──────┴─────┘
- ```
+```
+- Verified environments and models to diff along with the join on grain configured.
+- Showed me schema diffs between the environments.
+- Showed me row count diffs between the environments.
+- Showed me common rows stats between the environments.
+- Showed me sample data differences between the environments.
+- This is where your human judgement comes in to verify the changes are behaving as expected.
+
+Model definition:
+```sql linenums="1" hl_lines="6"
+-- models/full_model.sql
+MODEL (
+ name sqlmesh_example.full_model,
+ kind FULL,
+ cron '@daily',
+ grain item_id, -- grain is optional BUT necessary for table diffs to work correctly. It's your primary key that is unique and not null.
+ audits (assert_positive_order_ids),
+);
+
+SELECT
+ item_id,
+ COUNT(DISTINCT id) AS num_orders,
+ new_column
+FROM
+ sqlmesh_example.incremental_model
+GROUP BY item_id, new_column
+```
+
+Table diff:
+```bash
+> sqlmesh table_diff prod:dev sqlmesh_example.full_model --show-sample
+Table Diff
+├── Model:
+│ └── sqlmesh_example.full_model
+├── Environment:
+│ ├── Source: prod
+│ └── Target: dev
+├── Tables:
+│ ├── Source: db.sqlmesh_example.full_model
+│ └── Target: db.sqlmesh_example__dev.full_model
+└── Join On:
+ └── item_id
+
+Schema Diff Between 'PROD' and 'DEV' environments for model 'sqlmesh_example.full_model':
+└── Schemas match
+
+
+Row Counts:
+└── PARTIAL MATCH: 5 rows (100.0%)
+
+COMMON ROWS column comparison stats:
+ pct_match
+num_orders 100.0
+new_column 0.0
+
+
+COMMON ROWS sample data differences:
+Column: new_column
+┏━━━━━━━━━┳━━━━━━┳━━━━━┓
+┃ item_id ┃ PROD ┃ DEV ┃
+┡━━━━━━━━━╇━━━━━━╇━━━━━┩
+│ -11 │ 5 │ 7 │
+│ -3 │ 5 │ 7 │
+│ 1 │ 5 │ 7 │
+│ 3 │ 5 │ 7 │
+│ 9 │ 5 │ 7 │
+└─────────┴──────┴─────┘
+```
+```
+
+
### Apply Changes to Prod
After you feel confident about the changes, apply them to `prod`.
!!! warning "Apply the changes to prod"
- We recommend only applying changes to `prod` [**using CI/CD**](../integrations/github.md) as best practice.
+ We recommend only applying changes to `prod` **[using CI/CD](../integrations/github.md)** as best practice.
For learning purposes and hot fixes, you can manually apply the changes to prod by entering `y` at the prompt.
=== "SQLMesh"
- ```bash
- sqlmesh plan
- ```
+```
+```bash
+sqlmesh plan
+```
+```
=== "Tobiko Cloud"
- ```bash
- tcloud sqlmesh plan
- ```
+```
+```bash
+tcloud sqlmesh plan
+```
+```
??? "Example Output"
After I feel confident about the changes, I apply them to `prod`.
- SQLMesh:
+```
+SQLMesh:
- - Showed me the models impacted by the changes.
- - Showed me the changes that will be made to the models.
- - Showed me the models that need to be backfilled. None in this case as it was already backfilled earlier in `dev`.
- - Prompted me to apply the changes to `prod`.
- - Showed me physical layer and execution steps are skipped as the changes were already applied to `dev`.
- - Updated the virtual layer view pointers to reflect the changes.
+- Showed me the models impacted by the changes.
+- Showed me the changes that will be made to the models.
+- Showed me the models that need to be backfilled. None in this case as it was already backfilled earlier in `dev`.
+- Prompted me to apply the changes to `prod`.
+- Showed me physical layer and execution steps are skipped as the changes were already applied to `dev`.
+- Updated the virtual layer view pointers to reflect the changes.
- ```bash
- > sqlmesh plan
- Differences from the `prod` environment:
+```bash
+> sqlmesh plan
+Differences from the `prod` environment:
- Models:
- ├── Directly Modified:
- │ ├── sqlmesh_example.full_model
- │ └── sqlmesh_example.incremental_model
- └── Indirectly Modified:
- └── sqlmesh_example.view_model
+Models:
+├── Directly Modified:
+│ ├── sqlmesh_example.full_model
+│ └── sqlmesh_example.incremental_model
+└── Indirectly Modified:
+ └── sqlmesh_example.view_model
- ---
+---
- +++
++++
- @@ -9,7 +9,8 @@
+@@ -9,7 +9,8 @@
- SELECT
- item_id,
- COUNT(DISTINCT id) AS num_orders,
- - 5 AS new_column
- + new_column
- FROM sqlmesh_example.incremental_model
- GROUP BY
- - item_id
- + item_id,
- + new_column
+SELECT
+ item_id,
+ COUNT(DISTINCT id) AS num_orders,
+- 5 AS new_column
++ new_column
+FROM sqlmesh_example.incremental_model
+GROUP BY
+- item_id
++ item_id,
++ new_column
- Directly Modified: sqlmesh_example.full_model (Breaking)
+Directly Modified: sqlmesh_example.full_model (Breaking)
- ---
+---
- +++
++++
- @@ -15,7 +15,7 @@
+@@ -15,7 +15,7 @@
- id,
- item_id,
- event_date,
- - 5 AS new_column
- + 7 AS new_column
- FROM sqlmesh_example.seed_model
- WHERE
- event_date BETWEEN @start_date AND @end_date
+ id,
+ item_id,
+ event_date,
+- 5 AS new_column
++ 7 AS new_column
+FROM sqlmesh_example.seed_model
+WHERE
+ event_date BETWEEN @start_date AND @end_date
- Directly Modified: sqlmesh_example.incremental_model (Breaking)
- └── Indirectly Modified Children:
- └── sqlmesh_example.view_model (Indirect Breaking)
- Apply - Virtual Update [y/n]: y
+Directly Modified: sqlmesh_example.incremental_model (Breaking)
+└── Indirectly Modified Children:
+ └── sqlmesh_example.view_model (Indirect Breaking)
+Apply - Virtual Update [y/n]: y
- SKIP: No physical layer updates to perform
+SKIP: No physical layer updates to perform
- SKIP: No model batches to execute
+SKIP: No model batches to execute
- Updating virtual layer ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100.0% • 3/3 • 0:00:00
+Updating virtual layer ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100.0% • 3/3 • 0:00:00
- ✔ Virtual layer updated
- ```
+✔ Virtual layer updated
+```
+```
---
+
+
## Enhanced Testing Workflow
You'll use these commands to validate your changes are behaving as expected. Audits (data tests) are a great first step, and you'll want to grow from there to feel confident about your pipelines. The workflow is as follows:
@@ -370,6 +400,8 @@ You'll use these commands to validate your changes are behaving as expected. Aud
---
+
+
### Create and Audit External Models
Sometimes models `SELECT` from tables/views that are outside of SQLMesh's control. SQLMesh can automatically parse their fully qualified names from model definitions (ex: `bigquery-public-data`.`ga4_obfuscated_sample_ecommerce`.`events_20210131`) and determine their full schemas and column data types.
@@ -378,137 +410,145 @@ These "external model" schemas are used for column level lineage. You can also a
=== "SQLMesh"
- ```bash
- sqlmesh create_external_models
- ```
+```
+```bash
+sqlmesh create_external_models
+```
+```
=== "Tobiko Cloud"
- ```bash
- tcloud sqlmesh create_external_models
- ```
+```
+```bash
+tcloud sqlmesh create_external_models
+```
+```
??? "Example Output"
Note: this is an example from a separate Tobiko Cloud project, so you can't follow along in the Github repo.
- - Generated external models from the `bigquery-public-data`.`ga4_obfuscated_sample_ecommerce`.`events_20210131` table parsed in the model's SQL.
- - Added an audit to the external model to ensure `event_date` is not NULL.
- - Viewed a plan preview of the changes that will be made for the external model.
-
- ```sql linenums="1" hl_lines="29" title="models/external_model_example.sql"
- MODEL (
- name tcloud_demo.external_model
- );
-
- SELECT
- event_date,
- event_timestamp,
- event_name,
- event_params,
- event_previous_timestamp,
- event_value_in_usd,
- event_bundle_sequence_id,
- event_server_timestamp_offset,
- user_id,
- user_pseudo_id,
- privacy_info,
- user_properties,
- user_first_touch_timestamp,
- user_ltv,
- device,
- geo,
- app_info,
- traffic_source,
- stream_id,
- platform,
- event_dimensions,
- ecommerce
- /* items */
- FROM bigquery-public-data.ga4_obfuscated_sample_ecommerce.events_20210131 -- I fully qualified the external table name and sqlmesh will automatically create the external model
- ```
-
- `sqlmesh create_external_models` output file:
-
- ```yaml linenums="1" hl_lines="2 3 4" title="external_models.yaml"
- - name: '`bigquery-public-data`.`ga4_obfuscated_sample_ecommerce`.`events_20210131`'
- audits: # I added this audit manually to the external model YAML file
- - name: not_null
- columns: "[event_date]"
- columns:
- event_date: STRING
- event_timestamp: INT64
- event_name: STRING
- event_params: ARRAY>>
- event_previous_timestamp: INT64
- event_value_in_usd: FLOAT64
- event_bundle_sequence_id: INT64
- event_server_timestamp_offset: INT64
- user_id: STRING
- user_pseudo_id: STRING
- privacy_info: STRUCT
- user_properties: ARRAY>>
- user_first_touch_timestamp: INT64
- user_ltv: STRUCT
- device: STRUCT>
- geo: STRUCT
- app_info: STRUCT
- traffic_source: STRUCT
- stream_id: INT64
- platform: STRING
- event_dimensions: STRUCT
- ecommerce: STRUCT
- items: ARRAY>
- gateway: public-demo
- ```
-
- ```bash
- > sqlmesh plan dev_sung
- Differences from the `dev_sung` environment:
-
- Models:
- └── Metadata Updated:
- └── "bigquery-public-data".ga4_obfuscated_sample_ecommerce__dev_sung.events_20210131
-
- ---
-
- +++
-
- @@ -29,5 +29,6 @@
-
- ecommerce STRUCT,
- items ARRAY>
- ),
- + audits (not_null('columns' = [event_date])),
- gateway `public-demo`
- )
-
- Metadata Updated: "bigquery-public-data".ga4_obfuscated_sample_ecommerce__dev_sung.events_20210131
- Models needing backfill:
- └── "bigquery-public-data".ga4_obfuscated_sample_ecommerce__dev_sung.events_20210131: [full refresh]
- Apply - Backfill Tables [y/n]:
- ```
+```
+- Generated external models from the `bigquery-public-data`.`ga4_obfuscated_sample_ecommerce`.`events_20210131` table parsed in the model's SQL.
+- Added an audit to the external model to ensure `event_date` is not NULL.
+- Viewed a plan preview of the changes that will be made for the external model.
+
+```sql linenums="1" hl_lines="29" title="models/external_model_example.sql"
+MODEL (
+ name tcloud_demo.external_model
+);
+
+SELECT
+ event_date,
+ event_timestamp,
+ event_name,
+ event_params,
+ event_previous_timestamp,
+ event_value_in_usd,
+ event_bundle_sequence_id,
+ event_server_timestamp_offset,
+ user_id,
+ user_pseudo_id,
+ privacy_info,
+ user_properties,
+ user_first_touch_timestamp,
+ user_ltv,
+ device,
+ geo,
+ app_info,
+ traffic_source,
+ stream_id,
+ platform,
+ event_dimensions,
+ ecommerce
+/* items */
+FROM bigquery-public-data.ga4_obfuscated_sample_ecommerce.events_20210131 -- I fully qualified the external table name and sqlmesh will automatically create the external model
+```
+
+`sqlmesh create_external_models` output file:
+
+```yaml linenums="1" hl_lines="2 3 4" title="external_models.yaml"
+- name: '`bigquery-public-data`.`ga4_obfuscated_sample_ecommerce`.`events_20210131`'
+ audits: # I added this audit manually to the external model YAML file
+ - name: not_null
+ columns: "[event_date]"
+ columns:
+ event_date: STRING
+ event_timestamp: INT64
+ event_name: STRING
+ event_params: ARRAY>>
+ event_previous_timestamp: INT64
+ event_value_in_usd: FLOAT64
+ event_bundle_sequence_id: INT64
+ event_server_timestamp_offset: INT64
+ user_id: STRING
+ user_pseudo_id: STRING
+ privacy_info: STRUCT
+ user_properties: ARRAY>>
+ user_first_touch_timestamp: INT64
+ user_ltv: STRUCT
+ device: STRUCT>
+ geo: STRUCT
+ app_info: STRUCT
+ traffic_source: STRUCT
+ stream_id: INT64
+ platform: STRING
+ event_dimensions: STRUCT
+ ecommerce: STRUCT
+ items: ARRAY>
+ gateway: public-demo
+```
+
+```bash
+> sqlmesh plan dev_sung
+Differences from the `dev_sung` environment:
+
+Models:
+└── Metadata Updated:
+ └── "bigquery-public-data".ga4_obfuscated_sample_ecommerce__dev_sung.events_20210131
+
+---
+
++++
+
+@@ -29,5 +29,6 @@
+
+ ecommerce STRUCT,
+ items ARRAY>
+ ),
++ audits (not_null('columns' = [event_date])),
+ gateway `public-demo`
+)
+
+Metadata Updated: "bigquery-public-data".ga4_obfuscated_sample_ecommerce__dev_sung.events_20210131
+Models needing backfill:
+└── "bigquery-public-data".ga4_obfuscated_sample_ecommerce__dev_sung.events_20210131: [full refresh]
+Apply - Backfill Tables [y/n]:
+```
+```
+
+
### Automatically Generate Unit Tests
@@ -518,114 +558,121 @@ Unit tests run *before* a plan is applied automatically. This is great for testi
=== "SQLMesh"
- Create a unit test based on 5 rows from the upstream `sqlmesh_example.incremental_model`.
-
- ```bash
- sqlmesh create_test sqlmesh_example.full_model \
- --query sqlmesh_example.incremental_model \
- "select * from sqlmesh_example.incremental_model limit 5"
- ```
+```
+Create a unit test based on 5 rows from the upstream `sqlmesh_example.incremental_model`.
- ```bash
- sqlmesh create_test \
- --query \
- "select * from limit 5"
- ```
+```bash
+sqlmesh create_test sqlmesh_example.full_model \
+ --query sqlmesh_example.incremental_model \
+ "select * from sqlmesh_example.incremental_model limit 5"
+```
+```bash
+sqlmesh create_test \
+ --query \
+ "select * from limit 5"
+```
+```
=== "Tobiko Cloud"
- ```bash
- tcloud sqlmesh create_test demo.stg_payments \
- --query demo.seed_raw_payments \
- "select * from demo.seed_raw_payments limit 5"
- ```
+```
+```bash
+tcloud sqlmesh create_test demo.stg_payments \
+ --query demo.seed_raw_payments \
+ "select * from demo.seed_raw_payments limit 5"
+```
- ```bash
- tcloud sqlmesh create_test \
- --query \
- "select * from limit 5"
- ```
+```bash
+tcloud sqlmesh create_test \
+ --query \
+ "select * from limit 5"
+```
+```
??? "Example Output"
- SQLMesh:
-
- - Generated unit tests for the `sqlmesh_example.full_model` model by live querying the data.
- - Ran the tests and they passed locally in DuckDB.
- - If you're using a cloud data warehouse, this will transpile your SQL syntax to its equivalent in duckdb.
- - This runs fast and free on your local machine.
-
- Generated test definition file:
-
- ```yaml linenums="1" title="tests/test_full_model.yaml"
- test_full_model:
- model: '"db"."sqlmesh_example"."full_model"'
- inputs:
- '"db"."sqlmesh_example"."incremental_model"':
- - id: -11
- item_id: -11
- event_date: 2020-01-01
- new_column: 7
- - id: 1
- item_id: 1
- event_date: 2020-01-01
- new_column: 7
- - id: 3
- item_id: 3
- event_date: 2020-01-03
- new_column: 7
- - id: 4
- item_id: 1
- event_date: 2020-01-04
- new_column: 7
- - id: 5
- item_id: 1
- event_date: 2020-01-05
- new_column: 7
- outputs:
- query:
- - item_id: 3
- num_orders: 1
- new_column: 7
- - item_id: 1
- num_orders: 3
- new_column: 7
- - item_id: -11
- num_orders: 1
- new_column: 7
- ```
-
- Manually execute tests with `sqlmesh test`:
-
- ```bash
- (demo) ➜ demo git:(main) ✗ sqlmesh test
- .
- ----------------------------------------------------------------------
- Ran 1 test in 0.053s
-
- OK
- ```
-
- ```bash
- # what do we see if the test fails?
- (demo) ➜ demo git:(main) ✗ sqlmesh test
- F
- ======================================================================
- FAIL: test_full_model (/Users/sung/Desktop/git_repos/sqlmesh-cli-revamp/tests/test_full_model.yaml)
- None
- ----------------------------------------------------------------------
- AssertionError: Data mismatch (exp: expected, act: actual)
-
- new_column
- exp act
- 0 0.0 7.0
-
- ----------------------------------------------------------------------
- Ran 1 test in 0.020s
-
- FAILED (failures=1)
- ```
+```
+SQLMesh:
+
+- Generated unit tests for the `sqlmesh_example.full_model` model by live querying the data.
+- Ran the tests and they passed locally in DuckDB.
+- If you're using a cloud data warehouse, this will transpile your SQL syntax to its equivalent in duckdb.
+- This runs fast and free on your local machine.
+
+Generated test definition file:
+
+```yaml linenums="1" title="tests/test_full_model.yaml"
+test_full_model:
+ model: '"db"."sqlmesh_example"."full_model"'
+ inputs:
+ '"db"."sqlmesh_example"."incremental_model"':
+ - id: -11
+ item_id: -11
+ event_date: 2020-01-01
+ new_column: 7
+ - id: 1
+ item_id: 1
+ event_date: 2020-01-01
+ new_column: 7
+ - id: 3
+ item_id: 3
+ event_date: 2020-01-03
+ new_column: 7
+ - id: 4
+ item_id: 1
+ event_date: 2020-01-04
+ new_column: 7
+ - id: 5
+ item_id: 1
+ event_date: 2020-01-05
+ new_column: 7
+ outputs:
+ query:
+ - item_id: 3
+ num_orders: 1
+ new_column: 7
+ - item_id: 1
+ num_orders: 3
+ new_column: 7
+ - item_id: -11
+ num_orders: 1
+ new_column: 7
+```
+
+Manually execute tests with `sqlmesh test`:
+
+```bash
+(demo) ➜ demo git:(main) ✗ sqlmesh test
+.
+----------------------------------------------------------------------
+Ran 1 test in 0.053s
+
+OK
+```
+
+```bash
+# what do we see if the test fails?
+(demo) ➜ demo git:(main) ✗ sqlmesh test
+F
+======================================================================
+FAIL: test_full_model (/Users/sung/Desktop/git_repos/sqlmesh-cli-revamp/tests/test_full_model.yaml)
+None
+----------------------------------------------------------------------
+AssertionError: Data mismatch (exp: expected, act: actual)
+
+ new_column
+ exp act
+0 0.0 7.0
+
+----------------------------------------------------------------------
+Ran 1 test in 0.020s
+
+FAILED (failures=1)
+```
+```
+
+
### Run Ad-Hoc Queries
@@ -635,35 +682,32 @@ Pro tip: run this after `sqlmesh table_diff` to get a full picture of your chang
=== "SQLMesh"
- ```bash
- sqlmesh fetchdf "select * from sqlmesh_example__dev.full_model limit 5"
- ```
+```
+```bash
+sqlmesh fetchdf "select * from sqlmesh_example__dev.full_model limit 5"
+```
- ```bash
- # construct arbitrary query
- sqlmesh fetchdf "select * from . limit 5" # double underscore in schema name is important. Not needed for prod.
- ```
+```bash
+# construct arbitrary query
+sqlmesh fetchdf "select * from . limit 5" # double underscore in schema name is important. Not needed for prod.
+```
+```
=== "Tobiko Cloud"
- ```bash
- tcloud sqlmesh fetchdf "select * from sqlmesh_example__dev.full_model limit 5"
- ```
+```
+```bash
+tcloud sqlmesh fetchdf "select * from sqlmesh_example__dev.full_model limit 5"
+```
- ```bash
- # construct arbitrary query
- tcloud sqlmesh fetchdf "select * from . limit 5" # double underscore in schema name is important. Not needed for prod.
- ```
+```bash
+# construct arbitrary query
+tcloud sqlmesh fetchdf "select * from . limit 5" # double underscore in schema name is important. Not needed for prod.
+```
+```
??? "Example Output"
- ```bash
- item_id num_orders new_column
- 0 9 1 7
- 1 -11 1 7
- 2 3 1 7
- 3 -3 1 7
- 4 1 4 7
- ```
+ `bash item_id num_orders new_column 0 9 1 7 1 -11 1 7 2 3 1 7 3 -3 1 7 4 1 4 7`
### Linting
@@ -673,58 +717,66 @@ This is a great way to catch SQL issues before wasting runtime in your data ware
=== "SQLMesh"
- ```bash
- sqlmesh lint
- ```
+```
+```bash
+sqlmesh lint
+```
+```
=== "Tobiko Cloud"
- ```bash
- tcloud sqlmesh lint
- ```
+```
+```bash
+tcloud sqlmesh lint
+```
+```
??? "Example Output"
- You add linting rules in your `config.yaml` file.
-
- ```yaml linenums="1" hl_lines="13-17" title="config.yaml"
- gateways:
- duckdb:
- connection:
- type: duckdb
- database: db.db
-
- default_gateway: duckdb
-
- model_defaults:
- dialect: duckdb
- start: 2025-03-26
-
- linter:
- enabled: true
- rules: ["ambiguousorinvalidcolumn", "invalidselectstarexpansion"] # raise errors for these rules
- warn_rules: ["noselectstar", "nomissingaudits"]
- # ignored_rules: ["noselectstar"]
- ```
-
- ```bash
- > sqlmesh lint
- [WARNING] Linter warnings for /Users/sung/Desktop/git_repos/sqlmesh-cli-revamp/models/lint_warn.sql:
- - noselectstar: Query should not contain SELECT * on its outer most projections, even if it can be
- expanded.
- - nomissingaudits: Model `audits` must be configured to test data quality.
- [WARNING] Linter warnings for
- /Users/sung/Desktop/git_repos/sqlmesh-cli-revamp/models/incremental_by_partition.sql:
- - nomissingaudits: Model `audits` must be configured to test data quality.
- [WARNING] Linter warnings for /Users/sung/Desktop/git_repos/sqlmesh-cli-revamp/models/seed_model.sql:
- - nomissingaudits: Model `audits` must be configured to test data quality.
- [WARNING] Linter warnings for
- /Users/sung/Desktop/git_repos/sqlmesh-cli-revamp/models/incremental_by_unique_key.sql:
- - nomissingaudits: Model `audits` must be configured to test data quality.
- [WARNING] Linter warnings for
- /Users/sung/Desktop/git_repos/sqlmesh-cli-revamp/models/incremental_model.sql:
- - nomissingaudits: Model `audits` must be configured to test data quality.
- ```
+```
+You add linting rules in your `config.yaml` file.
+
+```yaml linenums="1" hl_lines="13-17" title="config.yaml"
+gateways:
+ duckdb:
+ connection:
+ type: duckdb
+ database: db.db
+
+default_gateway: duckdb
+
+model_defaults:
+ dialect: duckdb
+ start: 2025-03-26
+
+linter:
+ enabled: true
+ rules: ["ambiguousorinvalidcolumn", "invalidselectstarexpansion"] # raise errors for these rules
+ warn_rules: ["noselectstar", "nomissingaudits"]
+ # ignored_rules: ["noselectstar", "nomissingunittest"]
+```
+
+```bash
+> sqlmesh lint
+[WARNING] Linter warnings for /Users/sung/Desktop/git_repos/sqlmesh-cli-revamp/models/lint_warn.sql:
+- noselectstar: Query should not contain SELECT * on its outer most projections, even if it can be
+expanded.
+- nomissingaudits: Model `audits` must be configured to test data quality.
+[WARNING] Linter warnings for
+/Users/sung/Desktop/git_repos/sqlmesh-cli-revamp/models/incremental_by_partition.sql:
+- nomissingaudits: Model `audits` must be configured to test data quality.
+[WARNING] Linter warnings for /Users/sung/Desktop/git_repos/sqlmesh-cli-revamp/models/seed_model.sql:
+- nomissingaudits: Model `audits` must be configured to test data quality.
+[WARNING] Linter warnings for
+/Users/sung/Desktop/git_repos/sqlmesh-cli-revamp/models/incremental_by_unique_key.sql:
+- nomissingaudits: Model `audits` must be configured to test data quality.
+[WARNING] Linter warnings for
+/Users/sung/Desktop/git_repos/sqlmesh-cli-revamp/models/incremental_model.sql:
+- nomissingaudits: Model `audits` must be configured to test data quality.
+```
+```
+
+
## Debugging Workflow
@@ -734,95 +786,105 @@ You'll use these commands as needed to validate that your changes are behaving a
2. Run SQLMesh in verbose mode so you can verify its behavior.
3. View the logs easily in your terminal.
+
+
### Render your SQL Changes
This is a great way to verify that your model's SQL is looking as expected before applying the changes. It is especially important if you're migrating from one query engine to another (ex: postgres to databricks).
=== "SQLMesh"
- ```bash
- sqlmesh render sqlmesh_example.incremental_model
- ```
+```
+```bash
+sqlmesh render sqlmesh_example.incremental_model
+```
- ```bash
- sqlmesh render sqlmesh_example.incremental_model --dialect databricks
- ```
+```bash
+sqlmesh render sqlmesh_example.incremental_model --dialect databricks
+```
- ```bash
- sqlmesh render --dialect
- ```
+```bash
+sqlmesh render --dialect
+```
+```
=== "Tobiko Cloud"
- ```bash
- tcloud sqlmesh render sqlmesh_example.incremental_model
- ```
+```
+```bash
+tcloud sqlmesh render sqlmesh_example.incremental_model
+```
- ```bash
- tcloud sqlmesh render sqlmesh_example.incremental_model --dialect databricks
- ```
+```bash
+tcloud sqlmesh render sqlmesh_example.incremental_model --dialect databricks
+```
- ```bash
- tcloud sqlmesh render --dialect
- ```
+```bash
+tcloud sqlmesh render --dialect
+```
+```
??? "Example Output"
- Model definition:
+```
+Model definition:
+
+```sql linenums="1" title="models/incremental_model.sql"
+MODEL (
+ name sqlmesh_example.incremental_model,
+ kind INCREMENTAL_BY_TIME_RANGE (
+ time_column event_date
+ ),
+ start '2020-01-01',
+ cron '@daily',
+ grain (id, event_date)
+);
+
+SELECT
+ id,
+ item_id,
+ event_date,
+ 7 as new_column
+FROM
+ sqlmesh_example.seed_model
+WHERE
+ event_date BETWEEN @start_date AND @end_date
+```
+
+SQLMesh returns the full SQL code in the default or target dialect.
+
+```sql hl_lines="11"
+> sqlmesh render sqlmesh_example.incremental_model
+-- rendered sql in default dialect
+SELECT
+ "seed_model"."id" AS "id",
+ "seed_model"."item_id" AS "item_id",
+ "seed_model"."event_date" AS "event_date",
+ 7 AS "new_column"
+FROM "db"."sqlmesh__sqlmesh_example"."sqlmesh_example__seed_model__3294646944" AS "seed_model" /*
+db.sqlmesh_example.seed_model */
+WHERE
+ "seed_model"."event_date" <= CAST('1970-01-01' AS DATE) -- placeholder dates for date macros
+ AND "seed_model"."event_date" >= CAST('1970-01-01' AS DATE)
+```
+
+```sql
+> sqlmesh render sqlmesh_example.incremental_model --dialect databricks
+-- rendered sql in databricks dialect
+SELECT
+ `seed_model`.`id` AS `id`,
+ `seed_model`.`item_id` AS `item_id`,
+ `seed_model`.`event_date` AS `event_date`,
+ 7 AS `new_column`
+FROM `db`.`sqlmesh__sqlmesh_example`.`sqlmesh_example__seed_model__3294646944` AS `seed_model` /*
+db.sqlmesh_example.seed_model */
+WHERE
+ `seed_model`.`event_date` <= CAST('1970-01-01' AS DATE)
+ AND `seed_model`.`event_date` >= CAST('1970-01-01' AS DATE)
+```
+```
- ```sql linenums="1" title="models/incremental_model.sql"
- MODEL (
- name sqlmesh_example.incremental_model,
- kind INCREMENTAL_BY_TIME_RANGE (
- time_column event_date
- ),
- start '2020-01-01',
- cron '@daily',
- grain (id, event_date)
- );
- SELECT
- id,
- item_id,
- event_date,
- 7 as new_column
- FROM
- sqlmesh_example.seed_model
- WHERE
- event_date BETWEEN @start_date AND @end_date
- ```
-
- SQLMesh returns the full SQL code in the default or target dialect.
-
- ```sql hl_lines="11"
- > sqlmesh render sqlmesh_example.incremental_model
- -- rendered sql in default dialect
- SELECT
- "seed_model"."id" AS "id",
- "seed_model"."item_id" AS "item_id",
- "seed_model"."event_date" AS "event_date",
- 7 AS "new_column"
- FROM "db"."sqlmesh__sqlmesh_example"."sqlmesh_example__seed_model__3294646944" AS "seed_model" /*
- db.sqlmesh_example.seed_model */
- WHERE
- "seed_model"."event_date" <= CAST('1970-01-01' AS DATE) -- placeholder dates for date macros
- AND "seed_model"."event_date" >= CAST('1970-01-01' AS DATE)
- ```
-
- ```sql
- > sqlmesh render sqlmesh_example.incremental_model --dialect databricks
- -- rendered sql in databricks dialect
- SELECT
- `seed_model`.`id` AS `id`,
- `seed_model`.`item_id` AS `item_id`,
- `seed_model`.`event_date` AS `event_date`,
- 7 AS `new_column`
- FROM `db`.`sqlmesh__sqlmesh_example`.`sqlmesh_example__seed_model__3294646944` AS `seed_model` /*
- db.sqlmesh_example.seed_model */
- WHERE
- `seed_model`.`event_date` <= CAST('1970-01-01' AS DATE)
- AND `seed_model`.`event_date` >= CAST('1970-01-01' AS DATE)
- ```
### Apply Plan Changes in Verbose Mode
@@ -830,81 +892,89 @@ Verbose mode lets you see detailed operations in the physical and virtual layers
=== "SQLMesh"
- ```bash
- sqlmesh plan dev -vv
- ```
+```
+```bash
+sqlmesh plan dev -vv
+```
- ```bash
- sqlmesh plan -vv
- ```
+```bash
+sqlmesh plan -vv
+```
+```
=== "Tobiko Cloud"
- ```bash
- tcloud sqlmesh plan dev -vv
- ```
+```
+```bash
+tcloud sqlmesh plan dev -vv
+```
- ```bash
- tcloud sqlmesh plan -vv
- ```
+```bash
+tcloud sqlmesh plan -vv
+```
+```
??? "Example Output"
- ```bash hl_lines="48-50"
- > sqlmesh plan dev -vv
- [WARNING] Linter warnings for
- /Users/sung/Desktop/git_repos/sqlmesh-cli-revamp/models/incremental_by_partition.sql:
- - nomissingaudits: Model `audits` must be configured to test data quality.
- [WARNING] Linter warnings for /Users/sung/Desktop/git_repos/sqlmesh-cli-revamp/models/seed_model.sql:
- - nomissingaudits: Model `audits` must be configured to test data quality.
- [WARNING] Linter warnings for
- /Users/sung/Desktop/git_repos/sqlmesh-cli-revamp/models/incremental_by_unique_key.sql:
- - nomissingaudits: Model `audits` must be configured to test data quality.
- [WARNING] Linter warnings for
- /Users/sung/Desktop/git_repos/sqlmesh-cli-revamp/models/incremental_model.sql:
- - nomissingaudits: Model `audits` must be configured to test data quality.
+```
+```bash hl_lines="48-50"
+> sqlmesh plan dev -vv
+[WARNING] Linter warnings for
+/Users/sung/Desktop/git_repos/sqlmesh-cli-revamp/models/incremental_by_partition.sql:
+- nomissingaudits: Model `audits` must be configured to test data quality.
+[WARNING] Linter warnings for /Users/sung/Desktop/git_repos/sqlmesh-cli-revamp/models/seed_model.sql:
+- nomissingaudits: Model `audits` must be configured to test data quality.
+[WARNING] Linter warnings for
+/Users/sung/Desktop/git_repos/sqlmesh-cli-revamp/models/incremental_by_unique_key.sql:
+- nomissingaudits: Model `audits` must be configured to test data quality.
+[WARNING] Linter warnings for
+/Users/sung/Desktop/git_repos/sqlmesh-cli-revamp/models/incremental_model.sql:
+- nomissingaudits: Model `audits` must be configured to test data quality.
+
+Differences from the `dev` environment:
+
+Models:
+├── Directly Modified:
+│ └── db.sqlmesh_example__dev.incremental_model
+└── Indirectly Modified:
+ ├── db.sqlmesh_example__dev.full_model
+ └── db.sqlmesh_example__dev.view_model
- Differences from the `dev` environment:
+---
- Models:
- ├── Directly Modified:
- │ └── db.sqlmesh_example__dev.incremental_model
- └── Indirectly Modified:
- ├── db.sqlmesh_example__dev.full_model
- └── db.sqlmesh_example__dev.view_model
++++
- ---
+@@ -15,7 +15,7 @@
- +++
+ id,
+ item_id,
+ event_date,
+- 9 AS new_column
++ 7 AS new_column
+FROM sqlmesh_example.seed_model
+WHERE
+ event_date BETWEEN @start_date AND @end_date
- @@ -15,7 +15,7 @@
+Directly Modified: db.sqlmesh_example__dev.incremental_model (Breaking)
+└── Indirectly Modified Children:
+ ├── db.sqlmesh_example__dev.full_model (Breaking)
+ └── db.sqlmesh_example__dev.view_model (Indirect Breaking)
+Apply - Virtual Update [y/n]: y
- id,
- item_id,
- event_date,
- - 9 AS new_column
- + 7 AS new_column
- FROM sqlmesh_example.seed_model
- WHERE
- event_date BETWEEN @start_date AND @end_date
+SKIP: No physical layer updates to perform
- Directly Modified: db.sqlmesh_example__dev.incremental_model (Breaking)
- └── Indirectly Modified Children:
- ├── db.sqlmesh_example__dev.full_model (Breaking)
- └── db.sqlmesh_example__dev.view_model (Indirect Breaking)
- Apply - Virtual Update [y/n]: y
+SKIP: No model batches to execute
- SKIP: No physical layer updates to perform
+db.sqlmesh_example__dev.incremental_model updated # you'll notice that it's updated vs. promoted because we changed the existing view definition
+db.sqlmesh_example__dev.full_model updated
+db.sqlmesh_example__dev.view_model updated
+Updating virtual layer ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100.0% • 3/3 • 0:00:00
- SKIP: No model batches to execute
+✔ Virtual layer updated
+```
+```
- db.sqlmesh_example__dev.incremental_model updated # you'll notice that it's updated vs. promoted because we changed the existing view definition
- db.sqlmesh_example__dev.full_model updated
- db.sqlmesh_example__dev.view_model updated
- Updating virtual layer ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100.0% • 3/3 • 0:00:00
- ✔ Virtual layer updated
- ```
### View Logs Easily
@@ -927,35 +997,39 @@ bat --theme='ansi' $(ls -t logs/ | head -n 1 | sed 's/^/logs\//')
??? "Example Output"
- This is the log file for the `sqlmesh plan dev` command. If you want to see the log file directly, you can click on the file path in the output to open it in your code editor.
-
- ```bash
- ──────┬──────────────────────────────────────────────────────────────────────────────────────────────
- │ File: logs/sqlmesh_2025_04_18_12_34_35.log
- ──────┼──────────────────────────────────────────────────────────────────────────────────────────────
- 1 │ 2025-04-18 12:34:35,715 - MainThread - sqlmesh.core.config.connection - INFO - Creating new D
- │ uckDB adapter for data files: {'db.db'} (connection.py:319)
- 2 │ 2025-04-18 12:34:35,951 - MainThread - sqlmesh.core.console - WARNING - Linter warnings for /
- │ Users/sung/Desktop/git_repos/sqlmesh-cli-revamp/models/incremental_by_partition.sql:
- 3 │ - nomissingaudits: Model `audits` must be configured to test data quality. (console.py:1848)
- 4 │ 2025-04-18 12:34:35,953 - MainThread - sqlmesh.core.console - WARNING - Linter warnings for /
- │ Users/sung/Desktop/git_repos/sqlmesh-cli-revamp/models/seed_model.sql:
- 5 │ - nomissingaudits: Model `audits` must be configured to test data quality. (console.py:1848)
- 6 │ 2025-04-18 12:34:35,953 - MainThread - sqlmesh.core.console - WARNING - Linter warnings for /
- │ Users/sung/Desktop/git_repos/sqlmesh-cli-revamp/models/incremental_by_unique_key.sql:
- 7 │ - nomissingaudits: Model `audits` must be configured to test data quality. (console.py:1848)
- 8 │ 2025-04-18 12:34:35,953 - MainThread - sqlmesh.core.console - WARNING - Linter warnings for /
- │ Users/sung/Desktop/git_repos/sqlmesh-cli-revamp/models/incremental_model.sql:
- 9 │ - nomissingaudits: Model `audits` must be configured to test data quality. (console.py:1848)
- 10 │ 2025-04-18 12:34:35,954 - MainThread - sqlmesh.core.config.connection - INFO - Using existing
- │ DuckDB adapter due to overlapping data file: db.db (connection.py:309)
- 11 │ 2025-04-18 12:34:37,071 - MainThread - sqlmesh.core.snapshot.evaluator - INFO - Listing data
- │ objects in schema db.sqlmesh__sqlmesh_example (evaluator.py:338)
- 12 │ 2025-04-18 12:34:37,072 - MainThread - sqlmesh.core.engine_adapter.base - INFO - Executing SQ
- │ L: SELECT CURRENT_CATALOG() (base.py:2128)
- 13 │ 2025-04-18 12:34:37,072 - MainThread - sqlmesh.core.engine_adapter.base - INFO - Executing SQ
- │ L: SELECT CURRENT_CATALOG() (base.py:2128)
- ```
+```
+This is the log file for the `sqlmesh plan dev` command. If you want to see the log file directly, you can click on the file path in the output to open it in your code editor.
+
+```bash
+──────┬──────────────────────────────────────────────────────────────────────────────────────────────
+ │ File: logs/sqlmesh_2025_04_18_12_34_35.log
+──────┼──────────────────────────────────────────────────────────────────────────────────────────────
+ 1 │ 2025-04-18 12:34:35,715 - MainThread - sqlmesh.core.config.connection - INFO - Creating new D
+ │ uckDB adapter for data files: {'db.db'} (connection.py:319)
+ 2 │ 2025-04-18 12:34:35,951 - MainThread - sqlmesh.core.console - WARNING - Linter warnings for /
+ │ Users/sung/Desktop/git_repos/sqlmesh-cli-revamp/models/incremental_by_partition.sql:
+ 3 │ - nomissingaudits: Model `audits` must be configured to test data quality. (console.py:1848)
+ 4 │ 2025-04-18 12:34:35,953 - MainThread - sqlmesh.core.console - WARNING - Linter warnings for /
+ │ Users/sung/Desktop/git_repos/sqlmesh-cli-revamp/models/seed_model.sql:
+ 5 │ - nomissingaudits: Model `audits` must be configured to test data quality. (console.py:1848)
+ 6 │ 2025-04-18 12:34:35,953 - MainThread - sqlmesh.core.console - WARNING - Linter warnings for /
+ │ Users/sung/Desktop/git_repos/sqlmesh-cli-revamp/models/incremental_by_unique_key.sql:
+ 7 │ - nomissingaudits: Model `audits` must be configured to test data quality. (console.py:1848)
+ 8 │ 2025-04-18 12:34:35,953 - MainThread - sqlmesh.core.console - WARNING - Linter warnings for /
+ │ Users/sung/Desktop/git_repos/sqlmesh-cli-revamp/models/incremental_model.sql:
+ 9 │ - nomissingaudits: Model `audits` must be configured to test data quality. (console.py:1848)
+ 10 │ 2025-04-18 12:34:35,954 - MainThread - sqlmesh.core.config.connection - INFO - Using existing
+ │ DuckDB adapter due to overlapping data file: db.db (connection.py:309)
+ 11 │ 2025-04-18 12:34:37,071 - MainThread - sqlmesh.core.snapshot.evaluator - INFO - Listing data
+ │ objects in schema db.sqlmesh__sqlmesh_example (evaluator.py:338)
+ 12 │ 2025-04-18 12:34:37,072 - MainThread - sqlmesh.core.engine_adapter.base - INFO - Executing SQ
+ │ L: SELECT CURRENT_CATALOG() (base.py:2128)
+ 13 │ 2025-04-18 12:34:37,072 - MainThread - sqlmesh.core.engine_adapter.base - INFO - Executing SQ
+ │ L: SELECT CURRENT_CATALOG() (base.py:2128)
+```
+```
+
+
## Run on Production Schedule
@@ -985,55 +1059,63 @@ This command is intended be run on a schedule. It will skip the physical and vir
=== "SQLMesh"
- ```bash
- sqlmesh run
- ```
+```
+```bash
+sqlmesh run
+```
+```
=== "Tobiko Cloud"
- ```bash
- tcloud sqlmesh run
- ```
+```
+```bash
+tcloud sqlmesh run
+```
+```
??? "Example Output"
- This is what it looks like if models are ready to run.
-
- ```bash
- > sqlmesh run
- [1/1] sqlmesh_example.incremental_model [insert 2025-04-17 - 2025-04-17]
- 0.01s
- [1/1] sqlmesh_example.incremental_unique_model [insert/update rows]
- 0.01s
- [1/1] sqlmesh_example_v3.incremental_partition_model [insert partitions]
- 0.01s
- Executing model batches ━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━ 40.0% • 2/5 • 0:00:00
- sqlmesh_example_v3.incremental_partition_model .
- [WARNING] sqlmesh_example.full_model: 'assert_positive_order_ids' audit error: 2 rows failed. Learn
- more in logs: /Users/sung/Desktop/git_repos/sqlmesh-cli-revamp/logs/sqlmesh_2025_04_18_12_48_35.log
- [1/1] sqlmesh_example.full_model [full refresh, audits ❌1]
- 0.01s
- Executing model batches ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━ 80.0% • 4/5 • 0:00:00
- sqlmesh_example.view_model .
- [WARNING] sqlmesh_example.view_model: 'assert_positive_order_ids' audit error: 2 rows failed. Learn
- more in logs: /Users/sung/Desktop/git_repos/sqlmesh-cli-revamp/logs/sqlmesh_2025_04_18_12_48_35.log
- [1/1] sqlmesh_example.view_model [recreate view, audits ✔2 ❌1]
- 0.01s
- Executing model batches ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100.0% • 5/5 • 0:00:00
-
- ✔ Model batches executed
-
- Run finished for environment 'prod'
- ```
-
- This is what it looks like if no models are ready to run.
-
- ```bash
- > sqlmesh run
- No models are ready to run. Please wait until a model `cron` interval has elapsed.
-
- Next run will be ready at 2025-04-18 05:00PM PDT (2025-04-19 12:00AM UTC).
- ```
+```
+This is what it looks like if models are ready to run.
+
+```bash
+> sqlmesh run
+[1/1] sqlmesh_example.incremental_model [insert 2025-04-17 - 2025-04-17]
+0.01s
+[1/1] sqlmesh_example.incremental_unique_model [insert/update rows]
+0.01s
+[1/1] sqlmesh_example_v3.incremental_partition_model [insert partitions]
+0.01s
+Executing model batches ━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━ 40.0% • 2/5 • 0:00:00
+sqlmesh_example_v3.incremental_partition_model .
+[WARNING] sqlmesh_example.full_model: 'assert_positive_order_ids' audit error: 2 rows failed. Learn
+more in logs: /Users/sung/Desktop/git_repos/sqlmesh-cli-revamp/logs/sqlmesh_2025_04_18_12_48_35.log
+[1/1] sqlmesh_example.full_model [full refresh, audits ❌1]
+0.01s
+Executing model batches ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━ 80.0% • 4/5 • 0:00:00
+sqlmesh_example.view_model .
+[WARNING] sqlmesh_example.view_model: 'assert_positive_order_ids' audit error: 2 rows failed. Learn
+more in logs: /Users/sung/Desktop/git_repos/sqlmesh-cli-revamp/logs/sqlmesh_2025_04_18_12_48_35.log
+[1/1] sqlmesh_example.view_model [recreate view, audits ✔2 ❌1]
+0.01s
+Executing model batches ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100.0% • 5/5 • 0:00:00
+
+✔ Model batches executed
+
+Run finished for environment 'prod'
+```
+
+This is what it looks like if no models are ready to run.
+
+```bash
+> sqlmesh run
+No models are ready to run. Please wait until a model `cron` interval has elapsed.
+
+Next run will be ready at 2025-04-18 05:00PM PDT (2025-04-19 12:00AM UTC).
+```
+```
+
+
### Run Models with Incomplete Intervals (Warning)
@@ -1045,57 +1127,65 @@ You can run models that execute backfills each time you invoke a `run`, whether
=== "SQLMesh"
- ```bash
- sqlmesh run --ignore-cron
- ```
+```
+```bash
+sqlmesh run --ignore-cron
+```
+```
=== "Tobiko Cloud"
- ```bash
- tcloud sqlmesh run --ignore-cron
- ```
+```
+```bash
+tcloud sqlmesh run --ignore-cron
+```
+```
??? "Example Output"
- Model definition:
- ```sql linenums="1" hl_lines="15" title="models/incremental_model.sql"
- MODEL (
- name sqlmesh_example.incremental_model,
- kind INCREMENTAL_BY_TIME_RANGE (
- time_column event_date
- ),
- start '2020-01-01',
- cron '@daily',
- grain (id, event_date),
- audits( UNIQUE_VALUES(columns = (
- id,
- )), NOT_NULL(columns = (
- id,
- event_date
- ))),
- allow_partials true
- );
-
- SELECT
+```
+Model definition:
+```sql linenums="1" hl_lines="15" title="models/incremental_model.sql"
+MODEL (
+ name sqlmesh_example.incremental_model,
+ kind INCREMENTAL_BY_TIME_RANGE (
+ time_column event_date
+ ),
+ start '2020-01-01',
+ cron '@daily',
+ grain (id, event_date),
+ audits( UNIQUE_VALUES(columns = (
+ id,
+ )), NOT_NULL(columns = (
id,
- item_id,
- event_date,
- 16 as new_column
- FROM
- sqlmesh_example.seed_model
- WHERE
- event_date BETWEEN @start_date AND @end_date
- ```
+ event_date
+ ))),
+ allow_partials true
+);
+
+SELECT
+ id,
+ item_id,
+ event_date,
+ 16 as new_column
+FROM
+ sqlmesh_example.seed_model
+WHERE
+ event_date BETWEEN @start_date AND @end_date
+```
- ```bash
- > sqlmesh run --ignore-cron
- [1/1] sqlmesh_example.incremental_model [insert 2025-04-19 - 2025-04-19, audits ✔2] 0.05s
- Executing model batches ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100.0% • 1/1 • 0:00:00
+```bash
+> sqlmesh run --ignore-cron
+[1/1] sqlmesh_example.incremental_model [insert 2025-04-19 - 2025-04-19, audits ✔2] 0.05s
+Executing model batches ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100.0% • 1/1 • 0:00:00
+
+✔ Model batches executed
+
+Run finished for environment 'prod'
+```
+```
- ✔ Model batches executed
- Run finished for environment 'prod'
- ```
## Forward-Only Development Workflow
@@ -1111,133 +1201,140 @@ If you want to see a full walkthrough, [go here](incremental_time_full_walkthrou
=== "SQLMesh"
- ```bash
- sqlmesh plan dev --forward-only
- ```
+```
+```bash
+sqlmesh plan dev --forward-only
+```
- ```bash
- sqlmesh plan --forward-only
- ```
+```bash
+sqlmesh plan --forward-only
+```
+```
=== "Tobiko Cloud"
- ```bash
- tcloud sqlmesh plan dev --forward-only
- ```
+```
+```bash
+tcloud sqlmesh plan dev --forward-only
+```
- ```bash
- tcloud sqlmesh plan --forward-only
- ```
+```bash
+tcloud sqlmesh plan --forward-only
+```
+```
??? "Example Output"
- - I applied a change to a new column
- - It impacts 2 downstream models
- - I enforced a forward-only plan to avoid backfilling historical data for the incremental model (ex: `preview` language in the CLI output)
- - I previewed the changes in a clone of the incremental impacted (clones will NOT be reused in production) along with the full and view models (these are NOT clones).
+```
+- I applied a change to a new column
+- It impacts 2 downstream models
+- I enforced a forward-only plan to avoid backfilling historical data for the incremental model (ex: `preview` language in the CLI output)
+- I previewed the changes in a clone of the incremental impacted (clones will NOT be reused in production) along with the full and view models (these are NOT clones).
+
+```bash
+> sqlmesh plan dev
+Differences from the `dev` environment:
- ```bash
- > sqlmesh plan dev
- Differences from the `dev` environment:
+Models:
+├── Directly Modified:
+│ └── sqlmesh_example__dev.incremental_model
+└── Indirectly Modified:
+ ├── sqlmesh_example__dev.view_model
+ └── sqlmesh_example__dev.full_model
- Models:
- ├── Directly Modified:
- │ └── sqlmesh_example__dev.incremental_model
- └── Indirectly Modified:
- ├── sqlmesh_example__dev.view_model
- └── sqlmesh_example__dev.full_model
+---
- ---
++++
- +++
+@@ -16,7 +16,7 @@
- @@ -16,7 +16,7 @@
+ id,
+ item_id,
+ event_date,
+- 9 AS new_column
++ 10 AS new_column
+FROM sqlmesh_example.seed_model
+WHERE
+ event_date BETWEEN @start_date AND @end_date
- id,
- item_id,
- event_date,
- - 9 AS new_column
- + 10 AS new_column
- FROM sqlmesh_example.seed_model
- WHERE
- event_date BETWEEN @start_date AND @end_date
-
- Directly Modified: sqlmesh_example__dev.incremental_model (Forward-only)
- └── Indirectly Modified Children:
- ├── sqlmesh_example__dev.full_model (Forward-only)
- └── sqlmesh_example__dev.view_model (Forward-only)
- Models needing backfill:
- ├── sqlmesh_example__dev.full_model: [full refresh] (preview)
- ├── sqlmesh_example__dev.incremental_model: [2025-04-17 - 2025-04-17] (preview)
- └── sqlmesh_example__dev.view_model: [recreate view] (preview)
- Apply - Preview Tables [y/n]: y
-
- Updating physical layer ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100.0% • 3/3 • 0:00:00
-
- ✔ Physical layer updated
-
- [1/1] sqlmesh_example__dev.incremental_model [insert 2025-04-17 - 2025-04-17] 0.01s
- [1/1] sqlmesh_example__dev.full_model [full refresh, audits ✔1] 0.01s
- [1/1] sqlmesh_example__dev.view_model [recreate view, audits ✔3] 0.01s
- Executing model batches ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100.0% • 3/3 • 0:00:00
-
- ✔ Model batches executed
-
- Updating virtual layer ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100.0% • 3/3 • 0:00:00
-
- ✔ Virtual layer updated
- ```
-
- When the plan is applied to `prod`, it will only execute model batches for new intervals (new rows). This will NOT re-use `preview` models (backfilled data) in development.
-
- ```bash
- > sqlmesh plan
- Differences from the `prod` environment:
-
- Models:
- ├── Directly Modified:
- │ └── sqlmesh_example.incremental_model
- └── Indirectly Modified:
- ├── sqlmesh_example.view_model
- └── sqlmesh_example.full_model
-
- ---
-
- +++
-
- @@ -9,13 +9,14 @@
-
- disable_restatement FALSE,
- on_destructive_change 'ERROR'
- ),
- - grains ((id, event_date))
- + grains ((id, event_date)),
- + allow_partials TRUE
- )
- SELECT
- id,
- item_id,
- event_date,
- - 7 AS new_column
- + 10 AS new_column
- FROM sqlmesh_example.seed_model
- WHERE
- event_date BETWEEN @start_date AND @end_date
+Directly Modified: sqlmesh_example__dev.incremental_model (Forward-only)
+└── Indirectly Modified Children:
+ ├── sqlmesh_example__dev.full_model (Forward-only)
+ └── sqlmesh_example__dev.view_model (Forward-only)
+Models needing backfill:
+├── sqlmesh_example__dev.full_model: [full refresh] (preview)
+├── sqlmesh_example__dev.incremental_model: [2025-04-17 - 2025-04-17] (preview)
+└── sqlmesh_example__dev.view_model: [recreate view] (preview)
+Apply - Preview Tables [y/n]: y
+
+Updating physical layer ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100.0% • 3/3 • 0:00:00
- Directly Modified: sqlmesh_example.incremental_model (Forward-only)
- └── Indirectly Modified Children:
- ├── sqlmesh_example.full_model (Forward-only)
- └── sqlmesh_example.view_model (Forward-only)
- Apply - Virtual Update [y/n]: y
+✔ Physical layer updated
- SKIP: No physical layer updates to perform
+[1/1] sqlmesh_example__dev.incremental_model [insert 2025-04-17 - 2025-04-17] 0.01s
+[1/1] sqlmesh_example__dev.full_model [full refresh, audits ✔1] 0.01s
+[1/1] sqlmesh_example__dev.view_model [recreate view, audits ✔3] 0.01s
+Executing model batches ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100.0% • 3/3 • 0:00:00
- SKIP: No model batches to execute
+✔ Model batches executed
+
+Updating virtual layer ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100.0% • 3/3 • 0:00:00
+
+✔ Virtual layer updated
+```
+
+When the plan is applied to `prod`, it will only execute model batches for new intervals (new rows). This will NOT re-use `preview` models (backfilled data) in development.
+
+```bash
+> sqlmesh plan
+Differences from the `prod` environment:
+
+Models:
+├── Directly Modified:
+│ └── sqlmesh_example.incremental_model
+└── Indirectly Modified:
+ ├── sqlmesh_example.view_model
+ └── sqlmesh_example.full_model
+
+---
- Updating virtual layer ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100.0% • 3/3 • 0:00:00
++++
+
+@@ -9,13 +9,14 @@
+
+ disable_restatement FALSE,
+ on_destructive_change 'ERROR'
+ ),
+- grains ((id, event_date))
++ grains ((id, event_date)),
++ allow_partials TRUE
+)
+SELECT
+ id,
+ item_id,
+ event_date,
+- 7 AS new_column
++ 10 AS new_column
+FROM sqlmesh_example.seed_model
+WHERE
+ event_date BETWEEN @start_date AND @end_date
+
+Directly Modified: sqlmesh_example.incremental_model (Forward-only)
+└── Indirectly Modified Children:
+ ├── sqlmesh_example.full_model (Forward-only)
+ └── sqlmesh_example.view_model (Forward-only)
+Apply - Virtual Update [y/n]: y
+
+SKIP: No physical layer updates to perform
+
+SKIP: No model batches to execute
+
+Updating virtual layer ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100.0% • 3/3 • 0:00:00
+
+✔ Virtual layer updated
+```
+```
- ✔ Virtual layer updated
- ```
## Miscellaneous
@@ -1246,12 +1343,17 @@ If you notice you have a lot of old development schemas/data, you can clean them
=== "SQLMesh"
- ```bash
- sqlmesh janitor
- ```
+```
+```bash
+sqlmesh janitor
+```
+```
=== "Tobiko Cloud"
- ```bash
- tcloud sqlmesh janitor
- ```
\ No newline at end of file
+```
+```bash
+tcloud sqlmesh janitor
+```
+```
+
diff --git a/docs/guides/linter.md b/docs/guides/linter.md
index c9fc3407fe..0a2e3ea828 100644
--- a/docs/guides/linter.md
+++ b/docs/guides/linter.md
@@ -74,6 +74,7 @@ Here are all of SQLMesh's built-in linting rules:
| `invalidselectstarexpansion` | Correctness | The query's top-level selection may be `SELECT *`, but only if SQLMesh can expand the `SELECT *` into individual columns |
| `noselectstar` | Stylistic | The query's top-level selection may not be `SELECT *`, even if SQLMesh can expand the `SELECT *` into individual columns |
| `nomissingaudits` | Governance | SQLMesh did not find any `audits` in the model's configuration to test data quality. |
+| `nomissingunittest` | Governance | SQLMesh did not find any `unit tests` associated with the model to test |
### User-defined rules
From e8edddb837505c987ea728f807143e1b7bf0207f Mon Sep 17 00:00:00 2001
From: Cortland Goffena <30168413+cmgoffena13@users.noreply.github.com>
Date: Wed, 29 Jul 2026 18:19:30 -0600
Subject: [PATCH 2/2] chore(docs): restore crash course formatting
Co-authored-by: Cursor
Signed-off-by: Cortland Goffena <30168413+cmgoffena13@users.noreply.github.com>
Co-authored-by: Cursor
---
docs/examples/sqlmesh_cli_crash_course.md | 1898 ++++++++++-----------
1 file changed, 898 insertions(+), 1000 deletions(-)
diff --git a/docs/examples/sqlmesh_cli_crash_course.md b/docs/examples/sqlmesh_cli_crash_course.md
index c3f891c68a..0bf5780f12 100644
--- a/docs/examples/sqlmesh_cli_crash_course.md
+++ b/docs/examples/sqlmesh_cli_crash_course.md
@@ -1,6 +1,6 @@
# SQLMesh CLI Crash Course
-
+
This doc is designed to get you intimate with a **majority** of the SQLMesh workflows you’ll use to build *and* maintain transformation data pipelines. The goal is to get SQLMesh into muscle memory in 30 minutes or less.
@@ -12,11 +12,7 @@ If you're new to how SQLMesh uses virtual data environments, [watch this quick e
!!! tip
-```
-Put this page on your second monitor or in a side by side window to swiftly copy/paste into your terminal.
-```
-
-
+ Put this page on your second monitor or in a side by side window to swiftly copy/paste into your terminal.
## Development Workflow
@@ -29,8 +25,6 @@ You’ll use these commands 80% of the time because this is how you apply the ch
5. Run data diff against prod
6. Apply the changes to prod
-
-
### Preview, Apply, and Audit Changes in `dev`
You can make changes quickly and confidently through one simple command: `sqlmesh plan dev`
@@ -43,139 +37,131 @@ Note: If you run this without making any changes, SQLMesh will prompt you to mak
=== "SQLMesh"
-```
-```bash
-sqlmesh plan dev
-```
+ ```bash
+ sqlmesh plan dev
+ ```
-```bash
-sqlmesh plan
-```
+ ```bash
+ sqlmesh plan
+ ```
-If you want to move faster, you can add the `--auto-apply` flag to skip the manual prompt and apply the plan. You should do this when you're familiar with the plan output, and don't need to see tiny changes in the diff output before applying the plan.
+ If you want to move faster, you can add the `--auto-apply` flag to skip the manual prompt and apply the plan. You should do this when you're familiar with the plan output, and don't need to see tiny changes in the diff output before applying the plan.
-```bash
-sqlmesh plan --auto-apply
-```
-```
+ ```bash
+ sqlmesh plan --auto-apply
+ ```
=== "Tobiko Cloud"
-```
-```bash
-tcloud sqlmesh plan dev
-```
+ ```bash
+ tcloud sqlmesh plan dev
+ ```
-```bash
-tcloud sqlmesh plan
-```
+ ```bash
+ tcloud sqlmesh plan
+ ```
-If you want to move faster, you can add the `--auto-apply` flag to skip the manual prompt and apply the plan. You should do this when you're familiar with the plan output, and don't need to see tiny changes in the diff output before applying the plan.
+ If you want to move faster, you can add the `--auto-apply` flag to skip the manual prompt and apply the plan. You should do this when you're familiar with the plan output, and don't need to see tiny changes in the diff output before applying the plan.
-```bash
-tcloud sqlmesh plan --auto-apply
-```
-```
+ ```bash
+ tcloud sqlmesh plan --auto-apply
+ ```
??? "Example Output"
I made a breaking change to `incremental_model` and `full_model`.
-```
-SQLMesh:
-
- - Showed me the models impacted by the changes.
- - Showed me the changes that will be made to the models.
- - Showed me the models that need to be backfilled.
- - Prompted me to apply the changes to `dev`.
- - Showed me the audit failures that raise as warnings.
- - Updated the physical layer to validate the SQL.
- - Executed the model batches by inserting the data into the physical layer.
- - Updated the virtual layer's view pointers to reflect the changes.
-
-```bash
-> sqlmesh plan dev
-Differences from the `dev` environment:
-
-Models:
-├── Directly Modified:
-│ ├── sqlmesh_example__dev.incremental_model
-│ └── sqlmesh_example__dev.full_model
-└── Indirectly Modified:
- └── sqlmesh_example__dev.view_model
-
----
-
-+++
-
-@@ -9,7 +9,8 @@
-
- SELECT
- item_id,
- COUNT(DISTINCT id) AS num_orders,
-- 6 AS new_column
-+ new_column
- FROM sqlmesh_example.incremental_model
- GROUP BY
-- item_id
-+ item_id,
-+ new_column
-
-Directly Modified: sqlmesh_example__dev.full_model (Breaking)
-
----
-
-+++
-
-@@ -15,7 +15,7 @@
-
- id,
- item_id,
- event_date,
-- 5 AS new_column
-+ 7 AS new_column
- FROM sqlmesh_example.seed_model
- WHERE
- event_date BETWEEN @start_date AND @end_date
-
-Directly Modified: sqlmesh_example__dev.incremental_model (Breaking)
-└── Indirectly Modified Children:
- └── sqlmesh_example__dev.view_model (Indirect Breaking)
-Models needing backfill:
-├── sqlmesh_example__dev.full_model: [full refresh]
-├── sqlmesh_example__dev.incremental_model: [2020-01-01 - 2025-04-16]
-└── sqlmesh_example__dev.view_model: [recreate view]
-Apply - Backfill Tables [y/n]: y
-
-Updating physical layer ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100.0% • 2/2 • 0:00:00
-
-✔ Physical layer updated
-
-[1/1] sqlmesh_example__dev.incremental_model [insert 2020-01-01 - 2025-04-16] 0.03s
-Executing model batches ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0% • pending • 0:00:00
-sqlmesh_example__dev.incremental_model .
-[WARNING] sqlmesh_example__dev.full_model: 'assert_positive_order_ids' audit error: 2 rows failed. Learn more in logs:
-/Users/sung/Desktop/git_repos/sqlmesh-cli-revamp/logs/sqlmesh_2025_04_18_10_33_43.log
-[1/1] sqlmesh_example__dev.full_model [full refresh, audits ❌1] 0.01s
-Executing model batches ━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━ 33.3% • 1/3 • 0:00:00
-sqlmesh_example__dev.full_model .
-[WARNING] sqlmesh_example__dev.view_model: 'assert_positive_order_ids' audit error: 2 rows failed. Learn more in logs:
-/Users/sung/Desktop/git_repos/sqlmesh-cli-revamp/logs/sqlmesh_2025_04_18_10_33_43.log
-[1/1] sqlmesh_example__dev.view_model [recreate view, audits ✔2 ❌1] 0.01s
-Executing model batches ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100.0% • 3/3 • 0:00:00
-
-✔ Model batches executed
-
-Updating virtual layer ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100.0% • 3/3 • 0:00:00
-
-✔ Virtual layer updated
-```
-```
-
-
+ SQLMesh:
+
+ - Showed me the models impacted by the changes.
+ - Showed me the changes that will be made to the models.
+ - Showed me the models that need to be backfilled.
+ - Prompted me to apply the changes to `dev`.
+ - Showed me the audit failures that raise as warnings.
+ - Updated the physical layer to validate the SQL.
+ - Executed the model batches by inserting the data into the physical layer.
+ - Updated the virtual layer's view pointers to reflect the changes.
+
+ ```bash
+ > sqlmesh plan dev
+ Differences from the `dev` environment:
+
+ Models:
+ ├── Directly Modified:
+ │ ├── sqlmesh_example__dev.incremental_model
+ │ └── sqlmesh_example__dev.full_model
+ └── Indirectly Modified:
+ └── sqlmesh_example__dev.view_model
+
+ ---
+
+ +++
+
+ @@ -9,7 +9,8 @@
+
+ SELECT
+ item_id,
+ COUNT(DISTINCT id) AS num_orders,
+ - 6 AS new_column
+ + new_column
+ FROM sqlmesh_example.incremental_model
+ GROUP BY
+ - item_id
+ + item_id,
+ + new_column
+
+ Directly Modified: sqlmesh_example__dev.full_model (Breaking)
+
+ ---
+
+ +++
+
+ @@ -15,7 +15,7 @@
+
+ id,
+ item_id,
+ event_date,
+ - 5 AS new_column
+ + 7 AS new_column
+ FROM sqlmesh_example.seed_model
+ WHERE
+ event_date BETWEEN @start_date AND @end_date
+
+ Directly Modified: sqlmesh_example__dev.incremental_model (Breaking)
+ └── Indirectly Modified Children:
+ └── sqlmesh_example__dev.view_model (Indirect Breaking)
+ Models needing backfill:
+ ├── sqlmesh_example__dev.full_model: [full refresh]
+ ├── sqlmesh_example__dev.incremental_model: [2020-01-01 - 2025-04-16]
+ └── sqlmesh_example__dev.view_model: [recreate view]
+ Apply - Backfill Tables [y/n]: y
+
+ Updating physical layer ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100.0% • 2/2 • 0:00:00
+
+ ✔ Physical layer updated
+
+ [1/1] sqlmesh_example__dev.incremental_model [insert 2020-01-01 - 2025-04-16] 0.03s
+ Executing model batches ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0% • pending • 0:00:00
+ sqlmesh_example__dev.incremental_model .
+ [WARNING] sqlmesh_example__dev.full_model: 'assert_positive_order_ids' audit error: 2 rows failed. Learn more in logs:
+ /Users/sung/Desktop/git_repos/sqlmesh-cli-revamp/logs/sqlmesh_2025_04_18_10_33_43.log
+ [1/1] sqlmesh_example__dev.full_model [full refresh, audits ❌1] 0.01s
+ Executing model batches ━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━ 33.3% • 1/3 • 0:00:00
+ sqlmesh_example__dev.full_model .
+ [WARNING] sqlmesh_example__dev.view_model: 'assert_positive_order_ids' audit error: 2 rows failed. Learn more in logs:
+ /Users/sung/Desktop/git_repos/sqlmesh-cli-revamp/logs/sqlmesh_2025_04_18_10_33_43.log
+ [1/1] sqlmesh_example__dev.view_model [recreate view, audits ✔2 ❌1] 0.01s
+ Executing model batches ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100.0% • 3/3 • 0:00:00
+
+ ✔ Model batches executed
+
+ Updating virtual layer ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100.0% • 3/3 • 0:00:00
+
+ ✔ Virtual layer updated
+ ```
### Run Data Diff Against Prod
-
+
Run data diff against prod. This is a good way to verify the changes are behaving as expected **after** applying them to `dev`.
@@ -183,212 +169,196 @@ To make this easier and faster, you can run data diff against all models in the
=== "SQLMesh"
-```
-```bash
-sqlmesh table_diff prod:dev sqlmesh_example.full_model --show-sample
-```
+ ```bash
+ sqlmesh table_diff prod:dev sqlmesh_example.full_model --show-sample
+ ```
-```bash
-sqlmesh table_diff : --show-sample
-```
+ ```bash
+ sqlmesh table_diff : --show-sample
+ ```
-```bash
-sqlmesh table_diff prod:dev -m '*' --show-sample
-```
-```
+ ```bash
+ sqlmesh table_diff prod:dev -m '*' --show-sample
+ ```
=== "Tobiko Cloud"
-```
-```bash
-tcloud sqlmesh table_diff prod:dev sqlmesh_example.full_model --show-sample
-```
+ ```bash
+ tcloud sqlmesh table_diff prod:dev sqlmesh_example.full_model --show-sample
+ ```
-```bash
-tcloud sqlmesh table_diff : --show-sample
-```
+ ```bash
+ tcloud sqlmesh table_diff : --show-sample
+ ```
-```bash
-tcloud sqlmesh table_diff prod:dev -m '*' --show-sample
-```
-```
+ ```bash
+ tcloud sqlmesh table_diff prod:dev -m '*' --show-sample
+ ```
??? "Example Output"
I compare the `prod` and `dev` environments for `sqlmesh_example.full_model`.
-```
-- Verified environments and models to diff along with the join on grain configured.
-- Showed me schema diffs between the environments.
-- Showed me row count diffs between the environments.
-- Showed me common rows stats between the environments.
-- Showed me sample data differences between the environments.
-- This is where your human judgement comes in to verify the changes are behaving as expected.
-
-Model definition:
-```sql linenums="1" hl_lines="6"
--- models/full_model.sql
-MODEL (
- name sqlmesh_example.full_model,
- kind FULL,
- cron '@daily',
- grain item_id, -- grain is optional BUT necessary for table diffs to work correctly. It's your primary key that is unique and not null.
- audits (assert_positive_order_ids),
-);
-
-SELECT
- item_id,
- COUNT(DISTINCT id) AS num_orders,
- new_column
-FROM
- sqlmesh_example.incremental_model
-GROUP BY item_id, new_column
-```
-
-Table diff:
-```bash
-> sqlmesh table_diff prod:dev sqlmesh_example.full_model --show-sample
-Table Diff
-├── Model:
-│ └── sqlmesh_example.full_model
-├── Environment:
-│ ├── Source: prod
-│ └── Target: dev
-├── Tables:
-│ ├── Source: db.sqlmesh_example.full_model
-│ └── Target: db.sqlmesh_example__dev.full_model
-└── Join On:
- └── item_id
-
-Schema Diff Between 'PROD' and 'DEV' environments for model 'sqlmesh_example.full_model':
-└── Schemas match
-
-
-Row Counts:
-└── PARTIAL MATCH: 5 rows (100.0%)
-
-COMMON ROWS column comparison stats:
- pct_match
-num_orders 100.0
-new_column 0.0
-
-
-COMMON ROWS sample data differences:
-Column: new_column
-┏━━━━━━━━━┳━━━━━━┳━━━━━┓
-┃ item_id ┃ PROD ┃ DEV ┃
-┡━━━━━━━━━╇━━━━━━╇━━━━━┩
-│ -11 │ 5 │ 7 │
-│ -3 │ 5 │ 7 │
-│ 1 │ 5 │ 7 │
-│ 3 │ 5 │ 7 │
-│ 9 │ 5 │ 7 │
-└─────────┴──────┴─────┘
-```
-```
-
-
+ - Verified environments and models to diff along with the join on grain configured.
+ - Showed me schema diffs between the environments.
+ - Showed me row count diffs between the environments.
+ - Showed me common rows stats between the environments.
+ - Showed me sample data differences between the environments.
+ - This is where your human judgement comes in to verify the changes are behaving as expected.
+
+ Model definition:
+ ```sql linenums="1" hl_lines="6"
+ -- models/full_model.sql
+ MODEL (
+ name sqlmesh_example.full_model,
+ kind FULL,
+ cron '@daily',
+ grain item_id, -- grain is optional BUT necessary for table diffs to work correctly. It's your primary key that is unique and not null.
+ audits (assert_positive_order_ids),
+ );
+
+ SELECT
+ item_id,
+ COUNT(DISTINCT id) AS num_orders,
+ new_column
+ FROM
+ sqlmesh_example.incremental_model
+ GROUP BY item_id, new_column
+ ```
+
+ Table diff:
+ ```bash
+ > sqlmesh table_diff prod:dev sqlmesh_example.full_model --show-sample
+ Table Diff
+ ├── Model:
+ │ └── sqlmesh_example.full_model
+ ├── Environment:
+ │ ├── Source: prod
+ │ └── Target: dev
+ ├── Tables:
+ │ ├── Source: db.sqlmesh_example.full_model
+ │ └── Target: db.sqlmesh_example__dev.full_model
+ └── Join On:
+ └── item_id
+
+ Schema Diff Between 'PROD' and 'DEV' environments for model 'sqlmesh_example.full_model':
+ └── Schemas match
+
+
+ Row Counts:
+ └── PARTIAL MATCH: 5 rows (100.0%)
+
+ COMMON ROWS column comparison stats:
+ pct_match
+ num_orders 100.0
+ new_column 0.0
+
+
+ COMMON ROWS sample data differences:
+ Column: new_column
+ ┏━━━━━━━━━┳━━━━━━┳━━━━━┓
+ ┃ item_id ┃ PROD ┃ DEV ┃
+ ┡━━━━━━━━━╇━━━━━━╇━━━━━┩
+ │ -11 │ 5 │ 7 │
+ │ -3 │ 5 │ 7 │
+ │ 1 │ 5 │ 7 │
+ │ 3 │ 5 │ 7 │
+ │ 9 │ 5 │ 7 │
+ └─────────┴──────┴─────┘
+ ```
### Apply Changes to Prod
After you feel confident about the changes, apply them to `prod`.
!!! warning "Apply the changes to prod"
- We recommend only applying changes to `prod` **[using CI/CD](../integrations/github.md)** as best practice.
+ We recommend only applying changes to `prod` [**using CI/CD**](../integrations/github.md) as best practice.
For learning purposes and hot fixes, you can manually apply the changes to prod by entering `y` at the prompt.
=== "SQLMesh"
-```
-```bash
-sqlmesh plan
-```
-```
+ ```bash
+ sqlmesh plan
+ ```
=== "Tobiko Cloud"
-```
-```bash
-tcloud sqlmesh plan
-```
-```
+ ```bash
+ tcloud sqlmesh plan
+ ```
??? "Example Output"
After I feel confident about the changes, I apply them to `prod`.
-```
-SQLMesh:
+ SQLMesh:
-- Showed me the models impacted by the changes.
-- Showed me the changes that will be made to the models.
-- Showed me the models that need to be backfilled. None in this case as it was already backfilled earlier in `dev`.
-- Prompted me to apply the changes to `prod`.
-- Showed me physical layer and execution steps are skipped as the changes were already applied to `dev`.
-- Updated the virtual layer view pointers to reflect the changes.
+ - Showed me the models impacted by the changes.
+ - Showed me the changes that will be made to the models.
+ - Showed me the models that need to be backfilled. None in this case as it was already backfilled earlier in `dev`.
+ - Prompted me to apply the changes to `prod`.
+ - Showed me physical layer and execution steps are skipped as the changes were already applied to `dev`.
+ - Updated the virtual layer view pointers to reflect the changes.
-```bash
-> sqlmesh plan
-Differences from the `prod` environment:
+ ```bash
+ > sqlmesh plan
+ Differences from the `prod` environment:
-Models:
-├── Directly Modified:
-│ ├── sqlmesh_example.full_model
-│ └── sqlmesh_example.incremental_model
-└── Indirectly Modified:
- └── sqlmesh_example.view_model
+ Models:
+ ├── Directly Modified:
+ │ ├── sqlmesh_example.full_model
+ │ └── sqlmesh_example.incremental_model
+ └── Indirectly Modified:
+ └── sqlmesh_example.view_model
----
+ ---
-+++
+ +++
-@@ -9,7 +9,8 @@
+ @@ -9,7 +9,8 @@
-SELECT
- item_id,
- COUNT(DISTINCT id) AS num_orders,
-- 5 AS new_column
-+ new_column
-FROM sqlmesh_example.incremental_model
-GROUP BY
-- item_id
-+ item_id,
-+ new_column
+ SELECT
+ item_id,
+ COUNT(DISTINCT id) AS num_orders,
+ - 5 AS new_column
+ + new_column
+ FROM sqlmesh_example.incremental_model
+ GROUP BY
+ - item_id
+ + item_id,
+ + new_column
-Directly Modified: sqlmesh_example.full_model (Breaking)
+ Directly Modified: sqlmesh_example.full_model (Breaking)
----
+ ---
-+++
+ +++
-@@ -15,7 +15,7 @@
+ @@ -15,7 +15,7 @@
- id,
- item_id,
- event_date,
-- 5 AS new_column
-+ 7 AS new_column
-FROM sqlmesh_example.seed_model
-WHERE
- event_date BETWEEN @start_date AND @end_date
+ id,
+ item_id,
+ event_date,
+ - 5 AS new_column
+ + 7 AS new_column
+ FROM sqlmesh_example.seed_model
+ WHERE
+ event_date BETWEEN @start_date AND @end_date
-Directly Modified: sqlmesh_example.incremental_model (Breaking)
-└── Indirectly Modified Children:
- └── sqlmesh_example.view_model (Indirect Breaking)
-Apply - Virtual Update [y/n]: y
+ Directly Modified: sqlmesh_example.incremental_model (Breaking)
+ └── Indirectly Modified Children:
+ └── sqlmesh_example.view_model (Indirect Breaking)
+ Apply - Virtual Update [y/n]: y
-SKIP: No physical layer updates to perform
+ SKIP: No physical layer updates to perform
-SKIP: No model batches to execute
+ SKIP: No model batches to execute
-Updating virtual layer ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100.0% • 3/3 • 0:00:00
+ Updating virtual layer ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100.0% • 3/3 • 0:00:00
-✔ Virtual layer updated
-```
-```
+ ✔ Virtual layer updated
+ ```
---
-
-
## Enhanced Testing Workflow
You'll use these commands to validate your changes are behaving as expected. Audits (data tests) are a great first step, and you'll want to grow from there to feel confident about your pipelines. The workflow is as follows:
@@ -400,8 +370,6 @@ You'll use these commands to validate your changes are behaving as expected. Aud
---
-
-
### Create and Audit External Models
Sometimes models `SELECT` from tables/views that are outside of SQLMesh's control. SQLMesh can automatically parse their fully qualified names from model definitions (ex: `bigquery-public-data`.`ga4_obfuscated_sample_ecommerce`.`events_20210131`) and determine their full schemas and column data types.
@@ -410,145 +378,137 @@ These "external model" schemas are used for column level lineage. You can also a
=== "SQLMesh"
-```
-```bash
-sqlmesh create_external_models
-```
-```
+ ```bash
+ sqlmesh create_external_models
+ ```
=== "Tobiko Cloud"
-```
-```bash
-tcloud sqlmesh create_external_models
-```
-```
+ ```bash
+ tcloud sqlmesh create_external_models
+ ```
??? "Example Output"
Note: this is an example from a separate Tobiko Cloud project, so you can't follow along in the Github repo.
-```
-- Generated external models from the `bigquery-public-data`.`ga4_obfuscated_sample_ecommerce`.`events_20210131` table parsed in the model's SQL.
-- Added an audit to the external model to ensure `event_date` is not NULL.
-- Viewed a plan preview of the changes that will be made for the external model.
-
-```sql linenums="1" hl_lines="29" title="models/external_model_example.sql"
-MODEL (
- name tcloud_demo.external_model
-);
-
-SELECT
- event_date,
- event_timestamp,
- event_name,
- event_params,
- event_previous_timestamp,
- event_value_in_usd,
- event_bundle_sequence_id,
- event_server_timestamp_offset,
- user_id,
- user_pseudo_id,
- privacy_info,
- user_properties,
- user_first_touch_timestamp,
- user_ltv,
- device,
- geo,
- app_info,
- traffic_source,
- stream_id,
- platform,
- event_dimensions,
- ecommerce
-/* items */
-FROM bigquery-public-data.ga4_obfuscated_sample_ecommerce.events_20210131 -- I fully qualified the external table name and sqlmesh will automatically create the external model
-```
-
-`sqlmesh create_external_models` output file:
-
-```yaml linenums="1" hl_lines="2 3 4" title="external_models.yaml"
-- name: '`bigquery-public-data`.`ga4_obfuscated_sample_ecommerce`.`events_20210131`'
- audits: # I added this audit manually to the external model YAML file
- - name: not_null
- columns: "[event_date]"
- columns:
- event_date: STRING
- event_timestamp: INT64
- event_name: STRING
- event_params: ARRAY>>
- event_previous_timestamp: INT64
- event_value_in_usd: FLOAT64
- event_bundle_sequence_id: INT64
- event_server_timestamp_offset: INT64
- user_id: STRING
- user_pseudo_id: STRING
- privacy_info: STRUCT
- user_properties: ARRAY>>
- user_first_touch_timestamp: INT64
- user_ltv: STRUCT
- device: STRUCT>
- geo: STRUCT
- app_info: STRUCT
- traffic_source: STRUCT
- stream_id: INT64
- platform: STRING
- event_dimensions: STRUCT
- ecommerce: STRUCT
- items: ARRAY>
- gateway: public-demo
-```
-
-```bash
-> sqlmesh plan dev_sung
-Differences from the `dev_sung` environment:
-
-Models:
-└── Metadata Updated:
- └── "bigquery-public-data".ga4_obfuscated_sample_ecommerce__dev_sung.events_20210131
-
----
-
-+++
-
-@@ -29,5 +29,6 @@
-
- ecommerce STRUCT,
- items ARRAY>
- ),
-+ audits (not_null('columns' = [event_date])),
- gateway `public-demo`
-)
-
-Metadata Updated: "bigquery-public-data".ga4_obfuscated_sample_ecommerce__dev_sung.events_20210131
-Models needing backfill:
-└── "bigquery-public-data".ga4_obfuscated_sample_ecommerce__dev_sung.events_20210131: [full refresh]
-Apply - Backfill Tables [y/n]:
-```
-```
-
-
+ - Generated external models from the `bigquery-public-data`.`ga4_obfuscated_sample_ecommerce`.`events_20210131` table parsed in the model's SQL.
+ - Added an audit to the external model to ensure `event_date` is not NULL.
+ - Viewed a plan preview of the changes that will be made for the external model.
+
+ ```sql linenums="1" hl_lines="29" title="models/external_model_example.sql"
+ MODEL (
+ name tcloud_demo.external_model
+ );
+
+ SELECT
+ event_date,
+ event_timestamp,
+ event_name,
+ event_params,
+ event_previous_timestamp,
+ event_value_in_usd,
+ event_bundle_sequence_id,
+ event_server_timestamp_offset,
+ user_id,
+ user_pseudo_id,
+ privacy_info,
+ user_properties,
+ user_first_touch_timestamp,
+ user_ltv,
+ device,
+ geo,
+ app_info,
+ traffic_source,
+ stream_id,
+ platform,
+ event_dimensions,
+ ecommerce
+ /* items */
+ FROM bigquery-public-data.ga4_obfuscated_sample_ecommerce.events_20210131 -- I fully qualified the external table name and sqlmesh will automatically create the external model
+ ```
+
+ `sqlmesh create_external_models` output file:
+
+ ```yaml linenums="1" hl_lines="2 3 4" title="external_models.yaml"
+ - name: '`bigquery-public-data`.`ga4_obfuscated_sample_ecommerce`.`events_20210131`'
+ audits: # I added this audit manually to the external model YAML file
+ - name: not_null
+ columns: "[event_date]"
+ columns:
+ event_date: STRING
+ event_timestamp: INT64
+ event_name: STRING
+ event_params: ARRAY>>
+ event_previous_timestamp: INT64
+ event_value_in_usd: FLOAT64
+ event_bundle_sequence_id: INT64
+ event_server_timestamp_offset: INT64
+ user_id: STRING
+ user_pseudo_id: STRING
+ privacy_info: STRUCT
+ user_properties: ARRAY>>
+ user_first_touch_timestamp: INT64
+ user_ltv: STRUCT
+ device: STRUCT>
+ geo: STRUCT
+ app_info: STRUCT
+ traffic_source: STRUCT
+ stream_id: INT64
+ platform: STRING
+ event_dimensions: STRUCT
+ ecommerce: STRUCT
+ items: ARRAY>
+ gateway: public-demo
+ ```
+
+ ```bash
+ > sqlmesh plan dev_sung
+ Differences from the `dev_sung` environment:
+
+ Models:
+ └── Metadata Updated:
+ └── "bigquery-public-data".ga4_obfuscated_sample_ecommerce__dev_sung.events_20210131
+
+ ---
+
+ +++
+
+ @@ -29,5 +29,6 @@
+
+ ecommerce STRUCT,
+ items ARRAY>
+ ),
+ + audits (not_null('columns' = [event_date])),
+ gateway `public-demo`
+ )
+
+ Metadata Updated: "bigquery-public-data".ga4_obfuscated_sample_ecommerce__dev_sung.events_20210131
+ Models needing backfill:
+ └── "bigquery-public-data".ga4_obfuscated_sample_ecommerce__dev_sung.events_20210131: [full refresh]
+ Apply - Backfill Tables [y/n]:
+ ```
### Automatically Generate Unit Tests
@@ -558,121 +518,114 @@ Unit tests run *before* a plan is applied automatically. This is great for testi
=== "SQLMesh"
-```
-Create a unit test based on 5 rows from the upstream `sqlmesh_example.incremental_model`.
+ Create a unit test based on 5 rows from the upstream `sqlmesh_example.incremental_model`.
-```bash
-sqlmesh create_test sqlmesh_example.full_model \
- --query sqlmesh_example.incremental_model \
- "select * from sqlmesh_example.incremental_model limit 5"
-```
+ ```bash
+ sqlmesh create_test sqlmesh_example.full_model \
+ --query sqlmesh_example.incremental_model \
+ "select * from sqlmesh_example.incremental_model limit 5"
+ ```
+
+ ```bash
+ sqlmesh create_test \
+ --query \
+ "select * from limit 5"
+ ```
-```bash
-sqlmesh create_test \
- --query \
- "select * from limit 5"
-```
-```
=== "Tobiko Cloud"
-```
-```bash
-tcloud sqlmesh create_test demo.stg_payments \
- --query demo.seed_raw_payments \
- "select * from demo.seed_raw_payments limit 5"
-```
+ ```bash
+ tcloud sqlmesh create_test demo.stg_payments \
+ --query demo.seed_raw_payments \
+ "select * from demo.seed_raw_payments limit 5"
+ ```
-```bash
-tcloud sqlmesh create_test \
- --query \
- "select * from limit 5"
-```
-```
+ ```bash
+ tcloud sqlmesh create_test \
+ --query \
+ "select * from limit 5"
+ ```
??? "Example Output"
-```
-SQLMesh:
-
-- Generated unit tests for the `sqlmesh_example.full_model` model by live querying the data.
-- Ran the tests and they passed locally in DuckDB.
-- If you're using a cloud data warehouse, this will transpile your SQL syntax to its equivalent in duckdb.
-- This runs fast and free on your local machine.
-
-Generated test definition file:
-
-```yaml linenums="1" title="tests/test_full_model.yaml"
-test_full_model:
- model: '"db"."sqlmesh_example"."full_model"'
- inputs:
- '"db"."sqlmesh_example"."incremental_model"':
- - id: -11
- item_id: -11
- event_date: 2020-01-01
- new_column: 7
- - id: 1
- item_id: 1
- event_date: 2020-01-01
- new_column: 7
- - id: 3
- item_id: 3
- event_date: 2020-01-03
- new_column: 7
- - id: 4
- item_id: 1
- event_date: 2020-01-04
- new_column: 7
- - id: 5
- item_id: 1
- event_date: 2020-01-05
- new_column: 7
- outputs:
- query:
- - item_id: 3
- num_orders: 1
- new_column: 7
- - item_id: 1
- num_orders: 3
- new_column: 7
- - item_id: -11
- num_orders: 1
- new_column: 7
-```
-
-Manually execute tests with `sqlmesh test`:
-
-```bash
-(demo) ➜ demo git:(main) ✗ sqlmesh test
-.
-----------------------------------------------------------------------
-Ran 1 test in 0.053s
-
-OK
-```
-
-```bash
-# what do we see if the test fails?
-(demo) ➜ demo git:(main) ✗ sqlmesh test
-F
-======================================================================
-FAIL: test_full_model (/Users/sung/Desktop/git_repos/sqlmesh-cli-revamp/tests/test_full_model.yaml)
-None
-----------------------------------------------------------------------
-AssertionError: Data mismatch (exp: expected, act: actual)
-
- new_column
- exp act
-0 0.0 7.0
-
-----------------------------------------------------------------------
-Ran 1 test in 0.020s
-
-FAILED (failures=1)
-```
-```
-
-
+ SQLMesh:
+
+ - Generated unit tests for the `sqlmesh_example.full_model` model by live querying the data.
+ - Ran the tests and they passed locally in DuckDB.
+ - If you're using a cloud data warehouse, this will transpile your SQL syntax to its equivalent in duckdb.
+ - This runs fast and free on your local machine.
+
+ Generated test definition file:
+
+ ```yaml linenums="1" title="tests/test_full_model.yaml"
+ test_full_model:
+ model: '"db"."sqlmesh_example"."full_model"'
+ inputs:
+ '"db"."sqlmesh_example"."incremental_model"':
+ - id: -11
+ item_id: -11
+ event_date: 2020-01-01
+ new_column: 7
+ - id: 1
+ item_id: 1
+ event_date: 2020-01-01
+ new_column: 7
+ - id: 3
+ item_id: 3
+ event_date: 2020-01-03
+ new_column: 7
+ - id: 4
+ item_id: 1
+ event_date: 2020-01-04
+ new_column: 7
+ - id: 5
+ item_id: 1
+ event_date: 2020-01-05
+ new_column: 7
+ outputs:
+ query:
+ - item_id: 3
+ num_orders: 1
+ new_column: 7
+ - item_id: 1
+ num_orders: 3
+ new_column: 7
+ - item_id: -11
+ num_orders: 1
+ new_column: 7
+ ```
+
+ Manually execute tests with `sqlmesh test`:
+
+ ```bash
+ (demo) ➜ demo git:(main) ✗ sqlmesh test
+ .
+ ----------------------------------------------------------------------
+ Ran 1 test in 0.053s
+
+ OK
+ ```
+
+ ```bash
+ # what do we see if the test fails?
+ (demo) ➜ demo git:(main) ✗ sqlmesh test
+ F
+ ======================================================================
+ FAIL: test_full_model (/Users/sung/Desktop/git_repos/sqlmesh-cli-revamp/tests/test_full_model.yaml)
+ None
+ ----------------------------------------------------------------------
+ AssertionError: Data mismatch (exp: expected, act: actual)
+
+ new_column
+ exp act
+ 0 0.0 7.0
+
+ ----------------------------------------------------------------------
+ Ran 1 test in 0.020s
+
+ FAILED (failures=1)
+ ```
### Run Ad-Hoc Queries
@@ -682,32 +635,35 @@ Pro tip: run this after `sqlmesh table_diff` to get a full picture of your chang
=== "SQLMesh"
-```
-```bash
-sqlmesh fetchdf "select * from sqlmesh_example__dev.full_model limit 5"
-```
+ ```bash
+ sqlmesh fetchdf "select * from sqlmesh_example__dev.full_model limit 5"
+ ```
-```bash
-# construct arbitrary query
-sqlmesh fetchdf "select * from . limit 5" # double underscore in schema name is important. Not needed for prod.
-```
-```
+ ```bash
+ # construct arbitrary query
+ sqlmesh fetchdf "select * from . limit 5" # double underscore in schema name is important. Not needed for prod.
+ ```
=== "Tobiko Cloud"
-```
-```bash
-tcloud sqlmesh fetchdf "select * from sqlmesh_example__dev.full_model limit 5"
-```
+ ```bash
+ tcloud sqlmesh fetchdf "select * from sqlmesh_example__dev.full_model limit 5"
+ ```
-```bash
-# construct arbitrary query
-tcloud sqlmesh fetchdf "select * from . limit 5" # double underscore in schema name is important. Not needed for prod.
-```
-```
+ ```bash
+ # construct arbitrary query
+ tcloud sqlmesh fetchdf "select * from . limit 5" # double underscore in schema name is important. Not needed for prod.
+ ```
??? "Example Output"
- `bash item_id num_orders new_column 0 9 1 7 1 -11 1 7 2 3 1 7 3 -3 1 7 4 1 4 7`
+ ```bash
+ item_id num_orders new_column
+ 0 9 1 7
+ 1 -11 1 7
+ 2 3 1 7
+ 3 -3 1 7
+ 4 1 4 7
+ ```
### Linting
@@ -717,66 +673,58 @@ This is a great way to catch SQL issues before wasting runtime in your data ware
=== "SQLMesh"
-```
-```bash
-sqlmesh lint
-```
-```
+ ```bash
+ sqlmesh lint
+ ```
=== "Tobiko Cloud"
-```
-```bash
-tcloud sqlmesh lint
-```
-```
+ ```bash
+ tcloud sqlmesh lint
+ ```
??? "Example Output"
-```
-You add linting rules in your `config.yaml` file.
-
-```yaml linenums="1" hl_lines="13-17" title="config.yaml"
-gateways:
- duckdb:
- connection:
- type: duckdb
- database: db.db
-
-default_gateway: duckdb
-
-model_defaults:
- dialect: duckdb
- start: 2025-03-26
-
-linter:
- enabled: true
- rules: ["ambiguousorinvalidcolumn", "invalidselectstarexpansion"] # raise errors for these rules
- warn_rules: ["noselectstar", "nomissingaudits"]
- # ignored_rules: ["noselectstar", "nomissingunittest"]
-```
-
-```bash
-> sqlmesh lint
-[WARNING] Linter warnings for /Users/sung/Desktop/git_repos/sqlmesh-cli-revamp/models/lint_warn.sql:
-- noselectstar: Query should not contain SELECT * on its outer most projections, even if it can be
-expanded.
-- nomissingaudits: Model `audits` must be configured to test data quality.
-[WARNING] Linter warnings for
-/Users/sung/Desktop/git_repos/sqlmesh-cli-revamp/models/incremental_by_partition.sql:
-- nomissingaudits: Model `audits` must be configured to test data quality.
-[WARNING] Linter warnings for /Users/sung/Desktop/git_repos/sqlmesh-cli-revamp/models/seed_model.sql:
-- nomissingaudits: Model `audits` must be configured to test data quality.
-[WARNING] Linter warnings for
-/Users/sung/Desktop/git_repos/sqlmesh-cli-revamp/models/incremental_by_unique_key.sql:
-- nomissingaudits: Model `audits` must be configured to test data quality.
-[WARNING] Linter warnings for
-/Users/sung/Desktop/git_repos/sqlmesh-cli-revamp/models/incremental_model.sql:
-- nomissingaudits: Model `audits` must be configured to test data quality.
-```
-```
-
-
+ You add linting rules in your `config.yaml` file.
+
+ ```yaml linenums="1" hl_lines="13-17" title="config.yaml"
+ gateways:
+ duckdb:
+ connection:
+ type: duckdb
+ database: db.db
+
+ default_gateway: duckdb
+
+ model_defaults:
+ dialect: duckdb
+ start: 2025-03-26
+
+ linter:
+ enabled: true
+ rules: ["ambiguousorinvalidcolumn", "invalidselectstarexpansion"] # raise errors for these rules
+ warn_rules: ["noselectstar", "nomissingaudits"]
+ # ignored_rules: ["noselectstar"]
+ ```
+
+ ```bash
+ > sqlmesh lint
+ [WARNING] Linter warnings for /Users/sung/Desktop/git_repos/sqlmesh-cli-revamp/models/lint_warn.sql:
+ - noselectstar: Query should not contain SELECT * on its outer most projections, even if it can be
+ expanded.
+ - nomissingaudits: Model `audits` must be configured to test data quality.
+ [WARNING] Linter warnings for
+ /Users/sung/Desktop/git_repos/sqlmesh-cli-revamp/models/incremental_by_partition.sql:
+ - nomissingaudits: Model `audits` must be configured to test data quality.
+ [WARNING] Linter warnings for /Users/sung/Desktop/git_repos/sqlmesh-cli-revamp/models/seed_model.sql:
+ - nomissingaudits: Model `audits` must be configured to test data quality.
+ [WARNING] Linter warnings for
+ /Users/sung/Desktop/git_repos/sqlmesh-cli-revamp/models/incremental_by_unique_key.sql:
+ - nomissingaudits: Model `audits` must be configured to test data quality.
+ [WARNING] Linter warnings for
+ /Users/sung/Desktop/git_repos/sqlmesh-cli-revamp/models/incremental_model.sql:
+ - nomissingaudits: Model `audits` must be configured to test data quality.
+ ```
## Debugging Workflow
@@ -786,105 +734,95 @@ You'll use these commands as needed to validate that your changes are behaving a
2. Run SQLMesh in verbose mode so you can verify its behavior.
3. View the logs easily in your terminal.
-
-
### Render your SQL Changes
This is a great way to verify that your model's SQL is looking as expected before applying the changes. It is especially important if you're migrating from one query engine to another (ex: postgres to databricks).
=== "SQLMesh"
-```
-```bash
-sqlmesh render sqlmesh_example.incremental_model
-```
+ ```bash
+ sqlmesh render sqlmesh_example.incremental_model
+ ```
-```bash
-sqlmesh render sqlmesh_example.incremental_model --dialect databricks
-```
+ ```bash
+ sqlmesh render sqlmesh_example.incremental_model --dialect databricks
+ ```
-```bash
-sqlmesh render --dialect
-```
-```
+ ```bash
+ sqlmesh render --dialect
+ ```
=== "Tobiko Cloud"
-```
-```bash
-tcloud sqlmesh render sqlmesh_example.incremental_model
-```
+ ```bash
+ tcloud sqlmesh render sqlmesh_example.incremental_model
+ ```
-```bash
-tcloud sqlmesh render sqlmesh_example.incremental_model --dialect databricks
-```
+ ```bash
+ tcloud sqlmesh render sqlmesh_example.incremental_model --dialect databricks
+ ```
-```bash
-tcloud sqlmesh render --dialect
-```
-```
+ ```bash
+ tcloud sqlmesh render --dialect
+ ```
??? "Example Output"
-```
-Model definition:
-
-```sql linenums="1" title="models/incremental_model.sql"
-MODEL (
- name sqlmesh_example.incremental_model,
- kind INCREMENTAL_BY_TIME_RANGE (
- time_column event_date
- ),
- start '2020-01-01',
- cron '@daily',
- grain (id, event_date)
-);
-
-SELECT
- id,
- item_id,
- event_date,
- 7 as new_column
-FROM
- sqlmesh_example.seed_model
-WHERE
- event_date BETWEEN @start_date AND @end_date
-```
-
-SQLMesh returns the full SQL code in the default or target dialect.
-
-```sql hl_lines="11"
-> sqlmesh render sqlmesh_example.incremental_model
--- rendered sql in default dialect
-SELECT
- "seed_model"."id" AS "id",
- "seed_model"."item_id" AS "item_id",
- "seed_model"."event_date" AS "event_date",
- 7 AS "new_column"
-FROM "db"."sqlmesh__sqlmesh_example"."sqlmesh_example__seed_model__3294646944" AS "seed_model" /*
-db.sqlmesh_example.seed_model */
-WHERE
- "seed_model"."event_date" <= CAST('1970-01-01' AS DATE) -- placeholder dates for date macros
- AND "seed_model"."event_date" >= CAST('1970-01-01' AS DATE)
-```
-
-```sql
-> sqlmesh render sqlmesh_example.incremental_model --dialect databricks
--- rendered sql in databricks dialect
-SELECT
- `seed_model`.`id` AS `id`,
- `seed_model`.`item_id` AS `item_id`,
- `seed_model`.`event_date` AS `event_date`,
- 7 AS `new_column`
-FROM `db`.`sqlmesh__sqlmesh_example`.`sqlmesh_example__seed_model__3294646944` AS `seed_model` /*
-db.sqlmesh_example.seed_model */
-WHERE
- `seed_model`.`event_date` <= CAST('1970-01-01' AS DATE)
- AND `seed_model`.`event_date` >= CAST('1970-01-01' AS DATE)
-```
-```
+ Model definition:
+ ```sql linenums="1" title="models/incremental_model.sql"
+ MODEL (
+ name sqlmesh_example.incremental_model,
+ kind INCREMENTAL_BY_TIME_RANGE (
+ time_column event_date
+ ),
+ start '2020-01-01',
+ cron '@daily',
+ grain (id, event_date)
+ );
+ SELECT
+ id,
+ item_id,
+ event_date,
+ 7 as new_column
+ FROM
+ sqlmesh_example.seed_model
+ WHERE
+ event_date BETWEEN @start_date AND @end_date
+ ```
+
+ SQLMesh returns the full SQL code in the default or target dialect.
+
+ ```sql hl_lines="11"
+ > sqlmesh render sqlmesh_example.incremental_model
+ -- rendered sql in default dialect
+ SELECT
+ "seed_model"."id" AS "id",
+ "seed_model"."item_id" AS "item_id",
+ "seed_model"."event_date" AS "event_date",
+ 7 AS "new_column"
+ FROM "db"."sqlmesh__sqlmesh_example"."sqlmesh_example__seed_model__3294646944" AS "seed_model" /*
+ db.sqlmesh_example.seed_model */
+ WHERE
+ "seed_model"."event_date" <= CAST('1970-01-01' AS DATE) -- placeholder dates for date macros
+ AND "seed_model"."event_date" >= CAST('1970-01-01' AS DATE)
+ ```
+
+ ```sql
+ > sqlmesh render sqlmesh_example.incremental_model --dialect databricks
+ -- rendered sql in databricks dialect
+ SELECT
+ `seed_model`.`id` AS `id`,
+ `seed_model`.`item_id` AS `item_id`,
+ `seed_model`.`event_date` AS `event_date`,
+ 7 AS `new_column`
+ FROM `db`.`sqlmesh__sqlmesh_example`.`sqlmesh_example__seed_model__3294646944` AS `seed_model` /*
+ db.sqlmesh_example.seed_model */
+ WHERE
+ `seed_model`.`event_date` <= CAST('1970-01-01' AS DATE)
+ AND `seed_model`.`event_date` >= CAST('1970-01-01' AS DATE)
+ ```
### Apply Plan Changes in Verbose Mode
@@ -892,89 +830,81 @@ Verbose mode lets you see detailed operations in the physical and virtual layers
=== "SQLMesh"
-```
-```bash
-sqlmesh plan dev -vv
-```
+ ```bash
+ sqlmesh plan dev -vv
+ ```
-```bash
-sqlmesh plan -vv
-```
-```
+ ```bash
+ sqlmesh plan -vv
+ ```
=== "Tobiko Cloud"
-```
-```bash
-tcloud sqlmesh plan dev -vv
-```
+ ```bash
+ tcloud sqlmesh plan dev -vv
+ ```
-```bash
-tcloud sqlmesh plan -vv
-```
-```
+ ```bash
+ tcloud sqlmesh plan -vv
+ ```
??? "Example Output"
-```
-```bash hl_lines="48-50"
-> sqlmesh plan dev -vv
-[WARNING] Linter warnings for
-/Users/sung/Desktop/git_repos/sqlmesh-cli-revamp/models/incremental_by_partition.sql:
-- nomissingaudits: Model `audits` must be configured to test data quality.
-[WARNING] Linter warnings for /Users/sung/Desktop/git_repos/sqlmesh-cli-revamp/models/seed_model.sql:
-- nomissingaudits: Model `audits` must be configured to test data quality.
-[WARNING] Linter warnings for
-/Users/sung/Desktop/git_repos/sqlmesh-cli-revamp/models/incremental_by_unique_key.sql:
-- nomissingaudits: Model `audits` must be configured to test data quality.
-[WARNING] Linter warnings for
-/Users/sung/Desktop/git_repos/sqlmesh-cli-revamp/models/incremental_model.sql:
-- nomissingaudits: Model `audits` must be configured to test data quality.
-
-Differences from the `dev` environment:
-
-Models:
-├── Directly Modified:
-│ └── db.sqlmesh_example__dev.incremental_model
-└── Indirectly Modified:
- ├── db.sqlmesh_example__dev.full_model
- └── db.sqlmesh_example__dev.view_model
+ ```bash hl_lines="48-50"
+ > sqlmesh plan dev -vv
+ [WARNING] Linter warnings for
+ /Users/sung/Desktop/git_repos/sqlmesh-cli-revamp/models/incremental_by_partition.sql:
+ - nomissingaudits: Model `audits` must be configured to test data quality.
+ [WARNING] Linter warnings for /Users/sung/Desktop/git_repos/sqlmesh-cli-revamp/models/seed_model.sql:
+ - nomissingaudits: Model `audits` must be configured to test data quality.
+ [WARNING] Linter warnings for
+ /Users/sung/Desktop/git_repos/sqlmesh-cli-revamp/models/incremental_by_unique_key.sql:
+ - nomissingaudits: Model `audits` must be configured to test data quality.
+ [WARNING] Linter warnings for
+ /Users/sung/Desktop/git_repos/sqlmesh-cli-revamp/models/incremental_model.sql:
+ - nomissingaudits: Model `audits` must be configured to test data quality.
----
+ Differences from the `dev` environment:
-+++
+ Models:
+ ├── Directly Modified:
+ │ └── db.sqlmesh_example__dev.incremental_model
+ └── Indirectly Modified:
+ ├── db.sqlmesh_example__dev.full_model
+ └── db.sqlmesh_example__dev.view_model
-@@ -15,7 +15,7 @@
+ ---
- id,
- item_id,
- event_date,
-- 9 AS new_column
-+ 7 AS new_column
-FROM sqlmesh_example.seed_model
-WHERE
- event_date BETWEEN @start_date AND @end_date
+ +++
-Directly Modified: db.sqlmesh_example__dev.incremental_model (Breaking)
-└── Indirectly Modified Children:
- ├── db.sqlmesh_example__dev.full_model (Breaking)
- └── db.sqlmesh_example__dev.view_model (Indirect Breaking)
-Apply - Virtual Update [y/n]: y
+ @@ -15,7 +15,7 @@
-SKIP: No physical layer updates to perform
+ id,
+ item_id,
+ event_date,
+ - 9 AS new_column
+ + 7 AS new_column
+ FROM sqlmesh_example.seed_model
+ WHERE
+ event_date BETWEEN @start_date AND @end_date
-SKIP: No model batches to execute
+ Directly Modified: db.sqlmesh_example__dev.incremental_model (Breaking)
+ └── Indirectly Modified Children:
+ ├── db.sqlmesh_example__dev.full_model (Breaking)
+ └── db.sqlmesh_example__dev.view_model (Indirect Breaking)
+ Apply - Virtual Update [y/n]: y
-db.sqlmesh_example__dev.incremental_model updated # you'll notice that it's updated vs. promoted because we changed the existing view definition
-db.sqlmesh_example__dev.full_model updated
-db.sqlmesh_example__dev.view_model updated
-Updating virtual layer ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100.0% • 3/3 • 0:00:00
+ SKIP: No physical layer updates to perform
-✔ Virtual layer updated
-```
-```
+ SKIP: No model batches to execute
+ db.sqlmesh_example__dev.incremental_model updated # you'll notice that it's updated vs. promoted because we changed the existing view definition
+ db.sqlmesh_example__dev.full_model updated
+ db.sqlmesh_example__dev.view_model updated
+ Updating virtual layer ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100.0% • 3/3 • 0:00:00
+ ✔ Virtual layer updated
+ ```
### View Logs Easily
@@ -997,39 +927,35 @@ bat --theme='ansi' $(ls -t logs/ | head -n 1 | sed 's/^/logs\//')
??? "Example Output"
-```
-This is the log file for the `sqlmesh plan dev` command. If you want to see the log file directly, you can click on the file path in the output to open it in your code editor.
-
-```bash
-──────┬──────────────────────────────────────────────────────────────────────────────────────────────
- │ File: logs/sqlmesh_2025_04_18_12_34_35.log
-──────┼──────────────────────────────────────────────────────────────────────────────────────────────
- 1 │ 2025-04-18 12:34:35,715 - MainThread - sqlmesh.core.config.connection - INFO - Creating new D
- │ uckDB adapter for data files: {'db.db'} (connection.py:319)
- 2 │ 2025-04-18 12:34:35,951 - MainThread - sqlmesh.core.console - WARNING - Linter warnings for /
- │ Users/sung/Desktop/git_repos/sqlmesh-cli-revamp/models/incremental_by_partition.sql:
- 3 │ - nomissingaudits: Model `audits` must be configured to test data quality. (console.py:1848)
- 4 │ 2025-04-18 12:34:35,953 - MainThread - sqlmesh.core.console - WARNING - Linter warnings for /
- │ Users/sung/Desktop/git_repos/sqlmesh-cli-revamp/models/seed_model.sql:
- 5 │ - nomissingaudits: Model `audits` must be configured to test data quality. (console.py:1848)
- 6 │ 2025-04-18 12:34:35,953 - MainThread - sqlmesh.core.console - WARNING - Linter warnings for /
- │ Users/sung/Desktop/git_repos/sqlmesh-cli-revamp/models/incremental_by_unique_key.sql:
- 7 │ - nomissingaudits: Model `audits` must be configured to test data quality. (console.py:1848)
- 8 │ 2025-04-18 12:34:35,953 - MainThread - sqlmesh.core.console - WARNING - Linter warnings for /
- │ Users/sung/Desktop/git_repos/sqlmesh-cli-revamp/models/incremental_model.sql:
- 9 │ - nomissingaudits: Model `audits` must be configured to test data quality. (console.py:1848)
- 10 │ 2025-04-18 12:34:35,954 - MainThread - sqlmesh.core.config.connection - INFO - Using existing
- │ DuckDB adapter due to overlapping data file: db.db (connection.py:309)
- 11 │ 2025-04-18 12:34:37,071 - MainThread - sqlmesh.core.snapshot.evaluator - INFO - Listing data
- │ objects in schema db.sqlmesh__sqlmesh_example (evaluator.py:338)
- 12 │ 2025-04-18 12:34:37,072 - MainThread - sqlmesh.core.engine_adapter.base - INFO - Executing SQ
- │ L: SELECT CURRENT_CATALOG() (base.py:2128)
- 13 │ 2025-04-18 12:34:37,072 - MainThread - sqlmesh.core.engine_adapter.base - INFO - Executing SQ
- │ L: SELECT CURRENT_CATALOG() (base.py:2128)
-```
-```
-
-
+ This is the log file for the `sqlmesh plan dev` command. If you want to see the log file directly, you can click on the file path in the output to open it in your code editor.
+
+ ```bash
+ ──────┬──────────────────────────────────────────────────────────────────────────────────────────────
+ │ File: logs/sqlmesh_2025_04_18_12_34_35.log
+ ──────┼──────────────────────────────────────────────────────────────────────────────────────────────
+ 1 │ 2025-04-18 12:34:35,715 - MainThread - sqlmesh.core.config.connection - INFO - Creating new D
+ │ uckDB adapter for data files: {'db.db'} (connection.py:319)
+ 2 │ 2025-04-18 12:34:35,951 - MainThread - sqlmesh.core.console - WARNING - Linter warnings for /
+ │ Users/sung/Desktop/git_repos/sqlmesh-cli-revamp/models/incremental_by_partition.sql:
+ 3 │ - nomissingaudits: Model `audits` must be configured to test data quality. (console.py:1848)
+ 4 │ 2025-04-18 12:34:35,953 - MainThread - sqlmesh.core.console - WARNING - Linter warnings for /
+ │ Users/sung/Desktop/git_repos/sqlmesh-cli-revamp/models/seed_model.sql:
+ 5 │ - nomissingaudits: Model `audits` must be configured to test data quality. (console.py:1848)
+ 6 │ 2025-04-18 12:34:35,953 - MainThread - sqlmesh.core.console - WARNING - Linter warnings for /
+ │ Users/sung/Desktop/git_repos/sqlmesh-cli-revamp/models/incremental_by_unique_key.sql:
+ 7 │ - nomissingaudits: Model `audits` must be configured to test data quality. (console.py:1848)
+ 8 │ 2025-04-18 12:34:35,953 - MainThread - sqlmesh.core.console - WARNING - Linter warnings for /
+ │ Users/sung/Desktop/git_repos/sqlmesh-cli-revamp/models/incremental_model.sql:
+ 9 │ - nomissingaudits: Model `audits` must be configured to test data quality. (console.py:1848)
+ 10 │ 2025-04-18 12:34:35,954 - MainThread - sqlmesh.core.config.connection - INFO - Using existing
+ │ DuckDB adapter due to overlapping data file: db.db (connection.py:309)
+ 11 │ 2025-04-18 12:34:37,071 - MainThread - sqlmesh.core.snapshot.evaluator - INFO - Listing data
+ │ objects in schema db.sqlmesh__sqlmesh_example (evaluator.py:338)
+ 12 │ 2025-04-18 12:34:37,072 - MainThread - sqlmesh.core.engine_adapter.base - INFO - Executing SQ
+ │ L: SELECT CURRENT_CATALOG() (base.py:2128)
+ 13 │ 2025-04-18 12:34:37,072 - MainThread - sqlmesh.core.engine_adapter.base - INFO - Executing SQ
+ │ L: SELECT CURRENT_CATALOG() (base.py:2128)
+ ```
## Run on Production Schedule
@@ -1059,63 +985,55 @@ This command is intended be run on a schedule. It will skip the physical and vir
=== "SQLMesh"
-```
-```bash
-sqlmesh run
-```
-```
+ ```bash
+ sqlmesh run
+ ```
=== "Tobiko Cloud"
-```
-```bash
-tcloud sqlmesh run
-```
-```
+ ```bash
+ tcloud sqlmesh run
+ ```
??? "Example Output"
-```
-This is what it looks like if models are ready to run.
-
-```bash
-> sqlmesh run
-[1/1] sqlmesh_example.incremental_model [insert 2025-04-17 - 2025-04-17]
-0.01s
-[1/1] sqlmesh_example.incremental_unique_model [insert/update rows]
-0.01s
-[1/1] sqlmesh_example_v3.incremental_partition_model [insert partitions]
-0.01s
-Executing model batches ━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━ 40.0% • 2/5 • 0:00:00
-sqlmesh_example_v3.incremental_partition_model .
-[WARNING] sqlmesh_example.full_model: 'assert_positive_order_ids' audit error: 2 rows failed. Learn
-more in logs: /Users/sung/Desktop/git_repos/sqlmesh-cli-revamp/logs/sqlmesh_2025_04_18_12_48_35.log
-[1/1] sqlmesh_example.full_model [full refresh, audits ❌1]
-0.01s
-Executing model batches ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━ 80.0% • 4/5 • 0:00:00
-sqlmesh_example.view_model .
-[WARNING] sqlmesh_example.view_model: 'assert_positive_order_ids' audit error: 2 rows failed. Learn
-more in logs: /Users/sung/Desktop/git_repos/sqlmesh-cli-revamp/logs/sqlmesh_2025_04_18_12_48_35.log
-[1/1] sqlmesh_example.view_model [recreate view, audits ✔2 ❌1]
-0.01s
-Executing model batches ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100.0% • 5/5 • 0:00:00
-
-✔ Model batches executed
-
-Run finished for environment 'prod'
-```
-
-This is what it looks like if no models are ready to run.
-
-```bash
-> sqlmesh run
-No models are ready to run. Please wait until a model `cron` interval has elapsed.
-
-Next run will be ready at 2025-04-18 05:00PM PDT (2025-04-19 12:00AM UTC).
-```
-```
-
-
+ This is what it looks like if models are ready to run.
+
+ ```bash
+ > sqlmesh run
+ [1/1] sqlmesh_example.incremental_model [insert 2025-04-17 - 2025-04-17]
+ 0.01s
+ [1/1] sqlmesh_example.incremental_unique_model [insert/update rows]
+ 0.01s
+ [1/1] sqlmesh_example_v3.incremental_partition_model [insert partitions]
+ 0.01s
+ Executing model batches ━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━ 40.0% • 2/5 • 0:00:00
+ sqlmesh_example_v3.incremental_partition_model .
+ [WARNING] sqlmesh_example.full_model: 'assert_positive_order_ids' audit error: 2 rows failed. Learn
+ more in logs: /Users/sung/Desktop/git_repos/sqlmesh-cli-revamp/logs/sqlmesh_2025_04_18_12_48_35.log
+ [1/1] sqlmesh_example.full_model [full refresh, audits ❌1]
+ 0.01s
+ Executing model batches ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━ 80.0% • 4/5 • 0:00:00
+ sqlmesh_example.view_model .
+ [WARNING] sqlmesh_example.view_model: 'assert_positive_order_ids' audit error: 2 rows failed. Learn
+ more in logs: /Users/sung/Desktop/git_repos/sqlmesh-cli-revamp/logs/sqlmesh_2025_04_18_12_48_35.log
+ [1/1] sqlmesh_example.view_model [recreate view, audits ✔2 ❌1]
+ 0.01s
+ Executing model batches ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100.0% • 5/5 • 0:00:00
+
+ ✔ Model batches executed
+
+ Run finished for environment 'prod'
+ ```
+
+ This is what it looks like if no models are ready to run.
+
+ ```bash
+ > sqlmesh run
+ No models are ready to run. Please wait until a model `cron` interval has elapsed.
+
+ Next run will be ready at 2025-04-18 05:00PM PDT (2025-04-19 12:00AM UTC).
+ ```
### Run Models with Incomplete Intervals (Warning)
@@ -1127,65 +1045,57 @@ You can run models that execute backfills each time you invoke a `run`, whether
=== "SQLMesh"
-```
-```bash
-sqlmesh run --ignore-cron
-```
-```
+ ```bash
+ sqlmesh run --ignore-cron
+ ```
=== "Tobiko Cloud"
-```
-```bash
-tcloud sqlmesh run --ignore-cron
-```
-```
+ ```bash
+ tcloud sqlmesh run --ignore-cron
+ ```
??? "Example Output"
-```
-Model definition:
-```sql linenums="1" hl_lines="15" title="models/incremental_model.sql"
-MODEL (
- name sqlmesh_example.incremental_model,
- kind INCREMENTAL_BY_TIME_RANGE (
- time_column event_date
- ),
- start '2020-01-01',
- cron '@daily',
- grain (id, event_date),
- audits( UNIQUE_VALUES(columns = (
- id,
- )), NOT_NULL(columns = (
+ Model definition:
+ ```sql linenums="1" hl_lines="15" title="models/incremental_model.sql"
+ MODEL (
+ name sqlmesh_example.incremental_model,
+ kind INCREMENTAL_BY_TIME_RANGE (
+ time_column event_date
+ ),
+ start '2020-01-01',
+ cron '@daily',
+ grain (id, event_date),
+ audits( UNIQUE_VALUES(columns = (
+ id,
+ )), NOT_NULL(columns = (
+ id,
+ event_date
+ ))),
+ allow_partials true
+ );
+
+ SELECT
id,
- event_date
- ))),
- allow_partials true
-);
-
-SELECT
- id,
- item_id,
- event_date,
- 16 as new_column
-FROM
- sqlmesh_example.seed_model
-WHERE
- event_date BETWEEN @start_date AND @end_date
-```
+ item_id,
+ event_date,
+ 16 as new_column
+ FROM
+ sqlmesh_example.seed_model
+ WHERE
+ event_date BETWEEN @start_date AND @end_date
+ ```
-```bash
-> sqlmesh run --ignore-cron
-[1/1] sqlmesh_example.incremental_model [insert 2025-04-19 - 2025-04-19, audits ✔2] 0.05s
-Executing model batches ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100.0% • 1/1 • 0:00:00
-
-✔ Model batches executed
-
-Run finished for environment 'prod'
-```
-```
+ ```bash
+ > sqlmesh run --ignore-cron
+ [1/1] sqlmesh_example.incremental_model [insert 2025-04-19 - 2025-04-19, audits ✔2] 0.05s
+ Executing model batches ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100.0% • 1/1 • 0:00:00
+ ✔ Model batches executed
+ Run finished for environment 'prod'
+ ```
## Forward-Only Development Workflow
@@ -1201,140 +1111,133 @@ If you want to see a full walkthrough, [go here](incremental_time_full_walkthrou
=== "SQLMesh"
-```
-```bash
-sqlmesh plan dev --forward-only
-```
+ ```bash
+ sqlmesh plan dev --forward-only
+ ```
-```bash
-sqlmesh plan --forward-only
-```
-```
+ ```bash
+ sqlmesh plan --forward-only
+ ```
=== "Tobiko Cloud"
-```
-```bash
-tcloud sqlmesh plan dev --forward-only
-```
+ ```bash
+ tcloud sqlmesh plan dev --forward-only
+ ```
-```bash
-tcloud sqlmesh plan --forward-only
-```
-```
+ ```bash
+ tcloud sqlmesh plan --forward-only
+ ```
??? "Example Output"
-```
-- I applied a change to a new column
-- It impacts 2 downstream models
-- I enforced a forward-only plan to avoid backfilling historical data for the incremental model (ex: `preview` language in the CLI output)
-- I previewed the changes in a clone of the incremental impacted (clones will NOT be reused in production) along with the full and view models (these are NOT clones).
-
-```bash
-> sqlmesh plan dev
-Differences from the `dev` environment:
-
-Models:
-├── Directly Modified:
-│ └── sqlmesh_example__dev.incremental_model
-└── Indirectly Modified:
- ├── sqlmesh_example__dev.view_model
- └── sqlmesh_example__dev.full_model
-
----
-
-+++
+ - I applied a change to a new column
+ - It impacts 2 downstream models
+ - I enforced a forward-only plan to avoid backfilling historical data for the incremental model (ex: `preview` language in the CLI output)
+ - I previewed the changes in a clone of the incremental impacted (clones will NOT be reused in production) along with the full and view models (these are NOT clones).
-@@ -16,7 +16,7 @@
+ ```bash
+ > sqlmesh plan dev
+ Differences from the `dev` environment:
- id,
- item_id,
- event_date,
-- 9 AS new_column
-+ 10 AS new_column
-FROM sqlmesh_example.seed_model
-WHERE
- event_date BETWEEN @start_date AND @end_date
+ Models:
+ ├── Directly Modified:
+ │ └── sqlmesh_example__dev.incremental_model
+ └── Indirectly Modified:
+ ├── sqlmesh_example__dev.view_model
+ └── sqlmesh_example__dev.full_model
-Directly Modified: sqlmesh_example__dev.incremental_model (Forward-only)
-└── Indirectly Modified Children:
- ├── sqlmesh_example__dev.full_model (Forward-only)
- └── sqlmesh_example__dev.view_model (Forward-only)
-Models needing backfill:
-├── sqlmesh_example__dev.full_model: [full refresh] (preview)
-├── sqlmesh_example__dev.incremental_model: [2025-04-17 - 2025-04-17] (preview)
-└── sqlmesh_example__dev.view_model: [recreate view] (preview)
-Apply - Preview Tables [y/n]: y
+ ---
-Updating physical layer ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100.0% • 3/3 • 0:00:00
-
-✔ Physical layer updated
-
-[1/1] sqlmesh_example__dev.incremental_model [insert 2025-04-17 - 2025-04-17] 0.01s
-[1/1] sqlmesh_example__dev.full_model [full refresh, audits ✔1] 0.01s
-[1/1] sqlmesh_example__dev.view_model [recreate view, audits ✔3] 0.01s
-Executing model batches ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100.0% • 3/3 • 0:00:00
-
-✔ Model batches executed
-
-Updating virtual layer ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100.0% • 3/3 • 0:00:00
-
-✔ Virtual layer updated
-```
-
-When the plan is applied to `prod`, it will only execute model batches for new intervals (new rows). This will NOT re-use `preview` models (backfilled data) in development.
-
-```bash
-> sqlmesh plan
-Differences from the `prod` environment:
+ +++
-Models:
-├── Directly Modified:
-│ └── sqlmesh_example.incremental_model
-└── Indirectly Modified:
- ├── sqlmesh_example.view_model
- └── sqlmesh_example.full_model
+ @@ -16,7 +16,7 @@
----
-
-+++
-
-@@ -9,13 +9,14 @@
-
- disable_restatement FALSE,
- on_destructive_change 'ERROR'
- ),
-- grains ((id, event_date))
-+ grains ((id, event_date)),
-+ allow_partials TRUE
-)
-SELECT
- id,
- item_id,
- event_date,
-- 7 AS new_column
-+ 10 AS new_column
-FROM sqlmesh_example.seed_model
-WHERE
- event_date BETWEEN @start_date AND @end_date
-
-Directly Modified: sqlmesh_example.incremental_model (Forward-only)
-└── Indirectly Modified Children:
- ├── sqlmesh_example.full_model (Forward-only)
- └── sqlmesh_example.view_model (Forward-only)
-Apply - Virtual Update [y/n]: y
+ id,
+ item_id,
+ event_date,
+ - 9 AS new_column
+ + 10 AS new_column
+ FROM sqlmesh_example.seed_model
+ WHERE
+ event_date BETWEEN @start_date AND @end_date
+
+ Directly Modified: sqlmesh_example__dev.incremental_model (Forward-only)
+ └── Indirectly Modified Children:
+ ├── sqlmesh_example__dev.full_model (Forward-only)
+ └── sqlmesh_example__dev.view_model (Forward-only)
+ Models needing backfill:
+ ├── sqlmesh_example__dev.full_model: [full refresh] (preview)
+ ├── sqlmesh_example__dev.incremental_model: [2025-04-17 - 2025-04-17] (preview)
+ └── sqlmesh_example__dev.view_model: [recreate view] (preview)
+ Apply - Preview Tables [y/n]: y
+
+ Updating physical layer ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100.0% • 3/3 • 0:00:00
+
+ ✔ Physical layer updated
+
+ [1/1] sqlmesh_example__dev.incremental_model [insert 2025-04-17 - 2025-04-17] 0.01s
+ [1/1] sqlmesh_example__dev.full_model [full refresh, audits ✔1] 0.01s
+ [1/1] sqlmesh_example__dev.view_model [recreate view, audits ✔3] 0.01s
+ Executing model batches ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100.0% • 3/3 • 0:00:00
+
+ ✔ Model batches executed
+
+ Updating virtual layer ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100.0% • 3/3 • 0:00:00
+
+ ✔ Virtual layer updated
+ ```
+
+ When the plan is applied to `prod`, it will only execute model batches for new intervals (new rows). This will NOT re-use `preview` models (backfilled data) in development.
+
+ ```bash
+ > sqlmesh plan
+ Differences from the `prod` environment:
+
+ Models:
+ ├── Directly Modified:
+ │ └── sqlmesh_example.incremental_model
+ └── Indirectly Modified:
+ ├── sqlmesh_example.view_model
+ └── sqlmesh_example.full_model
+
+ ---
+
+ +++
+
+ @@ -9,13 +9,14 @@
+
+ disable_restatement FALSE,
+ on_destructive_change 'ERROR'
+ ),
+ - grains ((id, event_date))
+ + grains ((id, event_date)),
+ + allow_partials TRUE
+ )
+ SELECT
+ id,
+ item_id,
+ event_date,
+ - 7 AS new_column
+ + 10 AS new_column
+ FROM sqlmesh_example.seed_model
+ WHERE
+ event_date BETWEEN @start_date AND @end_date
-SKIP: No physical layer updates to perform
+ Directly Modified: sqlmesh_example.incremental_model (Forward-only)
+ └── Indirectly Modified Children:
+ ├── sqlmesh_example.full_model (Forward-only)
+ └── sqlmesh_example.view_model (Forward-only)
+ Apply - Virtual Update [y/n]: y
-SKIP: No model batches to execute
+ SKIP: No physical layer updates to perform
-Updating virtual layer ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100.0% • 3/3 • 0:00:00
+ SKIP: No model batches to execute
-✔ Virtual layer updated
-```
-```
+ Updating virtual layer ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100.0% • 3/3 • 0:00:00
+ ✔ Virtual layer updated
+ ```
## Miscellaneous
@@ -1343,17 +1246,12 @@ If you notice you have a lot of old development schemas/data, you can clean them
=== "SQLMesh"
-```
-```bash
-sqlmesh janitor
-```
-```
+ ```bash
+ sqlmesh janitor
+ ```
=== "Tobiko Cloud"
-```
-```bash
-tcloud sqlmesh janitor
-```
-```
-
+ ```bash
+ tcloud sqlmesh janitor
+ ```
\ No newline at end of file