Skip to main content

Snapshot configurations

Learn about using snapshot configurations in dbt, including snapshot-specific configurations and general configurations.

Learn by video!
For video tutorials on Snapshots, go to dbt Learn and check out the Snapshots course.

Available configurations​

Snapshot-specific configurations​

Resource-specific configurations are applicable to only one dbt resource type rather than multiple resource types. You can define these settings in the project file (dbt_project.yml), a property file (models/properties.yml for models, similarly for other resources), or within the resource’s file using the {{ config() }} macro.

The following resource-specific configurations are only available to Snapshots:

Snapshot configuration migration​

The latest snapshot configurations introduced in dbt Core v1.9 (such as snapshot_meta_column_names, dbt_valid_to_current, and hard_deletes) are best suited for new snapshots, but you can also adopt them in existing snapshots by migrating your table schema and configs carefully to avoid any inconsistencies in your snapshots.

Here's how you can do it:

  1. In your data platform, create a backup snapshot table. You can copy it to a new table:

    create table my_snapshot_table_backup as
    select * from my_snapshot_table;

    This allows you to restore your snapshot if anything goes wrong during migration.

  2. If you want to use the new configs, add required columns to your existing snapshot table using alter statements as needed. Here's an example of what to add if you're going to use dbt_valid_to_current and snapshot_meta_column_names:

    alter table my_snapshot_table
    add column dbt_valid_to_current string,
    add column dbt_valid_from timestamp,
    add column dbt_valid_to timestamp;

  3. Then update your snapshot config:

    snapshots:
      - name: orders_snapshot
        config:
          strategy: timestamp
          updated_at: updated_at
          unique_key: id
          dbt_valid_to_current: "to_date('9999-12-31')"
          snapshot_meta_column_names:
            dbt_valid_from: start_date
            dbt_valid_to: end_date

  4. Test each change before adopting multiple new configs by running dbt snapshot in development or staging.

  5. Confirm if the snapshot run completes without errors, the new columns are created, and historical logic behaves as you’d expect. The table should look like this:

    idstart_dateend_dateupdated_at
    12024-10-01 09:00:002024-10-03 08:00:002024-10-01 09:00:00
    22024-10-03 08:00:009999-12-31 00:00:002024-10-03 08:00:00
    32024-10-02 11:15:009999-12-31 00:00:002024-10-02 11:15:00

Note: The end_date column (defined by snapshot_meta_column_names) uses the configured value from dbt_valid_to_current (9999-12-31) for newly inserted records, instead of the default NULL. Existing records will have NULL for end_date.

warning

If you use one of the latest configs, such as dbt_valid_to_current, without migrating your data, you may have mixed old and new data, leading to an incorrect downstream result.

General configurations​

General configurations provide broader operational settings applicable across multiple resource types. Like resource-specific configurations, these can also be set in the project file, property files, or within resource-specific files.

dbt_project.yml

Configuring snapshots​

Snapshots can be configured in multiple ways:

Snapshot configurations are applied hierarchically in the order above with higher taking precedence. You can also apply tests to snapshots using the tests property.

Examples​

  • Apply configurations to all snapshots​

    To apply a configuration to all snapshots, including those in any installed packages, nest the configuration directly under the snapshots key:

    dbt_project.yml
    snapshots:
      +unique_key: id

  • Apply configurations to all snapshots in your project​

    To apply a configuration to all snapshots in your project only (for example, excluding any snapshots in installed packages), provide your project name as part of the resource path.

    For a project named jaffle_shop:

    dbt_project.yml
    snapshots:
      jaffle_shop:
        +unique_key: id

    Similarly, you can use the name of an installed package to configure snapshots in that package.

  • Apply configurations to one snapshot only​

    You can also use the full resource path (including the project name, and subdirectories) to configure an individual snapshot from your dbt_project.yml file.

    For a project named jaffle_shop, with a snapshot file within the snapshots/postgres_app/ directory, where the snapshot is named orders_snapshot (as above), this would look like:

    dbt_project.yml
    snapshots:
      jaffle_shop:
        postgres_app:
          orders_snapshot:
            +unique_key: id
            +strategy: timestamp
            +updated_at: updated_at

    You can also define some common configs in a snapshot's config block. However, we don't recommend this for a snapshot's required configuration.

    dbt_project.yml
    version: 2

    snapshots:
      - name: orders_snapshot
        +persist_docs:
          relation: true
          columns: true
0
Edit this page