Skip to main content

Snowflake Migration Guide

Availability
Available CoreAvailable StandardAvailable PlusAvailable ProAvailable Enterprise FlexAvailable Self-Managed EnterpriseAvailable PyAirbyte
Support Level
Airbyte
Connector Version
4.0.28(last updated 5 days ago)
Sync Success Rate
Usage Rate
Definition ID
424892c4-daac-4491-b35d-c6688ba547ba

Upgrading to 4.0.0​

This version upgrades the Snowflake destination from using typing and deduping to direct loading. This upgrade improves performance and reduces warehouse spend. If you have unusual requirements around record visibility or schema evolution, read the documentation for those methodologies for more information about how direct loading differs from typing and deduping.

This version also adds an option to enable CDC deletions as soft deletes.

Decide how to handle raw tables​

The exact steps to begin the migration depend on whether you interact with Airbyte's raw tables in Snowflake.

If you don't interact with raw tables​

If you don't interact with the raw tables, you can safely upgrade. There's no breaking change for you.

If you only interact with raw tables​

If you only interact with the raw tables, enable the Disable Final Tables option before upgrading. This enables the Legacy raw tables option after upgrading.

  1. In the navigation bar, click Destinations.

  2. Open your Snowflake connector.

  3. Click Edit configuration.

  4. Open the Advanced section.

  5. Turn on Disable Final Tables.

  6. Click Save.

note

After upgrading to version 4, this setting appears as Legacy raw tables and should remain enabled.

If you interact with both raw and final tables​

If you interact with both the raw and final tables, this use case is no longer supported. Instead, create two connectors. One with Disable Final Tables turned on, and one with it turned off. Starting now, you must run the two connections in parallel.

  1. In the navigation bar, click Destinations.

  2. For each Snowflake destination you have, add an identical second Snowflake destination.

  3. Ensure each pair of Snowflake connectors have opposite settings for Disable Final Tables (in version 3) or Legacy raw tables (in version 4). One connector should have this setting enabled, and the other should have it disabled.

  4. Configure distinct default schemas for each destination to avoid table name collisions.

  5. Update your connections to point to the appropriate destination:

    • Connections that need raw tables only should target the destination with Disable Final Tables or Legacy raw tables enabled.
    • Connections that need final tables should target the destination with this setting disabled.

  6. Run test syncs on both destinations to verify outputs:

    • The raw-only destination should write only to the internal schema (default airbyte_internal).
    • The standard destination should write only final tables to the target schema.

  7. After verifying that both destinations work correctly, continue running both connections in parallel going forward.

Review the following information to prepare for and execute your upgrade.

Review the changelog​

Before updating a connector, review the changelog to understand the changes and their potential impact on your existing connections. Find the changelog for any connector by navigating to the bottom of the documentation for that connector. Major version releases also include a migration guide.

Plan for major updates​

Major updates may require you to adjust connection settings or even make changes to your data pipelines. Allocate enough time and resources for this. Use the migration guide to ensure your transition process goes smoothly.

Airbyte provides tooling that guarantees safe connector version bumps and enforces automated version bumps for minor and patch updates. You always need to manually update for major version bumps.

Self-managed plans: pin a specific version if you can't update​

If you're unable to upgrade to the new version of a connector, you can pin that connector to a specific version.

  1. In the navigation bar:

    • If you're on the Self-Managed Enterprise plan, click Organization settings > Sources/Destinations.

    • If you're on any other plan, click Workspace settings > Sources/Destinations.

  2. Edit the entry for the connector you want to pin.

  3. Set the Default Version to the version you want to use.

Self-managed plans: update the local connector image​

If you self-manage Airbyte, you must manually update the connector image in your local registry before proceeding with the migration. Follow the steps below.

  1. In the navigation bar:

    • If you're on the Self-Managed Enterprise plan, click Organization settings > Sources/Destinations.

    • If you're on any other plan, click Workspace settings > Sources/Destinations.

  2. Find the connector you want to update in the list of connectors.

    note

    Airbyte lists two versions, the current in-use version and the latest version available.

  3. Click Change to update your OSS version to the latest available version.

Update the connector version​

Update each instance of the connector separately. If you have multiple instances of a connector, updating one doesn't affect the others.

  1. In the navigation bar:

    • If you're on the Self-Managed Enterprise plan, click Organization settings > Sources/Destinations.

    • If you're on any other plan, click Workspace settings > Sources/Destinations.

  2. Select the instance of the connector you wish to upgrade.

  3. Select Upgrade.

  4. Follow the prompt to confirm you are ready to upgrade to the new version.

Clear data from affected streams​

After upgrading a connector with a breaking change, you must refresh affected schemas and clear your data.

  1. In the nav bar, click Connections.

  2. Find the connection affected by the upgrade.

  3. Click the Schema tab.

  4. Click Refresh source schema (looks like ). When Airbyte finishes, it shows you any detected schema changes.

  5. Click OK.

  6. Click Save changes

  7. Clear the data for the streams affected by this upgrade.

Once the clear is complete, you can begin syncing your data again as usual.

Optional: clean up legacy raw tables​

The version 4.0 connector doesn't automatically remove tables created by earlier versions. After upgrading to version 4 and verifying your data, you can optionally remove the old raw tables.

For most users, you can find the raw tables in the schema configured as Airbyte Internal Table Dataset Name (which defaults to airbyte_internal). If you customized this setting, look in that schema instead.

The table names match these patterns depending on which version created them:

  • Version 2/3 (Typing and Deduping): raw_{namespace}__{stream} (for example, airbyte_internal.raw_public__users)
  • Version 4 (Legacy raw tables mode): {namespace}_raw__stream_{stream} (for example, airbyte_internal.public_raw__stream__users)

Note: The number of underscores between raw and stream may vary depending on the longest underscore sequence in your namespace and stream names.

note

Version 4 of the Snowflake destination uses the airbyte_internal schema for temporary scratch space (for example, streams running in dedup mode, truncate refreshes, and overwrite syncs). Dropping the entire airbyte_internal schema can interrupt active syncs and cause data loss. Only drop the specific raw tables you no longer need.

To remove the old raw tables:

  1. Pause or allow active syncs to complete before dropping any tables to avoid interrupting data transfers.

  2. List candidate raw tables to identify which tables to remove:

    -- For Version 2/3 raw tables:
    SHOW TABLES IN SCHEMA <DATABASE>.<INTERNAL_SCHEMA> LIKE 'RAW\_%';

    -- For Version 4 legacy raw tables:
    SHOW TABLES IN SCHEMA <DATABASE>.<INTERNAL_SCHEMA> LIKE '%_RAW__STREAM_%';

    Replace <DATABASE> with your Snowflake database name and <INTERNAL_SCHEMA> with your internal schema name (default airbyte_internal).

  3. Drop specific raw tables you no longer need:

    DROP TABLE IF EXISTS <DATABASE>.<INTERNAL_SCHEMA>.<TABLE_NAME>;

    Replace <TABLE_NAME> with the specific table name you want to remove. Use fully qualified names (database.schema.table) to avoid ambiguity.

Upgrading to 3.0.0​

This version introduces Destinations V2, which provides better error handling, incremental delivery of data for large syncs, and improved final table structures. To review the breaking changes, and how to upgrade, see here. These changes will likely require updates to downstream dbt / SQL models, which we walk through here. Selecting Upgrade will upgrade all connections using this destination at their next sync. You can manually sync existing connections prior to the next scheduled sync to start the upgrade early.

Worthy of specific mention, this version includes:

  • Per-record error handling
  • Clearer table structure
  • Removal of sub-tables for nested properties
  • Removal of SCD tables

Learn more about what's new in Destinations V2 here.

Upgrading to 2.0.0​

Snowflake no longer supports GCS/S3. Please migrate to the Internal Staging option. This is recommended by Snowflake and is cheaper and faster.

Was this page helpful?