Data migration · Data reconciliation

Migrate your warehouse with confidence.

Data reconciliation for platform teams moving a warehouse.

Reconcile row by row where it matters and know the data is right before it goes live.

Every table proves it matches, row by row

What the old system let through fails on the way in

Your data stays in your network

“We leveraged Soda to handle a lot of the reconciliation during migration.”

Gu Xie, Group 1001

How do you prove the migrated data matches?

“We want to see if the data matches between the source and the target.”

“Can we absolutely positively prove that there’s never a discrepancy if we don’t reconcile source to target?”

“Cleaning the data during our last migration was a lot of work without any tools.”

“If you want to decommission the old system, someone has to go recode the bloody thing.”

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
dataset: snowflake_ds/erp/orders
reconciliation:
  sources:
    - name: legacy
      dataset: legacy_ds/erp/orders
  checks:
    - row_count_diff:
        source: legacy
    - rows_diff:
        source: legacy
        source_key_columns: [order_id]
        target_key_columns: [order_id]
        threshold:
          must_be: 0
    - schema_diff:
        source: legacy

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
dataset: snowflake_ds/erp/orders
reconciliation:
  sources:
    - name: legacy
      dataset: legacy_ds/erp/orders
  checks:
    - row_count_diff:
        source: legacy
    - rows_diff:
        source: legacy
        source_key_columns: [order_id]
        target_key_columns: [order_id]
        threshold:
          must_be: 0
    - schema_diff:
        source: legacy

The data moved. Nobody can prove it matched.

When the proof is a check that runs, not a query someone ran once.

Three checks in the same file as the table’s quality rules.

Row counts for every table. Row by row where it matters.

✓

Row counts, sums and averages, within a tolerance.

✓

Row by row on a key: missing, changed, duplicated.

✓

Column by column: missing, changed type, renamed.

Old codes on the source, new codes on the target. One check.

✓

One filter for the legacy side, one for the new.

✓

Both systems reconcile to the target until cutover.

✓

Reconcile last night’s batch, not the whole table.

What your old system let through fails on the way in.

✓

Missing, invalid or duplicate rows fail in staging.

✓

Failed rows land in your warehouse. Nothing outside.

✓

Checks stay after cutover. Nothing to recode.

“Ironing out the migration over a few weeks before going live.”

Gu Xie, Group 1001. PostgreSQL to Snowflake. Row counts and schema alignment verified over a few weeks before going live.

“We combined Snowflake with Airflow at the time, and leveraged Soda to handle a lot of the reconciliation during migration. That was key to ensuring data quality was maintained while ironing out the migration over a few weeks before going live.”