Entertainment, client name withheld · 2026
Proving a Migration, Not Just Doing One
Migrated a player-cohort analytics platform off Snowflake onto a Databricks medallion architecture, and built the config-driven framework that proved every migrated table matched its source: 678M rows, 114 automated checks per run.
- Databricks
- Lakeflow Declarative Pipelines
- Lakehouse Federation
- Snowflake
- Delta Lake (MERGE, Liquid Clustering)
- Medallion architecture
- PySpark
- SQL
- AI/BI Dashboards
- rows migrated on the pilot
- 678M
- structural match on every validation run
- 100%
- automated checks per run
- 114
- average difference against source (2% tolerance)
- 0.01%
rows migrated on the pilot
structural match on every validation run
automated checks per run
average difference against source (2% tolerance)
The problem
The analytics behind retention, engagement, and monetization decisions ran in Snowflake across roughly ten source tables in multiple databases, on an expensive skeleton cross-join and MERGE pattern layered over stored procedures. It was costly and hard to maintain, but that was the smaller problem. The larger one was trust: as the workload moved to Databricks, nobody had a repeatable way to confirm that a migrated table was genuinely equivalent to its Snowflake source. Without that proof there was no defensible basis for the business to commit a revenue-driving analytics asset to a new platform. "The engineers checked it" is not a basis.
The approach
I built the migration and its proof as one system rather than two. Config-driven loads pull Snowflake into Bronze over Lakehouse Federation, with automatic merge and cluster-key detection, watermark incrementals, and a hardened MERGE using Liquid Clustering, so onboarding a new source table is one line of config, not new code. Lakeflow Declarative Pipelines then rebuild the model Bronze → Silver → Gold with the original stored procedures and UDFs migrated across, running two titles concurrently into one consolidated Gold table. Every landed table is regression-checked on schema, row count, and SHA-256 against its source, and the rebuilt Gold is confirmed by a tolerance-based statistical comparison, again with no per-table rule code. The last piece mattered most for adoption: an AI/BI dashboard that puts the evidence in front of non-technical stakeholders in language they can act on.
The outcome
678M rows migrated on the pilot with structure 100% identical on every validation run, and all 114 automated checks passing on settled data: 0.01% average difference, 1.1% at worst, against a 2% tolerance. The most useful result was the single run the framework flagged: not a migration error, but the Snowflake source caught mid-load, spiking 40–50× for a day while Databricks stayed correct. A validation framework that only ever confirms what you hoped is not a validation framework. Because the checks are config-driven, every table migrated since is proven the same way with no new code.