Proving a Migration, Not Just Doing One

The problem

The analytics behind retention, engagement, and monetization decisions ran in Snowflake across roughly ten source tables in multiple databases, on an expensive skeleton cross-join and MERGE pattern layered over stored procedures. It was costly and hard to maintain, but that was the smaller problem. The larger one was trust: as the workload moved to Databricks, nobody had a repeatable way to confirm that a migrated table was genuinely equivalent to its Snowflake source. Without that proof there was no defensible basis for the business to commit a revenue-driving analytics asset to a new platform. "The engineers checked it" is not a basis. And the pilot was one workload among roughly 500 pipelines across 20 domains. Migrating each by hand at pilot rigor does not scale, and an AI agent that migrates a flow on its own say-so is "the engineers checked it" with nobody left to ask.

The approach

I built the migration and its proof as one system rather than two. Config-driven loads pull Snowflake into Bronze over Lakehouse Federation, with automatic merge and cluster-key detection, watermark incrementals, and a hardened MERGE using Liquid Clustering, so onboarding a new source table is one line of config, not new code. Lakeflow Declarative Pipelines then rebuild the model Bronze → Silver → Gold with the original stored procedures and UDFs migrated across, running two titles concurrently into one consolidated Gold table. Every landed table is regression-checked on schema, row count, and SHA-256 against its source, and the rebuilt Gold is confirmed by a tolerance-based statistical comparison, again with no per-table rule code. The last piece mattered most for adoption: an AI/BI dashboard that puts the evidence in front of non-technical stakeholders in language they can act on. Cutover is reversible: a strangler-fig seam moves an incremental raw-to-Silver slice of each flow onto Databricks, and a reverse-sync bridge keeps downstream Snowflake consumers whole until the rest follows. For the wider estate I designed a gated agentic workflow that runs as two separate launches. The first reads only the legacy source and writes an object inventory, an architecture decision sheet with an empty approval box on every row, and the questions only the client can answer. A person ticks every row before any code exists, because approval takes days, and a workflow that paused to ask would either stall or make the call the gate exists to prevent. The second launch builds the migrated tree through five stages, each advancing only when its own verifier passes. Adversarial review follows, with a golden diff against a hand-built reference reported beside the verdict, never folded into it. One repair pass is allowed, and the re-validation decides the verdict: READY, NEEDS-CHANGES, or NOT-READY, written to a numbered run folder that is never overwritten. The work runs on 22 subagents tiered by model, with judging pinned to Sonnet or above because a Haiku judge over-confirms. A read firewall keeps the hand-built reference answers away from every producing agent, so the benchmark cannot be transcribed. The workflow deploys nothing.

The outcome

678M rows migrated on the pilot with structure 100% identical on every validation run, and all 114 automated checks passing on settled data: 0.01% average difference, 1.1% at worst, against a 2% tolerance. The most useful result was the single run the framework flagged: not a migration error, but the Snowflake source caught mid-load, spiking 40–50× for a day while Databricks stayed correct. A validation framework that only ever confirms what you hoped is not a validation framework. The limit worth naming: row-level parity is measured on the pilot only. Every other flow is asserted from static review, not yet executed against data, and the estate is mid-migration. Those are different strengths of claim, and I report them separately. The discipline carried into the agentic workflow intact: a repair can hand the reviewer a fixed tree, but it cannot turn a red verdict green, because an agent that grades its own work is not a gate. The tooling built to run this migration also became the practice accelerator.

What I owned

Mine

The target architecture, the config-driven load framework, the validation framework that became the engagement’s sign-off gate, and the dashboard that put its evidence in front of the business. Beyond the pilot, the gated agentic migration workflow end to end: the two-launch human gate, the verifier-gated build stages, the model tiering and its judging floor, and the read firewall.

Inherited

The Snowflake source model, the stored procedures and UDFs whose behavior the rebuilt Gold layer had to reproduce, and the client’s in-house orchestration platform that part of the estate deploys on.

The trade-off

The call

Prove the rebuilt Gold layer with a tolerance-based statistical comparison, not an exact match.

Instead of

Row-for-row equality on every table, which is the check a business expects a migration to pass.

Because

Gold is rebuilt from the source rather than copied, so exact equality was never going to hold, and a check that cannot pass is a check that gets waived on the first deadline. A stated tolerance with a measured margin can be argued about on the numbers, which is what a sign-off gate has to survive. Landed tables are still held to SHA-256.