Costing a Migration Nobody Could Log Into

The problem

The engagement was sold as a limited proof of concept: move a few BigQuery tables and one simple workflow into Databricks, and show it works. Two weeks in, the real object came into view. The warehouse was the easy half. What ran the business was a homegrown orchestration framework of roughly 2,000 tasks defined in Python and YAML, and every dependency and schedule the analytics ran on lived inside it. Moving the tables without it would have moved nothing. That turned the question from feasibility into cost, and cost was the thing nobody could answer: a migration of that size gets funded on a number, and no one on either side had counted what was in there. An unpriced migration is not one a client can approve.

The approach

Two things had to be true at once: a number defensible enough to fund a migration, and evidence that Databricks could carry the orchestration at all. I owned both. Warehouse access never arrived, so the analysis ran against the source of record instead, the transformation SQL and DAG definitions in the client’s Git repository. I stood up a Databricks Model Serving endpoint in our own workspace and scored that codebase through the practice’s migration estimator, which produced complexity across query structure, joins, UDFs, and scripting patterns. Out of it came a level-of-effort model with duration bands and risk factors by workload group, and the migration groupings and wave sequencing the plan would run on. The evidence came from a proof of concept I designed and built end to end. It carried the Databricks-native equivalent of the framework’s DAG model, a prototype pipeline demonstrating dependency ordering and execution, and a set of the framework’s Python tasks reimplemented as Databricks tasks. It shipped with a developer guide pitched at a junior developer and a recorded walkthrough, because a proof of concept nobody can extend proves feasibility once and then rots.

The outcome

The five-week assessment converted. The client executed the SOW and partner funding was approved, and the program moved from a proposal into funded delivery with a team onboarded. The number it was funded on was ~6,800 hours for the transformation component, and that is the figure I would defend hardest, because everything downstream was sized against it. What I would do differently: I priced a codebase I could read but never run. Static analysis over a repository gives you structure and complexity, and it cannot tell you which of those 2,000 tasks are dead, so some fraction of that estimate is work nobody needs done. The honest version of the number carries a runtime-usage cut alongside the static one. Getting warehouse access early is not a scheduling detail. It is the difference between an estimate and a measurement.

What I owned

Mine

The code analysis end to end, from complexity scoring through the level-of-effort model with its duration bands and risk factors, and the migration groupings and wave sequencing that came out of it. The proof of concept was mine end to end too: the Databricks-native orchestration design, the prototype DAG, the Python task reimplementations, and the developer guide and recording that shipped with it.

Inherited

A colleague produced the current-state inventory and the architecture diagram behind it, and drafted the three target-state options the client chose from. Beyond that, the migration strategy and the ROM cost and staffing model were team deliverables, and the executive readout was delivered jointly.

The trade-off

The call

Spend the proof of concept on replacing the orchestration framework, not on moving BigQuery tables.

Instead of

The proof of concept as sold: a small-scale migration of selected tables and one simple workflow, which is lower-risk to demonstrate and the thing the client had already agreed to buy.

Because

Moving tables proves the easy half. Nobody doubted Databricks could hold the data; the open question was whether it could carry a 2,000-task dependency graph without the framework around it, and that is the question the funding decision turned on. The cost was real: changing the demonstration mid-engagement spent client goodwill and put a harder problem on the same date, with the full dependency analysis pushed into the proof of concept to pay for it.