The Databricks Catalog Other Engineers Build Inside

The problem

The first version existed to deliver one Snowflake migration in Entertainment. Building it made the wider problem obvious: engagements across entertainment, finance, healthcare, and public sector were re-deriving the same Databricks decisions every time, on migration patterns, validation approach, governance conventions, and naming. The knowledge existed, but it lived in individual engineers’ heads and in whichever repo they last worked in. That has two costs that compound: onboarding an engineer to an engagement takes a week of tribal knowledge transfer, and delivery quality varies by who happens to be staffed. Neither is fixable by writing more documentation, because documentation is not where an engineer is at the moment they make the decision.

The approach

I built it as a system with an interface rather than a pile of documents. Every skill is a router: a name and a description carrying the condition that fires it and the condition that defers it to something else, so an agent reaches the right one and steps the wrong one aside. Depth sits behind that router in reference files pulled only when a step calls for them, which keeps the always-loaded half cheap to route on. Each skill ships a graded eval whose cases test correct deferral as well as correct firing, because a skill that fires on everything passes a suite that only tests firing. One source then renders to four surfaces: a marketplace plugin, a slash command, an MCP server that reaches coding agents mid-task, and an OKF export in vendor-neutral Markdown. That last one is generated from source and drift-guarded, so the mirror cannot quietly diverge from what it represents. Client identifiers are scrubbed to zero and re-checked before every commit, and a review agent gates the frontmatter contract, the scrub, and index currency before anything lands.

The outcome

It went practice-wide because the engineers who would have to use it asked for it, which is a better test of whether an accelerator earns generalizing than my own view of it was. It became the standard the engagements ran on, and it is why a second engagement in an industry started materially further along than the first. It carries five engagement shapes rather than one: a greenfield lakehouse build, an AI or agent delivery, a BI and data-app rollout, a platform-hardening pass, and a migration. What I would do differently: the telemetry loop came second. I wrote perhaps thirty skills on instinct before anything measured what got used, and the ledger later showed that some of them never did. Building the feedback loop first would have told me what to write instead of leaving me to guess.

What I owned

Mine

The skill contract and its firing and deferral clauses, the split between router and references, the graded eval format, the OKF export and its drift guard, and the skills themselves.

Inherited

The plugin backend that discovers each skill file, and the service-line convention it keys on. Both already existed, and the library is written to fit them rather than the other way round.

The trade-off

The call

Split every skill into a token-cheap router and reference files loaded only on demand.

Instead of

One self-contained document per skill, which is simpler to write and simpler to read.

Because

The router is loaded every time an agent decides whether a skill applies, so its size is a tax on every decision, while depth that is never read still costs. The price is a second fetch on the skills that do get used, and I would pay it again.