Personal project · 2025–2026
The Databricks Catalog Other Engineers Build Inside
A ~90-skill Databricks engineering catalog, served live to AI coding agents over MCP, with a usage-telemetry loop that decides what gets written next.
- Databricks
- Python
- FastMCP
- Google Cloud Run
- BigQuery
- OKF v0.1
- Databricks engineering skills codified
- ~90
- distribution surfaces from one source
- 4
- served to coding agents over MCP
- Live
Databricks engineering skills codified
distribution surfaces from one source
served to coding agents over MCP
The problem
Delivering migrations across entertainment, finance, healthcare, and public sector meant re-deriving the same Databricks decisions on every engagement: migration patterns, validation approach, governance conventions, naming. The knowledge existed, but it lived in individual engineers’ heads and in whichever repo they last worked in. That has two costs that compound: onboarding an engineer to an engagement takes a week of tribal knowledge transfer, and delivery quality varies by who happens to be staffed. Neither is fixable by writing more documentation, because documentation is not where an engineer is at the moment they make the decision.
The approach
I codified roughly 90 Databricks engineering skills across two trees and distributed them over four surfaces, so the guidance meets engineers where they already work rather than asking them to go and read something. One of those surfaces is a FastMCP server on Cloud Run that serves the catalog live to AI coding agents mid-task. Every invocation writes to a BigQuery usage ledger, and a harvest-learnings loop reads that ledger to decide which skills need revision and which are missing entirely, so the catalog is shaped by what engineers actually reach for, not by what I assumed they would. It exports to the open OKF v0.1 format rather than locking into a single vendor’s skill schema.
The outcome
The catalog is the standard the engagements run on, and it is why a second engagement in an industry starts materially further along than the first. What I would do differently: the telemetry loop came second. I wrote perhaps thirty skills on instinct before anything measured what got used, and the ledger later showed that some of them never did. Building the feedback loop first would have told me what to write instead of leaving me to guess.