NC Department of Information Technology · 2025
Statewide Transportation Data, Made Trustworthy
Real-time Delta Live Table ingestion that improved statewide safety reporting availability 30%, plus the metadata automation and glossary that made new sources fast to onboard and the numbers safe to publish.
- Databricks
- Delta Live Tables
- PySpark
- Azure OpenAI
- Data Vault 2.0
- Data Mesh
- improvement in statewide reporting availability
- 30%
- faster onboarding for a new data source
- 60%
- reduction in data errors
- 60%
- business metrics standardized in one glossary
- 100+
improvement in statewide reporting availability
faster onboarding for a new data source
reduction in data errors
business metrics standardized in one glossary
The problem
Statewide safety and mobility reporting depended on a legacy ETL that was quietly unreliable, and onboarding a new transportation dataset was slow enough to be a planning constraint in its own right. Both problems were worse than they looked. When a feed lapsed, the pipeline carried on and the dashboard still rendered, just with a gap nobody could see. For an agency making operational decisions about roads, an availability problem that presents as a plausible-looking number is considerably worse than an outage, because nothing signals that the answer is wrong.
The approach
I architected and deployed Delta Live Table pipelines in PySpark to automate real-time ingestion and transformation, replacing the legacy ETL path. Onboarding was a separate problem, so it got a separate answer: an Azure OpenAI solution with deliberate prompt design that automates metadata capture and Data Vault schema creation, taking the repetitive modeling work off engineers without handing over the modeling decisions. Then the part I care most about: dimensional models within a Data Mesh framework, standardizing 100+ business metrics through a centralized Business Model Glossary, and test-driven development for the pipelines so the assertion about what the data must look like is written before the transformation that produces it.
The outcome
Statewide safety reporting availability improved 30% over the legacy ETL, new sources onboarded 60% faster, and data errors fell 60%. The glossary mattered more than any single pipeline: cross-team reporting and executive self-service now resolve to one definition, which converts trust from a social property of the team into a property of the platform.