Personal project · 2025–2026
Proving Agent Skills Actually Work
A meta-toolkit for building agent skills and proving they work: 24 meta-skills, ten executable workflows, and a pass^k ≥ 0.95 graduation bar, shipped as a zero-dependency MCP server.
- Python
- Model Context Protocol
- OAuth 2.1
- uv workspace
- meta-skills for authoring and evaluation
- ~24
- graduation bar over k independent runs
- pass^k ≥ 0.95
- executable workflows
- 10
meta-skills for authoring and evaluation
graduation bar over k independent runs
executable workflows
The problem
Agent skills are easy to write and hard to trust. Almost all of them ship on the author’s say-so: the skill looks right in a demo, then fails intermittently in real use, and there is no way to tell a skill that works from one that happened to work once. Non-determinism makes the usual answer (write a test) insufficient, because a single passing run proves very little about the next one.
The approach
I built roughly 24 meta-skills for authoring and evaluating other skills, and set the graduation bar at pass^k ≥ 0.95: a skill has to pass k independent runs, not one, before it counts as working. Verification is deterministic and external: no skill certifies itself. It runs as a zero-dependency stdlib MCP server with OAuth 2.1, ten executable workflows, and a uv workspace split into core, runner, and mcp so the evaluation harness cannot quietly depend on the thing it is evaluating.
The outcome
The graduation bar is the whole point: it makes "this skill works" a measured claim rather than an assertion, and it rejected a meaningful fraction of skills that passed a single run convincingly. It now runs as a hosted, billable MCP server. The zero-dependency constraint was the decision I second-guessed most and would make again: it cost effort early and has cost nothing since.