Personal project · 2025–2026

Proving Agent Skills Actually Work

A meta-toolkit for building agent skills and proving they work: 24 meta-skills and ten executable workflows, graduated only past a pass^k ≥ 0.95 bar, shipped as a zero-dependency MCP server.

Python
Model Context Protocol
OAuth 2.1
uv workspace

meta-skills for authoring and evaluation: ~24
graduation bar over k independent runs: pass^k ≥ 0.95
executable workflows: 10

The problem

Agent skills are easy to write and hard to trust. Almost all of them ship on the author’s say-so: the skill looks right in a demo, then fails intermittently in real use, and there is no way to tell a skill that works from one that happened to work once. Non-determinism makes the usual answer (write a test) insufficient, because a single passing run proves very little about the next one.

The approach

I built roughly 24 meta-skills for authoring and evaluating other skills, and set the graduation bar at pass^k ≥ 0.95: a skill has to pass k independent runs, not one, before it counts as working. Verification is deterministic and external: no skill certifies itself. It runs as a zero-dependency stdlib MCP server with OAuth 2.1, ten executable workflows, and a uv workspace split into core, runner, and mcp so the evaluation harness cannot quietly depend on the thing it is evaluating.

The outcome

The graduation bar is the whole point: it makes "this skill works" a measured claim rather than an assertion, and it rejected a meaningful fraction of skills that passed a single run convincingly. It now runs as a hosted, billable MCP server. The zero-dependency constraint was the decision I second-guessed most and would make again: it cost effort early and has cost nothing since.

What I owned

Mine: The meta-skill set and the pass^k graduation bar, the deterministic external verification model, the zero-dependency stdlib MCP server with OAuth 2.1, and the uv workspace split into core, runner, and mcp.

The trade-off

The call: Build the MCP server on the Python standard library, with zero runtime dependencies.
Instead of: A framework-backed server built on the official MCP SDK, which is faster to stand up and has a larger ecosystem behind it.
Because: The server is what verifies every skill in the catalog, so a dependency in it is a dependency the evaluation harness cannot see past. Zero runtime dependencies removes an entire class of supply-chain and version-drift risk from a server other agents connect to. It cost more upfront and has cost nothing since.