agents-skill-eval
A production evaluator that scores agent skill packages on quality, cost and reliability.
One evaluator, two entry points
agents-skill-eval takes a SKILL.md package — either as a local folder or a GitHub URL — and runs it through the same production evaluator, returning a single structured report. It exists because "does this skill actually work well?" was being answered by gut feel instead of measurement.
Making quality legible and cheap to check
Skill packages fail in quiet ways: weak trigger phrasing, bloated token footprints, uncontrolled cost, brittle reliability, and security footguns. The goal was a deterministic core that anyone can run repeatably, with an optional AI pass for the judgement calls a rule set cannot make.
Deterministic first, AI optional
Every run goes through deterministic checks and produces a downloadable JSON report. An optional AI review can be layered on top with a provider of choice, kept strictly opt-in so a default run stays fast, private and reproducible.
- InputLocal folder or GitHub blob / tree URL
- Deterministic checksTriggers · tokens · cost · reliability · security
- Optional AI reviewGroq · Gemini · Anthropic · OpenAI
- ReportScored result + downloadable JSON
Both entry points run through the identical production evaluator, so a local run and a hosted run return the same report format.
- Trigger quality scoring
- Token-efficiency analysis
- Cost control checks
- Reliability evaluation
- Security review
- Local folder or GitHub URL input
- Optional AI review (opt-in)
- Downloadable JSON report