agents-skill-eval
A production evaluator that scores agent skill packages on quality, cost and reliability.
One evaluator, two entry points
agents-skill-eval takes a SKILL.md package — either as a local folder or a GitHub URL — and runs it through the same production evaluator, returning a single structured report. It exists because "does this skill actually work well?" was being answered by gut feel instead of measurement.
Making quality legible and cheap to check
Skill packages fail in quiet ways: drift from the spec, weak trigger phrasing, bloated token footprints, ambiguous instructions, and security footguns. The goal was a deterministic core that anyone can run repeatably, with an optional AI pass for the judgement calls a rule set cannot make.
Deterministic first, AI optional
Every run goes through deterministic checks and produces a downloadable JSON report. An optional AI review can be layered on top with a provider of choice, kept strictly opt-in so a default run stays fast, private and reproducible.
- 01InputLocal folder or GitHub blob / tree URL
- 02Deterministic checksSpec · security · tokens · effectiveness
- 03Optional AI reviewGroq · Gemini · Anthropic · OpenAI
- 04ReportScored result + downloadable JSON
Both entry points run through the identical production evaluator, so a local run and a hosted run return the same report format.
Spec compliance checks
Every skill gets checked against the spec it is supposed to follow, and anything non-standard is flagged immediately. That catches the kind of drift that works fine on one machine and quietly breaks on the next.
Security review
Overly broad permissions, unguarded deletes and hardcoded paths are flagged before a skill goes anywhere near real data. Nothing is sent to a third-party AI provider unless that is explicitly switched on for the run.
Token-efficiency analysis
Every skill costs a little context each time it loads, so bloat adds up fast across a team. The evaluator flags padding and duplication and names exactly which section to trim.
Effectiveness scoring
The biggest check is the simplest question: will this actually work? Ambiguous instructions, weak examples and vague trigger wording all get caught here — vague wording is the single most common reason a skill never fires.
Local folder or GitHub URL input
Point it at a folder on disk or paste a GitHub link — either way it runs through the same evaluator and gets the same report, so results are always comparable.
Deterministic by default
A default run uses no AI model at all, so it is free to run and returns the same result every time. That makes it safe to run on every commit instead of saving it for a release, and nothing leaves the machine unless asked to.
Optional AI review (opt-in)
Some judgement calls need more than a rule set. An optional AI pass, in a provider of choice, layers on top of the deterministic result — but only when it is switched on, so a default run stays fast, private and repeatable.
Downloadable JSON report
Every run returns a score out of 100, a plain-language rating, and a full breakdown of what failed and why. The report is machine-readable, so it drops straight into CI without extra glue code.