llm-evaluation Skill for Claude: How to Test LLM Apps

Gives Claude a playbook for measuring how well an LLM app works, using automated metrics, human review and LLM judges.

Installs: 13.1K; on skills.sh
Stars: 42.1K; on GitHub
Files: 1; besides SKILL.md
Updated: Oct 2026; last repo commit

llm-evaluation is a guidance skill from the wshobson/agents collection. When you ask Claude to test an LLM application, compare prompts or models, or set up an evaluation framework, it loads this skill and follows its three evaluation types: automated metrics, human evaluation and LLM-as-judge. The SKILL.md also includes a Python quick start that defines a small evaluation suite class.

The skill bundles no scripts. Worked examples for judging, A/B testing, regression checks, LangSmith and benchmarking sit in one reference file, references/details.md, which Claude reads when the main file is not enough. Those examples are code samples for you to adapt, and some of them call the Anthropic API.

Good for

Developers shipping an LLM feature who need a way to measure quality before release
Teams comparing two prompts or models and wanting a repeatable method
Engineers adding regression checks so prompt changes do not quietly make answers worse
Anyone building a RAG app who needs retrieval metrics such as MRR and NDCG

Install llm-evaluation

Claude Code

Copy this prompt.

Install the llm-evaluation skill in Claude Code. Run this command:

/plugin marketplace add wshobson/agents
/plugin install llm-application-dev@claude-code-workflows

Then check that it loaded and tell me how to use it.

2
Paste it into Claude Code. It shows the change and asks before running it.

Pick one route. The plugin routes install the whole llm-application-dev plugin: 3 agents, 3 commands and 8 skills (counted from the plugin folder in the repo tree, which has no hooks or MCP config files). The skills-only routes install just this skill, with no agents or commands.

From the project README. Source.

What's in the skill

When it loads

Implement comprehensive evaluation strategies for LLM applications using automated metrics, human feedback, and benchmarking. Use when testing LLM performance, measuring AI application quality, or establishing evaluation frameworks.

Sections in SKILL.md

When to Use This Skill
Core Evaluation Types
1. Automated Metrics
2. Human Evaluation
3. LLM-as-Judge
Quick Start
Detailed patterns and worked examples

Key facts

Author: wshobson
Repository: wshobson/agents
Installs on skills.sh: 13,064 as of 2026-11-02
GitHub stars: 42,081 for the whole repo, 2026-11-02
License: MIT (repo metadata)
Last commit to the repo: 2026-10-30
Files: 2 files including SKILL.md (SKILL.md and references/details.md)
Covers: Automated metrics, human evaluation, LLM-as-judge, A/B testing, regression testing, LangSmith, benchmarking

What it includes

Lists BLEU, ROUGE, BERTScore, accuracy and retrieval scores
Claude picks computed scores for the job: text generation, classification or retrieval (RAG). For RAG that means MRR, NDCG, Precision@K and Recall@K.
Try: “Which metrics should I use to score my RAG retrieval step?”
Automated metrics
Rates answers on accuracy, coherence, relevance, fluency, safety and helpfulness
Sets up manual review with named dimensions. The reference file adds annotation guidelines and inter-rater agreement.
Human evaluation
Has a stronger model score outputs: pointwise, pairwise or reference-based
Claude writes judge prompts that score one answer, compare two answers, or compare an answer to a gold standard. Reference-free judging is also listed.
Try: “Write a pairwise LLM judge to compare two versions of my support-bot prompt.”
LLM-as-judge
A Python EvaluationSuite class that runs metrics over test cases
Defines Metric and EvaluationSuite classes in SKILL.md. It loops over test cases, calls your model and averages each metric. You adapt it to your own model.
Quick start suite
Worked code for judging, A/B tests, regressions, LangSmith and benchmarks
Longer examples loaded only when needed. The judge examples import the Anthropic client, and the LangSmith section imports langsmith.
Read · references/details.md

From the official docs

Questions

What does the llm-evaluation skill do?

It teaches Claude how to measure an LLM application. It covers automated metrics, human review and LLM-as-judge, plus A/B testing, regression testing, LangSmith and benchmarking in a reference file.

How do I install llm-evaluation in Claude Code, the Claude app or Codex?

Use the install box at the top of this page. It installs the skill folder, which holds SKILL.md and references/details.md.

Does it need an API key?

The skill itself bundles no scripts and sets no key. Its example code in references/details.md imports the Anthropic client and langsmith, so code you build from those examples needs your own keys.

Is it safe?

It is instructions and sample code, with no bundled scripts to run. Read any code Claude writes from it before you run it, because the examples call outside services such as the Anthropic API and LangSmith.

Does it run my evaluations for me?

No script runs them. Claude writes or adapts evaluation code in your project from the skill, and you run it.

How is it different from the other skills in wshobson/agents?

It is one skill in the llm-application-dev plugin folder of the collection and focuses on measuring quality. The SKILL.md does not say it needs any sibling skill.

Information

updated Nov 2, 2026

Repository: wshobson/agents
Folder: plugins/llm-application-dev/skills/llm-evaluation

wshobson/agents skills.sh

Sources

Docs, issue trackers and discussions this page is built from.

Show all 4

llm-evaluation SKILL.md · github.com
wshobson/agents README · github.com
llm-application-dev plugin manifest · github.com
llm-evaluation references/details.md · github.com