agentic-eval Skill for Claude: What It Does and How to Install It

Gives Claude patterns for making AI agents check and improve their own output: reflection loops, evaluator-optimizer pipelines and rubrics.

Installs: 10.2K; on skills.sh
Stars: 40.7K; on GitHub
Updated: Oct 2026; last repo commit

This skill is a short guide to evaluation patterns, from the community awesome-copilot collection. It shows a generate, evaluate, critique, refine loop in three patterns: basic reflection, evaluator-optimizer and code-specific reflection.

It then lists three evaluation strategies (outcome-based, LLM-as-judge and rubric-based), a best-practices table of 5 rows and a setup checklist. The Python examples are sketches that call an llm() function. The skill bundles one file and no scripts.

Good for

Developers building agents that refine their own output
Teams adding LLM-as-judge or rubric scoring
Anyone adding test-driven refinement to code generation

Install agentic-eval

Claude Code

Copy this prompt.

Install the agentic-eval skill in Claude Code. Run this command:

npx skills add https://github.com/github/awesome-copilot --skill agentic-eval

Then check that it loaded and tell me how to use it.

2
Paste it into Claude Code. It shows the change and asks before running it.

Requires Node.js.

From its listing on skills.sh. Source.

What's in the skill

When it loads

Patterns and techniques for evaluating and improving AI agent outputs. Use this skill when: - Implementing self-critique and reflection loops - Building evaluator-optimizer pipelines for quality-critical generation - Creating test-driven code refinement workflows - Designing rubric-based or LLM-as-judge evaluation systems - Adding iterative improvement to agent outputs (code, reports, analysis) - Measuring and improving agent response quality

Sections in SKILL.md

Overview
When to Use
Pattern 1: Basic Reflection
Pattern 2: Evaluator-Optimizer
Pattern 3: Code-Specific Reflection
Evaluation Strategies
Outcome-Based
LLM-as-Judge
Rubric-Based
Best Practices
Quick Start Checklist
Evaluation Implementation Checklist
Setup
Implementation
Safety

Key facts

Author: GitHub (awesome-copilot community collection)
License: MIT (repo license from GitHub; SKILL.md states none)
Installs on skills.sh: 10,231 (Nov 2, 2026)
GitHub stars: 40,666 for the whole github/awesome-copilot repo (Nov 2, 2026)
Last repo commit: Oct 30, 2026
What it includes: 1 file: SKILL.md (no scripts or references)
Needs: No tools or keys are named. The Python examples call an llm() helper you supply

What it includes

Self-critique with PASS/FAIL JSON, up to 3 rounds
Claude drafts, rates its own output against criteria as JSON, and rewrites until all pass or the round limit is hit.
Try: “Add a reflection loop to my report generator that checks against these four criteria.”
Read · Pattern 1: Basic Reflection
Separate generate, evaluate and optimize steps with a score threshold
Splits the roles into three functions and stops when the overall score reaches the threshold (0.8 in the example).
Read · Pattern 2: Evaluator-Optimizer
Generate code and pytest tests, fix until tests pass
Shows a loop that writes code and tests, runs the tests, and feeds the error back for a fix.
Read · Pattern 3: Code-Specific Reflection
Outcome-based, LLM-as-judge, rubric-based
Three ways to score output: compare to an expected result, ask a model to rank two outputs, or score weighted dimensions.
Read · Evaluation strategies
Clear criteria, iteration limits, convergence, logs, JSON
A table of 5 practices with a reason for each.
Read · Best practices
Setup, implementation and safety steps
A checklist covering criteria, thresholds, max iterations, the three functions, convergence detection and logging.
Read · Quick start checklist

From the official docs

Questions

What does the agentic-eval skill do?

It gives Claude patterns for evaluating and improving agent output. Claude uses them when you build self-critique loops, evaluator-optimizer pipelines or rubric-based scoring.

How do I install agentic-eval?

Use the install box at the top of this page. The skill is one SKILL.md file with no scripts.

Does it run an evaluation for me?

No. It is a guide with code sketches that call an llm() helper. You or Claude write the real implementation in your project.

When does SKILL.md say to use it?

For quality-critical generation, tasks with clear evaluation criteria, and content that must meet a style, compliance or format standard.

How many improvement rounds does it suggest?

The examples default to 3 iterations, and the best-practices table recommends a limit of 3 to 5 to prevent infinite loops.

Is it safe?

It bundles no scripts. If you ask Claude to build one of these loops, the code it writes will call a model on every round, so check the cost of repeated rounds. Pattern 3 also runs model-written code through run_tests, so run that loop in a sandbox (our advice).

Information

updated Nov 2, 2026

Repository: github/awesome-copilot
Folder: skills/agentic-eval

github/awesome-copilot skills.sh

Sources

Docs, issue trackers and discussions this page is built from.

Show all 2

agentic-eval SKILL.md · github.com
github/awesome-copilot README · github.com