PlugWiki

google-agents-cli-eval Skill for Claude: Evaluate Your Agent With agents-cli

Teaches Claude to test an agent's behavior with agents-cli eval: datasets, metrics, grading, failure analysis and fix loops.

Installs
532.2K
on skills.sh
Stars
6.3K
on GitHub
Files
7
besides SKILL.md
Updated
Oct 2026
last repo commit

google-agents-cli-eval covers the evaluation step of the agents-cli suite. It runs Claude through a four-stage loop: prepare data, run the eval, analyze failures, then fix and re-run. The main command is agents-cli eval run, which chains eval generate and eval grade.

It says to start with one or two cases and to expect many iterations. It warns against lowering the bar to pass or skipping flaky cases, and it says unit tests should not assert on LLM output. Opt-in commands such as eval analyze and eval optimize hand work to Google's Eval Service.

Good for

  • Developers who want proof their agent behaves well
  • Teams debugging agent failures from eval results
  • Anyone comparing agent versions across eval runs

Install google-agents-cli-eval

Claude Code

  1. 1

    Copy this prompt.

    Install the google-agents-cli-eval skill in Claude Code. Run this command:

uvx google-agents-cli setup

Then check that it loaded and tell me how to use it.
  2. 2

    Paste it into Claude Code. It shows the change and asks before running it.

Requires Python 3.11+, uv, Node.js.

Pick one route. setup installs the agents-cli program and the skills into your coding agent. The README says that with the skills-only route your coding agent handles the rest. The seven skills in the suite point to each other, so take the whole set.

From the project README. Source.

What's in the skill

When it loads

This skill should be used when the user wants to "run an evaluation", "evaluate my agent", "evaluate my ADK agent", "write an eval dataset", "analyze eval failures", "compare eval results", "optimize agent", or needs guidance on the Gemini platform eval methodology and the Quality Flywheel. Covers eval metrics, dataset schema, LLM-as-judge scoring, and common failure causes. Applies to any agents-cli project, whatever framework the agent is written in. Do NOT use for agent API code patterns (ADK: use google-agents-cli-adk-code), deployment (use google-agents-cli-deploy), or project scaffolding (use google-agents-cli-scaffold).

Sections in SKILL.md

  1. Reference Files
  2. The Quality Flywheel
  3. 1. Prepare Data
  4. 2. Run the Eval (always run)
  5. 3. Analyze Failures
  6. 4. Optimize & Code Fix
  7. Running the loop
  8. Shortcuts That Waste Time
  9. Choosing the Right Metrics
  10. What to fix when scores fail
  11. Eval Commands
  12. eval run (default)
  13. eval generate
  14. eval grade
  15. eval compare
  16. eval dataset synthesize

Key facts

Author
Google
License
Apache-2.0 (stated in SKILL.md)
GitHub stars
6,276 for the whole google/agents-cli repo (Nov 2, 2026)
Last commit
Oct 27, 2026
What installs
SKILL.md plus 7 bundled files (guides and examples); no scripts in the file list
Needs
agents-cli (uv tool install google-agents-cli); a scaffolded project; the other skills in the suite
Part of
The google-agents-cli plugin in google/agents-cli
skills.sh audits
Gen Agent Trust Hub pass, Socket pass, Snyk warn
Works in
Antigravity CLI, Claude Code, Codex and other coding agents (README)

What it includes

  • Runs the agent over a dataset and grades the traces

    Writes results_<timestamp>.json and .html to artifacts/grade_results/.

    Runs code · agents-cli eval run

  • The two halves of eval run, for custom traces

    Generate runs the agent; grade scores saved traces.

    Runs code · agents-cli eval generate and grade

  • Compares two result files

    Use it to confirm a fix did not hurt other metrics.

    Runs code · agents-cli eval compare

  • Creates multi-turn test scenarios with a simulated user

    Opt-in, for ADK projects.

    Runs code · agents-cli eval dataset synthesize

  • Clusters failures and tunes prompts against a metric

    Opt-in commands that use Google's Eval Service.

    Runs code · eval analyze and eval optimize

  • Built-in metrics such as final_response_quality, safety, hallucination

    agents-cli eval metric list shows all built-ins.

    Read · Metrics guide

  • Format for datasets and eval_config.yaml

    Read · Dataset and config schema

From the official docs

Permissions

Medium risk

Claude runs your agent and uses Google's Eval Service to grade it. The SKILL.md calls eval optimize long-running and expensive, so run it only when you ask for it.

What it can do

  • Read your data
  • Create and change things
  • Delete things
  • Read-only mode

Watch out for

  • Evals run your agent and use Google's Eval Service

    Each run executes your agent, which may call models and tools. Start with one or two cases, as the skill advises.

  • eval optimize is long-running and expensive

    The SKILL.md says it makes many LLM calls and says not to run it unless you explicitly ask for prompt optimization.

  • Keep sensitive prompts out of test data

    Eval datasets and traces are processed by the grading service, so use test inputs you are allowed to share.

  • skills.sh audit: Snyk warn

    Gen Agent Trust Hub and Socket pass; Snyk is listed as warn. Read the audit on skills.sh before installing.

Technical details
Scopes

  • Runs your agent, which may call models and tools

  • Grading and the opt-in eval commands use Google's Eval Service

  • Writes eval results into artifacts/

Questions

What does google-agents-cli-eval do?

It teaches Claude the agents-cli evaluation loop: prepare a dataset, run agents-cli eval run, analyze failures, fix the agent and compare results.

How do I install it?

Use the install box at the top of this page. The setup command installs the agents-cli program and all seven skills in the suite, which is what you want. The README says that with the skills-only route your coding agent handles the rest. You need Python 3.11+, uv and Node.js first.

Does it work on its own?

It needs a project from agents-cli. The seven skills in the suite cross-reference each other, so install the whole set.

Does it need an API key?

The SKILL.md names no API key. It needs the agents-cli program and a Google Cloud sign-in, which agents-cli login handles; the README says login also supports AI Studio.

Is it safe?

It runs your agent, which may call models and tools, and it uses Google's Eval Service for grading. The SKILL.md calls eval optimize long-running and expensive and says to run it only when you ask. skills.sh lists Gen Agent Trust Hub and Socket as pass and Snyk as warn.

Why not just write unit tests?

The SKILL.md says unit tests check code correctness, while agents-cli eval checks agent behavior. It warns against tests that assert on LLM output because output varies.

Does it work in the Claude app, Codex or Cursor?

The README names Antigravity CLI, Claude Code and Codex, and says any other coding agent works too. It does not name the Claude desktop or web app or Cursor.

Information

updated Nov 2, 2026

Repository
google/agents-cli
Folder
skills/google-agents-cli-eval

google/agents-cliskills.sh

Sources

Docs, issue trackers and discussions this page is built from.

Show all 3
  1. google/agents-cli README · github.com
  2. google-agents-cli-eval SKILL.md · github.com
  3. google-agents-cli-eval on skills.sh · skills.sh

Updated Nov 2, 2026.