agent-platform-eval-flywheel Skill (Google's own repo): Evaluate AI Agents and Models

• Builds a dataset from EvalCases, a DataFrame, ADK traces or synthetic scenarios.

  Cold start calls generate_conversation_scenarios on the server and needs your confirmation. The dataset, schema and ADK trace conversion are described in references/dataset_schema.md.

  Write · Stage 1: Prepare data

• Fills in responses or traces by calling an agent, a model ID or a managed agent.

  Skipped when traces are already complete. run_inference overwrites an existing response column, so the skill avoids it when you supply responses.

  Runs code · Stage 2: Run inference

• Runs client.evals.evaluate with metrics, then saves JSON and HTML results.

  Always run. Predefined metrics include multi_turn_task_success, general_quality, hallucination, grounding and safety. Custom LLM and code metrics are also supported.

  Runs code · Stage 3: Grade

• Reads scores, filters failures, and clusters 10+ failures on one metric into themes.

  The clustering calls generate_loss_clusters and covers multi_turn_task_success and multi_turn_tool_use_quality only.

  Network · Stage 4: Analyze failures

• Applies a fix, re-grades, and compares baseline with candidate for regressions.

  The SKILL.md expects 5 to 10 or more iterations per failing case.

  Runs code · Stage 5: Optimize and iterate

• Read-only scripts run freely; paid evaluation calls need your Yes after a dry-run preview.

  Every evaluation, re-evaluation, metric change or scenario generation needs its own preview. The agent ends its turn and waits for your reply.

  Confirmation tiers

• Evaluates a deployed endpoint or a Model-as-a-Service model; can deploy a model first.

  references/deployment.md uses gcloud ai model-garden models deploy and two scripts. It warns that this can take about 30 minutes per 50 samples.

  Runs code · Endpoint and MaaS evaluation

• Seven Python scripts: validate, parse, inspect, compare, render, endpoint, MaaS.

  validate_dataset, parse_adk_traces, inspect_results, compare_results, render_html_report, endpoint_evaluation and maas_evaluation.

  Runs code · Bundled scripts

Watch out for

• Evaluation runs spend Google Cloud compute

  run_inference, evaluate, generate_conversation_scenarios and generate_loss_clusters invoke LLMs or remote services that incur costs. The SKILL.md requires a preview and your explicit Yes before each one; do not approve blindly. github.com

• It can deploy a model to an endpoint

  references/deployment.md deploys with gcloud ai model-garden models deploy. It tells the assistant to stop and ask you for the machine type and accelerator, and to confirm the full command before deploying. github.com

• Scripts use your gcloud login

  endpoint_evaluation.py runs gcloud auth print-access-token and gsutil cat, then posts prompts to the endpoint. Use a project and account scoped to this work. github.com

• It installs Python packages if they are missing

  Setup probes the imports and runs pip install for google-cloud-aiplatform[evaluation] and google-genai. It says not to create a virtual environment. github.com