Skip to main content
This page is the shortest onboarding path for evaluating a Skill: prepare the environment first, then run a full evaluation in two commands. For the complete mechanism, failure attribution, and report interpretation, see the advanced content.

​
What Problem Evaluation Solves

Before publishing a Skill, you need to answer one question: can this Skill, as a product capability, pass a real task evaluation? comet eval is the tool that answers this. It wraps the underlying details (pytest, Docker, task registration, profiles, reports) so you can run evaluations from the project root directory. The evaluation result is mandatory evidence for /comet-any before publishing — a Skill that hasn’t passed evaluation cannot be published.

​
What You Need

  • A Skill already generated via /comet-any (the artifact contains comet/eval.yaml), or
  • A local Skill directory (for early smoke testing).
If you don’t have a Skill yet, first read Composing Any Skill Quickstart.

​
What You Need to Prepare Before Running

The core Comet runtime for comet eval does not need these, but because eval runs real model tasks, it depends on the following environment. If not prepared, eval will silently skip instead of erroring, so be sure to confirm.

​
Required

DependencyDescriptionHow to check
uvPython package manager; comet eval uses it internally to drive pytestuv --version
Python 3.11+Runtime for the eval harnesspython --version
DockerUsed to build task images, isolate the running Claude and validatorsdocker info
claude CLIOn PATH; the harness invokes it inside Dockerclaude --version
ANTHROPIC_API_KEY or ANTHROPIC_AUTH_TOKENModel call credentials. When neither is present the entire suite is skippedecho $ANTHROPIC_API_KEY
When neither API key is present, the harness won’t fail — it will skip all cases. If you see an evaluation “pass instantly” with no real run, it’s most likely the environment isn’t ready. Confirm all four items above are in place first.

​
Install uv and Python

If you don’t have uv yet: 
# macOS / Linux
curl -LsSf https://astral.sh/uv/install.sh | sh

# Windows (PowerShell)
powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"
uv will automatically manage Python versions. On first run it creates a .venv under eval/ and installs dependencies (pytest, pyyaml, pydantic, etc.).

​
Configure the API key

# Recommended: set directly in the environment
export ANTHROPIC_API_KEY=sk-ant-...

# Or use proxy-type credentials (BigModel / OpenRouter, etc.)
export ANTHROPIC_AUTH_TOKEN=...
You can also configure it in eval/.env (the harness will load it): 
# eval/.env
ANTHROPIC_API_KEY=sk-ant-...
If you provide the key via a system environment variable, don’t copy the empty assignment like ANTHROPIC_API_KEY= from .env.example verbatim into .env. Empty assignments can override an existing system environment variable in some loading paths, causing the key to “disappear”. For recordings or demos, you can skip creating .env (or create a completely empty file) and leave the key in the system environment variable.

​
Optional

Environment variablePurpose
BENCH_CC_MODELOverride the Claude model
BENCH_CC_VERSIONClaude Code version in the Docker image (default latest)
BENCH_LLM_JUDGE=1Enable the optional LLM-as-judge override scoring
BENCH_SIMULATOR_PROMPT_FILECustom user simulator prompt file (default eval/simulator-instruction.md), overrides the user behavior profile simulated in auto_user evaluation
COMET_EVAL_REPORT_CONFIGReport output config path (replaces --report-config)
Using an Anthropic-compatible proxy? When ANTHROPIC_API_KEY is not set, write ANTHROPIC_AUTH_TOKEN + ANTHROPIC_BASE_URL (BigModel / mimo / OpenRouter, etc.) into eval/.env, and the harness will have the claude inside Docker authenticate with them. For the complete variable table, see Eval harness · Environment Variable Reference.

​
Shortest Path: Two-Step Evaluation

Once the environment is ready, evaluation takes just two steps.

​
Step 1: Discovery pre-check (collect)

First confirm that eval can discover your Skill and tasks. This step consumes no model calls and has the lowest cost: 
comet eval ./generated-skill/comet/eval.yaml --collect
If the output has no errors, the manifest, paths, and tasks can all be correctly discovered.
Only have a local Skill directory, no comet/eval.yaml yet? Use the early smoke path:
comet eval ./my-skill --skill-name my-skill --quick
This uses the generic-skill-smoke task for a smoke test, which is not equivalent to pre-release evidence.

​
Step 2: Execute the real evaluation (run)

After confirming discovery is fine, run the real evaluation and generate a browsable HTML report: 
comet eval ./generated-skill/comet/eval.yaml --html
After running, the CLI prints Report path pointing to the generated report. Open it to see whether the result passed.
Without —collect, the model is actually invoked, consuming tokens and time. The duration of one evaluation depends on the number of tasks and how many outer round-trips of “subject Agent reaches a decision point, user simulator replies, subject Agent continues” are allowed under auto_user mode. maxTurns limits this round-trip ceiling, not the Agent’s internal tool call count. Using —collect to troubleshoot first avoids waste.

Little fish first placing the Skill card into the --collect low-cost pre-check tray, then moving it to the --html real evaluation report tray

First use --collect to catch path, manifest, and task discovery issues, then use --html to execute the real evaluation and generate a browsable report

​
Only Three Things to Focus On in the Report

  1. Whether the evaluation passed — the report header or CLI output will tell you clearly.
  2. Failure attribution — if it didn’t pass, see whether the failure is attributed to harness, workflow, task, or model. This determines what you should fix.
  3. Report location — usually eval/local/logs/experiments/<experiment-id>/summary.html.
For the complete report structure, see Reading Evaluation Reports.

​
What to Do After the Evaluation Passes

The passing evaluation result becomes /comet-any’s pre-release evidence. Hand the result back to /comet-any to continue, or use the creator command to check readiness: 
comet creator status my-skill --project . --json
comet creator next my-skill --project . --json
See Publishing and Distributing a Skill.

​
What Happens Inside a Single Evaluation

Understanding this section helps you judge whether an evaluation result is trustworthy. comet eval ... --html does the following internally: Key points:
  • The model runs inside a Docker container, isolated from your working directory to avoid contamination.
  • Expected artifacts are a hard check — the files the task requires must exist.
  • rubric scoring is informational (the [RUBRIC] lines) and does not directly determine pass/fail; the real pass/fail is determined by the validators and whether the “required skill” was invoked.

​
FAQ

Check whether the path is correct and confirm the comet/eval.yaml file exists. If you’re not in the Comet repo root directory, add --project <dir> to point to the correct root.
Look at the Experiment and Report path in the CLI output. If the path contains <experiment-id>, look it up in the eval/local/logs/experiments/ directory using the actual experiment id.
Almost certainly the environment is not ready: Docker isn’t running, no ANTHROPIC_API_KEY, or the claude CLI isn’t on PATH. In these cases the harness will skip rather than fail. First confirm What you need to prepare before running.
To evaluate whether a Skill’s product capability can pass, use comet eval. To check whether a specific Skill run is missing artifacts or state, use comet skill check. Pre-release evidence only needs comet eval. See Runtime check.
For isolation. Each task has its own Dockerfile and environment; Claude produces files inside the container and the validator also checks inside the container, so it won’t contaminate your working directory. The image is built only once per session.

​
Advanced Content

The basic path ends here. If you need a deeper understanding of the evaluation system, read these pages:
Advanced topicSuitable scenario
Evaluation System OverviewUnderstand where eval fits in Skill creation, validation, and publish readiness
Scoring Metrics and Dual-Agent Evaluationrubric dimension details, pass@k/pass^k, dual-agent automatic interaction
Eval harnessUnderstand the collect/run internal mechanism, profiles, task selection, environment variables
Reading Evaluation ReportsComplete failure attribution, report signals, experiment id, and next-step judgment
Runtime checkDistinguish comet eval and comet skill check
comet eval commandComplete options, arguments, and troubleshooting reference
Last modified on July 6, 2026