comet eval is the general-purpose Skill evaluation entry point. It wraps the eval harness so you don’t have to manually cd into eval/ or assemble pytest arguments.
It provides pre-release evidence for Skills: it can evaluate artifacts produced by /comet-any, and it can also do early smoke testing on a local Skill directory.
Command shape
comet eval is a single-entry command. You pass a Skill directory, SKILL.md, or comet/eval.yaml as the target, then choose with options whether to collect or run the real evaluation.
| Command | Purpose | Runs model tasks |
|---|
comet eval ./generated-skill/comet/eval.yaml --collect | Only discovery and pre-checks, no model or Docker tasks | No |
comet eval ./generated-skill/comet/eval.yaml --html | Runs the local eval and can produce an HTML report | Yes |
There is also a similarly named comet skill check, which is a completely different command — it inspects the runtime checks of a single Skill run, not a general Skill evaluation. See Runtime check. Prerequisites
Because comet eval runs real model tasks, it depends on the following environment (the core Comet runtime does not need these):
- uv (
comet eval uses it internally to drive pytest)
- Python 3.11+
- Docker (builds task images, runs in isolation)
claude CLI on PATHANTHROPIC_API_KEY or ANTHROPIC_AUTH_TOKEN
When neither API key is present, the harness silently skips instead of erroring. For the full setup steps, see Evaluation quickstart · What you need before running.
Using proxy-style credentials (BigModel / OpenRouter, etc.): when ANTHROPIC_API_KEY is not set, configure ANTHROPIC_AUTH_TOKEN + ANTHROPIC_BASE_URL and related variables in eval/.env. The harness will make claude inside Docker use proxy authentication. For the full variable list, see Eval harness · Environment variable reference. Recommended path
When you have a comet/eval.yaml (full pre-release evaluation):
comet eval ./generated-skill/comet/eval.yaml --collect
comet eval ./generated-skill/comet/eval.yaml --html
comet eval ./my-skill --skill-name my-skill --quick
Run —collect first for a low-cost discovery pre-check, then use —html to run the real evaluation. —collect does not consume any model calls.
Command options
comet eval [target] [options]
| Option | Description | Default |
|---|
[target] | Local Skill directory, SKILL.md, or comet/eval.yaml | None |
--project <dir> | Repository root containing eval/ | . (current directory) |
--manifest <path> | Path to comet/eval.yaml | None |
--skill-path <path> | Local Skill directory or SKILL.md | None |
--skill-name <name> | Skill name for --skill-path | None |
--profile <name> | Override the eval profile | generic |
--task <task> | Specify the eval task | See table below |
--report-config <path> | Custom report config path | Auto-generated temp file when --html is used |
--html | Generate an HTML report (also produces markdown) | Off |
--quick | Use the default quick smoke task | Off |
--collect | Only discovery and pre-checks, no model or Docker tasks | Off |
--manifest and --skill-path are mutually exclusive
A single evaluation can only choose one entry point:
- Only
--manifest: evaluates comet/eval.yaml.
- Only
--skill-path: evaluates a local Skill directory.
- A
[target]: the CLI decides based on the path whether it is a manifest or a local Skill.
- Both or neither passed: error.
Default task and profile
| Mode | Default profile | Default task |
|---|
--manifest | From the manifest (usually authoring-skill) | recommended (reads the manifest’s recommendedTasks) |
--skill-path --quick | generic | generic-skill-smoke |
--skill-path (no --quick) | generic | recommended |
--task and --profile.
Execution info
Before running, comet eval prints a set of execution info to help you locate the report:
Eval root: the actual eval/ root it launched fromMode: collect or runTarget: whether the current evaluation targets a manifest or a local Skill directoryExperiment: the experiment id for this runProfile: the profile used in this evaluationTask: the evaluation task for this runReport path: report locationReport config: the temporary report config used when --html is enabled
In run mode it also notes that failure attribution will be recorded into the generated eval summary.
Report location
--html requires the report to produce both markdown and HTML. Reports are typically located at:
eval/local/logs/experiments/<experiment-id>/summary.html
<task>_<YYYYMMDD_HHMMSS>. See Reading evaluation reports.
Choosing between manifest path and skill-path
| Scenario | Recommended entry |
|---|
You have a comet/eval.yaml | --manifest |
| Only a local directory, still in early debugging | --skill-path --quick |
It’s a /comet-any artifact | --manifest |
| You want to reach release readiness | --manifest |
For release readiness, prefer the manifest path. —skill-path —quick is only suitable for early smoke testing — do not treat it as the final release evaluation.
How to decide the next step on failure
collect failure
Check first:
- Whether the manifest path is correct
- Whether
comet/eval.yaml exists
- Whether the recommended task in the manifest exists
- Whether you are in the Comet repository root or passed the correct
--project
run failure
Look at the failure attribution in the report first:
| Attribution | Meaning | Next step |
|---|
harness | eval harness, dependencies, Docker, path, or environment issues | Check dependencies, Docker, network, or local environment |
workflow | The Skill execution flow did not meet expectations | Go back to /comet-any to refine the Skill |
task | Task definition, validation conditions, or fixture issues | Check the eval task definition and fixtures |
model | Model behavior, tool usage, or unstable output | Re-run or reduce reliance on non-deterministic behavior |
HTML report not found
Check the Experiment and Report path in the CLI output first. If the path contains placeholders, look in the eval/local/logs/experiments/ directory using the actual experiment id.
comet eval vs comet skill check
These two commands have similar names but different purposes:
comet eval: the user entry point to the shared eval harness, evaluating a Skill package or comet/eval.yaml.comet skill check: a completeness check for a single Skill run, determining whether a run / change satisfies the runtime checks in comet/checks.yaml.
If your question is “can this Skill pass evaluation as a product capability”, use comet eval. If your question is “does this Skill run have missing artifacts or state”, use comet skill check. See Runtime check.
What users need to remember at minimum
/comet-any artifacts should use comet/eval.yaml first- Run
--collect first, then --html
- The eval result is evidence for
/comet-any release readiness, not the release action itself
Recommended commands:
comet eval ./generated-skill/comet/eval.yaml --collect
comet eval ./generated-skill/comet/eval.yaml --html
comet creator next <name> --json
Next steps
