comet eval evaluation system, environment preparation, report interpretation, and publish evidence.
Basic Concepts
What exactly does comet eval evaluate
What exactly does comet eval evaluate
comet eval executes real model tasks through a shared eval harness to verify whether a Skill, as a product capability, can pass evaluation, producing a reviewable report. It discovers authored tasks from comet/eval.yaml, or automatically generates and caches 2–4 bounded cases from the Skill content when no tasks are declared. See Evaluation System Overview.What is the difference between comet eval and comet skill check
What is the difference between comet eval and comet skill check
comet eval evaluates a Skill’s product capability and produces a reviewable
report. It can read comet/eval.yaml, or take any local Skill directory
directly. A normal directory run uses declared tasks when available and
otherwise generates 2–4 cases from the Skill content; --quick explicitly
selects the generic-skill-smoke smoke. comet skill check checks whether a
specific Skill run is missing artifacts or state, does not execute model
tasks, and does not produce an evaluation report (reads comet/checks.yaml).
See Runtime check.Do I need to know pytest or Docker
Do I need to know pytest or Docker
No.
comet eval wraps the underlying details — you only need to know whether
to use --manifest or --skill-path. The underlying details are handled by
the harness. See Eval harness.What do you mean by two evaluation systems
What do you mean by two evaluation systems
Authoring-time evaluation (
comet eval, reads a local Skill directory or comet/eval.yaml) and runtime checks (comet skill check, reads comet/checks.yaml). The former is publish evidence; the latter is run-completeness checking. See Evaluation System Overview · Two Evaluation Systems.Environment Preparation
What environment do I need to run eval
What environment do I need to run eval
You need
uv, Python 3.11+, Docker, the selected Agent CLI, and that Agent’s credentials. The core Comet runtime does not need these — only a real comet eval run does. For the supported Agents and credentials, see Evaluation Quickstart · Supported Evaluation Agents.The eval "passes" instantly but it doesn't feel like it really ran
The eval "passes" instantly but it doesn't feel like it really ran
Almost certainly the environment is not ready: Docker isn’t running, the
selected Agent CLI isn’t on PATH, or its credentials are missing. In these
cases the harness will skip rather than fail. Confirm the environment is
ready first.
How do I install uv
How do I install uv
macOS/Linux:
curl -LsSf https://astral.sh/uv/install.sh | sh. Windows
PowerShell: powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex". uv will automatically manage
Python versions and eval/.venv.What is the difference between a missing Eval harness and missing uv
What is the difference between a missing Eval harness and missing uv
Eval harness is missing at ... means the bundled eval/ is incomplete or
--project points to the wrong repository; reinstall @rpamis/comet or
correct the path. uv is not installed or not in PATH means the harness was
found and only uv needs installation. npm users normally do not need a
separate Comet clone.How do I configure the API key
How do I configure the API key
export ANTHROPIC_API_KEY=sk-ant-..., or configure it in the user-level %USERPROFILE%\\.comet\\eval\\.env / ~/.comet/eval/.env (automatically loaded by comet eval). Use ANTHROPIC_AUTH_TOKEN for proxy-type credentials (BigModel / OpenRouter).Two Entry Points
Should I use --manifest or --skill-path
Should I use --manifest or --skill-path
To evaluate any local Skill directory, just pass the directory. Comet discovers
comet/eval.yaml under it when present; without declared tasks, a normal run generates and caches 2–4 cases from the Skill content. Use --quick when you explicitly want the fixed generic-skill-smoke task. To evaluate a Skill bundle through its comet/eval.yaml, pass the manifest directly. The two entry points are mutually exclusive when selected explicitly.Do I replace current-bundle-hash in a generated manifest
Do I replace current-bundle-hash in a generated manifest
No. Before collect or run,
comet eval resolves <current-bundle-hash> to the current Bundle draft and writes a temporary manifest. It does not modify the source Bundle or comet/eval.yaml. Resolution fails only when the manifest has left its Bundle, bundle.yaml is missing, or the draft cannot be loaded.What is --quick and what is the default task
What is --quick and what is the default task
--quick is an explicit low-cost smoke mode for a Skill path; it selects the
fixed generic-skill-smoke task. A normal directory run follows the
manifest’s tasks or automatically generates 2–4 cases from the Skill content.
Use --quick for a fast injection/invocation check and a normal run for
broader Skill-specific coverage.Why collect first, then run
Why collect first, then run
collect only does discovery pre-checks (validates manifest, tasks, paths),
consumes no model calls, has the lowest cost, and is suitable for
troubleshooting right after generating a Skill. run executes the real
evaluation. Collecting first can quickly surface configuration problems and
avoid wasting model calls. See Quickstart.Reports and Failures
Where to find the report
Where to find the report
The CLI output prints
Report path, usually .comet/eval/runs/<experiment-id>/summary.html. The experiment id format is <task>_<YYYYMMDD_HHMMSS>. See Reading Evaluation Reports.The evaluation failed, how do I tell where the problem is
The evaluation failed, how do I tell where the problem is
Look at the failure attribution in the report.
harness indicates an
environment/dependency/path problem, workflow indicates the Skill flow
didn’t meet expectations, task indicates a task definition/fixture problem,
and model indicates unstable model behavior. The attribution determines what
you should fix. See Reading Evaluation Reports.collect errors saying it can't find the target
collect errors saying it can't find the target
Check whether the path is correct: a local Skill directory must contain
SKILL.md, and comet/eval.yaml must point to a file that actually exists. If you’re not in the Comet repo root directory, add --project <dir> to point to the correct root.Should I rerun after a model attribution failure
Should I rerun after a model attribution failure
model attribution means the model behavior or tool usage was unstable —
rerunning usually helps. If it fails repeatedly, consider reducing the Skill’s
reliance on non-deterministic behavior.The Rubric score is very low but all checks passed, does it count as a pass
The Rubric score is very low but all checks passed, does it count as a pass
It counts as a pass. The rubric is an informational score (
[RUBRIC] lines and RubricAvg) and does not directly determine pass or fail. The real pass/fail is determined by the validators and required Skill invocations. A low rubric score is a diagnostic signal you can use to improve the Skill, but it doesn’t affect the publish gate.Publish Evidence
Does passing eval mean I can publish
Does passing eval mean I can publish
Passing eval is only one of the conditions.
/comet-any or the backend will incorporate eval evidence into readiness: no evidence, failures, or matching an old hash all prevent publishing. Only passing with a matching hash lets you proceed to review/publish. See Evaluation System Overview.What does it mean that eval evidence matches an old hash
What does it mean that eval evidence matches an old hash
It means the Skill was modified after the last evaluation (the draft hash
changed), so the old evaluation result is no longer valid. You need to rerun
comet eval ... --html to generate evidence bound to the current hash.Can I manually write the report path into the publish status
Can I manually write the report path into the publish status
No.
/comet-any records structured evidence through the Bundle backend.
Manually editing Bundle state or internal JSON will break the hash binding and
readiness validation.Can quick smoke be used as publish evidence
Can quick smoke be used as publish evidence
No.
--skill-path --quick is only early smoke testing with limited coverage. Before publishing you must generate comet/eval.yaml via /comet-any, then run the full evaluation with --manifest.