Skip to main content
This is advanced content. If you just want to quickly run an evaluation, start with Quickstart: Evaluating a Skill.
This page is for source maintainers who need to inspect Eval internals. Ordinary users do not need to clone Comet or enter its eval/ directory; use the installed comet eval and user-level .env. Clone the source only when modifying the harness, using --project, or reproducing an experiment.
comet eval is Comet’s general-purpose Skill evaluation entry point. It first answers one question: the Skill you have in hand — as a product capability — can it pass a real task evaluation? This overview is split into two parts:
  1. First, how your own Skill gets evaluated: entry points, tasks, scoring, reports, and failure attribution.
  2. Then, how /comet-any connects eval results into publish readiness.

​
What Problem It Solves

Ordinary users don’t need to understand pytest, the task registry, profiles, treatments, or Docker details. comet eval wraps the local eval harness’s launch path, task discovery, profile selection, and report generation, letting you run evaluations from the project root directory instead of manually switching to the eval/ directory to assemble command-line parameters.

​
Two Evaluation Systems, Don’t Confuse Them

Comet has two evaluation systems with similar names but completely different purposes: comet eval answers “can this Skill, as a product capability, pass the evaluation”, executing real model tasks through a shared eval harness. comet skill check answers “is this Skill run missing files or state”, checking only runtime check items without running the model. See Runtime check.

Little fish separating comet eval's publish evidence desk from comet skill check's run-completeness desk, holding up a sign reminding not to mix them

comet eval produces pre-release evidence; comet skill check only checks whether a specific Skill run is complete — do not mix them

​
Part One: How Your Own Skill Gets Evaluated

From the user’s perspective, comet eval does four things:
  1. Find your Skill.
  2. Find which evaluation tasks should run.
  3. Have the model execute tasks in an isolated environment, and check the results with validators.
  4. Generate a report that tells you whether it passed, why it failed, and what to do next.

​
How to Choose an Entry Point

comet eval [target] decides the entry point from the target: a directory or SKILL.md goes through skill-path, a comet/eval.yaml goes through manifest. You can also specify the entry point explicitly with --skill-path / --manifest; explicit entry points are mutually exclusive. When you pass a directory, Comet discovers comet/eval.yaml under it when present. Without a manifest task set, a normal run automatically generates and caches 2–4 bounded tasks from the Skill content; use --quick when you explicitly want the fixed generic-skill-smoke smoke. No Comet repo clone is needed — the npm package bundles the eval harness.
The fixed —quick smoke is early validation: it only verifies that “the Skill can be injected, invoked, and produce files.” A normal directory run uses declared or automatically generated Skill-specific tasks. Publish readiness, when applicable, still requires evaluating the full package generated by /comet-any (with comet/eval.yaml).

​
Why collect First

collect is the user’s cheapest troubleshooting entry point. It only does discovery and pre-checks, executing no model or Docker tasks, suitable for quickly surfacing path, manifest, and task registration problems.
It mainly answers:
  • Whether the comet/eval.yaml path is correct
  • Whether the eval harness can read this manifest
  • Whether the recommended tasks in the manifest can be discovered
  • Whether the current repo’s eval dependency paths are available
It should not run the full model evaluation first, nor consume long-running tasks. On failure, you usually fix manifest, path, or task discovery issues first.

​
comet/eval.yaml Manifest Format

comet/eval.yaml is the manifest for the full pre-release evaluation. It tells the eval harness: where the Skill is, which profile to use, which tasks to recommend running, and which evidence and artifacts are expected. Its format (parsed by the harness’s manifests.py):
apiVersion and kind are strictly validated: not equaling comet.eval/v1alpha1 / comet.eval/SkillEvalManifest will error directly. Ordinary users usually don’t need to hand-write this file — /comet-any generates it.
The eval.yaml generated by /comet-any defaults to the authoring-skill profile. An ordinary workflow-kernel will recommend generic-skill-smoke, authoring-skill-smoke, and workflow-route-conformance; a /comet-based overlay additionally recommends workflow-overlay-contract and classic Comet workflow tasks to check the Output Schema, expected evidence, and overlay routing.

​
Profile System

The eval harness has three built-in profiles, each determining rubric dimensions, default interaction mode, and scorer: Profile resolution priority: --profile override > manifest’s skill.profile > task’s evaluation.profile > generic.
maxTurns is not the number of internal Agent messages or tool calls. It only takes effect in auto_user mode, limiting how many outer round-trips of “subject Agent reaches a decision point -> user simulator replies -> subject Agent continues with —resume” may occur.
Tasks starting with comet-* or with metadata.category=comet are automatically inferred as the comet-workflow profile, and the interaction mode is automatically switched to auto_user ( two Agents interacting automatically: one runs the Skill under test, the other simulates the user replying at decision points).

​
Scoring Metrics: rubric + pass@k/pass^k

eval is a metric-driven evaluation that doesn’t just give pass/fail:
  • rubric multi-dimension scoring: breaks Skill quality down into multiple dimensions (e.g. the five-phase main_flow/gate_guard, generic Skill’s safety_boundary), each dimension 0.0–1.0, weighted and aggregated into weighted_score. Informational, for diagnosis.
  • pass@k / pass^k: distinguishes the capability ceiling (at least one success out of k runs) from the reliability floor (all k runs succeed). Computed from multiple repeated runs (--count N). Informational.
  • Task validator pass/fail: whether this implementation is actually correct (target_artifacts + test_scripts). This is the hard pass/fail.
For the complete dimension details, weights, formulas, and dual-agent interaction loop, see Scoring Metrics and Dual-Agent Evaluation.

​
Task System

The eval harness has a built-in set of tasks, each task being a directory (containing instruction.md, task.toml, environment/, validation/). Common tasks: recommended is not a task name but the CLI’s default resolution path: when using --manifest it reads the manifest’s recommendedTasks; without a manifest it runs each task’s default_treatments.

​
What the Default skill-path Entry Runs

When you pass a local Skill directory, comet eval uses the manifest’s tasks when available. If no usable task is declared, a normal run generates and caches 2–4 bounded tasks from the Skill content:
The normal run verifies:
  • Whether the Skill directory is readable
  • Whether the eval harness can inject it as a dynamic Skill
  • Whether the generated or declared tasks can run and validate their expected results
Use comet eval ./my-skill --quick --html when you specifically want the fixed generic-skill-smoke smoke. The normal directory entry point is a lightweight but real Skill evaluation; publish readiness, when applicable, requires evaluating the full package generated by /comet-any (with comet/eval.yaml).

​
Part Two: How /comet-any Connects to eval

/comet-any is responsible for creating or improving a Skill, and comet eval is responsible for verifying that the Skill can be discovered, run, and produce a report by the eval harness. The connection point between the two is the comet/eval.yaml in the artifact and the Eval evidence after evaluation. Complete chain:
comet eval is not responsible for publishing. Publishing is still handled by the creator / publish commands: creation and state recovery are exposed to ordinary users as comet creator, and publishing and distribution as comet publish. eval’s responsibility is to provide pre-release evidence. After /comet-any generates a Skill, first look for this file:
Then run in two steps:
The first step, collect, only confirms “can the tasks be discovered”, suitable as a low-cost pre-check right after generating a Skill. The second step, run --html, executes the real evaluation and generates a browsable report.

​
How Eval Results Enter Publish Readiness

After recording the Eval result, /comet-any or the creator / publish backend incorporates it into publish readiness. Users only need to know two things:
  1. The results produced by comet eval become the evidence source for Publish readiness:.
  2. When the current hash is missing Eval evidence, User next steps: must first point to completing the evaluation, not continuing to publish.
The usual sequence is:
comet creator next only outputs the single recommended next user command; comet publish review shows Publish readiness:, User next steps:, Readiness:, Blockers:, Warnings:, and Evidence: to the user.

​
How /comet-any Uses eval Results

From the user’s perspective, after eval finishes, hand the results back to /comet-any to continue. /comet-any will incorporate the eval evidence into readiness: Users don’t need to manually edit internal state, nor should they manually write report paths into JSON. /comet-any records structured evidence through the backend.

​
The Minimum You Need to Remember

  1. The core question of comet eval is: can this Skill, as a product capability, pass a real task evaluation.
  2. Any local Skill directory can run a Docker evaluation via comet eval ./your-skill; when you want to publish, evaluate the full package generated by /comet-any (with comet/eval.yaml).
  3. First collect, then run --html.
  4. /comet-any artifacts connect eval results into publish readiness, but eval itself is not a publish action.
  5. comet eval (authoring-time) and comet skill check (runtime) are two different systems — don’t mix them.

​
Next steps

Last modified on August 13, 2026