> ## Documentation Index
> Fetch the complete documentation index at: https://docs.comet.rpamis.com/llms.txt
> Use this file to discover all available pages before exploring further.

# comet eval

> Evaluate a local Skill or comet/eval.yaml through the shared eval harness and generate a report.

`comet eval` is the general-purpose Skill evaluation entry point. It wraps the eval harness so you don't have to manually `cd` into `eval/` or assemble pytest arguments.

It provides pre-release evidence for Skills: it can evaluate artifacts produced by `/comet-any`, and it can also do early smoke testing on a local Skill directory.

## Command shape

`comet eval` is a single-entry command. You pass a Skill directory, `SKILL.md`, or `comet/eval.yaml` as the target, then choose with options whether to collect or run the real evaluation.

| Command                                                  | Purpose                                                 | Runs model tasks |
| -------------------------------------------------------- | ------------------------------------------------------- | ---------------- |
| `comet eval ./generated-skill/comet/eval.yaml --collect` | Only discovery and pre-checks, no model or Docker tasks | No               |
| `comet eval ./generated-skill/comet/eval.yaml --html`    | Runs the local eval and can produce an HTML report      | Yes              |

<Note>
  There is also a similarly named <code>comet skill check</code>, which is a <strong>completely different command</strong> — it inspects the runtime checks of a single Skill run, not a general Skill evaluation. See <a href="/en/eval/runtime">Runtime check</a>.
</Note>

## Prerequisites

Because `comet eval` runs real model tasks, it depends on the following environment (the core Comet runtime does not need these):

* **uv** (`comet eval` uses it internally to drive pytest)
* **Python 3.11+**
* **Docker** (builds task images, runs in isolation)
* **`claude` CLI** on PATH
* **`ANTHROPIC_API_KEY`** or **`ANTHROPIC_AUTH_TOKEN`**

When neither API key is present, the harness **silently skips** instead of erroring. For the full setup steps, see [Evaluation quickstart · What you need before running](/en/eval/quickstart#what-you-need-before-running).

<Note>
  <strong>Using proxy-style credentials</strong> (BigModel / OpenRouter, etc.): when <code>ANTHROPIC\_API\_KEY</code> is not set, configure <code>ANTHROPIC\_AUTH\_TOKEN</code> + <code>ANTHROPIC\_BASE\_URL</code> and related variables in <code>eval/.env</code>. The harness will make <code>claude</code> inside Docker use proxy authentication. For the full variable list, see <a href="/en/eval/harness#environment-variable-reference">Eval harness · Environment variable reference</a>.
</Note>

## Recommended path

When you have a `comet/eval.yaml` (full pre-release evaluation):

```bash theme={null}
comet eval ./generated-skill/comet/eval.yaml --collect
comet eval ./generated-skill/comet/eval.yaml --html
```

When you only have a local Skill directory (early smoke test):

```bash theme={null}
comet eval ./my-skill --skill-name my-skill --quick
```

<Tip>
  Run <code>--collect</code> first for a low-cost discovery pre-check, then use <code>--html</code> to run the real evaluation. <code>--collect</code> does not consume any model calls.
</Tip>

## Command options

```bash theme={null}
comet eval [target] [options]
```

| Option                   | Description                                             | Default                                        |
| ------------------------ | ------------------------------------------------------- | ---------------------------------------------- |
| `[target]`               | Local Skill directory, `SKILL.md`, or `comet/eval.yaml` | None                                           |
| `--project <dir>`        | Repository root containing `eval/`                      | `.` (current directory)                        |
| `--manifest <path>`      | Path to `comet/eval.yaml`                               | None                                           |
| `--skill-path <path>`    | Local Skill directory or `SKILL.md`                     | None                                           |
| `--skill-name <name>`    | Skill name for `--skill-path`                           | None                                           |
| `--profile <name>`       | Override the eval profile                               | `generic`                                      |
| `--task <task>`          | Specify the eval task                                   | See table below                                |
| `--report-config <path>` | Custom report config path                               | Auto-generated temp file when `--html` is used |
| `--html`                 | Generate an HTML report (also produces markdown)        | Off                                            |
| `--quick`                | Use the default quick smoke task                        | Off                                            |
| `--collect`              | Only discovery and pre-checks, no model or Docker tasks | Off                                            |

### `--manifest` and `--skill-path` are mutually exclusive

A single evaluation can only choose one entry point:

* Only `--manifest`: evaluates `comet/eval.yaml`.
* Only `--skill-path`: evaluates a local Skill directory.
* A `[target]`: the CLI decides based on the path whether it is a manifest or a local Skill.
* Both or neither passed: error.

### Default task and profile

| Mode                          | Default profile                               | Default task                                            |
| ----------------------------- | --------------------------------------------- | ------------------------------------------------------- |
| `--manifest`                  | From the manifest (usually `authoring-skill`) | `recommended` (reads the manifest's `recommendedTasks`) |
| `--skill-path --quick`        | `generic`                                     | `generic-skill-smoke`                                   |
| `--skill-path` (no `--quick`) | `generic`                                     | `recommended`                                           |

You can override the defaults with `--task` and `--profile`.

## Execution info

Before running, `comet eval` prints a set of execution info to help you locate the report:

* `Eval root`: the actual `eval/` root it launched from
* `Mode`: `collect` or `run`
* `Target`: whether the current evaluation targets a manifest or a local Skill directory
* `Experiment`: the experiment id for this run
* `Profile`: the profile used in this evaluation
* `Task`: the evaluation task for this run
* `Report path`: report location
* `Report config`: the temporary report config used when `--html` is enabled

In `run` mode it also notes that failure attribution will be recorded into the generated eval summary.

## Report location

`--html` requires the report to produce both markdown and HTML. Reports are typically located at:

```text theme={null}
eval/local/logs/experiments/<experiment-id>/summary.html
```

The experiment id format is `<task>_<YYYYMMDD_HHMMSS>`. See [Reading evaluation reports](/en/eval/reports).

## Choosing between manifest path and skill-path

| Scenario                                         | Recommended entry      |
| ------------------------------------------------ | ---------------------- |
| You have a `comet/eval.yaml`                     | `--manifest`           |
| Only a local directory, still in early debugging | `--skill-path --quick` |
| It's a `/comet-any` artifact                     | `--manifest`           |
| You want to reach release readiness              | `--manifest`           |

<Warning>
  For release readiness, prefer the manifest path. <code>--skill-path --quick</code> is only suitable for early smoke testing — do not treat it as the final release evaluation.
</Warning>

## How to decide the next step on failure

### collect failure

Check first:

* Whether the manifest path is correct
* Whether `comet/eval.yaml` exists
* Whether the recommended task in the manifest exists
* Whether you are in the Comet repository root or passed the correct `--project`

### run failure

Look at the failure attribution in the report first:

| Attribution | Meaning                                                         | Next step                                                 |
| ----------- | --------------------------------------------------------------- | --------------------------------------------------------- |
| `harness`   | eval harness, dependencies, Docker, path, or environment issues | Check dependencies, Docker, network, or local environment |
| `workflow`  | The Skill execution flow did not meet expectations              | Go back to `/comet-any` to refine the Skill               |
| `task`      | Task definition, validation conditions, or fixture issues       | Check the eval task definition and fixtures               |
| `model`     | Model behavior, tool usage, or unstable output                  | Re-run or reduce reliance on non-deterministic behavior   |

### HTML report not found

Check the `Experiment` and `Report path` in the CLI output first. If the path contains placeholders, look in the `eval/local/logs/experiments/` directory using the actual experiment id.

## comet eval vs comet skill check

These two commands have similar names but different purposes:

* `comet eval`: the user entry point to the shared eval harness, evaluating a Skill package or `comet/eval.yaml`.
* `comet skill check`: a completeness check for a single Skill run, determining whether a run / change satisfies the runtime checks in `comet/checks.yaml`.

If your question is "can this Skill pass evaluation as a product capability", use `comet eval`. If your question is "does this Skill run have missing artifacts or state", use `comet skill check`. See [Runtime check](/en/eval/runtime).

## What users need to remember at minimum

1. `/comet-any` artifacts should use `comet/eval.yaml` first
2. Run `--collect` first, then `--html`
3. The eval result is evidence for `/comet-any` release readiness, not the release action itself

Recommended commands:

```bash theme={null}
comet eval ./generated-skill/comet/eval.yaml --collect
comet eval ./generated-skill/comet/eval.yaml --html
comet creator next <name> --json
```

## Next steps

* [Evaluation system overview](/en/eval/overview) — where eval fits in the workflow and the eval.yaml format
* [Reading evaluation reports](/en/eval/reports) — understanding report signals
* [comet publish](/en/cli/publish) — how eval evidence drives publishing
* [comet skill](/en/cli/skill) — `comet skill check` (runtime check) reference