comet eval

comet eval is Comet’s standalone Skill-evaluation entry. You can evaluate an existing local Skill directly, without running /comet-any first or preparing comet/eval.yaml ahead of time.

Shortest path

# Check the Skill, manifest, and task discovery only
comet eval ./my-skill --collect

# Run a low-cost smoke evaluation
comet eval ./my-skill --quick --html

# Run the full local evaluation
comet eval ./my-skill --html

The target can be a Skill directory, a direct SKILL.md, or comet/eval.yaml. When you pass a directory, Comet discovers its manifest automatically. Without a manifest, a normal run generates 2–4 bounded tasks from a Skill snapshot without rewriting the Skill source. Three ways to point at a target cover different situations:

Way	When to use	Example
Positional `target`	Evaluating an existing local Skill directory or manifest (most common)	`comet eval ./my-skill --html`
`--manifest <path>`	Running one specific manifest when the Skill lives elsewhere	`comet eval --manifest ./evals/smoke.yaml`
`--skill-path` + `--skill-name`	Fixed inputs in scripts or CI, independent of relative paths	`comet eval --skill-path ./skills --skill-name my-skill`

--collect performs static discovery and configuration checks without starting an Agent, Docker, plugins, credentials, or network requests.
--quick uses the fixed generic-skill-smoke task for early smoke testing.
--html produces both Markdown and HTML reports.
Reports and run state are written under .comet/eval/runs/ in the Skill or the project passed with --project.

If the Skill is outside the repository that contains the eval harness, specify that project explicitly:

comet eval ./my-skill --project ./my-project --html

User-level `.env` configuration

Published Comet CLI users do not need to pull the source repository. Place the configuration file beside the existing user-level Eval adapter directory:

Unix:    ~/.comet/eval/.env
Windows: %USERPROFILE%\.comet\eval\.env

Keep the subject (Bench) and LLM-as-judge model, endpoint, and credentials independent:

BENCH_API_KEY=subject-key
BENCH_BASE_URL=https://subject.example/v1
BENCH_MODEL=subject-model

BENCH_JUDGE_API_KEY=judge-key
BENCH_JUDGE_BASE_URL=https://judge.example/v1
BENCH_JUDGE_MODEL=judge-model

The current process environment takes precedence over .env; CLI options and the manifest take precedence over environment defaults. Claude Code, Codex, and CodeBuddy map the common values to their native variables; Codex uses its isolated runtime config.toml, and Qoder uses only its officially supported authentication and service configuration. Explicit Agent-native variables override the common fallbacks. Security boundary: Eval never writes API keys into the published package, manifest, report, or Skill workspace. During Docker Agent runs, Codex, Qoder, and CodeBuddy use isolated container-local temporary config roots. Codex’s config.toml contains only an environment-variable reference, CodeBuddy’s settings.json uses only apiKeyHelper, and the actual key exists only in the current container process and is destroyed with the temporary config root after the run.

Task selection

Tasks are selected in this order: explicit --task, --quick, evaluation.tasks in the manifest, recommendedTasks, and finally generated tasks. Generated tasks are cached by the Skill snapshot, Agent, and evaluation configuration so later runs can reuse them. For stable acceptance conditions, define an inline task in comet/eval.yaml:

evaluation:
  tasks:
    - name: writes-summary
      prompt: Create summary.md.
      expect:
        files:
          - summary.md
        contains:
          summary.md:
            - '# Summary'

You can also use source to reference a task package containing task.toml and instruction.md inside the Skill package. Inline tasks support file, text, JSON, and command checks. The task workspace and expected artifacts must remain inside the allowed Skill package or evaluation workspace.

Choose an evaluation Agent

The default Agent is claude-code. You can also select codex, qoder, or codebuddy:

comet eval ./my-skill --agent codex --model subject-model

Configure the subject Agent and the LLM-as-Judge independently:

execution:
  agent: codex
  model: subject-model
  baseUrl: https://subject.example/v1

judge:
  agent: claude-code
  model: judge-model
  baseUrl: https://judge.example/v1

CLI options take precedence over the manifest. When a Judge is enabled, provide a separate Judge model and credentials; it does not inherit the subject Agent’s credentials. Custom Agents must be registered explicitly. Comet does not discover an adapter merely because its executable is on PATH. Place the adapter at:

~/.comet/eval/adapters/<agent-id>/adapter.yaml

Then select it with the same option:

comet eval ./my-skill --agent my-agent

Choose an evaluation suite

local is the default and is suitable for local development and HTML reports. For team tracing, use the existing LangSmith suite:

comet eval ./my-skill --suite langsmith --html

To send task traces, scores, and experiment summaries to Langfuse, use the Langfuse suite added in beta18:

LANGFUSE_PUBLIC_KEY=pk-lf-... LANGFUSE_SECRET_KEY=sk-lf-... \
  comet eval ./my-skill --suite langfuse --html

The Langfuse suite still produces the local report. --collect --suite langfuse does not initialize the SDK, make network requests, or download plugins.

Common options

comet eval [target] [options]

Option	Purpose
`--project <dir>`	Select the project root containing `eval/`
`--manifest <path>`	Pass `comet/eval.yaml` directly
`--skill-path <path>`	Pass a local Skill directory or `SKILL.md`
`--skill-name <name>`	Set the Skill name
`--agent <agent>`	Select the subject evaluation Agent
`--model <model>`	Override the subject model
`--base-url <url>`	Override the subject API base URL
`--judge-agent <agent>`	Select the independent Judge Agent
`--judge-model <model>`	Select the independent Judge model
`--judge-base-url <url>`	Override the Judge API base URL
`--profile <name>`	Select an evaluation profile
`--report-config <path>`	Pass a report configuration file
`--suite <suite>`	Select `local`, `langsmith`, or `langfuse`
`--task <task>`	Run only the selected task
`--collect`	Perform static discovery and pre-checks only
`--quick`	Use the fixed smoke task
`--html`	Generate an HTML report

Reading the result

Before running, the CLI prints the target, suite, task, experiment, and report path. The report attributes failures to harness, workflow, task, or model:

harness: dependency, Docker, path, or environment problem;
workflow: the Skill flow did not meet its intended behavior;
task: the task definition or acceptance condition is incorrect;
model: model behavior or invocation instability.

comet eval evaluates a Skill as a product capability. comet skill check checks whether one Skill Run satisfies its runtime checks; they serve different purposes.

Next steps

Eval Agent startup configuration — prepare the CLI, authentication, and model protocol for Claude Code, Codex, or Qoder
Evaluation system overview — evaluation configuration and scoring
Reading evaluation reports — pass rates and failure attribution
comet publish — use evaluation evidence for publishing decisions

Install and health checks

Skills, evals, and publishing

Core scripts

Shortest path

User-level `.env` configuration

Task selection

Choose an evaluation Agent

Choose an evaluation suite

Common options

Reading the result

Next steps

​Shortest path

​User-level .env configuration

​Task selection

​Choose an evaluation Agent

​Choose an evaluation suite

​Common options

​Reading the result

​Next steps

Shortest path

User-level `.env` configuration

Task selection

Choose an evaluation Agent

Choose an evaluation suite

Common options

Reading the result

Next steps