Quickstart: Evaluating a Skill

All you need is a local directory containing SKILL.md to evaluate a Skill with comet eval. Even if you have not written evaluation cases in advance, Eval can automatically generate 2–4 evaluation cases from the Skill content and run them. This page follows the practical order: get one run working, choose an Agent and evaluation backend, configure tasks, generate reports, and diagnose failures.

What You Need

Required

@rpamis/comet installed. The npm package includes the eval harness; you do not need to clone the Comet repository.
A local Skill directory containing SKILL.md.
For a real evaluation run: uv, Python 3.11+, Docker, the selected Agent CLI, and its model credentials.

--collect only performs static discovery and configuration checks. It does not start the Agent, Docker, plugins, credentials, or network requests, so you can check a Skill before preparing the full model environment. Before choosing an Agent other than Claude Code, read Eval Agent startup configuration to confirm its CLI, authentication method, and model protocol.

User-level `.env` configuration

Ordinary users do not need to clone the Comet repository or enter its eval/ directory. On the first comet eval command, the CLI automatically creates the complete user-level configuration file:

Windows: %USERPROFILE%\\.comet\\eval\\.env
macOS/Linux: ~/.comet/eval/.env

The CLI creates the file only when it is missing and never overwrites an existing file. The first run prints the actual path; edit that file as needed and run the command again. Environment variables already present in the current shell take precedence over .env. The generated template contains all user-configurable parameters, grouped as follows:

Parameter group	Main variables
Subject (Bench)	`BENCH_EVAL_AGENT`, `BENCH_API_KEY`, `BENCH_BASE_URL`, `BENCH_MODEL`
Independent Judge	`BENCH_LLM_JUDGE`, `BENCH_JUDGE_AGENT`, `BENCH_JUDGE_API_KEY`, `BENCH_JUDGE_AUTH_TOKEN`, `BENCH_JUDGE_BASE_URL`, `BENCH_JUDGE_MODEL`
Claude Code	`ANTHROPIC_API_KEY`, `ANTHROPIC_AUTH_TOKEN`, `ANTHROPIC_BASE_URL`, `ANTHROPIC_MODEL`, `ANTHROPIC_DEFAULT_*`, `CLAUDE_CODE_SUBAGENT_MODEL`, `BENCH_CC_MODEL`, `BENCH_CC_VERSION`
Codex	`OPENAI_API_KEY`, `OPENAI_BASE_URL`, `OPENAI_MODEL`, `CODEX_API_KEY`, `CODEX_BASE_URL`, `CODEX_MODEL`, `BENCH_CODEX_*`
Qoder	`QODER_PERSONAL_ACCESS_TOKEN`, `QODER_BASE_URL`, `QODER_MODEL`, `BENCH_QODER_*`
CodeBuddy	`CODEBUDDY_API_KEY`, `CODEBUDDY_AUTH_TOKEN`, `CODEBUDDY_BASE_URL`, `CODEBUDDY_MODEL`, `CODEBUDDY__MODEL`, `CODEBUDDY_CUSTOM_HEADERS`, `CODEBUDDY_INTERNET_ENVIRONMENT`, `BENCH_CODEBUDDY_`
Custom Agent	`COMET_EVAL_ADAPTERS_DIR`, plus the credential, model, and endpoint variables declared by `adapter.yaml`
Other evaluation and tracing	`BENCH_SIMULATOR_PROMPT_FILE`, `LANGSMITH_`, `TRACE_TO_LANGSMITH`, `LANGFUSE_`

All parameters in the template are commented out by default, so unused entries do not change the default behavior. Keep real API keys only in the user-level .env or current shell; never put them in a Skill, manifest, report, or public repository.

Supported Evaluation Agents

comet eval uses claude-code by default. The subject Agent, user simulator, and optional Judge can use the following Agents:

Agent	Suitable for	Common credentials
`claude-code`	Default choice; runs the Skill with Claude Code	`ANTHROPIC_API_KEY` or `ANTHROPIC_AUTH_TOKEN`
`codex`	Codex CLI or OpenAI-compatible models	`OPENAI_API_KEY` or `CODEX_API_KEY`
`qoder`	Qoder CLI	`QODER_PERSONAL_ACCESS_TOKEN`
`codebuddy`	CodeBuddy CLI or API	`CODEBUDDY_API_KEY` or `CODEBUDDY_AUTH_TOKEN`

Custom Agent adapters are a user-level advanced capability and do not require a Comet source checkout. Clone the source only when you need to modify the built-in Eval harness, tasks, or Docker environment; see advanced configuration.

For the registration path, credentials, and CLI contract for a non-built-in Agent, see Extend Eval with a custom Agent. Putting an executable on PATH is not enough to enable it. If an evaluation finishes instantly without a real run, Docker, the Agent CLI, or model credentials are usually not ready. In these cases the harness normally skips instead of fabricating a successful run.

First Run

Start with a model-free pre-check, then run a low-cost smoke test, and finally run the configured or generated task set:

# 1. Check Skill, manifest, and task discovery
comet eval ./my-skill --collect

# 2. Run the fixed low-cost smoke task
comet eval ./my-skill --quick --html

# 3. Run the configured or automatically generated tasks
comet eval ./my-skill --html

When there is no comet/eval.yaml, or the manifest has neither evaluation.tasks nor recommendedTasks, a normal run automatically generates 2–4 evaluation cases from the Skill content, freezes and caches them, and then executes them. You do not need to write tasks before starting an evaluation. --quick does not use these generated cases; it always runs generic-skill-smoke, which checks that the Skill can be injected, invoked, and produce result.md.

Passing the Target

You can pass a directory, a direct SKILL.md, or a manifest as the target:

# Skill directory: automatically discovers comet/eval.yaml if present
comet eval ./my-skill --collect

# Direct SKILL.md
comet eval ./my-skill/SKILL.md --html

# Direct manifest
comet eval ./my-skill/comet/eval.yaml --html

Without a manifest, Comet creates the basic configuration in memory and does not rewrite the Skill source. If the Skill is outside the current repository, use --project to specify the project directory where runtime state and reports are stored:

comet eval ./my-skill --project ./my-project --html

Configure the Evaluation Agent and Judge

The simplest way to choose the subject Agent is on the command line:

comet eval ./my-skill --agent codex --model subject-model --html

You can also create comet/eval.yaml under the root of the Skill being evaluated to configure the default Agent. The layout is:

my-skill/
├── SKILL.md
└── comet/
    └── eval.yaml

When you run comet eval ./my-skill, Comet automatically discovers ./my-skill/comet/eval.yaml. You can also pass the file directly as the target:

comet eval ./my-skill/comet/eval.yaml --collect

Configure the default Agent in eval.yaml. CLI options take precedence over the manifest:

execution:
  agent: codex
  model: subject-model
  baseUrl: https://subject.example/v1

judge:
  agent: claude-code
  model: judge-model
  baseUrl: https://judge.example/v1

The subject Agent and the LLM-as-Judge can use different Agents, models, API endpoints, and credentials. The Judge does not silently inherit the subject Agent’s model or credentials; when you enable it, provide a separate Judge configuration.

Choose an Evaluation Backend

Use --suite to select the evaluation backend. All three backends use the same target, task, and Agent configuration; they differ in whether results are synchronized to an external evaluation service:

Backend	Command	Result location and purpose
`local`	`comet eval ./my-skill --html`	Default backend; creates local reports for everyday development
`langsmith`	`comet eval ./my-skill --suite langsmith --html`	Creates a local report and synchronizes traces, scores, and experiment data to LangSmith
`langfuse`	`comet eval ./my-skill --suite langfuse --html`	Creates a local report and synchronizes task traces, rubric scores, pass metrics, and experiment summaries to Langfuse

Langfuse example:

LANGFUSE_PUBLIC_KEY=pk-lf-... LANGFUSE_SECRET_KEY=sk-lf-... \
  comet eval ./my-skill --suite langfuse --html

If you only want to check task and configuration discovery, any backend can be used with --collect:

comet eval ./my-skill --suite langfuse --collect

--collect --suite langfuse does not initialize an SDK, connect to the network, or download plugins.

Report Types

Evaluation state and reports are written to the following directory under the target project by default:

.comet/eval/runs/<experiment-id>/
├── summary.md       # Markdown summary; always generated
├── summary.html     # HTML summary; generated with --html
├── metadata.json    # Metadata for this experiment
├── events/          # Agent runtime events
├── raw/             # Raw output
├── reports/         # report.json for each run
└── artifacts/       # Task artifacts and validation evidence

Without --html, the evaluation still generates summary.md. To generate HTML, run:

comet eval ./my-skill --html

You can also customize report outputs with --report-config <path> or COMET_EVAL_REPORT_CONFIG. The CLI prints Experiment and Report path; use those values when locating a specific report. When reading a report, start with three questions:

Did the evaluation pass?
Is the failure attributed to harness, workflow, task, or model?
Are expected artifacts missing, or are token, cost, or duration values unusual?

In general, harness means an environment or dependency problem, workflow means the Skill flow did not meet expectations, task means the task definition or validation condition needs attention, and model means model behavior or invocation was unstable.

Customize Tasks

Tasks are selected in this order:

The task specified with --task.
generic-skill-smoke selected by --quick.
evaluation.tasks in the manifest.
recommendedTasks or a task package referenced by source.
2–4 evaluation cases automatically generated and cached from the Skill content when no usable task is available.

If the automatically generated tasks do not fit your Skill well, create or edit comet/eval.yaml under the root of the Skill being evaluated, and declare an inline task under evaluation.tasks. For example:

evaluation:
  tasks:
    - name: writes-summary
      prompt: Create summary.md with the requested result.
      expect:
        files:
          - summary.md
        contains:
          summary.md:
            - '# Summary'

Reference a Task Package Inside the Skill

If a task needs its own task.toml, instruction.md, Docker environment, or validation script, keep the task package inside the Skill directory and reference it with source. A recommended layout is:

my-skill/
├── SKILL.md
├── comet/
│   └── eval.yaml
└── eval-tasks/
    └── writes-summary/
        ├── task.toml
        ├── instruction.md
        ├── environment/
        │   └── Dockerfile
        └── validation/
            └── test_summary.py

In comet/eval.yaml at the Skill root, use evaluation.tasks[].source to point to the task package. The source path is relative to the Skill package root (the directory containing SKILL.md), not to the comet/ directory:

# my-skill/comet/eval.yaml
skill:
  name: my-skill
  source: ..

evaluation:
  tasks:
    - name: writes-summary
      source: eval-tasks/writes-summary

Here is minimal example content for the files in this task package. For the complete task.toml fields, validation scripts, profiles, and Docker configuration, see Configuring Evaluations and Custom Tasks. task.toml declares the task metadata, Docker environment, expected artifact, and validation script:

[metadata]
name = "writes-summary"
description = "Create summary.md"

[environment]
dockerfile = "environment/Dockerfile"

[validation]
test_scripts = ["test_summary.py"]
target_artifacts = ["summary.md"]

instruction.md contains the task instruction sent to the Agent:

Create `summary.md` in the current workspace.

Requirements:

- Include the heading `# Summary`
- Summarize the result in at least three bullet points

environment/Dockerfile provides the base environment for the task and its validation:

FROM python:3.11-slim

WORKDIR /workspace

validation/test_summary.py runs inside the task container and checks whether the Agent produced the expected file:

from pathlib import Path

from scaffold.python.validation.core import write_test_results


passed = []
failed = []
summary = Path("summary.md")

if not summary.exists():
    failed.append("summary.md is missing")
else:
    text = summary.read_text(encoding="utf-8")
    if "# Summary" in text:
        passed.append("summary heading is present")
    else:
        failed.append("summary heading is missing")

write_test_results({"passed": passed, "failed": failed})

eval-tasks/writes-summary/task.toml describes the task environment and validation, while instruction.md contains the task instruction sent to the Agent. Put any Dockerfile or deterministic validation script in the task package’s environment/ and validation/ directories. A source task cannot also define the inline task fields prompt or expect, and Comet checks that the package contains both task.toml and instruction.md. You can pass the Skill directory and let Comet discover comet/eval.yaml, or pass the manifest directly:

# Automatically discovers my-skill/comet/eval.yaml
comet eval ./my-skill --collect
comet eval ./my-skill --task writes-summary --html

# Pass the manifest directly
comet eval ./my-skill/comet/eval.yaml --collect
comet eval ./my-skill/comet/eval.yaml --task writes-summary --html

Tasks can check files, text, JSON, or command results; the task workspace and expected artifacts must remain inside the allowed Skill package or evaluation workspace. Only continue to Configuring Evaluations and Custom Tasks when you need a custom Docker environment, validation scripts, a profile, or a treatment.

Recommended Evaluation Flow

Prepare a directory containing SKILL.md
comet eval ./my-skill --collect
Fix path, manifest, or task discovery issues
comet eval ./my-skill --quick --html
Configure an Agent, Judge, suite, or custom task as needed
comet eval ./my-skill --html
Read .comet/eval/runs/<experiment-id>/summary.md or summary.html

Topic	Suitable scenario
Evaluation System Overview	Understand tasks, profiles, scoring, and the execution model
Scoring Metrics and Dual-Agent Evaluation	Understand rubric, pass@k/pass^k, and multi-turn Agent interaction
Eval harness	Troubleshoot the harness, environment variables, and suite internals
Reading Evaluation Reports	Analyze reports and failure attribution in depth
comet eval command	See the complete CLI options and troubleshooting reference

Quickstart

Methodology: Why evaluation drives Skill evolution

Real evaluation experiments

Advanced

Related commands

Quickstart: Evaluating a Skill

What You Need

Required

User-level `.env` configuration

Supported Evaluation Agents

First Run

Passing the Target

Configure the Evaluation Agent and Judge

Choose an Evaluation Backend

Report Types

Customize Tasks

Reference a Task Package Inside the Skill

Recommended Evaluation Flow

Further Reading

​What You Need

​Required

​User-level .env configuration

​Supported Evaluation Agents

​First Run

​Passing the Target

​Configure the Evaluation Agent and Judge

​Choose an Evaluation Backend

​Report Types

​Customize Tasks

​Reference a Task Package Inside the Skill

​Recommended Evaluation Flow

​Further Reading

What You Need

Required

User-level `.env` configuration

Supported Evaluation Agents

First Run

Passing the Target

Configure the Evaluation Agent and Judge

Choose an Evaluation Backend

Report Types

Customize Tasks

Reference a Task Package Inside the Skill

Recommended Evaluation Flow

Further Reading