Quick Start: Evaluating a Skill

# 1. Check Skill, manifest and task discovery
comet eval ./my-skill --collect

# 2. Run fixed low-cost smoking tasks
comet eval ./my-skill --quick --html

# 3. Run evaluations using configured or automatically generated tasks
comet eval ./my-skill --html

# Skill directory: Automatically discover comet/eval.yaml within it (if it exists)
comet eval ./my-skill --collect

# Just pass SKILL.md directly
comet eval ./my-skill/SKILL.md --html

# Pass the manifest directly
comet eval ./my-skill/comet/eval.yaml --html

comet eval ./my-skill --project ./my-project --html

comet eval ./my-skill --agent codex --model subject-model --html

my-skill/
├── SKILL.md
└── comet/
    └── eval.yaml

comet eval ./my-skill/comet/eval.yaml --collect

execution:
  agent: codex
  model: subject-model
  baseUrl: https://subject.example/v1

judge:
  agent: claude-code
  model: judge-model
  baseUrl: https://judge.example/v1

LANGFUSE_PUBLIC_KEY=pk-lf-... LANGFUSE_SECRET_KEY=sk-lf-... \
  comet eval ./my-skill --suite langfuse --html

comet eval ./my-skill --suite langfuse --collect

.comet/eval/runs/<experiment-id>/
├─ Summary.md # Markdown summary reports are always generated
├── Summary.html # HTML summary report generated when using -- HTML
├── Metadata.json # The metadata of this experiment
├── events/ # Agent runs events
├── raw/ # Original output
├── reports/ # report.json for each run
└── artifacts/ # Task products and validation evidence

comet eval ./my-skill --html

evaluation:
  tasks:
    - name: writes-summary
      prompt: Create summary.md with the requested result.
      expect:
        files:
          - summary.md
        contains:
          summary.md:
            - '# Summary'

my-skill/
├── SKILL.md
├── comet/
│   └── eval.yaml
└── eval-tasks/
    └── writes-summary/
        ├── task.toml
        ├── instruction.md
        ├── environment/
        │   └── Dockerfile
        └── validation/
            └── test_summary.py

# my-skill/comet/eval.yaml
skill:
  name: my-skill
  source: ..

evaluation:
  tasks:
    - name: writes-summary
      source: eval-tasks/writes-summary

[metadata]
name = "writes-summary"
description = "Create summary.md"

[environment]
dockerfile = "environment/Dockerfile"

[validation]
test_scripts = ["test_summary.py"]
target_artifacts = ["summary.md"]

Create `summary.md` in the current workspace.

Requirements:

- Include the heading `# Summary`
- Summarize the result in at least three bullet points

FROM python:3.11-slim

WORKDIR /workspace

from pathlib import Path

from scaffold.python.validation.core import write_test_results


passed = []
failed = []
summary = Path("summary.md")

if not summary.exists():
    failed.append("summary.md is missing")
else:
    text = summary.read_text(encoding="utf-8")
    if "# Summary" in text:
        passed.append("summary heading is present")
    else:
        failed.append("summary heading is missing")

write_test_results({"passed": passed, "failed": failed})

# Automatically discover my-skill/comet/eval.yaml
comet eval ./my-skill --collect
comet eval ./my-skill --task writes-summary --html

# Pass the manifest directly
comet eval ./my-skill/comet/eval.yaml --collect
comet eval ./my-skill/comet/eval.yaml --task writes-summary --html

1. Prepare a directory containing SKILL.md
2. comet eval ./my-skill --collect
3. Fix issues with path, manifest or task discovery
4. comet eval ./my-skill --quick --html
5. Configure Agents, judges, Suites or custom tasks as needed
6. comet eval ./my-skill --html
7. Reading. Comet/eval/runs/<experiment-id>/summary. The md or summary. HTML