/
├── summary.md # Main report (always generated)
├── summary.html # HTML version (with --html)
├── metadata.json # Experiment metadata
├── events/ # stream-json events per run
├── raw/ # Raw stdout/stderr
├── reports/ # report.json per run
└── artifacts/ # Files produced by Claude
```
### summary.md contains
1. Header: Experiment ID, start/completion time.
2. **Results table**: one row per treatment, with columns including Checks, Turns, Duration, Tools, Tokens, Cost, RubricAvg.
3. **Summary**: total runs, checks passed X/Y (percentage).
4. **Treatment Details**: detailed metrics for each run of each treatment, skills invoked, scripts used, passed and failed check lists.
### report.json per run
Fields include: `passed`, `checks_passed[]`, `checks_failed[]`, `events_summary` (duration, turns, tool\_calls, tokens, cost, files\_created, skills\_invoked, failure\_attribution).
## Inside a Single Evaluation: Docker Isolation + Dual Agent + Rubric
Understanding this section helps you judge whether an evaluation result is trustworthy. `comet eval ... --html` runs internally along `treatment × task × reps`, and each run does:
```mermaid theme={null}
flowchart TD
A["Inject treatment
(Skills + CLAUDE.md)"] --> B["Build isolated Docker environment
(per-task Dockerfile)"]
B --> Interact{"interaction.mode?"}
Interact -->|none / generic| C1["Single-turn: claude -p runs to completion"]
Interact -->|auto_user / comet, authoring| C2["Multi-turn dual-agent loop
subject Agent + user simulator Agent"]
C1 --> D["Parse stream-json events"]
C2 --> D
D --> E["Run task validators
(target_artifacts + test_scripts)"]
E --> F["Run profile rubric
output [RUBRIC] dimension scores"]
F --> G["Compute failure attribution
harness/workflow/task/model"]
G --> H["Write report.json + summary"]
```
A real evaluation runs Agent interactions in an isolated environment, then records evidence via validators and rubric
Key points:
* The model runs inside a **Docker container**, isolated from your working directory.
* **Dual-agent loop** (`auto_user` mode): the subject Agent runs the Skill under test, and at each decision point a **user simulator Agent** replies (approves reasonable proposals, picks defaults, drives progress — never refuses, never writes code). The subject Agent continues the same session with `--resume`, up to `max_turns` outer round-trips (comet-workflow typically 12, authoring-skill typically 8), ending early when a "complete" signal is hit. The `max_turns` here is not the number of internal messages or tool calls of the Agent under test. This lets multi-stage workflows **automatically run the entire chain**. For the complete loop and decision-point detection, see [Scoring Metrics and Dual-Agent Evaluation](/en/eval/scoring).
* **Rubric scoring** runs after the validators and appends results as `[RUBRIC]` informational checks (comet-workflow rubric never produces a hard failure; generic/authoring produce hard failures for specific missing items).
* The real **pass/fail** is determined by the task validators (expected artifacts exist + test\_scripts pass); rubric scores and pass\@k are diagnostic information.
## Report Output Configuration
Report output is controlled by `ReportOutputConfig`, with priority:
1. `--report-config ` (JSON or YAML)
2. `COMET_EVAL_REPORT_CONFIG` environment variable
3. Default (markdown only)
Config format (top-level or nested both accepted):
```json theme={null}
{"report_outputs": {"markdown": true, "html": false}}
```
`--html` is equivalent to `{"markdown": true, "html": true}` and writes a temporary file passed to pytest.
## Failure Attribution
The report helps distinguish sources of failure. The attribution logic (`attribution.py`) judges each failed check in this order:
| Attribution | Judgment condition | Next step |
| ----------- | -------------------------------------------------------------------------------------------------------- | --------------------------------------------------------- |
| `harness` | Required skill not invoked at all; or generic profile with no skill run | Check dependencies, Docker, network, or local environment |
| `task` | Check involves artifact path / validator / task directory | Check eval task definition and fixtures |
| `workflow` | Check involves `.comet.yaml` / guard / state / transition / archive; or skill invocation contract failed | Go back to `/comet-any` to improve the Skill |
| `model` | Other default fallback (task failed after an observable workflow execution) | Rerun or reduce reliance on non-deterministic behavior |
This attribution is used to decide whether to fix the Skill, fix the eval config, or rerun the environment. See [Reading Evaluation Reports](/en/eval/reports).
## Environment Variable Reference
| Variable | Purpose |
| ------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `ANTHROPIC_API_KEY` / `ANTHROPIC_AUTH_TOKEN` | **Required**; when neither is present the suite is skipped |
| `BENCH_CC_MODEL` | Override the Claude model |
| `BENCH_CC_VERSION` | Claude Code version in the Docker image (default `latest`) |
| `BENCH_SIMULATOR_PROMPT_FILE` | Custom user simulator prompt file (default `eval/simulator-instruction.md`; relative paths resolved from `eval/`). The `--simulator-prompt` command-line argument takes higher priority |
| `BENCH_LLM_JUDGE=1` | Enable the optional LLM-as-judge override scoring |
| `COMET_EVAL_REPORT_CONFIG` | Report output config path |
| `BENCH_SUITE_ROOT` / `BENCH_TASKS_DIR` / `BENCH_TREATMENTS_DIR` / `BENCH_SKILLS_DIR` / `BENCH_LOGS_DIR` | Path overrides |
| `BENCH_TEST_CONTEXT` / `BENCH_TEST_RESULTS` | Override host↔Docker transfer file names |
The LangSmith suite additionally requires `LANGSMITH_API_KEY`, `LANGSMITH_TRACING=true`, and `TRACE_TO_LANGSMITH=true`.
### Authenticating with an Anthropic-compatible proxy
When `ANTHROPIC_API_KEY` is not set, the `claude` inside Docker switches to authenticating via an **Anthropic-compatible proxy** (BigModel / mimo / OpenRouter, etc.). Required variables:
| Variable | Purpose |
| --------------------------------------------------------------------------------------------------- | ----------------------------------------- |
| `ANTHROPIC_AUTH_TOKEN` | Proxy-type auth token |
| `ANTHROPIC_BASE_URL` | The proxy's Anthropic-compatible endpoint |
| `ANTHROPIC_MODEL` | Default model |
| `ANTHROPIC_DEFAULT_HAIKU_MODEL` / `ANTHROPIC_DEFAULT_SONNET_MODEL` / `ANTHROPIC_DEFAULT_OPUS_MODEL` | Per-tier model mappings |
| `ANTHROPIC_DEFAULT_SONNET_MODEL_NAME` / `ANTHROPIC_DEFAULT_OPUS_MODEL_NAME` | Per-tier model display names |
| `CLAUDE_CODE_SUBAGENT_MODEL` | Subagent model |
All of these variables have placeholder entries in `eval/.env.example`. Write them into `eval/.env` and the harness will inject them when starting Docker.
### Custom User Simulator Prompt
In `auto_user` mode evaluation, the user simulator Agent's instructions are driven by a prompt file. It reads `eval/simulator-instruction.md` by default:
```text theme={null}
You are simulating a developer user in an automated eval. ...
- Approves the proposed approach / name / plan when asked to confirm
- Picks the most reasonable default option when asked to choose
- Asks for clarification only if the question is truly ambiguous ...
Never refuse; always let the workflow move forward. Do not write code or files.
```
If you want to swap in a different user behavior (e.g. more demanding, asks for more clarification), write your version to a file and point `BENCH_SIMULATOR_PROMPT_FILE` to it:
```bash theme={null}
# eval/.env
BENCH_SIMULATOR_PROMPT_FILE=my-simulator-prompt.md
```
Relative paths are resolved from `eval/`; the file is read only if it exists. The command-line `--simulator-prompt "..."` has the highest priority and overrides file contents. See [Scoring Metrics and Dual-Agent Evaluation · User Simulator Agent Instructions](/en/eval/scoring#user-simulator-agent-instructions).
## Don't Confuse the Harness with Runtime Check
`comet eval` and `comet skill check` have similar names but different purposes:
* `comet eval`: evaluates a Skill bundle or `comet/eval.yaml`, answering "can this Skill, as a product capability, pass the evaluation".
* `comet skill check`: checks whether a specific Skill run is missing artifacts or state, answering "is this run complete".
See [Runtime check](/en/eval/runtime).
## Next steps
* [Scoring Metrics and Dual-Agent Evaluation](/en/eval/scoring) — rubric dimension details, pass\@k/pass^k, dual-agent interaction loop
* [Reading Evaluation Reports](/en/eval/reports) — learn to read report signals and failure attribution
* [comet eval command](/en/cli/eval) — complete options and subcommand reference
* [Evaluation System Overview](/en/eval/overview) — where eval fits in the workflow and the eval.yaml format