` format, with each reason ≤25 words and citing specific content.
## pass\@k and pass^k
These two metrics measure **capability vs reliability** and are key to evaluating whether a Skill can run stably and repeatedly. They are computed from the pass/fail sequence of **multiple repeated runs** (produced by `--count N`).
### Definitions
| Metric | Formula | Meaning |
| -------------------------------- | ----------------------------------------- | ----------------------------------------------------------------------------------------------- |
| **pass\@k** (capability ceiling) | `1 - C(n-c, k) / C(n, k)` | Probability of **at least one** success in k independent attempts (HumanEval unbiased estimate) |
| **pass^k** (reliability floor) | `1.0` if and only if `c == n`, else `0.0` | Probability that **all** k attempts succeed |
Where n = total runs, c = successful runs. "Success" = that run's task validator had zero failures.
### Why Both Are Needed
```mermaid theme={null}
flowchart LR
subgraph Cap["Capability ceiling pass@k"]
A1["Attempt 1 failed"] --- A2["Attempt 2 succeeded"]
end
subgraph Rel["Reliability floor pass^k"]
B1["Attempt 1 succeeded"] --- B2["Attempt 2 succeeded"]
end
Cap -->|"High pass@k but low pass^k"| Gap["gap = unstable
can do it but result varies each time"]
Rel -->|"High pass^k"| Trust["Trustworthy: gets it right every time"]
```
pass\@k looks at the capability ceiling, pass^k looks at the reliability floor — the larger the gap, the more unstable
* **High pass\@k, low pass^k**: capability is sufficient, but **unstable** — "can do it, but can't guarantee doing it right every time". For a Skill that users run repeatedly, this is a danger signal.
* **Both high**: can do it and **gets it right every time** — trustworthy.
**The gap (pass\@k − pass^k) quantifies instability.** The report explicitly highlights this gap.
### How to Get Multiple Runs
`--count N` (a pytest option) repeats each `(task, treatment)` combination N times, producing N independent pass/fail results. The comparison report first filters to the analysis set: clear environment or runner noise is excluded, while flagged runs still participate in the main statistics and are called out. Those N booleans in the analysis set are the input to pass\@k/pass^k.
```bash theme={null}
# Repeat 5 times to make pass@k/pass^k meaningful
# (actually driven indirectly via comet eval; equivalent to --count 5 under the hood)
```
The report warns when there aren't enough runs: "Only N run per treatment — pass\@k/pass^k for k>1 need ≥2 runs to be meaningful. Use --count 5." A single run can only compute pass\@1/pass^1; k>1 metrics require multiple repeats.
### How to Choose k
The report takes k values from `{1, 2, 5}` that **don't exceed the actual number of runs n** (k is clamped to n), falling back to `[1]` when insufficient. Report columns: `pass@1 [pass@2 pass@5]` and `pass^1 [pass^2 pass^5]`.
### Where They Appear
pass\@k/pass^k are not in `summary.md` but in the **comparison report** (produced by `compare_baselines.py` as `comparison_report.md`) in the `## pass@k / pass^k — capability vs reliability` section. They are **informational**, not gates.
## Dual-Agent Automatic Interaction Evaluation
For multi-stage workflow-type Skills (`comet-workflow` and `authoring-skill` profiles), the evaluation is completed by **two Agents interacting automatically**, with no human involvement needed.
### Two Roles
| Role | What it does | Session |
| ------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------- |
| **Subject Agent (subject)** | Runs the Skill under test (e.g. the `/comet` five-phase) | A single continuous session, resumed with `--resume` each turn |
| **User simulator Agent (simulator)** | Simulates user replies at decision points (approves reasonable proposals, picks reasonable defaults, clarifies only when truly ambiguous) | A one-shot `claude -p` call per decision point, stateless |
### Interaction Loop
```mermaid theme={null}
flowchart TD
Start["Turn 1: subject Agent receives task prompt
runs the Skill in a new session"] --> Parse["Parse output, extract session_id and result text"]
Parse --> Decision{"Is it a decision point?
(? / confirm / choose /
approve / which ... )"}
Decision -->|Yes| Sim["User simulator Agent generates reply
approve proposal / pick default / move forward"]
Decision -->|No| Complete{"Workflow complete?
(archive complete /
all 5 phases / done)"}
Sim --> Resume["Next turn: subject Agent --resume same session
feed in the simulated user reply"]
Resume --> Parse
Complete -->|Yes| End["End loop"]
Complete -->|No| Nudge["Send continue prompt to push forward"]
Nudge --> Resume
```
The loop runs up to `max_turns` outer round-trips (comet-workflow typically 12, authoring-skill typically 8), ending early when "complete" is hit (`archive complete` / `workflow complete` / `all 5 phases`, etc.).
Every subject Agent turn also runs with `--output-format stream-json --verbose`. The loop driver only concatenates the subject Agent's stream-json stdout for the harness to parse; the user simulator Agent's one-shot replies do not enter the main event stream. As a result, event statistics reflect the subject Agent's observable behavior, not the simulator's behavior.
max\_turns is not the number of real internal working turns of the subject Agent, nor the number of tool calls. One outer round-trip means: the subject Agent runs to a decision point, the user simulator Agent replies, then the subject Agent continues with --resume. Within the same round-trip there may still be multiple assistant messages, tool calls, and file operations.
### Decision Point Detection
The subject Agent's output text is judged as a decision point when it matches any of these signals (case-insensitive):
```text theme={null}
? | confirm | choose | proceed | continue | approve | select an option |
which (option|approach|name) | enter the | provide | would you | shall we |
do you want | preferred
```
Custom `--decision-pattern` is also supported.
### User Simulator Agent Instructions
The simulator Agent receives the simulator prompt + the subject Agent's last message, and is asked to:
* When asked to confirm, **approve** the proposed plan/name/approach
* When asked to choose, **pick the most reasonable default**
* Ask for clarification only when the question is genuinely ambiguous about "what to do"
* **Never refuse**; always let the workflow move forward
* **Don't write code or files**
comet-workflow uses `COMET_SIMULATOR_PROMPT`; generic/authoring uses `GENERIC_SIMULATOR_PROMPT` (slightly simpler wording). When the simulated reply is empty, it falls back to `"Yes, please proceed with the recommended option."`.
#### Customizing the Simulator Prompt
The simulator instructions for auto\_user evaluation can be overridden. By default it reads from `eval/simulator-instruction.md`, whose content is the standard template of the principles above. Two override methods (from highest to lowest priority):
| Method | How to set | Scope |
| ------------------------------------ | ------------------------------------------------------ | ------------------- |
| `--simulator-prompt "..."` | pytest command-line argument | Single run |
| `BENCH_SIMULATOR_PROMPT_FILE=` | Environment variable (can be written into `eval/.env`) | All auto\_user runs |
Relative paths for `BENCH_SIMULATOR_PROMPT_FILE` are resolved from `eval/`, with a default value of `simulator-instruction.md`; the file is read only if it exists. This lets you swap in a different user behavior profile (e.g. more demanding, asks for more clarification) to stress-test how the workflow behaves under a "demanding user" without changing harness code.
### Why It's Designed This Way
Workflow-type Skills pause at decision points waiting for user confirmation. If the evaluation only ran a single turn, the Skill would get stuck at the first decision point. The dual-agent loop lets the evaluation **automatically run the entire workflow** (open→...→archive), while ensuring the user input at decision points is "reasonable and forward-moving" rather than hardcoded replies. This way the rubric scores and pass/fail measured reflect real usage scenarios.
### Single-turn vs Multi-turn
* **`generic` profile** (`interaction.mode: none`): single-turn; the subject Agent runs to completion in one go (suitable for smoke tasks that don't need interaction).
* **`comet-workflow` / `authoring-skill` profile** (`interaction.mode: auto_user`): multi-turn; enables the dual-agent loop.
Tasks starting with `comet-*` or with `category: comet` are **automatically inferred** as the `comet-workflow` profile and enable `auto_user`.
## The Three Axes of Evaluation: treatment × task × reps
An evaluation is the Cartesian product of three axes:
| Axis | Meaning | Example |
| ------------- | ----------------------------------------------------------------------- | ---------------------------------------------------------- |
| **task** | An evaluation task (directory, with instruction/environment/validation) | `comet-fix-median` (the bug-fix dimension) |
| **treatment** | Experimental condition (which Skills / CLAUDE.md to inject) | `CONTROL` (no-Skill baseline), `COMET_FULL` (inject comet) |
| **reps** | Number of repeats (`--count N`) | `--count 5` runs each (task,treatment) 5 times |
### treatment Enables A/B Comparison
Running multiple treatments on the same task lets you measure the **marginal effect of the Comet Skill**:
| Treatment | Injects | Purpose |
| ---------------- | -------------------------------------------------------------------- | --------------------- |
| `CONTROL` | No-Skill baseline | Control |
| `COMET_FULL` | comet main + 7 phase sub-Skills + brainstorming (v3, `.mjs` runtime) | Current version |
| `COMET_FULL_039` | 0.3.9 frozen baseline (`.sh` runtime) | Regression comparison |
The comparison report (`compare_baselines.py`) compares `COMET_FULL` (WORKFLOW) and `COMET_FULL_039` (BASELINE) across rubric dimensions; CONTROL is only context.
## How Metrics Get Into the Report
| Report location | Content |
| -------------------------------------------------- | ------------------------------------------------------------------------------------------------------------- |
| `summary.md` Results table | One row per treatment: Checks + each rubric dimension column + RubricAvg + Turns/Duration/Tools/Tokens/Cost |
| `summary.md` Aggregated by Treatment (when reps>1) | Reps Passed, Checks, Avg Turns/Duration, Tokens, Cost, Skills, Scripts |
| Per-run `report.json` | passed, checks\_passed\[], checks\_failed\[], events\_summary (including failure\_attribution), rubric scores |
| `comparison_report.md` | pass\@k/pass^k table (capability vs reliability) + rubric dimension cross-treatment comparison |
See [Reading Evaluation Reports](/en/eval/reports).
## Next steps
* [Reading Evaluation Reports](/en/eval/reports) — report structure and failure attribution
* [Eval harness](/en/eval/harness) — collect/run internal mechanism and environment variables
* [Evaluation System Overview](/en/eval/overview) — where eval fits in the publish workflow
* [comet eval command](/en/cli/eval) — complete options reference