Eval Harness (Advanced)

​

Harness Directory Structure

eval/
├── pyproject.toml          # Python dependencies (pytest, pyyaml, pydantic, etc.)
├── uv.lock
├── .env / .env.example
├── scaffold/               # Shared harness code
│   ├── python/
│   │   ├── attribution.py      # Failure attribution
│   │   ├── logging.py          # ExperimentLogger, summary.md generation
│   │   ├── manifests.py        # comet/eval.yaml parsing
│   │   ├── profiles.py         # Profile registry
│   │   ├── report_outputs.py   # Report config + HTML rendering
│   │   ├── tasks.py            # task.toml loading
│   │   ├── treatments.py       # treatments YAML loading
│   │   └── validation/         # Rubric and Docker validators
│   └── shell/
│       ├── docker.sh           # Wrapper for running Claude in Docker
│       └── run-claude-loop.sh  # Multi-turn auto_user driver
├── local/                  # Local suite (no LangSmith credentials needed)
│   ├── tasks/              # Task definitions
│   ├── treatments/         # Control / experiment groups
│   ├── tests/
│   │   └── tasks/test_tasks.py   # Parameterized tests actually invoked by the CLI
│   └── logs/experiments/   # Output: reports + artifacts per run
└── langsmith/             # LangSmith suite (reuses local tasks)

​

What It Wraps

• Launches the harness from the <project>/eval root directory (using uv run)
• Converts --manifest or --skill-path into pytest parameters
• Selects the default profile (generic) or task (recommended, generic-skill-smoke)
• Generates a temporary report config on demand (when using --html)
• Prints a set of execution info to help locate the report

uv run pytest local/tests/tasks/test_tasks.py [args]

​

collect and run

​

manifest mode

comet eval ./generated-skill/comet/eval.yaml --collect
comet eval ./generated-skill/comet/eval.yaml --html

​

skill-path mode

comet eval ./my-skill --skill-name my-skill --quick

​

Execution Info

• Eval root: which eval/ root directory is actually launched from
• Mode: collect or run
• Target: whether the current evaluation target is a manifest or a local Skill directory
• Experiment: the experiment id for this run
• Profile: the profile used in this evaluation
• Task: the evaluation task
• Report path: report location
• Report config: the temporary report config used when --html is enabled

​

Where the Report Is and What It Looks Like

​

Experiment ID

eval/local/logs/experiments/<experiment-id>/
  ├── summary.md            # Main report (always generated)
  ├── summary.html          # HTML version (with --html)
  ├── metadata.json         # Experiment metadata
  ├── events/               # stream-json events per run
  ├── raw/                  # Raw stdout/stderr
  ├── reports/              # report.json per run
  └── artifacts/            # Files produced by Claude

​

summary.md contains

1. Header: Experiment ID, start/completion time.
2. Results table: one row per treatment, with columns including Checks, Turns, Duration, Tools, Tokens, Cost, RubricAvg.
3. Summary: total runs, checks passed X/Y (percentage).
4. Treatment Details: detailed metrics for each run of each treatment, skills invoked, scripts used, passed and failed check lists.

​

report.json per run

​

Inside a Single Evaluation: Docker Isolation + Dual Agent + Rubric

A real evaluation runs Agent interactions in an isolated environment, then records evidence via validators and rubric

• The model runs inside a Docker container, isolated from your working directory.
• Dual-agent loop (auto_user mode): the subject Agent runs the Skill under test, and at each decision point a user simulator Agent replies (approves reasonable proposals, picks defaults, drives progress — never refuses, never writes code). The subject Agent continues the same session with --resume, up to max_turns outer round-trips (comet-workflow typically 12, authoring-skill typically 8), ending early when a “complete” signal is hit. The max_turns here is not the number of internal messages or tool calls of the Agent under test. This lets multi-stage workflows automatically run the entire chain. For the complete loop and decision-point detection, see Scoring Metrics and Dual-Agent Evaluation.
• Rubric scoring runs after the validators and appends results as [RUBRIC] informational checks (comet-workflow rubric never produces a hard failure; generic/authoring produce hard failures for specific missing items).
• The real pass/fail is determined by the task validators (expected artifacts exist + test_scripts pass); rubric scores and pass@k are diagnostic information.

​

Report Output Configuration

1. --report-config <path> (JSON or YAML)
2. COMET_EVAL_REPORT_CONFIG environment variable
3. Default (markdown only)

{"report_outputs": {"markdown": true, "html": false}}

​

Failure Attribution

​

Environment Variable Reference

​

Authenticating with an Anthropic-compatible proxy

​

Custom User Simulator Prompt

You are simulating a developer user in an automated eval. ...
- Approves the proposed approach / name / plan when asked to confirm
- Picks the most reasonable default option when asked to choose
- Asks for clarification only if the question is truly ambiguous ...
Never refuse; always let the workflow move forward. Do not write code or files.

# eval/.env
BENCH_SIMULATOR_PROMPT_FILE=my-simulator-prompt.md

​

Don’t Confuse the Harness with Runtime Check

• comet eval: evaluates a Skill bundle or comet/eval.yaml, answering “can this Skill, as a product capability, pass the evaluation”.
• comet skill check: checks whether a specific Skill run is missing artifacts or state, answering “is this run complete”.

​

Next steps

• Scoring Metrics and Dual-Agent Evaluation — rubric dimension details, pass@k/pass^k, dual-agent interaction loop
• Reading Evaluation Reports — learn to read report signals and failure attribution
• comet eval command — complete options and subcommand reference
• Evaluation System Overview — where eval fits in the workflow and the eval.yaml format