Configuring Evaluations and Custom Tasks

comet eval works out of the box to evaluate built-in tasks and any local Skill — evaluating your own Skill takes just one command, comet eval ./your-skill --html (see Quickstart). If you only need stable acceptance conditions for a local Skill, prefer declaring an inline task in that Skill package’s comet/eval.yaml, or reference a package-local task with source. Only create and register a task under eval/local/tasks/ when you are extending Comet’s built-in harness. This article walks you through the complete customization path: defining a task, configuring an evaluation profile, and running a report.

This article is advanced customization. If you just want to evaluate an existing Skill, you don’t need to read it — just use comet eval ./your-skill —html. For a Skill-specific inline task, edit comet/eval.yaml under the Skill root. This article assumes you can already run comet eval ./your-skill —collect successfully.

The eval/local/tasks/, pytest, Dockerfile, and registration workflow in this page are advanced source-maintainer content and require a Comet source checkout to reproduce. Ordinary users only need an inline task in the Skill’s comet/eval.yaml, or the comet eval command in Quickstart.

The Big Picture of Eval Configuration

A complete eval is assembled from three parts, and understanding their division of labor is the foundation of customization:

Little fish placing Task, Treatment and Profile configuration cards into the run tray, and organizing score, report and failure attribution outputs

Task decides what to test, Treatment decides what to inject, Profile and Rubric decide how to score

Configuration	Question it answers	Location
Task	What scenario to test? What to produce? How to verify correctness?	`local/tasks/<name>/`
Treatment	Which Skills to inject in this run (including the no-Skill baseline)?	`local/treatments/`
Profile + Rubric	Which dimensions to score by?	`[evaluation]` in task.toml

Let’s explain each one below.

Define a Task

Each task is a directory under local/tasks/ containing four files:

local/tasks/my-task/

task.toml

instruction.md

environment/

Dockerfile

validation/

test_my_task.py

For an ordinary local Skill, put inline tasks in comet/eval.yaml under that Skill’s root, or use source for a task package stored in the Skill package. The directory convention in this section comes from the eval harness (eval/), not your business repo: tasks under eval/local/tasks/ are for extending the Comet repository’s built-in harness and must be registered in eval/local/tasks/index.yaml.

1. Write `task.toml`

task.toml is the core configuration of a task, divided into four sections:

[metadata]
name = "my-task"
description = "Create a markdown result file"
difficulty = "medium"           # easy / medium / hard
category = "generic"            # generic or comet
default_treatments = ["CONTROL"]

[environment]
description = "Validation environment"
dockerfile = "environment/Dockerfile"
timeout_sec = 600               # Docker execution timeout (seconds)

[validation]
test_scripts = ["test_my_task.py"]   # Validation scripts run inside Docker
target_artifacts = ["result.md"]     # Expected artifacts (existence check)
timeout = 120                        # Validation script timeout (seconds)

[evaluation]
profile = "generic"                       # Evaluation profile
expected_artifacts = ["result.md"]        # Artifacts checked by the rubric
required_skills = ["my-skill"]            # Skills that must be invoked
require_skill_invocation = true           # Produces a hard failure when not invoked

# Custom rubric criteria (passed to the LLM judge, optional)
rubric_criteria = [
    "Output includes error handling",
    "Code follows project naming conventions",
]

[interaction]
mode = "none"                   # none (single-turn) or auto_user (multi-turn simulation)
max_turns = 12                  # Outer round-trip limit between subject/simulator under auto_user

The role of each section:

Section	Role
`[metadata]`	Task identity. `category = "comet"` or a name starting with `comet-` will be automatically inferred as the `comet-workflow` profile
`[environment]`	What Docker environment the validation runs in, and the timeout
`[validation]`	Which scripts to validate with, which artifacts are expected
`[evaluation]`	Which profile to score with, which Skills must be invoked, custom judging criteria
`[interaction]`	Single-turn or multi-turn simulated interaction

Key fields in `[evaluation]`

These fields directly determine the scoring result and deserve separate explanation:

Field	Role
`profile`	Select the rubric: `generic` (7 dimensions) / `comet-workflow` (9 dimensions) / `authoring-skill` (11 dimensions). See Scoring Metrics and Dual-Agent Evaluation for dimension details
`expected_artifacts`	The `artifact_presence` dimension of the rubric checks whether these artifacts exist (supports glob)
`required_skills`	Lists Skills that must be invoked; the `skill_invocation` dimension scores based on this
`require_skill_invocation`	When set to `true`, a required Skill not being invoked produces a hard failure (the run is directly judged as not passed)
`rubric_criteria`	Custom judging criteria, passed to the LLM judge as additional dimensions (`custom_0`, `custom_1`…)

require_skill_invocation: true is useful but use it with caution: it requires the agent to have actually invoked the specified Skill (read from Claude Code events, not inferred from artifacts). If your Skill name is misspelled or not injected, every run will hard-fail.

2. Write `instruction.md`

This is the task instruction for the agent, supporting template variables like {run_id}:

Create a file `result.md` in the current workspace.

Requirements:

- Include the heading `# My Task Result`
- Describe your implementation approach
- Include at least three key points

3. Write the validation script

The validation script runs inside Docker and writes results to _test_results.json. This is the source of truth for “did this run get it right”:

from pathlib import Path
from scaffold.python.validation.core import write_test_results

def main():
    passed = []
    failed = []

    # Check whether the artifact exists
    result = Path("result.md")
    if not result.exists():
        write_test_results({"passed": [], "failed": ["result.md missing"]})
        return

    text = result.read_text(encoding="utf-8")

    # Check the content
    if "# My Task Result" in text:
        passed.append("heading present")
    else:
        failed.append("heading missing")

    write_test_results({"passed": passed, "failed": failed})

if __name__ == "__main__":
    main()

The definition of a “pass” for a run is clear: the failed list reported by the validation script is empty. Both pass@k and pass^k are based on this judgment.

4. Write the Dockerfile

The Dockerfile provides the runtime and dependencies needed for validation. A minimal example:

FROM python:3.11-slim

WORKDIR /workspace
# Install dependencies needed by the validation script (if any)
# COPY requirements.txt .
# RUN pip install -r requirements.txt

The code produced by the agent and your validation script run inside the built image.

5. Register the task

Finally, register it in local/tasks/index.yaml:

tasks:
  - name: my-task
    category: generic
    default_treatments:
      - CONTROL
    description: My custom task

Once registered, you can run it with --task my-task.

Configure Treatment

Treatment answers “which Skills to inject in this run”. Located at local/treatments/:

# local/treatments/common/control.yaml
CONTROL:
  description: "No-Skill baseline"
  skills: []

# local/treatments/comet/comet_full_040_beta.yaml
COMET_FULL_040_BETA:
  description: "Full Comet workflow"
  skills:
    - name: comet
      skill: 040-beta/comet
      variant: all
      base: benchmarks

Treatment	Meaning
`CONTROL`	No-Skill baseline — no Skill injected, tests the agent’s bare capability floor
`COMET_FULL_040_BETA`	Injects the full Comet workflow Skill bundle
Custom	Inject your own Skill to compare “with Skill vs without Skill”

The standard approach for a comparative experiment is CONTROL (no Skill) vs your Skill treatment. The difference between the two is the gain your Skill brings.

Choose a Profile

The profile determines which rubric to score with. Choosing the wrong profile makes the dimension scores meaningless:

Profile	Suitable for	Dimensions	Interaction mode
`generic`	Smoke-testing any generic Skill	7	`none` (single-turn)
`comet-workflow`	Classic `/comet` five-phase	9	`auto_user` (multi-turn)
`authoring-skill`	`/comet-any` artifacts	11	`auto_user` (multi-turn)

Profile resolution priority: —profile override > manifest’s skill.profile > task’s evaluation.profile > generic. Comet-type tasks (category = “comet”) are automatically inferred as comet-workflow.

For the dimensions, weights and checking logic of the three rubrics, see Scoring Metrics and Dual-Agent Evaluation.

Configure LLM-as-judge

LLM-as-judge is an optional quality coverage layer. It does not replace rule-based [RUBRIC] scores and does not directly change weighted_score. It lets an independent judge model read the run artifacts and append [RUBRIC-JUDGE] scores, helping you catch cases where rule checks miss whether the artifact has real substance. Use it when:

You evaluate documents, plans, code explanations, designs, or other artifacts that are hard to judge completely with scripts.
You already have validation scripts, but want to know whether the artifact is empty, shallow, or template-like.
You want comparison reports to show the gap between rule scores and judge scores.

Judge configuration is isolated from the subject Agent configuration. The subject Agent can keep using ANTHROPIC_MODEL, ANTHROPIC_BASE_URL, and its own token. The judge must explicitly set BENCH_JUDGE_MODEL so the same configuration does not silently act as both contestant and judge.

1. Enable judge in `eval/.env`

Add judge variables to the Comet repository’s eval/.env:

# eval/.env
BENCH_LLM_JUDGE=1
BENCH_JUDGE_MODEL=your-judge-model

BENCH_LLM_JUDGE=1 is the switch. BENCH_JUDGE_MODEL is required. If you enable the switch without setting a model, the report records a skipped status:

[RUBRIC-JUDGE] status: skipped - BENCH_JUDGE_MODEL is required when BENCH_LLM_JUDGE=1

2. Choose judge authentication

If the local host claude CLI is already authenticated, you can set only BENCH_JUDGE_MODEL. The harness calls the host claude CLI for judge runs. If you want the judge to use an independent Anthropic-compatible proxy, configure a dedicated endpoint and token:

# eval/.env
BENCH_LLM_JUDGE=1
BENCH_JUDGE_MODEL=your-judge-model
BENCH_JUDGE_BASE_URL=https://your-anthropic-compatible-endpoint
BENCH_JUDGE_AUTH_TOKEN=your-judge-token

You can also use an API key:

# eval/.env
BENCH_LLM_JUDGE=1
BENCH_JUDGE_MODEL=your-judge-model
BENCH_JUDGE_BASE_URL=https://your-anthropic-compatible-endpoint
BENCH_JUDGE_API_KEY=your-judge-api-key

Priority:

Configuration	Behavior
`BENCH_JUDGE_BASE_URL` + `BENCH_JUDGE_AUTH_TOKEN`	Prefer Anthropic Messages HTTP; the HTTP header uses the bearer token
`BENCH_JUDGE_BASE_URL` + `BENCH_JUDGE_API_KEY`	Prefer Anthropic Messages HTTP; the HTTP header uses the API key
Only `BENCH_JUDGE_MODEL`	Fall back to the host `claude` CLI

If both BENCH_JUDGE_AUTH_TOKEN and BENCH_JUDGE_API_KEY are set, the auth token wins. The judge subprocess clears inherited main ANTHROPIC_* provider variables before mapping BENCH_JUDGE_*, so do not expect it to reuse the subject Agent provider configuration automatically.

3. Add custom judge criteria for generic tasks

Generic Skills (generic / authoring-skill) make the judge output three default dimensions:

Dimension	What it judges
`task_completion`	Whether the task requirements are satisfied, artifacts exist and are correct, and basic validation passed
`output_quality`	Whether the artifact is complete, structured, and substantive
`instruction_adherence`	Whether the run followed instructions, constraints, and tool boundaries

If your task has task-specific quality criteria, put them in [evaluation] in task.toml:

[evaluation]
profile = "generic"
expected_artifacts = ["result.md"]
required_skills = ["my-skill"]
require_skill_invocation = true
rubric_criteria = [
  "The output explains key tradeoffs instead of only listing steps",
  "Error handling covers empty input and external command failure",
]

These criteria are emitted as additional judge dimensions named custom_0, custom_1. They only affect the [RUBRIC-JUDGE] coverage layer and do not replace your validation scripts. The comet-workflow profile uses a different judge dimension set: artifact_quality, spec_drift, and main_flow. These dimensions review workflow artifact depth, whether the spec drifted, and whether the five-phase main flow completed.

4. Run and confirm judge output

First use --collect to confirm tasks, manifests, and paths can be discovered:

comet eval ./generated-skill/comet/eval.yaml --collect

Then run the real evaluation:

comet eval ./generated-skill/comet/eval.yaml --html

Open the report printed in Report path and look for [RUBRIC-JUDGE] lines in each run’s checks. A successful run looks like this:

[RUBRIC-JUDGE] output_quality: 0.75 - cites concrete artifact content
[RUBRIC-JUDGE] instruction_adherence: 1.00 - follows stated constraints
[RUBRIC-JUDGE] status: enabled_and_successful

If you only see skipped or failed, use this table:

Symptom	Common cause	Fix
`BENCH_JUDGE_MODEL is required`	`BENCH_LLM_JUDGE=1` is set, but `BENCH_JUDGE_MODEL` is missing	Add `BENCH_JUDGE_MODEL`
`(judge error: HTTP ...)`	Judge endpoint, token, or model name does not match	Check `BENCH_JUDGE_BASE_URL`, token, and model name
No `[RUBRIC-JUDGE]` lines	`BENCH_LLM_JUDGE=1` is not enabled, or this run did not reach profile rubric scoring	Confirm `eval/.env` is loaded and fix earlier harness/task failures first

Multi-turn interaction: `auto_user` mode

A single-turn task (mode: none) runs only once, suitable for simple scenarios of “give an instruction and see the result”. But workflow-type Skills need to pause at decision points, ask the user, and continue after receiving a reply — single-turn can’t test this. auto_user mode solves this: it uses two Agents interacting automatically — one runs the Skill under test (subject), the other simulates the user replying at decision points (simulator). The simulator’s behavior is controlled by a prompt file. It reads eval/simulator-instruction.md by default, and you can point BENCH_SIMULATOR_PROMPT_FILE to your own version to simulate “a more demanding user” or “a user who asks for clarification”:

export BENCH_SIMULATOR_PROMPT_FILE=my-simulator-prompt.md

max_turns controls the loop limit (comet-workflow typically 12 outer round-trips, authoring-skill typically 8 outer round-trips). It is not the number of internal messages or tool calls of the Agent under test; one outer round-trip means the Agent under test reaches a decision point, the user simulator replies, and the Agent under test continues with --resume. Hitting a “complete” signal (such as archive complete) ends the loop early.

A complete custom evaluation from scratch

Putting the above steps together, to evaluate your own Skill from scratch:

# 1. Define the task (the four files above + register in index.yaml)

# 2. First collect to troubleshoot, without consuming tokens
comet eval --task my-task --collect

# 3. Run for real (inject your Skill treatment)
comet eval --task my-task --treatment MY_SKILL --html

# 4. View the report
# .comet/eval/runs/<experiment-id>/summary.html

In the report, focus on three things:

pass@1 and pass^k: capability ceiling vs reliability floor. See Scoring Metrics.
Rubric dimension scores: which dimension dragged you down?
Failure attribution (harness/workflow/task/model): is the failure a Skill problem, or a task/environment problem?

Evaluating `/comet-any` generated bundles

A Skill bundle generated by /comet-any comes with its own comet/eval.yaml manifest, so you don’t need to hand-write a task.toml. Just run with the manifest:

# Discovery pre-check
comet eval ./generated-skill/comet/eval.yaml --collect

# Real evaluation
comet eval ./generated-skill/comet/eval.yaml --html

The manifest automatically selects the authoring-skill profile, injects recommended tasks and quality gates. For the complete manifest format, see Evaluation System Overview.

Next steps

Scoring Metrics and Dual-Agent Evaluation — dimensions, weights and pass@k/pass^k of the three rubrics
Evaluation System Overview — manifest format, profile system and task system
comet eval command — complete command-line argument reference

Quickstart

Methodology: Why evaluation drives Skill evolution

Real evaluation experiments

Advanced

Related commands

Configuring Evaluations and Custom Tasks

The Big Picture of Eval Configuration

Define a Task

1. Write `task.toml`

Key fields in `[evaluation]`

2. Write `instruction.md`

3. Write the validation script

4. Write the Dockerfile

5. Register the task

Configure Treatment

Choose a Profile

Configure LLM-as-judge

1. Enable judge in `eval/.env`

2. Choose judge authentication

3. Add custom judge criteria for generic tasks

4. Run and confirm judge output

Multi-turn interaction: `auto_user` mode

A complete custom evaluation from scratch

Evaluating `/comet-any` generated bundles

Next steps

​The Big Picture of Eval Configuration

​Define a Task

​1. Write task.toml

​Key fields in [evaluation]

​2. Write instruction.md

​3. Write the validation script

​4. Write the Dockerfile

​5. Register the task

​Configure Treatment

​Choose a Profile

​Configure LLM-as-judge

​1. Enable judge in eval/.env

​2. Choose judge authentication

​3. Add custom judge criteria for generic tasks

​4. Run and confirm judge output

​Multi-turn interaction: auto_user mode

​A complete custom evaluation from scratch

​Evaluating /comet-any generated bundles

​Next steps

The Big Picture of Eval Configuration

Define a Task

1. Write `task.toml`

Key fields in `[evaluation]`

2. Write `instruction.md`

3. Write the validation script

4. Write the Dockerfile

5. Register the task

Configure Treatment

Choose a Profile

Configure LLM-as-judge

1. Enable judge in `eval/.env`

2. Choose judge authentication

3. Add custom judge criteria for generic tasks

4. Run and confirm judge output

Multi-turn interaction: `auto_user` mode

A complete custom evaluation from scratch

Evaluating `/comet-any` generated bundles

Next steps