Skip to main content
comet eval works out of the box to evaluate built-in tasks and any local Skill. But when you want to evaluate how your own Skill performs in your own scenario, you need a custom task. This article walks you through the entire process step by step: from defining a task, to configuring an evaluation profile, to running a report.
If you haven’t run an eval yet, first read Evaluating Any Skill · Quickstart to set up the environment. This article assumes you can already run comet eval —collect successfully.

​
The Big Picture of Eval Configuration

A complete eval is assembled from three parts, and understanding their division of labor is the foundation of customization:

Little fish placing Task, Treatment and Profile configuration cards into the run tray, and organizing score, report and failure attribution outputs

Task decides what to test, Treatment decides what to inject, Profile and Rubric decide how to score

ConfigurationQuestion it answersLocation
TaskWhat scenario to test? What to produce? How to verify correctness?local/tasks/<name>/
TreatmentWhich Skills to inject in this run (including the no-Skill baseline)?local/treatments/
Profile + RubricWhich dimensions to score by?[evaluation] in task.toml
Let’s explain each one below.

​
Define a Task

Each task is a directory under local/tasks/ containing four files:
local/tasks/my-task/
task.toml
instruction.md
environment/
Dockerfile
validation/
test_my_task.py
The task directory convention comes from the eval harness (eval/), not your business repo. Custom tasks live under eval/local/tasks/ in the Comet repo and are registered in eval/local/tasks/index.yaml.

​
1. Write task.toml

task.toml is the core configuration of a task, divided into four sections: 
[metadata]
name = "my-task"
description = "Create a markdown result file"
difficulty = "medium"           # easy / medium / hard
category = "generic"            # generic or comet
default_treatments = ["CONTROL"]

[environment]
description = "Validation environment"
dockerfile = "environment/Dockerfile"
timeout_sec = 600               # Docker execution timeout (seconds)

[validation]
test_scripts = ["test_my_task.py"]   # Validation scripts run inside Docker
target_artifacts = ["result.md"]     # Expected artifacts (existence check)
timeout = 120                        # Validation script timeout (seconds)

[evaluation]
profile = "generic"                       # Evaluation profile
expected_artifacts = ["result.md"]        # Artifacts checked by the rubric
required_skills = ["my-skill"]            # Skills that must be invoked
require_skill_invocation = true           # Produces a hard failure when not invoked

# Custom rubric criteria (passed to the LLM judge, optional)
rubric_criteria = [
    "Output includes error handling",
    "Code follows project naming conventions",
]

[interaction]
mode = "none"                   # none (single-turn) or auto_user (multi-turn simulation)
max_turns = 12                  # Outer round-trip limit between subject/simulator under auto_user
The role of each section:
SectionRole
[metadata]Task identity. category = "comet" or a name starting with comet- will be automatically inferred as the comet-workflow profile
[environment]What Docker environment the validation runs in, and the timeout
[validation]Which scripts to validate with, which artifacts are expected
[evaluation]Which profile to score with, which Skills must be invoked, custom judging criteria
[interaction]Single-turn or multi-turn simulated interaction

​
Key fields in [evaluation]

These fields directly determine the scoring result and deserve separate explanation:
FieldRole
profileSelect the rubric: generic (7 dimensions) / comet-workflow (9 dimensions) / authoring-skill (11 dimensions). See Scoring Metrics and Dual-Agent Evaluation for dimension details
expected_artifactsThe artifact_presence dimension of the rubric checks whether these artifacts exist (supports glob)
required_skillsLists Skills that must be invoked; the skill_invocation dimension scores based on this
require_skill_invocationWhen set to true, a required Skill not being invoked produces a hard failure (the run is directly judged as not passed)
rubric_criteriaCustom judging criteria, passed to the LLM judge as additional dimensions (custom_0, custom_1…)
require_skill_invocation: true is useful but use it with caution: it requires the agent to have actually invoked the specified Skill (read from Claude Code events, not inferred from artifacts). If your Skill name is misspelled or not injected, every run will hard-fail.

​
2. Write instruction.md

This is the task instruction for the agent, supporting template variables like {run_id}: 
Create a file `result.md` in the current workspace.

Requirements:
- Include the heading `# My Task Result`
- Describe your implementation approach
- Include at least three key points

​
3. Write the validation script

The validation script runs inside Docker and writes results to _test_results.json. This is the source of truth for “did this run get it right”: 
from pathlib import Path
from scaffold.python.validation.core import write_test_results

def main():
    passed = []
    failed = []

    # Check whether the artifact exists
    result = Path("result.md")
    if not result.exists():
        write_test_results({"passed": [], "failed": ["result.md missing"]})
        return

    text = result.read_text(encoding="utf-8")

    # Check the content
    if "# My Task Result" in text:
        passed.append("heading present")
    else:
        failed.append("heading missing")

    write_test_results({"passed": passed, "failed": failed})

if __name__ == "__main__":
    main()
The definition of a “pass” for a run is clear: the failed list reported by the validation script is empty. Both pass@k and pass^k are based on this judgment.

​
4. Write the Dockerfile

The Dockerfile provides the runtime and dependencies needed for validation. A minimal example: 
FROM python:3.11-slim

WORKDIR /workspace
# Install dependencies needed by the validation script (if any)
# COPY requirements.txt .
# RUN pip install -r requirements.txt
The code produced by the agent and your validation script run inside the built image.

​
5. Register the task

Finally, register it in local/tasks/index.yaml: 
tasks:
  - name: my-task
    category: generic
    default_treatments:
      - CONTROL
    description: My custom task
Once registered, you can run it with --task my-task.

​
Configure Treatment

Treatment answers “which Skills to inject in this run”. Located at local/treatments/: 
# local/treatments/common/control.yaml
CONTROL:
  description: "No-Skill baseline"
  skills: []

# local/treatments/comet/comet_full_040_beta.yaml
COMET_FULL_040_BETA:
  description: "Full Comet workflow"
  skills:
    - name: comet
      skill: 040-beta/comet
      variant: all
      base: benchmarks
TreatmentMeaning
CONTROLNo-Skill baseline — no Skill injected, tests the agent’s bare capability floor
COMET_FULL_040_BETAInjects the full Comet workflow Skill bundle
CustomInject your own Skill to compare “with Skill vs without Skill”
The standard approach for a comparative experiment is CONTROL (no Skill) vs your Skill treatment. The difference between the two is the gain your Skill brings.

​
Choose a Profile

The profile determines which rubric to score with. Choosing the wrong profile makes the dimension scores meaningless:
ProfileSuitable forDimensionsInteraction mode
genericSmoke-testing any generic Skill7none (single-turn)
comet-workflowClassic /comet five-phase9auto_user (multi-turn)
authoring-skill/comet-any artifacts11auto_user (multi-turn)
Profile resolution priority: —profile override > manifest’s skill.profile > task’s evaluation.profile > generic. Comet-type tasks (category = “comet”) are automatically inferred as comet-workflow.
For the dimensions, weights and checking logic of the three rubrics, see Scoring Metrics and Dual-Agent Evaluation.

​
Multi-turn interaction: auto_user mode

A single-turn task (mode: none) runs only once, suitable for simple scenarios of “give an instruction and see the result”. But workflow-type Skills need to pause at decision points, ask the user, and continue after receiving a reply — single-turn can’t test this. auto_user mode solves this: it uses two Agents interacting automatically — one runs the Skill under test (subject), the other simulates the user replying at decision points (simulator). The simulator’s behavior is controlled by a prompt file. It reads eval/simulator-instruction.md by default, and you can point BENCH_SIMULATOR_PROMPT_FILE to your own version to simulate “a more demanding user” or “a user who asks for clarification”: 
export BENCH_SIMULATOR_PROMPT_FILE=my-simulator-prompt.md
max_turns controls the loop limit (comet-workflow typically 12 outer round-trips, authoring-skill typically 8 outer round-trips). It is not the number of internal messages or tool calls of the Agent under test; one outer round-trip means the Agent under test reaches a decision point, the user simulator replies, and the Agent under test continues with --resume. Hitting a “complete” signal (such as archive complete) ends the loop early.

​
A complete custom evaluation from scratch

Putting the above steps together, to evaluate your own Skill from scratch: 
# 1. Define the task (the four files above + register in index.yaml)

# 2. First collect to troubleshoot, without consuming tokens
comet eval --task my-task --collect

# 3. Run for real (inject your Skill treatment)
comet eval --task my-task --treatment MY_SKILL --html

# 4. View the report
# eval/local/logs/experiments/<experiment-id>/summary.html
In the report, focus on three things:
  1. pass@1 and pass^k: capability ceiling vs reliability floor. See Scoring Metrics.
  2. Rubric dimension scores: which dimension dragged you down?
  3. Failure attribution (harness/workflow/task/model): is the failure a Skill problem, or a task/environment problem?

​
Evaluating /comet-any generated bundles

A Skill bundle generated by /comet-any comes with its own comet/eval.yaml manifest, so you don’t need to hand-write a task.toml. Just run with the manifest: 
# Discovery pre-check
comet eval ./generated-skill/comet/eval.yaml --collect

# Real evaluation
comet eval ./generated-skill/comet/eval.yaml --html
The manifest automatically selects the authoring-skill profile, injects recommended tasks and quality gates. For the complete manifest format, see Evaluation System Overview.

​
Next steps

Last modified on July 6, 2026