Eval Agent Startup Configuration

comet eval starts the selected Agent inside an isolated Docker container. Agent login and model protocols are not interchangeable: Claude Code can use an Anthropic-compatible gateway, Codex requires an OpenAI Responses API provider, Qoder requires its own login session or Personal Access Token, and CodeBuddy uses its own API configuration for custom models. This page covers the Agent prerequisites only. The subject (Bench) and LLM-as-judge still use separate BENCH_* and BENCH_JUDGE_* settings; see Eval harness for the complete variable reference.

Never put real credentials in a Skill, manifest, public repository, Dockerfile, or report. Prefer the current shell environment, or the user-level %USERPROFILE%\.comet\eval\.env / ~/.comet/eval/.env; these files are excluded from the published package.

Where the user-level `.env` lives

Ordinary users only need the .env in their user directory. You do not need to clone the Comet repository or edit the eval/.env inside an installed package:

Windows: %USERPROFILE%\\.comet\\eval\\.env
macOS/Linux: ~/.comet/eval/.env

When comet eval runs for the first time and this file is missing, the CLI creates a commented template containing all Eval parameters and prints the exact path. Edit it and run the command again. Existing files are never overwritten, and environment variables in the current shell have higher priority. The complete template is grouped into subject, independent Judge, Claude Code, Codex, Qoder, CodeBuddy, LangSmith, and Langfuse settings; edit the generated template directly instead of creating a parameter list by hand.

Choose an Agent

# Claude Code (default)
comet eval ./my-skill --agent claude-code --quick --html

# OpenAI Codex
comet eval ./my-skill --agent codex --quick --html

# Qoder
comet eval ./my-skill --agent qoder --quick --html

# CodeBuddy
comet eval ./my-skill --agent codebuddy --quick --html

Verify each Agent directly on the host before running Eval. This separates “the Agent is not logged in” from “the Skill evaluation failed.”

Claude Code

What to prepare

The claude CLI available on PATH;
either an interactive first-use login or ANTHROPIC_API_KEY;
when using an Anthropic-compatible gateway, ANTHROPIC_AUTH_TOKEN, ANTHROPIC_BASE_URL, and a model name.

The official Claude Code quickstart says that the first claude launch opens the login flow; setting ANTHROPIC_API_KEY skips the login prompt and asks you to approve the key instead. For another LLM gateway, the official configuration uses ANTHROPIC_AUTH_TOKEN and ANTHROPIC_BASE_URL. Official references: Claude Code overview and LLM gateway.

Verify it on the host

claude --version
claude -p "Reply with exactly OK" --output-format json

Use an Anthropic-compatible gateway

BENCH_EVAL_AGENT=claude-code
BENCH_MODEL=your-anthropic-compatible-model
BENCH_BASE_URL=https://your-anthropic-gateway.example/v1
BENCH_API_KEY=your-subject-key

# Or use Claude Code's native variables directly.
ANTHROPIC_AUTH_TOKEN=your-subject-key
ANTHROPIC_BASE_URL=https://your-anthropic-gateway.example/v1
ANTHROPIC_MODEL=your-anthropic-compatible-model

Eval maps the common settings to Claude Code’s native variables. Explicit ANTHROPIC_* values take precedence over common BENCH_* fallbacks.

Codex

What to prepare

The codex CLI available on PATH;
an API key or login session that can access an OpenAI Responses API;
a model name accepted by that provider;
for a custom provider, a user-level ~/.codex/config.toml entry declaring model_provider, base_url, env_key, and wire_api = "responses".

Codex custom providers are configured in the user-level config.toml. The official reference defines base_url as the provider API address, env_key as the environment variable supplying the key, and responses as the supported wire_api value. Do not give Codex an Anthropic Messages endpoint; the provider must expose the Responses API, normally at <base-url>/responses. Official reference: Codex configuration reference.

Verify it on the host

codex --version
codex exec --json --yolo "Reply with exactly OK"

Configure a custom Responses provider

Set the key before starting Codex:

# macOS / Linux
export OPENAI_API_KEY=your-subject-key

# Windows PowerShell
$env:OPENAI_API_KEY = "your-subject-key"

Example user-level ~/.codex/config.toml:

model = "your-responses-model"
model_provider = "comet-eval"

[model_providers.comet-eval]
name = "Comet Eval provider"
base_url = "https://your-openai-compatible-gateway.example/v1"
env_key = "OPENAI_API_KEY"
wire_api = "responses"
requires_openai_auth = false

The corresponding Comet Eval .env is:

BENCH_EVAL_AGENT=codex
OPENAI_API_KEY=your-subject-key
OPENAI_BASE_URL=https://your-openai-compatible-gateway.example/v1
OPENAI_MODEL=your-responses-model

# You can also use the common subject settings; Codex maps them to OPENAI_*.
# BENCH_API_KEY=your-subject-key
# BENCH_BASE_URL=https://your-openai-compatible-gateway.example/v1
# BENCH_MODEL=your-responses-model

At runtime, Eval creates a temporary container-local config.toml that contains only env_key = "OPENAI_API_KEY"; it does not save the real key. The provider must still support the Responses API. An endpoint that only supports /chat/completions or Anthropic /messages cannot be used directly by Codex.

Qoder

What to prepare

The qodercli CLI available on PATH;
a Qoder login session or a Qoder Personal Access Token (PAT);
for automation and Eval, QODER_PERSONAL_ACCESS_TOKEN is recommended.

Qoder’s official documentation supports two CLI login paths: run qodercli, then enter /login to use browser login or paste a Qoder PAT; for non-interactive startup, set QODER_PERSONAL_ACCESS_TOKEN. Create a PAT at Qoder Account → Integrations. Official references: Qoder CLI quick start and Qoder CLI authentication.

Verify it on the host

qodercli --version
qodercli
# At the Qoder prompt, enter:
/login

For a non-interactive check:

# macOS / Linux
QODER_PERSONAL_ACCESS_TOKEN=your-qoder-pat \
  qodercli -p "Reply with exactly OK" --output-format stream-json --yolo

# Windows PowerShell
$env:QODER_PERSONAL_ACCESS_TOKEN = "your-qoder-pat"
qodercli -p "Reply with exactly OK" --output-format stream-json --yolo

Configure Comet Eval

BENCH_EVAL_AGENT=qoder
QODER_PERSONAL_ACCESS_TOKEN=your-qoder-pat
QODER_MODEL=your-qoder-model

# Or use the common model setting.
# BENCH_MODEL=your-qoder-model

The Qoder PAT is Qoder’s own login credential, not an Anthropic or OpenAI API key. Do not put the Claude Code ANTHROPIC_AUTH_TOKEN or another model gateway key into QODER_PERSONAL_ACCESS_TOKEN; Qoder will reject it during startup. The current Eval setup does not require Claude Code’s ANTHROPIC_BASE_URL for Qoder. Eval injects QODER_PERSONAL_ACCESS_TOKEN only into the current container process and uses an isolated temporary Qoder config root; it does not read or mount the host’s Qoder login files.

CodeBuddy

What to prepare

The codebuddy CLI available on PATH;
CODEBUDDY_AUTH_TOKEN when using CodeBuddy platform authentication;
CODEBUDDY_API_KEY, CODEBUDDY_BASE_URL, and a provider-supported model name when using a third-party or OpenAI-compatible model.

Select a CodeBuddy custom model with --model <model-id>. CODEBUDDY_MODEL sets the default model and CODEBUDDY_BASE_URL overrides the request endpoint. Do not reuse Claude Code’s ANTHROPIC_BASE_URL directly for CodeBuddy, especially an /anthropic or /messages endpoint; the model ID must also be one accepted by CodeBuddy/the provider rather than a Claude-specific model suffix. CodeBuddy’s host-side custom model registry is ~/.codebuddy/models.json or the workspace-level .codebuddy/models.json. Its url should be the complete OpenAI-compatible /chat/completions URL, and apiKey can reference ${CODEBUDDY_API_KEY}. Never put the real key in a published file. For example:

{
  "models": [
    {
      "id": "your-model-id",
      "name": "Your Model",
      "vendor": "custom",
      "apiKey": "${CODEBUDDY_API_KEY}",
      "url": "https://your-openai-compatible-gateway.example/v1/chat/completions"
    }
  ],
  "availableModels": ["your-model-id"]
}

Eval does not mount the host’s models.json or CodeBuddy login directory. It creates a temporary settings.json inside the container and passes the native environment variables together with --model. Verify the provider directly first:

CODEBUDDY_API_KEY=your-subject-key \
CODEBUDDY_BASE_URL=https://your-openai-compatible-gateway.example/v1 \
CODEBUDDY_MODEL=your-model-id \
  codebuddy -p "Reply with exactly OK" --model your-model-id --output-format json -y

Then put the same values in the user-level Eval .env:

BENCH_EVAL_AGENT=codebuddy
CODEBUDDY_API_KEY=your-subject-key
CODEBUDDY_BASE_URL=https://your-openai-compatible-gateway.example/v1
CODEBUDDY_MODEL=your-model-id

# You can also use the common subject settings; Eval maps them to CODEBUDDY_*.
# BENCH_API_KEY=your-subject-key
# BENCH_BASE_URL=https://your-openai-compatible-gateway.example/v1
# BENCH_MODEL=your-model-id

# Optional: override CodeBuddy's model roles.
# CODEBUDDY_SMALL_FAST_MODEL=your-fast-model-id
# CODEBUDDY_BIG_SLOW_MODEL=your-reasoning-model-id
# CODEBUDDY_CODE_SUBAGENT_MODEL=your-subagent-model-id

Official references: CodeBuddy models, environment variables, and CLI reference. At runtime, Eval injects credentials only into the current container process. The temporary settings.json uses apiKeyHelper to read the environment, and the container-local config directory is destroyed after the run; real keys do not enter reports, manifests, or the Skill workspace.

Extend Eval with a custom Agent

An Agent that is not one of the built-ins is registered explicitly with an adapter.yaml in the user directory. Putting an executable on PATH is not enough, and the adapter does not require a Comet source checkout:

# macOS/Linux
~/.comet/eval/adapters/<agent-id>/adapter.yaml

# Windows
%USERPROFILE%\\.comet\\eval\\adapters\\<agent-id>\\adapter.yaml

You can set COMET_EVAL_ADAPTERS_DIR in the user-level %USERPROFILE%\\.comet\\eval\\.env / ~/.comet/eval/.env to override the registry directory. <agent-id> must start with a lowercase letter, contain only lowercase letters, digits, and hyphens, and be 2–32 characters long.

Minimal complete `adapter.yaml`

apiVersion: comet.eval.agent/v1alpha1
kind: EvalAgentAdapter
metadata:
  id: my-agent
  version: 1.0.0
runtime:
  executable: my-agent
  install:
    kind: npm
    package: my-agent
    version: latest
credentials:
  - MY_AGENT_API_KEY
  - MY_AGENT_AUTH_TOKEN
modelEnv: MY_AGENT_MODEL
baseUrlEnv: MY_AGENT_BASE_URL
capabilities:
  singleTurn: true
  resume: true
  structuredEvents: true
  telemetry: false
  skillInvocationEvidence: true

Field rules:

runtime.executable is the command started inside the container; an npm/pip package must expose this executable. runtime.install.kind supports npm, pip, and none; none means the command is already available in the base image.
credentials may declare at most two environment-variable names, never credential values. For a subject Agent, only these names are forwarded into the container. For a Judge, BENCH_JUDGE_API_KEY maps to the first name and BENCH_JUDGE_AUTH_TOKEN maps to the second name.
modelEnv and baseUrlEnv are optional custom environment-variable names. A custom Agent does not automatically inherit BENCH_API_KEY, BENCH_MODEL, or BENCH_BASE_URL; use the declared names, CLI options, or manifest configuration.
resume enables auto_user multi-turn interaction; structuredEvents requires JSONL output; skillInvocationEvidence requires explicit Skill-invocation events; and telemetry: false allows token/cost fields to be N/A. A real evaluation requires singleTurn, resume, structuredEvents, and skillInvocationEvidence to be true.

The custom CLI must accept Eval’s invocation contract:

<executable> -p "<prompt>" --output-format stream-json --model <model> [--resume <session-id>]

Write one structured event per JSONL line to stdout. To let the rubric confirm that a Skill was invoked, the output must also contain an event such as:

{"type":"skill_invocation","skill":"my-skill"}

Add the variables declared by adapter.yaml to the generated user-level .env:

BENCH_EVAL_AGENT=my-agent
MY_AGENT_API_KEY=your-subject-key
MY_AGENT_MODEL=your-model
MY_AGENT_BASE_URL=https://your-provider.example/v1

# Use separate credentials when it is also the independent Judge:
# BENCH_LLM_JUDGE=1
# BENCH_JUDGE_AGENT=my-agent
# BENCH_JUDGE_API_KEY=your-judge-key
# BENCH_JUDGE_MODEL=your-judge-model
# BENCH_JUDGE_BASE_URL=https://your-judge-provider.example/v1

Run a static check first, then a real evaluation:

comet eval ./my-skill --collect --agent my-agent
comet eval ./my-skill --quick --agent my-agent --model your-model

An adapter can be distributed as an npm/pip package or wrapper CLI, but adapter.yaml currently does not support custom argument templates, host configuration-directory mounts, or arbitrary Docker commands. Real credentials must remain in each user’s own .env or current shell.

Keep Bench and Judge separate

When LLM-as-judge is enabled, the Judge needs its own model and credentials:

BENCH_LLM_JUDGE=1
BENCH_JUDGE_MODEL=your-judge-model
BENCH_JUDGE_API_KEY=your-judge-key
BENCH_JUDGE_BASE_URL=https://your-judge-gateway.example/v1

Do not omit BENCH_JUDGE_MODEL just because the subject Agent can run, and do not silently reuse the subject credentials. The subject and Judge may use different Agents, but each credential must match the selected Agent.

Common mismatches

Symptom	Usual cause	Action
Claude Code returns 401	No `ANTHROPIC_API_KEY` / `ANTHROPIC_AUTH_TOKEN`, or the endpoint and model do not match	Run `claude -p` first, then check `ANTHROPIC_*`
Codex returns 404	The gateway only implements Anthropic Messages or Chat Completions	Use a provider that supports Responses API and set `wire_api = "responses"`
Qoder `auth.exchangeJobToken` fails	Another vendor’s API key was used as a Qoder PAT	Create and use a Qoder PAT from Qoder Integrations
CodeBuddy returns 401/404	A Claude Code Anthropic endpoint was used, or the provider rejected the model ID	Use `CODEBUDDY_BASE_URL`, `CODEBUDDY_API_KEY`, and the provider’s model ID; run `codebuddy -p` first
Eval skips immediately	Agent credentials never reached the Eval process	Check the user-level `.env` or current shell; a GUI client’s config alone is not enough

CodeBuddy custom models use CodeBuddy’s OpenAI-compatible configuration; Claude Code’s ANTHROPIC_* settings do not automatically become CodeBuddy settings.

Next steps

Eval quickstart — run your first evaluation from a Skill directory
Eval harness — environment variables, Docker, and task execution
comet eval command — CLI options and report paths

Quickstart

Methodology: Why evaluation drives Skill evolution

Real evaluation experiments

Advanced

Related commands

Eval Agent Startup Configuration

Where the user-level `.env` lives

Choose an Agent

Claude Code

What to prepare

Verify it on the host

Use an Anthropic-compatible gateway

Codex

What to prepare

Verify it on the host

Configure a custom Responses provider

Qoder

What to prepare

Verify it on the host

Configure Comet Eval

CodeBuddy

What to prepare

Extend Eval with a custom Agent

Minimal complete `adapter.yaml`

Keep Bench and Judge separate

Common mismatches

Next steps

​Where the user-level .env lives

​Choose an Agent

​Claude Code

​What to prepare

​Verify it on the host

​Use an Anthropic-compatible gateway

​Codex

​What to prepare

​Verify it on the host

​Configure a custom Responses provider

​Qoder

​What to prepare

​Verify it on the host

​Configure Comet Eval

​CodeBuddy

​What to prepare

​Extend Eval with a custom Agent

​Minimal complete adapter.yaml

​Keep Bench and Judge separate

​Common mismatches

​Next steps

Where the user-level `.env` lives

Choose an Agent

Claude Code

What to prepare

Verify it on the host

Use an Anthropic-compatible gateway

Codex

What to prepare

Verify it on the host

Configure a custom Responses provider

Qoder

What to prepare

Verify it on the host

Configure Comet Eval

CodeBuddy

What to prepare

Extend Eval with a custom Agent

Minimal complete `adapter.yaml`

Keep Bench and Judge separate

Common mismatches

Next steps