> ## Documentation Index
> Fetch the complete documentation index at: https://docs.comet.rpamis.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Eval Agent Startup Configuration

> Prepare the CLI, authentication, endpoint, and model settings for Claude Code, Codex, Qoder, or CodeBuddy before running comet eval.

`comet eval` starts the selected Agent inside an isolated Docker container. Agent login and model protocols are not interchangeable: Claude Code can use an Anthropic-compatible gateway, Codex requires an OpenAI Responses API provider, Qoder requires its own login session or Personal Access Token, and CodeBuddy uses its own API configuration for custom models.

This page covers the Agent prerequisites only. The subject (Bench) and LLM-as-judge still use separate `BENCH_*` and `BENCH_JUDGE_*` settings; see [Eval harness](/en/eval/harness#environment-variable-reference) for the complete variable reference.

<Warning>
  Never put real credentials in a Skill, manifest, public repository, Dockerfile, or report. Prefer
  the current shell environment, or the user-level <code>%USERPROFILE%\\.comet\eval\\.env</code> /{' '}
  <code>\~/.comet/eval/.env</code>; these files are excluded from the published package.
</Warning>

## Where the user-level `.env` lives

Ordinary users only need the `.env` in their user directory. You do not need to clone the Comet
repository or edit the `eval/.env` inside an installed package:

* Windows: `%USERPROFILE%\\.comet\\eval\\.env`
* macOS/Linux: `~/.comet/eval/.env`

When `comet eval` runs for the first time and this file is missing, the CLI creates a commented
template containing all Eval parameters and prints the exact path. Edit it and run the command
again. Existing files are never overwritten, and environment variables in the current shell have
higher priority. The complete template is grouped into subject, independent Judge, Claude Code,
Codex, Qoder, CodeBuddy, LangSmith, and Langfuse settings; edit the generated template directly
instead of creating a parameter list by hand.

## Choose an Agent

```bash theme={null}
# Claude Code (default)
comet eval ./my-skill --agent claude-code --quick --html

# OpenAI Codex
comet eval ./my-skill --agent codex --quick --html

# Qoder
comet eval ./my-skill --agent qoder --quick --html

# CodeBuddy
comet eval ./my-skill --agent codebuddy --quick --html
```

Verify each Agent directly on the host before running Eval. This separates “the Agent is not logged in” from “the Skill evaluation failed.”

## Claude Code

### What to prepare

* The `claude` CLI available on PATH;
* either an interactive first-use login or `ANTHROPIC_API_KEY`;
* when using an Anthropic-compatible gateway, `ANTHROPIC_AUTH_TOKEN`, `ANTHROPIC_BASE_URL`, and a model name.

The official Claude Code quickstart says that the first `claude` launch opens the login flow; setting `ANTHROPIC_API_KEY` skips the login prompt and asks you to approve the key instead. For another LLM gateway, the official configuration uses `ANTHROPIC_AUTH_TOKEN` and `ANTHROPIC_BASE_URL`.

Official references: [Claude Code overview](https://code.claude.com/docs/en/overview) and [LLM gateway](https://code.claude.com/docs/en/llm-gateway).

### Verify it on the host

```bash theme={null}
claude --version
claude -p "Reply with exactly OK" --output-format json
```

### Use an Anthropic-compatible gateway

```dotenv theme={null}
BENCH_EVAL_AGENT=claude-code
BENCH_MODEL=your-anthropic-compatible-model
BENCH_BASE_URL=https://your-anthropic-gateway.example/v1
BENCH_API_KEY=your-subject-key

# Or use Claude Code's native variables directly.
ANTHROPIC_AUTH_TOKEN=your-subject-key
ANTHROPIC_BASE_URL=https://your-anthropic-gateway.example/v1
ANTHROPIC_MODEL=your-anthropic-compatible-model
```

Eval maps the common settings to Claude Code's native variables. Explicit `ANTHROPIC_*` values take precedence over common `BENCH_*` fallbacks.

## Codex

### What to prepare

* The `codex` CLI available on PATH;
* an API key or login session that can access an OpenAI Responses API;
* a model name accepted by that provider;
* for a custom provider, a user-level `~/.codex/config.toml` entry declaring `model_provider`, `base_url`, `env_key`, and `wire_api = "responses"`.

Codex custom providers are configured in the user-level `config.toml`. The official reference defines `base_url` as the provider API address, `env_key` as the environment variable supplying the key, and `responses` as the supported `wire_api` value. Do not give Codex an Anthropic Messages endpoint; the provider must expose the Responses API, normally at `<base-url>/responses`.

Official reference: [Codex configuration reference](https://developers.openai.com/codex/config-reference/).

### Verify it on the host

```bash theme={null}
codex --version
codex exec --json --yolo "Reply with exactly OK"
```

### Configure a custom Responses provider

Set the key before starting Codex:

```bash theme={null}
# macOS / Linux
export OPENAI_API_KEY=your-subject-key

# Windows PowerShell
$env:OPENAI_API_KEY = "your-subject-key"
```

Example user-level `~/.codex/config.toml`:

```toml theme={null}
model = "your-responses-model"
model_provider = "comet-eval"

[model_providers.comet-eval]
name = "Comet Eval provider"
base_url = "https://your-openai-compatible-gateway.example/v1"
env_key = "OPENAI_API_KEY"
wire_api = "responses"
requires_openai_auth = false
```

The corresponding Comet Eval `.env` is:

```dotenv theme={null}
BENCH_EVAL_AGENT=codex
OPENAI_API_KEY=your-subject-key
OPENAI_BASE_URL=https://your-openai-compatible-gateway.example/v1
OPENAI_MODEL=your-responses-model

# You can also use the common subject settings; Codex maps them to OPENAI_*.
# BENCH_API_KEY=your-subject-key
# BENCH_BASE_URL=https://your-openai-compatible-gateway.example/v1
# BENCH_MODEL=your-responses-model
```

At runtime, Eval creates a temporary container-local `config.toml` that contains only `env_key = "OPENAI_API_KEY"`; it does not save the real key. The provider must still support the Responses API. An endpoint that only supports `/chat/completions` or Anthropic `/messages` cannot be used directly by Codex.

## Qoder

### What to prepare

* The `qodercli` CLI available on PATH;
* a Qoder login session or a Qoder Personal Access Token (PAT);
* for automation and Eval, `QODER_PERSONAL_ACCESS_TOKEN` is recommended.

Qoder's official documentation supports two CLI login paths: run `qodercli`, then enter `/login` to use browser login or paste a Qoder PAT; for non-interactive startup, set `QODER_PERSONAL_ACCESS_TOKEN`. Create a PAT at [Qoder Account → Integrations](https://qoder.com/account/integrations).

Official references: [Qoder CLI quick start](https://docs.qoder.com/en/cli/quick-start) and [Qoder CLI authentication](https://docs.qoder.com/en/cli/sdk/authentication).

### Verify it on the host

```bash theme={null}
qodercli --version
qodercli
# At the Qoder prompt, enter:
/login
```

For a non-interactive check:

```bash theme={null}
# macOS / Linux
QODER_PERSONAL_ACCESS_TOKEN=your-qoder-pat \
  qodercli -p "Reply with exactly OK" --output-format stream-json --yolo

# Windows PowerShell
$env:QODER_PERSONAL_ACCESS_TOKEN = "your-qoder-pat"
qodercli -p "Reply with exactly OK" --output-format stream-json --yolo
```

### Configure Comet Eval

```dotenv theme={null}
BENCH_EVAL_AGENT=qoder
QODER_PERSONAL_ACCESS_TOKEN=your-qoder-pat
QODER_MODEL=your-qoder-model

# Or use the common model setting.
# BENCH_MODEL=your-qoder-model
```

The Qoder PAT is Qoder's own login credential, not an Anthropic or OpenAI API key. Do not put the Claude Code `ANTHROPIC_AUTH_TOKEN` or another model gateway key into `QODER_PERSONAL_ACCESS_TOKEN`; Qoder will reject it during startup. The current Eval setup does not require Claude Code's `ANTHROPIC_BASE_URL` for Qoder.

Eval injects `QODER_PERSONAL_ACCESS_TOKEN` only into the current container process and uses an isolated temporary Qoder config root; it does not read or mount the host's Qoder login files.

## CodeBuddy

### What to prepare

* The `codebuddy` CLI available on PATH;
* `CODEBUDDY_AUTH_TOKEN` when using CodeBuddy platform authentication;
* `CODEBUDDY_API_KEY`, `CODEBUDDY_BASE_URL`, and a provider-supported model name when using a third-party or OpenAI-compatible model.

Select a CodeBuddy custom model with `--model <model-id>`. `CODEBUDDY_MODEL` sets the default model and `CODEBUDDY_BASE_URL` overrides the request endpoint. Do not reuse Claude Code's `ANTHROPIC_BASE_URL` directly for CodeBuddy, especially an `/anthropic` or `/messages` endpoint; the model ID must also be one accepted by CodeBuddy/the provider rather than a Claude-specific model suffix.

CodeBuddy's host-side custom model registry is `~/.codebuddy/models.json` or the workspace-level `.codebuddy/models.json`. Its `url` should be the complete OpenAI-compatible `/chat/completions` URL, and `apiKey` can reference `${CODEBUDDY_API_KEY}`. Never put the real key in a published file. For example:

```json theme={null}
{
  "models": [
    {
      "id": "your-model-id",
      "name": "Your Model",
      "vendor": "custom",
      "apiKey": "${CODEBUDDY_API_KEY}",
      "url": "https://your-openai-compatible-gateway.example/v1/chat/completions"
    }
  ],
  "availableModels": ["your-model-id"]
}
```

Eval does not mount the host's `models.json` or CodeBuddy login directory. It creates a temporary `settings.json` inside the container and passes the native environment variables together with `--model`. Verify the provider directly first:

```bash theme={null}
CODEBUDDY_API_KEY=your-subject-key \
CODEBUDDY_BASE_URL=https://your-openai-compatible-gateway.example/v1 \
CODEBUDDY_MODEL=your-model-id \
  codebuddy -p "Reply with exactly OK" --model your-model-id --output-format json -y
```

Then put the same values in the user-level Eval `.env`:

```dotenv theme={null}
BENCH_EVAL_AGENT=codebuddy
CODEBUDDY_API_KEY=your-subject-key
CODEBUDDY_BASE_URL=https://your-openai-compatible-gateway.example/v1
CODEBUDDY_MODEL=your-model-id

# You can also use the common subject settings; Eval maps them to CODEBUDDY_*.
# BENCH_API_KEY=your-subject-key
# BENCH_BASE_URL=https://your-openai-compatible-gateway.example/v1
# BENCH_MODEL=your-model-id

# Optional: override CodeBuddy's model roles.
# CODEBUDDY_SMALL_FAST_MODEL=your-fast-model-id
# CODEBUDDY_BIG_SLOW_MODEL=your-reasoning-model-id
# CODEBUDDY_CODE_SUBAGENT_MODEL=your-subagent-model-id
```

Official references: [CodeBuddy models](https://www.codebuddy.cn/docs/cli/models), [environment variables](https://www.codebuddy.cn/docs/cli/env-vars), and [CLI reference](https://www.codebuddy.cn/docs/cli/cli-reference).

At runtime, Eval injects credentials only into the current container process. The temporary `settings.json` uses `apiKeyHelper` to read the environment, and the container-local config directory is destroyed after the run; real keys do not enter reports, manifests, or the Skill workspace.

## Extend Eval with a custom Agent

An Agent that is not one of the built-ins is registered explicitly with an `adapter.yaml` in the user
directory. Putting an executable on `PATH` is not enough, and the adapter does not require a Comet source
checkout:

```text theme={null}
# macOS/Linux
~/.comet/eval/adapters/<agent-id>/adapter.yaml

# Windows
%USERPROFILE%\\.comet\\eval\\adapters\\<agent-id>\\adapter.yaml
```

You can set `COMET_EVAL_ADAPTERS_DIR` in the user-level `%USERPROFILE%\\.comet\\eval\\.env` /
`~/.comet/eval/.env` to override the registry directory. `<agent-id>` must start with a lowercase letter,
contain only lowercase letters, digits, and hyphens, and be 2–32 characters long.

### Minimal complete `adapter.yaml`

```yaml theme={null}
apiVersion: comet.eval.agent/v1alpha1
kind: EvalAgentAdapter
metadata:
  id: my-agent
  version: 1.0.0
runtime:
  executable: my-agent
  install:
    kind: npm
    package: my-agent
    version: latest
credentials:
  - MY_AGENT_API_KEY
  - MY_AGENT_AUTH_TOKEN
modelEnv: MY_AGENT_MODEL
baseUrlEnv: MY_AGENT_BASE_URL
capabilities:
  singleTurn: true
  resume: true
  structuredEvents: true
  telemetry: false
  skillInvocationEvidence: true
```

Field rules:

* `runtime.executable` is the command started inside the container; an npm/pip package must expose
  this executable. `runtime.install.kind` supports `npm`, `pip`, and `none`; `none` means the command
  is already available in the base image.
* `credentials` may declare at most two environment-variable names, never credential values. For a
  subject Agent, only these names are forwarded into the container. For a Judge,
  `BENCH_JUDGE_API_KEY` maps to the first name and `BENCH_JUDGE_AUTH_TOKEN` maps to the second name.
* `modelEnv` and `baseUrlEnv` are optional custom environment-variable names. A custom Agent does not
  automatically inherit `BENCH_API_KEY`, `BENCH_MODEL`, or `BENCH_BASE_URL`; use the declared names,
  CLI options, or manifest configuration.
* `resume` enables `auto_user` multi-turn interaction; `structuredEvents` requires JSONL output;
  `skillInvocationEvidence` requires explicit Skill-invocation events; and `telemetry: false` allows
  token/cost fields to be `N/A`. A real evaluation requires `singleTurn`, `resume`, `structuredEvents`,
  and `skillInvocationEvidence` to be `true`.

The custom CLI must accept Eval's invocation contract:

```text theme={null}
<executable> -p "<prompt>" --output-format stream-json --model <model> [--resume <session-id>]
```

Write one structured event per JSONL line to stdout. To let the rubric confirm that a Skill was invoked,
the output must also contain an event such as:

```json theme={null}
{"type":"skill_invocation","skill":"my-skill"}
```

Add the variables declared by `adapter.yaml` to the generated user-level `.env`:

```dotenv theme={null}
BENCH_EVAL_AGENT=my-agent
MY_AGENT_API_KEY=your-subject-key
MY_AGENT_MODEL=your-model
MY_AGENT_BASE_URL=https://your-provider.example/v1

# Use separate credentials when it is also the independent Judge:
# BENCH_LLM_JUDGE=1
# BENCH_JUDGE_AGENT=my-agent
# BENCH_JUDGE_API_KEY=your-judge-key
# BENCH_JUDGE_MODEL=your-judge-model
# BENCH_JUDGE_BASE_URL=https://your-judge-provider.example/v1
```

Run a static check first, then a real evaluation:

```bash theme={null}
comet eval ./my-skill --collect --agent my-agent
comet eval ./my-skill --quick --agent my-agent --model your-model
```

An adapter can be distributed as an npm/pip package or wrapper CLI, but `adapter.yaml` currently does not
support custom argument templates, host configuration-directory mounts, or arbitrary Docker commands.
Real credentials must remain in each user's own `.env` or current shell.

## Keep Bench and Judge separate

When LLM-as-judge is enabled, the Judge needs its own model and credentials:

```dotenv theme={null}
BENCH_LLM_JUDGE=1
BENCH_JUDGE_MODEL=your-judge-model
BENCH_JUDGE_API_KEY=your-judge-key
BENCH_JUDGE_BASE_URL=https://your-judge-gateway.example/v1
```

Do not omit `BENCH_JUDGE_MODEL` just because the subject Agent can run, and do not silently reuse the subject credentials. The subject and Judge may use different Agents, but each credential must match the selected Agent.

## Common mismatches

| Symptom                             | Usual cause                                                                             | Action                                                                                               |
| ----------------------------------- | --------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------- |
| Claude Code returns 401             | No `ANTHROPIC_API_KEY` / `ANTHROPIC_AUTH_TOKEN`, or the endpoint and model do not match | Run `claude -p` first, then check `ANTHROPIC_*`                                                      |
| Codex returns 404                   | The gateway only implements Anthropic Messages or Chat Completions                      | Use a provider that supports Responses API and set `wire_api = "responses"`                          |
| Qoder `auth.exchangeJobToken` fails | Another vendor's API key was used as a Qoder PAT                                        | Create and use a Qoder PAT from Qoder Integrations                                                   |
| CodeBuddy returns 401/404           | A Claude Code Anthropic endpoint was used, or the provider rejected the model ID        | Use `CODEBUDDY_BASE_URL`, `CODEBUDDY_API_KEY`, and the provider's model ID; run `codebuddy -p` first |
| Eval skips immediately              | Agent credentials never reached the Eval process                                        | Check the user-level `.env` or current shell; a GUI client's config alone is not enough              |

<Info>
  CodeBuddy custom models use CodeBuddy's OpenAI-compatible configuration; Claude Code's
  `ANTHROPIC_*` settings do not automatically become CodeBuddy settings.
</Info>

## Next steps

* [Eval quickstart](/en/eval/quickstart) — run your first evaluation from a Skill directory
* [Eval harness](/en/eval/harness) — environment variables, Docker, and task execution
* [comet eval command](/en/cli/eval) — CLI options and report paths