> ## Documentation Index
> Fetch the complete documentation index at: https://docs.comet.rpamis.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Evaluate SDK applications

> Pin complete applications and cases, check actual workflow behavior, and distinguish local, model, and host evidence.

An SDK application evaluation targets the complete `application.json` package. It includes the workflow, entry, actual Skill dependencies, and validation conditions. Evaluating entry text alone cannot establish the behavior of the complete application.

## Start an evaluation

Prepare the [Eval Agent environment](/en/eval/agent-setup.md), then run:

```bash theme={null}
comet eval ./weekly-report/application.json --agent codex --html
```

You can also select an application target and create an evaluation in the local Dashboard's **Eval** workspace. Review the Agent, model, and budget for this run. Comet Any's optional Eval is bound to the actual compiled version.

## Pin and reuse cases

Comet generates and freezes 2–4 cases. Subsequent retries and application revisions reuse the original cases so differences can be compared against the same inputs.

User decisions, rejection, revision, and recovery follow the actual cases. Case generation failures, invalid inputs, and host execution failures are recorded as such. Work that never ran does not count as a pass.

## Read three kinds of evidence separately

| Evidence | What it establishes |
| - | - |
| Local validation | Whether the pinned package, actual files, output contracts, and declared business samples pass |
| Model Eval | Actual execution behavior of the selected Agent, model, and frozen cases |
| Host validation | Actual loading, Hooks, handoffs, and task execution on a specified platform |

Model Eval shows the steps and paths actually covered. Branches, external operations, and platforms that did not run remain uncovered. A small set of cases cannot establish coverage of the entire business scope.

## Inspect and retry

Dashboard provides status, stages, cases, logs, reports, and configuration snapshots. You can stop executions you started, retry failed items with the original cases, and compare results against a compatible baseline.

After opening a linked Eval from Comet Any, you can return to the original authoring or application Run's evidence page. Skipping model Eval is explicitly recorded and does not affect mandatory isolated artifact validation.

Credentials come only from the current execution environment or managed user configuration. They are not written to application packages, case inputs, reports, or generated Agent configuration.

## Complete results and failed-case retries

Business failures and harness errors are recorded separately. Setup, cleanup, or execution errors leave the overall result incomplete and ineligible as a baseline even when cases have scores. Failed-case retries retain original case identities, preserve the full evaluation, and cannot independently establish full application verification. See [reports and baselines](/en/eval/reports.md#results-baselines-and-application-distribution).


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.