An SDK application evaluation targets the complete application.json package. It includes the workflow, entry, actual Skill dependencies, and validation conditions. Evaluating entry text alone cannot establish the behavior of the complete application.
Start an evaluation
Prepare the Eval Agent environment, then run:
You can also select an application target and create an evaluation in the local Dashboard’s Eval workspace. Review the Agent, model, and budget for this run. Comet Any’s optional Eval is bound to the actual compiled version.
Pin and reuse cases
Comet generates and freezes 2–4 cases. Subsequent retries and application revisions reuse the original cases so differences can be compared against the same inputs.
User decisions, rejection, revision, and recovery follow the actual cases. Case generation failures, invalid inputs, and host execution failures are recorded as such. Work that never ran does not count as a pass.
Read three kinds of evidence separately
Model Eval shows the steps and paths actually covered. Branches, external operations, and platforms that did not run remain uncovered. A small set of cases cannot establish coverage of the entire business scope.
Inspect and retry
Dashboard provides status, stages, cases, logs, reports, and configuration snapshots. You can stop executions you started, retry failed items with the original cases, and compare results against a compatible baseline.
After opening a linked Eval from Comet Any, you can return to the original authoring or application Run’s evidence page. Skipping model Eval is explicitly recorded and does not affect mandatory isolated artifact validation.
Credentials come only from the current execution environment or managed user configuration. They are not written to application packages, case inputs, reports, or generated Agent configuration.
Complete results and failed-case retries
Business failures and harness errors are recorded separately. Setup, cleanup, or execution errors leave the overall result incomplete and ineligible as a baseline even when cases have scores. Failed-case retries retain original case identities, preserve the full evaluation, and cannot independently establish full application verification. See reports and baselines. Last modified on October 22, 2026
