> ## Documentation Index > Fetch the complete documentation index at: https://docs.comet.rpamis.com/llms.txt > Use this file to discover all available pages before exploring further. # Reading eval reports (advanced) > Learn about the report locations, experiment id, key fields, failure attribution, rubric scoring, and next-step decisions produced by comet eval --html. This is advanced content. Come here to check failure attribution when a report fails. For day-to-day evaluation, the "You only need to focus on three things when reading reports" section in [Quickstart](/en/eval/quickstart) is enough. `comet eval ... --html` generates a browsable report. You don't need to read the underlying logs line by line — look at results, attribution, and artifacts first, then decide the next step. ## Report location `--html` asks the report to produce both markdown and HTML. The CLI output includes `Experiment` and `Report path`, commonly at: ```text theme={null} eval/local/logs/experiments//summary.html ``` The actual format of `` is `_`, for example `authoring_skill_smoke_20260625_143000`. The experiment name comes from the task name of the first parameterized test (`-` converted to `_`, defaulting to `experiment` when there are no tests). If the CLI output shows a placeholder, use the `Experiment` value from the same output to look it up. You can also find the corresponding experiment by browsing this directory: ```text theme={null} eval/local/logs/experiments/ ``` ## Report directory structure * `reports/_rep_report.json`: the complete result of each run, including `passed`, `checks_passed[]`, `checks_failed[]`, `events_summary` (tokens/cost/skills\_invoked/failure\_attribution). * `metadata.json`: `experiment_id`, `started_at`, `completed_at`, `total_runs`, `total_passed`, `treatments[]`, `report_outputs`. ## What to read first Once you open the report, prioritize these kinds of information rather than reading the underlying logs line by line: * **Whether the evaluation passed** (top of summary / CLI output) * **Failure attribution** — harness, workflow, task, or model * Whether the failed cases are related to the Skill's goal * Whether any expected artifact is missing (hard check) * Whether it's a path, manifest, or environment problem * Whether token / cost / duration are abnormal

Xiaoyu follows summary, report.json, and failure attribution clues to decide the next action

Read the summary first, then inspect individual reports, and finally use failure attribution to decide the next step

## Failure attribution The output of `comet eval ... --html` reports failure attribution: the report buckets failures into harness, workflow, task, and model. The attribution logic decides in order: | Attribution | Trigger condition | Meaning | | ----------- | ------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------ | | `harness` | required skill was never invoked; or generic profile with no skill run | Environment/dependency/path problem — the Skill never actually ran | | `task` | check involves artifact path / validator / task directory | Task definition or fixture problem | | `workflow` | check involves `.comet.yaml`/guard/state/transition/archive; or skill invocation contract failed | The Skill flow didn't meet expectations | | `model` | default fallback (task failed after observable workflow execution) | Unstable model behavior | ## Rubric scoring (informational) The report contains `[RUBRIC] : - ` lines and an aggregated `RubricAvg` column. This is an **informational** score — it doesn't directly determine pass/fail. The dimensions differ by profile: | Profile | Main dimensions | | ----------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `generic` | completion、skill\_invocation、artifact\_presence、instruction\_following、interaction\_compliance、efficiency、safety\_boundary | | `comet-workflow` | main\_flow、gate\_guard、skill\_invocation、spec\_drift、completion、efficiency、decision\_point\_compliance、artifact\_quality、recovery\_resilience | | `authoring-skill` | completion、generated\_package、resolved\_skill\_evidence、engine\_contract、workflow\_route\_conformance、authoring\_lanes、review\_gate、review\_readiness、skill\_invocation、artifact\_presence、safety\_boundary | ### Rubric columns in summary.md The Results table has one row per treatment, with columns in this order: `Checks` → individual rubric dimension columns → `RubricAvg` → `Turns/Duration/Tools/Tokens/Cost`. * **Dimension columns**: for a single run, shows that dimension's score (0.00–1.00); for multiple runs (reps), shows the mean. * **RubricAvg**: the **simple average** of all dimension scores for that run (including the `weighted_score` row) — a summary for quick horizontal comparison, computed differently from the `weighted_score` that each rubric produces using its own weights. * **weighted\_score**: the weighted total score output by the rubric itself (`Σ(dimension score × weight)/Σ(weights)`), appearing as a column. ### Aggregation when reps > 1 When you use `--count N` to run multiple repetitions, the summary adds an **"Aggregated by Treatment"** table: `Reps Passed` (number of passing repetitions), `Checks`, `Avg Turns/Duration`, `Tokens`, `Cost`, `Skills`, `Scripts`. Each repetition's pass/fail (zero `checks_failed` means pass) is aggregated and used to compute pass\@k. Rubric is a diagnostic tool, not a pass condition. Actual pass/fail is determined by validators (expected artifacts, test\_scripts) and whether the "required skill was invoked." A low rubric score with all checks passing still counts as a pass. ## pass\@k / pass^k (comparison reports) pass\@k / pass^k is not in `summary.md` but in the **comparison report** (`comparison_report.md`, produced by `compare_baselines.py`) under the `## pass@k / pass^k — capability vs reliability` section: * **pass\@k**: probability of at least one success out of k runs (capability ceiling) * **pass^k**: probability of all k runs succeeding (reliability floor) * **gap (pass\@k − pass^k)**: instability — can do it, but can't guarantee doing it right every time Multiple repetitions (`--count`) are required for this to be meaningful; a single run can only compute k=1. For the full formulas, meanings, and how to read the comparison table, see [Scoring metrics and dual-Agent evaluation](/en/eval/scoring). ## How to decide the next step | Report signal | Meaning | Next step | | ------------------------- | --------------------------------------------------------- | ------------------------------------------------------- | | Manifest read failed | Path or file doesn't exist | Fix `comet/eval.yaml` or the path | | `harness` attribution | Environment, dependencies, Docker, network | Check Docker, `ANTHROPIC_API_KEY`, `claude` CLI | | `workflow` attribution | Skill execution flow didn't meet expectations | Go back to `/comet-any` to refine the Skill | | `task` attribution | Task definition, validation condition, or fixture problem | Check the eval task definition and fixture | | `model` attribution | Unstable model behavior or tool usage | Re-run or reduce reliance on non-deterministic behavior | | Missing expected artifact | Task didn't produce the expected file | Check the task definition or Skill output logic | ## How to tell which stage failed ### collect failed Check first: * Whether the manifest path is correct * Whether `comet/eval.yaml` exists * Whether the recommended tasks in the manifest exist * Whether you're in the Comet repository root or passed the correct `--project` ### run failed First look at the failure attribution in the report and locate it by the buckets above. Then inspect `events_summary` in `reports/_rep_report.json`: * `skills_invoked` is empty → harness attribution, the Skill didn't run * `files_created` is missing the expected artifact → the Skill didn't produce the expected file * `total_tokens` is abnormally low → the Skill may have terminated early ### "Instant pass" in evaluation Almost certainly the environment wasn't ready and the suite was skipped. Check Docker, API key, `claude` CLI. ### HTML report not found First check the `Experiment` and `Report path` in the CLI output. If the path contains a placeholder, use the actual experiment id to look it up in the `eval/local/logs/experiments/` directory. ## How reports enter the release process **Do not** manually edit Bundle state, and **do not** manually write report paths into internal JSON. Let `/comet-any` or the Bundle backend record eval results, and let `comet creator status` read readiness. Rules for eval evidence entering readiness: | Situation | Can publish? | | ---------------------------------------- | ---------------------------------------- | | No eval evidence | No | | Eval failed | No | | Eval evidence corresponds to an old hash | No | | Eval passed and hash matches | Can proceed to review / publish decision | ## Next steps * [Runtime check](/en/eval/runtime) — Distinguishing `comet eval` and `comet skill check` * [comet eval command](/en/cli/eval) — Complete options and subcommand reference * [Publishing and distributing Skills](/en/skill-creator/publishing) — How eval evidence drives readiness