Comet baseline evaluation experiment

This page summarizes a real Comet workflow baseline evaluation. The experiment used more than 10 billion tokens to compare Comet version baselines. The subject Agent model was mimo-2.5 pro, the judge model was glm-5.2, and the Pass@5 experiment count was 240 runs in total. The goal is not to introduce a feature, but to show how to read a comparison report: confirm the data scope first, then inspect task outcomes, reliability metrics, rubric dimensions, cost, and failure attribution.

​

Experiment scope

This experiment compares three treatments: The experiment includes 16 Comet workflow tasks. Each treatment has 80 runs in the analysis set, which is roughly 5 repeated runs per task. The report covers task outcomes, rubric dimensions, pass@k / pass^k, cost, runtime overhead, and run-level failed checks.

​

Key data

The report conclusion says COMET_FULL_040_BETA has an overall workflow score of 0.89, above COMET_FULL_039 at 0.82, with no dimension regressing beyond the 0.05 tolerance. Strict overall pass shows another signal: COMET_FULL_040_BETA is 71/80, lower than COMET_FULL_039 at 76/80. This difference mainly comes from run-level workflow contract failures, not business failures in the task outcome matrix. Read the report by metric family: 0.4.0 beta is higher on weighted workflow quality, recovery, and some process-evidence dimensions; run-level Skill invocation contracts still need inspection.

​

pass@k and pass^k

pass@k is the probability that at least one of k attempts succeeds. It is a capability ceiling. pass^k is the probability that all k attempts succeed. It is a reliability floor. The main signal is the gap: COMET_FULL_040_BETA has pass@5 = 1.00 but pass^5 = 0. Success is observable across repeated attempts, but all-attempt reliability is not yet reached.

​

Task outcomes

The task matrix reflects whether the business task passed: The only task-level failure is comet-api-cache-ttl under COMET_FULL_039. By task outcome, 0.4.0 beta covers all 16 tasks. The task matrix only answers whether the task completed. It does not answer whether the Skill was invoked as expected, whether enough workflow evidence was preserved, or whether decision points were handled consistently. Use rubric dimensions and failed checks for those questions.

​

Rubric dimensions

0.4.0 beta has a higher weighted score, mainly from main_flow, gate_guard, and recovery_resilience: recovery_resilience has the largest delta in this report. It shows higher scoring for interruption recovery, state preservation, and recovery evidence in 0.4.0 beta. decision_point_compliance and skill_invocation did not improve. They point to two follow-up checks: whether decision points are reliably surfaced to the user, and whether dependency Skill invocation evidence reliably enters the report.

​

Cost and runtime

0.4.0 beta has lower total tokens, total cost, and average cost than 0.3.9: Runtime overhead is close, while tool calls differ: 0.4.0 beta uses more average turns, fewer tool calls, and roughly the same elapsed time as 0.3.9.

​

Failed checks

Run-level failed checks are concentrated around Skill invocation contracts: These failures are not always the same as business task failures. They mean the report did not reliably observe expected Skill invocation evidence. For workflow Skills, this matters because evaluation checks the final result and whether the process is traceable and recoverable.

​

LLM judge overlay

The LLM judge reread artifacts and independently scored three qualitative dimensions: These readings show that the rule-based rubric observes better process and recovery evidence in 0.4.0 beta; artifact content quality is close to 0.3.9 and slightly lower on some dimensions. Future changes can consider both signals: structured workflow evidence and the information density of proposal, design, tasks, and verify artifacts.

​

How to use these results

Treat this report as a baseline reading, not as a single pass/fail conclusion. If you continue 0.4.0 beta work, check two areas first: whether dependency Skill invocation evidence reliably enters stream-json, and whether decision points reliably ask for user confirmation. After changes, rerun the comparison with the same tasks and repetition count so task or sample differences do not look like version differences.

​

Integration with LangSmith/LangFuse

Comet Eval’s automated dual-agent architecture can integrate online with LangSmith/LangFuse environments, making experiments traceable and skills evolvable.

Manage your Skill baseline in LangSmith and view detailed performance metrics, latency, and token consumption.

Trace your Claude Code in LangSmith

Trace custom Rubric metrics with Pytest in LangSmith

​

Real Mimo token consumption

Before the experiment

After the experiment

One oddity is that the total consumed package quota appears to exceed 10 billion tokens, while the detailed figure below shows only 1.5 billion-plus. The reason is unknown. Since the actual package consumption exceeded the 10 billion-token quota, this page treats it as a 10 billion-plus token run.

​

Raw report

The embedded HTML report snapshot includes the full charts, task matrix, source evidence, raw vs analysis sensitivity, failed checks, and LLM judge overlay.

​

Next steps

• Scoring metrics and two-agent evaluation - Understand pass@k, pass^k, rubric scoring, and LLM judge.
• Reading eval reports - Learn how to use summary output, report JSON, and failed checks.
• Eval harness - Understand how evaluation runs and how stream-json evidence is collected.