mimo-2.5 pro, the judge model was glm-5.2, and the Pass@5 experiment count was 240 runs in total. The goal is not to introduce a feature, but to show how to read a comparison report: confirm the data scope first, then inspect task outcomes, reliability metrics, rubric dimensions, cost, and failure attribution.
This page is based on one report snapshot. In the report,
CONTROL is a business-completion baseline without the Comet Skill. It does not require Comet workflow artifacts, so it is useful as a business baseline but not as evidence that the Comet workflow executed correctly.Experiment scope
This experiment compares three treatments:
The experiment includes 16 Comet workflow tasks. Each treatment has 80 runs in the analysis set, which is roughly 5 repeated runs per task. The report covers task outcomes, rubric dimensions,
pass@k / pass^k, cost, runtime overhead, and run-level failed checks.
Key data
The report conclusion saysCOMET_FULL_040_BETA has an overall workflow score of 0.89, above COMET_FULL_039 at 0.82, with no dimension regressing beyond the 0.05 tolerance.
Strict overall pass shows another signal: COMET_FULL_040_BETA is 71/80, lower than COMET_FULL_039 at 76/80. This difference mainly comes from run-level workflow contract failures, not business failures in the task outcome matrix.
Read the report by metric family: 0.4.0 beta is higher on weighted workflow quality, recovery, and some process-evidence dimensions; run-level Skill invocation contracts still need inspection.
pass@k and pass^k
pass@k is the probability that at least one of k attempts succeeds. It is a capability ceiling. pass^k is the probability that all k attempts succeed. It is a reliability floor.
The main signal is the gap:
COMET_FULL_040_BETA has pass@5 = 1.00 but pass^5 = 0. Success is observable across repeated attempts, but all-attempt reliability is not yet reached.
Task outcomes
The task matrix reflects whether the business task passed:
The only task-level failure is
comet-api-cache-ttl under COMET_FULL_039. By task outcome, 0.4.0 beta covers all 16 tasks.
The task matrix only answers whether the task completed. It does not answer whether the Skill was invoked as expected, whether enough workflow evidence was preserved, or whether decision points were handled consistently. Use rubric dimensions and failed checks for those questions.
Rubric dimensions
0.4.0 beta has a higher weighted score, mainly frommain_flow, gate_guard, and recovery_resilience:
recovery_resilience has the largest delta in this report. It shows higher scoring for interruption recovery, state preservation, and recovery evidence in 0.4.0 beta.
decision_point_compliance and skill_invocation did not improve. They point to two follow-up checks: whether decision points are reliably surfaced to the user, and whether dependency Skill invocation evidence reliably enters the report.
Cost and runtime
0.4.0 beta has lower total tokens, total cost, and average cost than 0.3.9:
Runtime overhead is close, while tool calls differ:
0.4.0 beta uses more average turns, fewer tool calls, and roughly the same elapsed time as 0.3.9.
Failed checks
Run-level failed checks are concentrated around Skill invocation contracts:
These failures are not always the same as business task failures. They mean the report did not reliably observe expected Skill invocation evidence. For workflow Skills, this matters because evaluation checks the final result and whether the process is traceable and recoverable.
LLM judge overlay
The LLM judge reread artifacts and independently scored three qualitative dimensions:
These readings show that the rule-based rubric observes better process and recovery evidence in 0.4.0 beta; artifact content quality is close to 0.3.9 and slightly lower on some dimensions. Future changes can consider both signals: structured workflow evidence and the information density of proposal, design, tasks, and verify artifacts.
How to use these results
Treat this report as a baseline reading, not as a single pass/fail conclusion.
If you continue 0.4.0 beta work, check two areas first: whether dependency Skill invocation evidence reliably enters stream-json, and whether decision points reliably ask for user confirmation. After changes, rerun the comparison with the same tasks and repetition count so task or sample differences do not look like version differences.
Integration with LangSmith/LangFuse
Comet Eval’s automated dual-agent architecture can integrate online with LangSmith/LangFuse environments, making experiments traceable and skills evolvable.
Manage your Skill baseline in LangSmith and view detailed performance metrics, latency, and token consumption.
Trace your Claude Code in LangSmith
Trace custom Rubric metrics with Pytest in LangSmith
Real Mimo token consumption
Before the experiment
After the experiment
The total consumption of Credits here exceeds 10 billion. Below, it is equivalent to 150 million + Tokens.Raw report
The embedded HTML report snapshot includes the full charts, task matrix, source evidence, raw vs analysis sensitivity, failed checks, and LLM judge overlay.Next steps
- Scoring metrics and two-agent evaluation - Understand
pass@k,pass^k, rubric scoring, and LLM judge. - Reading eval reports - Learn how to use summary output, report JSON, and failed checks.
- Eval harness - Understand how evaluation runs and how stream-json evidence is collected.