The RC1 experiment was conducted on August 28, 2026. The 48 baseline samples of beta16 are from August 27-28, 2026
The existing experiments were directly reused this time without re-running. Both sides use the same 16 tasks and 3 repetitions, but
Native products and trajectory checks follow the contracts of their respective versions.
The final RC1 analysis set consists of 48 unique
task + run records. The previous four recovery attempts only produced Docker noise shells of harness_trigger_suspect/exit-125, and thus were excluded. The corresponding missing nodes have been precisely replaced by subsequent supplementary runs. It will not count repeatedly.The original RC1 snapshot was once recorded as 47/48 because the parser recognized the shell glob in find ... -path '*/.claude/skills/*' as an unexpected Skill named *. After fixing the parser and re-reading the same raw stdout, only comet-native is explicitly called. Therefore, this paper uses the corrected 48/48 analysis set without modifying the original snapshot.How to align in the experiment
The business tasks and business validators remain consistent; workflow checks respectively follow the real Native contracts of beta16 and RC1.
“strict pass” indicates that both the business validator and the corresponding version of the Native workflow check have passed. Business verification is listed separately to avoid mistaking the failure of workflow evidence for code or task failure.
Core result
Both task-level
pass@3 on both sides are 100%, meaning that each of the 16 tasks has been successful at least once. RC1 achieved two more strict passes and two more tasks passed all three times. The reduction in time consumption can only be regarded as directional evidence, as beta16 uses reused historical runs, and the two groups of samples are not controlled re-runs in the same time window.
Why look at pass@3 and pass^3 at the same time
pass@3 measures at least one success among three attempts and is suitable for observing ability coverage. The pass^3 measurement shows that all three observation runs were successful, making it suitable for observing the stability of repeated runs.
These two indicators distinguish between “the version can complete the task” and “the version stably completes the task every time”. This time, both sides cover all 16 tasks, but RC1 has 16/16 tasks that pass strictly three times, while beta16 has 14/16.
The execution efficiency of completing tasks
The efficiency main view only counts 46 groups of identical tasks that are strictly passed on both sides, avoiding misting early failure for low consumption. The number of model starts/restores indicates the number of times the evaluator enters the model under test. The Agent rounds are the cumulative rounds within these calls.
In the paired view with strict pass on both sides, the model of RC1 has fewer entries and a lower cumulative time consumption. However, the complete table also shows the costs: the Agent rounds, tool calls, total tokens, and average costs of RC1 are all higher. Therefore, this is a “shorter but not cheaper” execution portrait, and not all efficiency indicators have improved.
Both 48/48 trajectories on both sides retain peak context telemetry. The average peak context is RC1 102,470 tokens and beta16 104,208 tokens. This conclusion does not use the qualitative text of LLM Judge to replace hard checks.
The corrected model is time-consuming
The time consumption is recalculated from raw stdout: accumulate all the top layers ofresult.duration_ms for each sample. The calculation scope excludes the Docker preparation and business validator running time.
- RC1:48 observable samples, with an average of 391.260 seconds and a median of 369.053 seconds.
- beta16:48 observable samples, with an average of 503.552 seconds and a median of 476.481 seconds.
Task-level stability matrix
RC1 has 16/16 tasks with three strict passes; beta16 is 14/16. Both sides cover at least one success of all 16 tasks.
Failure attribution
The corrected RC1 analysis has no remaining strict failure; beta16 has two times. One failure in the original RC1 snapshot has been verified as a false positive by the same raw stdout and the fixed shell-glob parser:
beta16 had one pure workflow failure and one simultaneous failure of both business and workflow. Therefore, the business verification lines are RC1 48/48 and beta16 47/48 respectively.
About LLM Judge
This comparison only adopts deterministic tasks and workflow verification, the finalpassed state of each run, and the raw result time consumption. Do not use qualitative LLM Judge text instead of these checks.
How should we interpret this result
- The task coverage remains unchanged. Both the task-level
pass@3of RC1 and beta16 are 16/16. - RC1 was slightly more stable in this group of samples. ** strict pass@1 is 4.17 percentage points higher,
pass^3is 12.50 percentage points higher. - RC1 is faster but not cheaper. The average cost and total Token in the successfully paired samples increased, although the average and median of raw model duration decreased by approximately 22%. These are empirical trade-offs, not a single “comprehensively better”.
A strict retest after disabling memory and project rules
To eliminate the deviation caused by inconsistent model specifications in the previous round, the supplementary re-test only selects the most difficultcomet-full-workflow and runs it once. The main model was fixed as deepseek-v4-flash[1m], which was consistent with the RC1-on control. Close memory.learning and memory.retrieval simultaneously, and confirm plugins: [] in the startup snapshot. The re-test passed 18 out of 18 checks, and the business baseline was 6/6.
This result does not support the judgment that “the Token actually increases after the memory and rules are turned off.” The previous abnormal increase was due to the mixed comparison of
deepseek-v4-flash (200K context) and deepseek-v4-flash[1m] (1M context). After the model specification correction, the total Token and cost of the RC1-off sample were actually lower than the average of the three RC1-on samples.
This is still not a causal cost experiment for the project rule plugin: the current source code has removed comet.project-rules, and this startup snapshot does not load the plugin either. Therefore, it can only indicate the corrected RC1-off reference value and cannot infer from a sample how many tokens the plugin itself has added. The complete evidence and configuration snapshots can be found in the HTML report of the same format below.
HTML Visualization Report
The following embedded report focuses on presenting headline metrics, reliability metrics, paired execution efficiency, correction time consumption, the three-run matrix of 16 tasks, failure attribution, and comparison boundaries.Next step
- Run RC1 and beta16 again under the same service window and the same concurrent conditions, and further strengthen the conclusion of time consumption.
- Track the tool calls, rounds, tokens and cost increases of RC1 separately and evaluate them separately from the decrease in time consumption.
- Comet Baseline True Evaluation Experiment - View earlier Classic baseline comparisons
- Scoring Metrics and Dual-Agent Evaluation - Understand the differences between
pass@k,pass^k, hard validation, and LLM Judge