Real Evaluation of Comet Native and 0.4.0 Classic

This experiment answers a focused question: when the model is already strong enough (such as Fable 5 and GPT-5.6), can Comet Native stay lighter than 0.4.0 Classic while retaining requirements, state, verification, and archive evidence? We aligned the Native sample size to the pass@3 experiment that was once run in 0.4.0: using the same 16 business tasks, each task was repeated three times. There were a total of 96 runs in the two groups.

​

How to align in the experiment

Alignment experiments do not mean applying the original checks of Classic to Native. The business goals and business validators remain consistent, while workflow checks respectively follow the real contracts of the two modes. Some task prompts still contain Classic terms such as Open, Design, Build, Verify, and Archive. Native treatment explicitly requires the model to retain business requirements but map them to Shape, Build, Verify, and Archive. Do not create OpenSpec, Superpowers, or the old change-local .comet process products. This can reduce the prior bias of the task copy on the Classic. All 48 samples of 0.4.0 were conservatively marked as flagged due to the words “outer runner”, “API”, or “environment failure” in the log, but still all entered the analysis set. flagged is merely an observability marker and does not indicate that the sample has been excluded or the determination has failed. This paper retains its strict pass results and incorporates this observability difference into the comparison constraints.

​

Core result

This set of results can be viewed from two perspectives. Both modes’ pass@3 is 100%, indicating that each task has been successful at least once out of the three attempts. If Native’s pass^3 is higher, it indicates that it has better stability in repeated runs: 14 out of 16 tasks pass three consecutive times, while 0.4.0 Classic has 12.

​

Why look at pass@3 and pass^3 at the same time

pass@3 measures at least one success in three attempts and is suitable for observing ability coverage. pass^3 measures that all three attempts were successful and is suitable for observing the lower limit of reliability. If you only look at pass@3 = 100%, you won’t see the actual failures that exist in the 48 runs. Native has two strict failures; There are five times for 0.4.0 Classic. The two indicators must be examined together to distinguish between “the model is capable of completing” and “the model can complete stably every time”.

​

The execution efficiency of completing tasks

The efficiency main view only counts the repetition of the same task that both Native and Classic have passed, totaling 41 groups. This way, operations that fail prematurely and thus consume less will not be mistaken for efficiency improvements. A “model start/recovery” indicates that the evaluator starts the model under test or re-enters the same task after the user responds and undergoes a cold recovery. It is different from calling the internal cumulative Agent rounds. The 48 runs of Native all retained message-by-message context usage: the average peak was 34,771 tokens, the median peak was 34,568, and the maximum was 42,012. The average peak accounts for 17.4% of the 200k window. Only 9/48 of the old Classic trajectory saved data of the same granularity, so the context occupancy improvement ratio of the two Workflows cannot be reliably calculated.

​

Correcting the time-consuming caliber

In the early report, Duration only retained the last segment of the result event. When an automatic interaction involves multiple rounds of model calls, this will miss the time consumption of the previous rounds, resulting in unreasonable 4-second or 5-second samples. This comparison is changed to accumulate all the top layers of result.duration_ms for each sample:

• Native: 48 observable samples, with an average of 183.971 seconds and a median of 174.493 seconds.
• 0.4.0 Classic: 47 observable samples, with an average of 352.375 seconds and a median of 328.192 seconds; Another run failed before generating a result.
• Neither of them is included in the Docker preparation time and the task validator runtime.

Under this standard, the average time consumption of Native is reduced by 47.8%, and the median time consumption is reduced by 46.8%. This is in line with the design goals of Native to reduce phase switching, dependency skills, and repetitive process reads.

​

Failure attribution

Neither of the two failures of Native was a failure in business implementation: The five failures of 0.4.0 Classic include one business failure and four process contract failures: This reflects the different failure situations of the two sets of models. The Classic process is more dependent. Failure may occur at any stage of the autonomous Skill, phase, OpenSpec, or dependent Skill. The Native process has fewer steps, but it still retains the final state evidence check. If the code is completed but the brief, specification or verification is incomplete, it will still not be counted as a strict pass.

​

About LLM Judge

This conclusion does not use the LLM Judge qualitative text from the old Native report. Upon review, it was found that when generating this report, Judge would read the _test_results.json transport file before adaptation, thereby mistakenly identifying the old path check within it as the Native final product. Therefore, this article only adopts:

• Hard checks for task validator and workflow validator;
• The final passed status of each run;
• The cumulative time consumption in the original top-level result event;
• Traceable causes of task-level failure.

The subsequent evaluation code has excluded _test_context.json and _test_results.json, but the fixed code will not be used to reverse rewrite the qualitative Judge conclusion of this historical experiment.

​

How should we interpret this result

This alignment experiment supports the following judgments:

1. Light execution does not sacrifice task coverage. The task-level pass@3 for both Native and 0.4.0 Classic is 100%.
2. The repetitive operation of Native is more stable. ** strict pass@1 is 6.25 percentage points higher, pass^3 is 12.5 percentage points higher.
3. The model execution of Native is significantly shorter. The revised average and median have both decreased by approximately 47%, but they still need to be retested in exactly the same running window to form a stronger delayed causal conclusion.

It does not support the conclusion that “Classic is no longer necessary”. Native serves Fable 5, GPT-5.6 and similar strong models: These models can perform complex code reasoning on their own, but still require requirement clarification, status, checking and archiving. Classic continues to serve models and tasks with lower capability levels that require more detailed stage guidance and stronger process constraints.

​

HTML Visualization Report

The following embedded report centrally presents headline metric, reliability metrics, rounds of task completion, tool invocation, Token, cost, correction time consumption, three-run matrix of 16 tasks, failure attribution, and comparison boundaries.

​

Next step

• Running the two modes again under the same running window and the same concurrent conditions further tightens the conclusion of time consumption.
• Continue to enhance the integrity of Native final state evidence, with a focus on checking the archiving consistency of brief, specification, and verification.
• Comet Baseline True Evaluation Experiment - View the complete baseline comparison between 0.3.9 and 0.4.0 Classic
• Scoring Metrics and Dual-Agent Evaluation - Understand the differences between pass@k, pass^k, hard validation, and LLM Judge