pass@3 experiment previously run for 0.4.0: the same 16 business tasks, repeated 3 times each. The comparison contains 96 runs in total.
Native and Classic are separate workflows for different model capability levels. Native does not
upgrade into Classic. This page compares business completion, reliability, and model duration on
the same tasks; it does not require both modes to produce identical workflow artifacts.
How the experiment is aligned
Alignment does not mean applying Classic checks unchanged to Native. The business objectives and task validators stay the same. Workflow checks follow the actual contract of each mode.
Some task prompts still use Classic terms such as Open, Design, Build, Verify, and Archive. The Native treatment tells the model to preserve the business requirements while mapping them to Shape, Build, Verify, and Archive. It must not create OpenSpec, Superpowers, or
.comet workflow artifacts. This reduces prompt bias toward Classic without changing the task itself.
All 48 Classic samples were conservatively marked flagged because their logs mention outer-runner, API, or environment failures. They still belong to the analysis set; flagged does not mean excluded or failed. This page preserves their strict pass results and treats the observability difference as a limitation.
Headline results
These results have two layers. Both modes reach
pass@3 = 100%, so every task succeeds at least once in three attempts. Native has the higher pass^3, which indicates better repeated-run reliability: 14 of 16 tasks pass all three Native runs, compared with 12 of 16 for 0.4.0 Classic.
Why pass@3 and pass^3 both matter
pass@3 measures whether at least one of three attempts succeeds. It describes capability coverage. pass^3 measures whether all three observed attempts succeed. It describes a strict reliability floor.
Looking only at pass@3 = 100% would hide failures among the 48 runs. Native has two strict failures; 0.4.0 Classic has five. The two metrics must be read together to distinguish “the model can complete this task” from “the model completes it consistently.”
Completed-task execution efficiency
The primary efficiency view includes only the 41 matching task repetitions that both Native and Classic passed. This prevents an early failure from looking artificially efficient. One “model start/resume” means that the evaluator starts the subject model or re-enters the same task after a user answer or cold resume; it is distinct from the Agent turns accumulated inside those invocations.
All 48 Native runs preserve per-message context usage: the mean peak is 34,771 tokens, the median peak is 34,568, and the maximum is 42,012. The mean peak occupies 17.4% of a 200k context window. Only 9 of 48 historical Classic traces preserve equivalent telemetry, so the experiment cannot support a reliable workflow-to-workflow context-usage improvement percentage.
Corrected duration methodology
The earlier report retained only the finalresult event in its Duration column. When an automatic interaction contains multiple model calls, that omits the earlier calls and can produce implausible four- or five-second samples.
This comparison instead sums every top-level result.duration_ms within a sample:
- Native: 48 observable samples, mean 183.971 seconds and median 174.493 seconds.
- 0.4.0 Classic: 47 observable samples, mean 352.375 seconds and median 328.192 seconds; one run failed before producing a result event.
- Neither figure includes Docker setup or task-validator execution time.
Failure attribution
Neither Native failure is a business-implementation failure:
The five 0.4.0 Classic failures contain one business failure and four workflow-contract failures:
This shows the different failure surfaces. Classic has more workflow dependencies, so a run can fail at the main Skill, a phase, OpenSpec, or a dependency Skill. Native has a narrower workflow surface but keeps a terminal-evidence check: completed code without complete brief, specification, or verification evidence is still not a strict pass.
About the LLM Judge
This comparison does not use the qualitative Judge text from the old Native report. Review showed that the Judge could read a pre-adaptation_test_results.json transport file and mistake legacy path checks for final Native artifacts.
The page therefore uses only:
- deterministic task and workflow validators;
- the final
passedstate of each run; - cumulative duration from raw top-level result events;
- traceable task-level failure reasons.
_test_context.json and _test_results.json from Judge evidence. That fix is not used to rewrite the qualitative conclusions of this historical experiment.
How to read the result
The aligned experiment supports three conclusions:- Lighter execution did not reduce task coverage. Native and 0.4.0 Classic both reach task-level
pass@3 = 100%. - Native is more stable across repeated runs. Its strict pass@1 is 6.25 percentage points higher and its
pass^3is 12.5 points higher. - Native model execution is substantially shorter. Corrected mean and median duration both fall by about 47%, although a same-window rerun is needed for a stronger latency claim.
HTML visualization
The embedded report presents the headline metrics, reliability measures, completed-task turns, tool calls, tokens, cost, corrected duration, the three-run matrix for all 16 tasks, failure attribution, and comparison boundaries.Next steps
- Rerun both modes in the same service window and under the same concurrency to strengthen the duration comparison.
- Improve Native terminal-evidence completeness, especially consistency across brief, specification, verification, and archive artifacts.
- Comet baseline evaluation experiment - Read the full 0.3.9 vs. 0.4.0 Classic baseline comparison.
- Scoring and dual-Agent evaluation - Understand
pass@k,pass^k, deterministic validators, and the LLM Judge.