> ## Documentation Index
> Fetch the complete documentation index at: https://docs.comet.rpamis.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Comet Native vs. 0.4.0 Classic evaluation

> Compare Comet Native with 0.4.0 Classic on the same 16 tasks at pass@3 scale, including strict pass rate, repeated-run reliability, and corrected model duration.

export const RawHtmlReportFrame = ({src, title, height = 720}) => {
  const [html, setHtml] = useState('');
  const [error, setError] = useState('');
  useEffect(() => {
    let cancelled = false;
    setHtml('');
    setError('');
    fetch(src).then(response => {
      if (!response.ok) {
        throw new Error(`Failed to load report: ${response.status}`);
      }
      return response.text();
    }).then(body => {
      if (!cancelled) {
        const payload = src.endsWith('.json') ? JSON.parse(body) : null;
        setHtml(payload?.html || body);
      }
    }).catch(loadError => {
      if (!cancelled) {
        setError(loadError.message);
      }
    });
    return () => {
      cancelled = true;
    };
  }, [src]);
  return <div className="not-prose overflow-hidden rounded-xl border border-zinc-200 bg-white dark:border-zinc-800">
      {html ? <iframe srcDoc={html} title={title} className="block w-full bg-white" height={height} loading="lazy" sandbox="allow-downloads allow-forms allow-popups allow-scripts" /> : <div className="flex min-h-48 items-center justify-center px-6 py-12 text-sm text-zinc-600 dark:text-zinc-300">
          {error || 'Loading report...'}
        </div>}
    </div>;
};

This experiment asks a narrower question: when the model is already capable, can Comet Native preserve requirements, state, verification, and archive evidence while running a lighter workflow than 0.4.0 Classic?

We aligned Native with the `pass@3` experiment previously run for 0.4.0: the same 16 business tasks, repeated 3 times each. The comparison contains 96 runs in total.

<Note>
  Native and Classic are separate workflows for different model capability levels. Native does not
  upgrade into Classic. This page compares business completion, reliability, and model duration on
  the same tasks; it does not require both modes to produce identical workflow artifacts.
</Note>

## How the experiment is aligned

Alignment does not mean applying Classic checks unchanged to Native. The business objectives and task validators stay the same. Workflow checks follow the actual contract of each mode.

| Observation         | Native                                                             | 0.4.0 Classic                                                          |
| ------------------- | ------------------------------------------------------------------ | ---------------------------------------------------------------------- |
| Business tasks      | Same 16 tasks                                                      | Same 16 tasks                                                          |
| Repetitions         | 3 per task, 48 runs                                                | 3 per task, 48 runs                                                    |
| Business validation | Same task validators                                               | Same task validators                                                   |
| Workflow validation | Native Skill, `comet/` artifacts, state, trajectory, and isolation | Classic Skill, OpenSpec, phases, state, and dependency-Skill contracts |
| Execution window    | Current local concurrent run                                       | Historical run from July 4–5, 2026                                     |
| Experiment ID       | `experiment_20260716_104344`                                       | `combined_comet_workflow_full_k3_20260705_v3_rerun_failed`             |
| Analysis set        | 48/48 included, high confidence                                    | 48/48 included, all flagged at medium confidence                       |

Some task prompts still use Classic terms such as Open, Design, Build, Verify, and Archive. The Native treatment tells the model to preserve the business requirements while mapping them to Shape, Build, Verify, and Archive. It must not create OpenSpec, Superpowers, or `.comet` workflow artifacts. This reduces prompt bias toward Classic without changing the task itself.

All 48 Classic samples were conservatively marked `flagged` because their logs mention outer-runner, API, or environment failures. They still belong to the analysis set; `flagged` does not mean excluded or failed. This page preserves their strict pass results and treats the observability difference as a limitation.

## Headline results

| Metric                    |        Native | 0.4.0 Classic |      Difference |
| ------------------------- | ------------: | ------------: | --------------: |
| strict pass\@1            | 46/48, 95.83% | 43/48, 89.58% | Native +6.25 pp |
| pass\@3                   |   16/16, 100% |   16/16, 100% |            Tied |
| pass^3                    |  14/16, 87.5% |    12/16, 75% | Native +12.5 pp |
| Business validator pass   |         48/48 |         47/48 |   Native +1 run |
| Corrected mean duration   |      183.971s |      352.375s |   Native -47.8% |
| Corrected median duration |      174.493s |      328.192s |   Native -46.8% |

These results have two layers. Both modes reach `pass@3 = 100%`, so every task succeeds at least once in three attempts. Native has the higher `pass^3`, which indicates better repeated-run reliability: 14 of 16 tasks pass all three Native runs, compared with 12 of 16 for 0.4.0 Classic.

## Why pass\@3 and pass^3 both matter

`pass@3` measures whether at least one of three attempts succeeds. It describes capability coverage. `pass^3` measures whether all three observed attempts succeed. It describes a strict reliability floor.

Looking only at `pass@3 = 100%` would hide failures among the 48 runs. Native has two strict failures; 0.4.0 Classic has five. The two metrics must be read together to distinguish “the model can complete this task” from “the model completes it consistently.”

## Completed-task execution efficiency

The primary efficiency view includes only the 41 matching task repetitions that both Native and Classic passed. This prevents an early failure from looking artificially efficient. One “model start/resume” means that the evaluator starts the subject model or re-enters the same task after a user answer or cold resume; it is distinct from the Agent turns accumulated inside those invocations.

| Metric                       |  Native | 0.4.0 Classic | Native difference |
| ---------------------------- | ------: | ------------: | ----------------: |
| Model starts/resumes         |    1.80 |          2.85 |            -36.8% |
| Agent turns                  |   39.02 |         91.66 |            -57.4% |
| Tool calls                   |   36.22 |         81.80 |            -55.7% |
| Cumulative model duration    |  180.8s |        343.9s |            -47.4% |
| Non-cache input tokens       |  16,647 |        74,903 |            -77.8% |
| Output tokens                |   5,471 |        13,788 |            -60.3% |
| Total tokens including cache | 891,631 |     3,840,975 |            -76.8% |
| Model cost                   | \$0.658 |       \$2.638 |            -75.1% |

All 48 Native runs preserve per-message context usage: the mean peak is 34,771 tokens, the median peak is 34,568, and the maximum is 42,012. The mean peak occupies 17.4% of a 200k context window. Only 9 of 48 historical Classic traces preserve equivalent telemetry, so the experiment cannot support a reliable workflow-to-workflow context-usage improvement percentage.

## Corrected duration methodology

The earlier report retained only the final `result` event in its `Duration` column. When an automatic interaction contains multiple model calls, that omits the earlier calls and can produce implausible four- or five-second samples.

This comparison instead sums every top-level `result.duration_ms` within a sample:

* Native: 48 observable samples, mean 183.971 seconds and median 174.493 seconds.
* 0.4.0 Classic: 47 observable samples, mean 352.375 seconds and median 328.192 seconds; one run failed before producing a result event.
* Neither figure includes Docker setup or task-validator execution time.

Under this methodology, Native reduces mean duration by 47.8% and median duration by 46.8%. This is consistent with Native's goal of reducing phase switching, dependency Skills, and repeated workflow reads.

<Warning>
  The 0.4.0 data comes from a historical run, while the Native data comes from a current concurrent
  run. Model, tasks, and sample size are aligned, but machine load and service windows are not fully
  controlled. The duration difference is strong directional evidence, not a controlled causal
  estimate.
</Warning>

## Failure attribution

Neither Native failure is a business-implementation failure:

| Task                           | Run | Failed check                                                                 | Business validator |
| ------------------------------ | --- | ---------------------------------------------------------------------------- | ------------------ |
| `comet-api-cache-ttl`          | r2  | `native_state`: brief, specification, or verification evidence is incomplete | Passed             |
| `comet-graph-execution-review` | r3  | `native_state`: brief, specification, or verification evidence is incomplete | Passed             |

The five 0.4.0 Classic failures contain one business failure and four workflow-contract failures:

| Type                      | Count | Primary signal                                                             |
| ------------------------- | ----: | -------------------------------------------------------------------------- |
| Business implementation   |     1 | `full-workflow` did not implement the requested `--sentences` feature      |
| Dependency-Skill contract |     3 | Required Superpowers dependency Skill was not observed                     |
| Main flow and phases      |     1 | Main Comet Skill was not invoked and only 3/5 Classic phases were observed |

This shows the different failure surfaces. Classic has more workflow dependencies, so a run can fail at the main Skill, a phase, OpenSpec, or a dependency Skill. Native has a narrower workflow surface but keeps a terminal-evidence check: completed code without complete brief, specification, or verification evidence is still not a strict pass.

## About the LLM Judge

This comparison does not use the qualitative Judge text from the old Native report. Review showed that the Judge could read a pre-adaptation `_test_results.json` transport file and mistake legacy path checks for final Native artifacts.

The page therefore uses only:

* deterministic task and workflow validators;
* the final `passed` state of each run;
* cumulative duration from raw top-level result events;
* traceable task-level failure reasons.

The eval code now excludes `_test_context.json` and `_test_results.json` from Judge evidence. That fix is not used to rewrite the qualitative conclusions of this historical experiment.

## How to read the result

The aligned experiment supports three conclusions:

1. **Lighter execution did not reduce task coverage.** Native and 0.4.0 Classic both reach task-level `pass@3 = 100%`.
2. **Native is more stable across repeated runs.** Its strict pass\@1 is 6.25 percentage points higher and its `pass^3` is 12.5 points higher.
3. **Native model execution is substantially shorter.** Corrected mean and median duration both fall by about 47%, although a same-window rerun is needed for a stronger latency claim.

It does not support the conclusion that Classic is obsolete. Native serves capable models that can reason through complex code but still benefit from clarification, state, verification, and archive evidence. Classic continues to serve models and tasks that need finer phase guidance, stronger process constraints, and explicit dependency-Skill orchestration.

## HTML visualization

The embedded report presents the headline metrics, reliability measures, completed-task turns, tool calls, tokens, cost, corrected duration, the three-run matrix for all 16 tasks, failure attribution, and comparison boundaries.

<RawHtmlReportFrame src="/assets/eval-reports/comet-native-vs-040-20260716/native-benchmark-report.json" title="Comet Native vs. 0.4.0 Classic comparison report" height={860} />

## Next steps

* Rerun both modes in the same service window and under the same concurrency to strengthen the duration comparison.
* Improve Native terminal-evidence completeness, especially consistency across brief, specification, verification, and archive artifacts.
* [Comet baseline evaluation experiment](/en/eval/comet-baseline-experiment) - Read the full 0.3.9 vs. 0.4.0 Classic baseline comparison.
* [Scoring and dual-Agent evaluation](/en/eval/scoring) - Understand `pass@k`, `pass^k`, deterministic validators, and the LLM Judge.