> ## Documentation Index
> Fetch the complete documentation index at: https://docs.comet.rpamis.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Native 0.5.3 vs. Native 0.4.0-rc.1: measured results

> With identical tasks, repetitions, and DeepSeek configurations, both Native versions maintained a 100% pass rate; mean cumulative Agent execution time was 32.1% lower with 0.5.3 than 0.4.0-rc.1.

On this page

<ol>
  <li>
    [Experiment setup](#experiment-setup)
  </li>

  <li>
    [Success rates and repeat-run stability](#success-rates-and-repeat-run-stability)
  </li>

  <li>
    [Execution overhead when both pass](#execution-overhead-when-both-pass)
  </li>

  <li>
    [Interpretation limits](#interpretation-limits)
  </li>

  <li>
    [Full HTML report](#full-html-report)
  </li>
</ol>

[← Evaluation experiments](https://docs.comet.rpamis.com/en/tech-blog.md#eval-experiments)

2026-11-12

Evaluation experiments

# Native 0.5.3 vs. Native 0.4.0-rc.1: measured results

With identical tasks, repetitions, and DeepSeek configurations, both Native versions
maintained a 100% pass rate; mean cumulative Agent execution time was 32.1% lower with
0.5.3 than 0.4.0-rc.1.

By Comet team

Across the same 16 benchmark tasks, both Native 0.5.3 and Native 0.4.0-rc.1 achieved 48/48 strict passes (100% pass rate). Native 0.5.3 recorded an average cumulative Agent execution time of 159.8 seconds, a 32.1% reduction compared to 235.2 seconds for 0.4.0-rc.1.

Both benchmark cohorts ran under the Native workflow. Version 0.4.0-rc.1 used its frozen published package, adhering to the confirmation, verification, and Archive requirements of that release.

## Experiment setup

| Item | Shared setup |
| - | - |
| Tasks | The same 16 Comet workflow tasks |
| Repetitions | 3 runs per task, 48 runs per treatment (96 scored runs in total) |
| Agent | Claude Code 2.1.318 |
| Model | DeepSeek-V4.1-Flash-1017; invocation alias `deepseek-flash[1M]` |
| Task validation | Identical task definitions, original prompt instructions, test data, and assertions |
| Workflow validation | Follows the respective version's Native protocol contracts |

Telemetry in both groups reported `deepseek-flash[1m]` and `deepseek-v4-flash` labels. The report preserves both labels as recorded rather than merging them.

## Success rates and repeat-run stability

Strict pass requires satisfying two conditions simultaneously: the task functionality checks pass, and the execution satisfies the respective version's Native workflow contract.

| Metric | Native 0.5.3 | Native 0.4.0-rc.1 |
| - | -: | -: |
| Strict pass\@1 | 48/48, 100% | 48/48, 100% |
| pass\@3 | 16/16, 100% | 16/16, 100% |
| pass^3 | 16/16, 100% | 16/16, 100% |
| Task checks passed | 48/48 | 48/48 |

`pass@3` indicates at least one success across 3 attempts per task, representing problem-solving coverage; `pass^3` indicates all 3 attempts succeeded, representing execution stability. Both versions achieved 100% on both metrics, tying in this benchmark.

## Execution overhead when both pass

The following averages reflect all 48 matched pairs that strictly passed in both treatments. Execution time includes model calls, tool execution, and session resumption, excluding container image builds.

| Mean per run | Native 0.5.3 | Native 0.4.0-rc.1 | Relative change |
| - | -: | -: | -: |
| Cumulative Agent execution time | 159.8 s | 235.2 s | −32.1% |
| Tool calls | 53.44 | 86.54 | −38.3% |
| Observed total tokens, including cache reads | 2,314,560 | 2,633,238 | −12.1% |
| Model starts / resumptions | 2.96 | 2.71 | +9.2% |
| Uncached input tokens | 66,456 | 63,978 | +3.9% |

While maintaining a 100% pass rate, 0.5.3 showed notable efficiency improvements: cumulative execution time dropped by 32.1%, tool calls decreased by 38.3%, and observed token volume was reduced by 12.1%. Model starts / resumptions and uncached input tokens rose slightly. The full report below details these metrics.

## Interpretation limits

Because the two treatments ran across different time windows, model provider load fluctuations were not controlled; timing differences should not be attributed entirely to code adjustments. Conclusions apply strictly to this fixed task set.

Token statistics cover only observed calls from the primary Agent session, excluding independent sub-agents. Costs reported are CLI estimates and do not represent actual provider invoices.

Transient environment preparation issues before model launch were not counted as model errors; affected positions were cleanly rerun according to plan without duplicating valid scores. Historical MiMo baseline figures in the appendix serve as background reference only and are not pooled with the DeepSeek dataset.

## Full HTML report

The report below presents the per-task result matrix, pairwise efficiency, peak context, execution batches, and benchmark limits. Source links open in a new tab.

Native 0.5.3 vs. Native 0.4.0-rc.1: full report

[Per-sample data](https://github.com/rpamis/comet/blob/0f6877fd6245cbd62a2cd783abca0c20403d5931/docs/research/2026-11-11-native-053-vs-040/comparison-data.json) · [Execution record, in Chinese](https://github.com/rpamis/comet/blob/0f6877fd6245cbd62a2cd783abca0c20403d5931/docs/research/2026-11-11-native-053-vs-040/execution-record.zh.md)

Related experiment: [Native 0.5.3 vs. native Goal](https://docs.comet.rpamis.com/en/eval/comet-native-053-vs-goal-experiment.md).

[← Back to evaluation experiments](https://docs.comet.rpamis.com/en/tech-blog.md#eval-experiments)

[Read in the evaluation docs ↗](https://docs.comet.rpamis.com/en/eval/comet-native-053-vs-040-rc1-experiment.md)


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.