> ## Documentation Index
> Fetch the complete documentation index at: https://docs.comet.rpamis.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Real Evaluation of Comet Native RC1 and beta16

> Using the same 16 tasks and pass@3 sample size, compare the strict pass rate, repeat run stability, correction time consumption, and execution cost of Native RC1 with beta16.

On this page

<ol>
  <li>
    [Experiment alignment](#experiment-alignment)
  </li>

  <li>
    [Key results](#key-results)
  </li>

  <li>
    [Why measure both pass@3 and pass^3](#why-measure-both-pass%403-and-pass%5E3)
  </li>

  <li>
    [Execution efficiency on successful tasks](#execution-efficiency-on-successful-tasks)
  </li>

  <li>
    [Corrected model execution time](#corrected-model-execution-time)
  </li>

  <li>
    [Task-level stability matrix](#task-level-stability-matrix)
  </li>

  <li>
    [Failure attribution](#failure-attribution)
  </li>

  <li>
    [About LLM Judge](#about-llm-judge)
  </li>

  <li>
    [Interpreting the results](#interpreting-the-results)
  </li>

  <li>
    [Strict retest with memory and project rules disabled](#strict-retest-with-memory-and-project-rules-disabled)
  </li>

  <li>
    [Full HTML report](#full-html-report)
  </li>

  <li>
    [Next steps](#next-steps)
  </li>
</ol>

[← Evaluation experiments](https://docs.comet.rpamis.com/en/tech-blog.md#eval-experiments)

2026-08-28

Evaluation experiments

# Real Evaluation of Comet Native RC1 and beta16

Using the same 16 tasks and pass\@3 sample size, compare the strict pass rate, repeat run
stability, correction time consumption, and execution cost of Native RC1 with beta16.

By Comet team

> [!WARNING]
>
> This is an experiment report for maintainers, not a running tutorial for ordinary users.
> Reproducing the experiment requires the corresponding Comet Source code, task set, model
> configuration and report snapshot; Ordinary users, please use [Quick Start ](https://docs.comet.rpamis.com/en/eval/quickstart.md)

> [!NOTE]
>
> The RC1 experiment was conducted on August 28, 2026. The 48 baseline samples of beta16 are from
> August 27-28, 2026 The existing experiments were directly reused this time without re-running.
> Both sides use the same 16 tasks and 3 repetitions, but Native products and trajectory checks
> follow the contracts of their respective versions.

This experiment answers a specific question: with fixed business tasks and repetition counts, does Native RC1 improve reliability and execution cost compared with beta16?

Both sides use the same 16 business tasks, with each task repeated 3 times: 48 runs for each version, totaling 96 runs for comparison.

> [!NOTE]
>
> The final RC1 analysis set consists of 48 unique `task + run` records. The previous four recovery attempts only produced Docker environment noise of `harness_trigger_suspect`/exit-125, and thus were excluded; the corresponding missing runs have been filled in by subsequent make-up runs, with no double counting.
>
> The original RC1 snapshot was once recorded as 47/48 because the parser recognized the shell glob in `find ... -path '*/.claude/skills/*'` as an unexpected Skill named `*`. After fixing the parser and re-reading the same raw stdout, only `comet-native` is explicitly called. Therefore, this paper uses the corrected 48/48 analysis set without modifying the original snapshot.

## Experiment alignment

The business tasks and business validators remain consistent; workflow checks respectively follow the real Native contracts of beta16 and RC1.

| Observation item | RC1 | beta16 |
| - | - | - |
| Business tasks | The same 16 tasks | The same 16 tasks |
| Repetition times | Do each task 3 times, for a total of 48 times | Do each task 3 times, for a total of 48 times |
| Business verification | The same task validator | The same task validator |
| workflow verification | RC1 Native Skill, product, state, loop, isolation | beta16 Native Skill, product, state, trajectory, isolation |
| Model selection | `deepseek-v4-flash[1m]` | `deepseek-v4-flash[1m]` |
| Interactive mode | `auto_user`, up to 12 wheels | `auto_user`, up to 12 wheels |
| Experimental identification | `native_rc1_20260828_0952` + precise make-up run batch | `native_beta16_20260827_2308` |
| Analysis set | 48/48 included，high confidence | 48/48 included，high confidence |

"strict pass" indicates that both the business validator and the corresponding version of the Native workflow check have passed. Business verification is listed separately to avoid mistaking the failure of workflow evidence for code or task failure.

## Key results

| Indicator | RC1 | beta16 | "Difference |
| - | -: | -: | -: |
| strict pass\@1 | 48/48，100.00% | 46/48，95.83% | RC1 +4.17 pp |
| pass\@3 | 16/16，100.00% | 16/16，100.00% | "Flat" |
| pass^3 | 16/16，100.00% | 14/16，87.50% | RC1 +12.50 pp |
| Business verification passed | 48/48 | 47/48 | RC1 +1 time |
| Correct the average time consumption | 391.3s | 503.6s | RC1 −22.3% |
| Correcting the median is time-consuming | 369.1s | 476.5s | RC1 −22.5% |

Both task-level `pass@3` on both sides are 100%, meaning that each of the 16 tasks has been successful at least once. RC1 achieved two more strict passes and two more tasks passed all three times. The reduction in time consumption can only be regarded as directional evidence, as beta16 uses reused historical runs, and the two groups of samples are not controlled re-runs in the same time window.

## Why measure both pass\@3 and pass^3

`pass@3` measures at least one success among three attempts and is suitable for observing ability coverage. The `pass^3` measurement shows that all three observation runs were successful, making it suitable for observing the stability of repeated runs.

These two indicators distinguish between "the version can complete the task" and "the version stably completes the task every time". This time, both sides cover all 16 tasks, but RC1 has 16/16 tasks that pass strictly three times, while beta16 has 14/16.

## Execution efficiency on successful tasks

The efficiency main view only counts 46 groups of identical tasks that are strictly passed on both sides, avoiding misting early failure for low consumption. The number of model starts/restores indicates the number of times the evaluator enters the model under test. The Agent rounds are the cumulative rounds within these calls.

| Indicator | RC1 | beta16 | RC1 difference |
| - | -: | -: | -: |
| The number of times the model is started/restored | 1.15 | 1.89 | −39.1% |
| Agent round | 83.39 | 77.54 | +7.5% |
| Tool invocation | 97.35 | 74.65 | +30.4% |
| Cumulative model time consumption | 396.3s | 501.6s | −21.0% |
| Non-cached input Token | 75,102 | 73,632 | +2.0% |
| Output Token | 35,445 | 46,604 | −23.9% |
| Total Token (including cache reads) | 4,474,779 | 3,730,902 | +19.9% |
| Model cost | \$4.003 | \$3.348 | +19.6% |

In the paired view with strict pass on both sides, the model of RC1 has fewer entries and a lower cumulative time consumption. However, the complete table also shows the costs: the Agent rounds, tool calls, total tokens, and average costs of RC1 are all higher. Therefore, this is a "shorter but not cheaper" execution portrait, and not all efficiency indicators have improved.

Both 48/48 trajectories on both sides retain peak context telemetry. The average peak context is RC1 102,470 tokens and beta16 104,208 tokens. This conclusion does not use the qualitative text of LLM Judge to replace hard checks.

## Corrected model execution time

The time consumption is recalculated from raw stdout: accumulate all the top layers of `result.duration_ms` for each sample. The calculation scope excludes the Docker preparation and business validator running time.

* RC1:48 observable samples, with an average of 391.260 seconds and a median of 369.053 seconds.
* beta16:48 observable samples, with an average of 503.552 seconds and a median of 476.481 seconds.

Under this metric, the average model time consumption of RC1 is reduced by 22.3%, and the median time consumption is reduced by 22.5%. This is a comparison of two running Windows and not a completely controlled causal conclusion.

> [!WARNING]
>
> RC1 and beta16 Not running in the same service window. The task set, model selection, interaction
> mode and repetition count are aligned, but the machine load, service provider latency and
> version-specific are The Workflow Contract is not fully controlled.

## Task-level stability matrix

| Task | RC1 | beta16 |
| - | -: | -: |
| `agent-memory-routing` | 3/3 | 3/3 |
| `api-cache-ttl` | 3/3 | 3/3 |
| `cross-file-refactor` | 3/3 | 2/3 |
| `dependency-confusion` | 3/3 | 3/3 |
| `fix-median` | 3/3 | 3/3 |
| `framework-selection` | 3/3 | 3/3 |
| `full-workflow` | 3/3 | 3/3 |
| `graph-execution-review` | 3/3 | 3/3 |
| `human-approval-flow` | 3/3 | 3/3 |
| `layered-streaming-fix` | 3/3 | 3/3 |
| `noise-distractor` | 3/3 | 2/3 |
| `observability-env-template` | 3/3 | 3/3 |
| `perf-dedupe` | 3/3 | 3/3 |
| `persistence-threading` | 3/3 | 3/3 |
| `refactor-counter` | 3/3 | 3/3 |
| `robust-config` | 3/3 | 3/3 |

RC1 has 16/16 tasks with three strict passes; beta16 is 14/16. Both sides cover at least one success of all 16 tasks.

## Failure attribution

The corrected RC1 analysis has no remaining strict failure; beta16 has two times. One failure in the original RC1 snapshot has been verified as a false positive by the same raw stdout and the fixed shell-glob parser:

| Version | Task/Run | Failure signal | Business verification |
| - | - | - | - |
| beta16 | `cross-file-refactor` r3 | The `.comet/config.yaml`, the final archive state and the complete Native trajectory are missing | "Through |
| beta16 | `noise-distractor` r2 | The invoice tax rate calculation failed, and at the same time, there is a lack of product/state /trajectory evidence | Failure |

beta16 had one pure workflow failure and one simultaneous failure of both business and workflow. Therefore, the business verification lines are RC1 48/48 and beta16 47/48 respectively.

## About LLM Judge

This comparison only adopts deterministic tasks and workflow verification, the final `passed` state of each run, and the raw result time consumption. Do not use qualitative LLM Judge text instead of these checks.

## Interpreting the results

1. The task coverage remains unchanged. Both the task-level `pass@3` of RC1 and beta16 are 16/16.
2. RC1 was slightly more stable in this group of samples. \*\* strict pass\@1 is 4.17 percentage points higher, `pass^3` is 12.50 percentage points higher.
3. RC1 is faster but not cheaper. The average cost and total Token in the successfully paired samples increased, although the average and median of raw model duration decreased by approximately 22%. These are experience-based trade-offs and do not support a "comprehensively better" conclusion.

This experiment supports understanding RC1 as a Native version that is more reliable and has a shorter execution time on this task set, while retaining beta16 as a valid historical baseline. It does not prove that RC1 has a comprehensive advantage in all task and run Windows, nor can it replace controlled re-running in the same window.

## Strict retest with memory and project rules disabled

To eliminate the deviation caused by inconsistent model specifications in the previous round, the supplementary re-test only selects the most difficult `comet-full-workflow` and runs it once. The main model was fixed as `deepseek-v4-flash[1m]`, which was consistent with the RC1-on control. Close `memory.learning` and `memory.retrieval` simultaneously, and confirm `plugins: []` in the startup snapshot. The re-test passed 18 out of 18 checks, and the business baseline was 6/6.

| Indicator | RC1-on (Triple mean) | RC1-off (once) | Change |
| - | -: | -: | -: |
| Agent round | 92.00 | 91 | −1.09% |
| Tool invocation | 118.00 | 118 | 0.00% |
| Non-cached input Token | 75,644 | 79,670 | +5.32% |
| Output Token | 45,193 | 39,737 | −12.07% |
| Cache reads Token | 5,349,675 | 4,894,464 | −8.51% |
| Total Token (including cache reads) | 5,470,512 | 5,013,871 | −8.35% |
| Model cost | \$4.7515 | \$4.4721 | −5.88% |
| Cumulative model time consumption | 579.491s | 492.341s | −15.04% |

This result does not support the judgment that "the Token actually increases after the memory and rules are turned off." The previous abnormal increase was due to the mixed comparison of `deepseek-v4-flash` (200K context) and `deepseek-v4-flash[1m]` (1M context). After the model specification correction, the total Token and cost of the RC1-off sample were actually lower than the average of the three RC1-on samples.

This is still not a causal cost experiment for the project rule plugin: the current source code has removed `comet.project-rules`, and this startup snapshot does not load the plugin either. Therefore, it can only indicate the corrected RC1-off reference value and cannot infer from a sample how many tokens the plugin itself has added. The complete evidence and configuration snapshots can be found in the HTML report of the same format below.

RC1 strict retest with memory and project rules disabled

## Full HTML report

The following embedded report focuses on presenting headline metrics, reliability metrics, paired execution efficiency, correction time consumption, the three-run matrix of 16 tasks, failure attribution, and comparison boundaries.

Comet Native RC1 vs. beta16 comparison report

## Next steps

* Run RC1 and beta16 again under the same service window and the same concurrent conditions, and further strengthen the conclusion of time consumption.
* Track the tool calls, rounds, tokens and cost increases of RC1 separately and evaluate them separately from the decrease in time consumption.
* [Comet Baseline True Evaluation Experiment ](https://docs.comet.rpamis.com/en/eval/comet-baseline-experiment.md) - View earlier Classic baseline comparisons
* [Scoring Metrics and Dual-Agent Evaluation ](https://docs.comet.rpamis.com/en/eval/scoring.md) - Understand the differences between `pass@k`, `pass^k`, hard validation, and LLM Judge

[← Back to evaluation experiments](https://docs.comet.rpamis.com/en/tech-blog.md#eval-experiments)

[Read in the evaluation docs ↗](https://docs.comet.rpamis.com/en/eval/comet-native-vs-rc1-beta16-experiment.md)


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.