> ## Documentation Index
> Fetch the complete documentation index at: https://docs.comet.rpamis.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Native Batch vs. Sequential clarification evaluation

> Ten repeated runs per Native clarification mode, comparing pass@3, interaction turns, tool calls, duration, completion, and failure attribution.

On this page

<ol>
  <li>
    [Core data](#core-data)
  </li>

  <li>
    [Reading the result](#reading-the-result)
  </li>

  <li>
    [HTML report](#html-report)
  </li>

  <li>
    [Method](#method)
  </li>
</ol>

[← Evaluation experiments](https://docs.comet.rpamis.com/en/tech-blog.md#eval-experiments)

2026-07-22

Evaluation experiments

# Native Batch vs. Sequential clarification evaluation

Ten repeated runs per Native clarification mode, comparing pass\@3, interaction turns, tool
calls, duration, completion, and failure attribution.

By Comet team

> [!WARNING]
>
> This is a historical experiment report kept for source maintainers, not a runbook for ordinary
> users. Reproducing it requires the corresponding Comet source, tasks, and report snapshot;
> ordinary users should use [Quickstart](https://docs.comet.rpamis.com/en/eval/quickstart.md).

This experiment compares two Comet Native clarification modes. `batch` asks independent questions together and confirms shared understanding separately; `sequential` resolves one decision at a time.

We repeated the same `wordcount-cli` task, which contains three independent product decisions, ten times per treatment. The model, runner, validators, and environment were identical. All 20 samples completed and entered the analysis set.

## Core data

| Metric | Batch | Sequential | Batch delta |
| - | -: | -: | -: |
| strict pass\@1 | 60% (6/10) | 70% (7/10) | -10 pp |
| pass\@3 | 96.7% | 99.2% | -2.5 pp |
| pass^3 | 0% | 0% | tied |
| Mean outer interaction turns | 2.7 | 4.5 | -40.0% |
| Mean cumulative model duration | 617s | 712s | -13.3% |

`pass@3` uses the HumanEval at-least-one-success estimator: Batch passed 6 of 10 observations and Sequential passed 7. The ten groups are repeated runs of one task, not ten distinct tasks.

## Reading the result

Batch materially reduced user interaction. Across the five paired repetitions where both treatments strictly passed, it also used 13.1% fewer Agent turns, 12.5% fewer tool calls, 16.6% less cumulative model time, and 22.8% fewer total tokens. However, Batch strict success was 10 percentage points lower in this sample, mainly because four runs merged the shared-understanding confirmation into the first batch-answer round.

Sequential had a different failure surface: one run added an extra decision round; two others completed the expected three decision points but produced incorrect behavior or incomplete archived evidence. Fewer turns are therefore not sufficient evidence of improvement without the business result, clarification protocol, and terminal evidence.

> [!WARNING]
>
> This is a small experiment with one task and ten repetitions per mode. It exposes interaction cost
> and failure modes for this task, but does not establish universal reliability for either mode.

## HTML report

The embedded report covers pass\@k, paired efficiency, the repetition matrix, completion quality, and failure attribution.

Comet Native Batch vs. Sequential clarification evaluation

## Method

* Experiment: `experiment_20260722_002221`
* Model: `mimo-v2.5-pro`
* Samples: 10 Batch and 10 Sequential runs, 10 strict pairs
* Data quality: 20/20 included, all high confidence, no exclusions
* Duration: additive top-level result events; Docker setup and validator time excluded
* Primary efficiency view: the five same-repetition pairs where both treatments strictly passed

Continue with [Native vs. 0.4.0 Classic](https://docs.comet.rpamis.com/en/eval/comet-native-vs-040-experiment.md) and [scoring and dual-agent evaluation](https://docs.comet.rpamis.com/en/eval/scoring.md).

[← Back to evaluation experiments](https://docs.comet.rpamis.com/en/tech-blog.md#eval-experiments)

[Read in the evaluation docs ↗](https://docs.comet.rpamis.com/en/eval/comet-native-clarification-experiment.md)


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.