This experiment compares Comet Native’s two clarification modes. batch asks all independent questions together and confirms the shared understanding separately; sequential resolves one decision at a time.
We repeated the same wordcount-cli task, which contains three independent product decisions, ten times per treatment. The model, runner, validators, and environment were identical. All 20 samples completed and entered the analysis set.
Core data
pass@3 uses the HumanEval at-least-one-success estimator: Batch passed 6 of 10 observations and Sequential passed 7. The ten groups are repeated runs of one task, not ten distinct tasks.
Reading the result
Batch materially reduced user interaction. Across the five paired repetitions where both treatments strictly passed, it also used 13.1% fewer Agent turns, 12.5% fewer tool calls, 16.6% less cumulative model time, and 22.8% fewer total tokens. However, Batch strict success was 10 percentage points lower in this sample, mainly because four runs merged the shared-understanding confirmation into the first batch-answer round.
Sequential had a different failure surface: one run added an extra decision round; two others completed the expected three decision points but produced incorrect behavior or incomplete archived evidence. Fewer turns are therefore not sufficient evidence of improvement without the business result, clarification protocol, and terminal evidence.
This is a small experiment with one task and ten repetitions per mode. It exposes interaction cost
and failure modes for this task, but does not establish universal reliability for either mode.
HTML report
The embedded report covers pass@k, paired efficiency, the repetition matrix, completion quality, and failure attribution.
Method
- Experiment:
experiment_20260722_002221
- Model:
mimo-v2.5-pro
- Samples: 10 Batch and 10 Sequential runs, 10 strict pairs
- Data quality: 20/20 included, all high confidence, no exclusions
- Duration: additive top-level result events; Docker setup and validator time excluded
- Primary efficiency view: the five same-repetition pairs where both treatments strictly passed
Continue with Native vs. 0.4.0 Classic and scoring and dual-agent evaluation. 
