Across the same 16 benchmark tasks, both Native 0.5.3 and Native 0.4.0-rc.1 achieved 48/48 strict passes (100% pass rate). Native 0.5.3 recorded an average cumulative Agent execution time of 159.8 seconds, a 32.1% reduction compared to 235.2 seconds for 0.4.0-rc.1.
Both benchmark cohorts ran under the Native workflow. Version 0.4.0-rc.1 used its frozen published package, adhering to the confirmation, verification, and Archive requirements of that release.
Experiment setup
Telemetry in both groups reported deepseek-flash[1m] and deepseek-v4-flash labels. The report preserves both labels as recorded rather than merging them.
Success rates and repeat-run stability
Strict pass requires satisfying two conditions simultaneously: the task functionality checks pass, and the execution satisfies the respective version’s Native workflow contract.
pass@3 indicates at least one success across 3 attempts per task, representing problem-solving coverage; pass^3 indicates all 3 attempts succeeded, representing execution stability. Both versions achieved 100% on both metrics, tying in this benchmark.
Execution overhead when both pass
The following averages reflect all 48 matched pairs that strictly passed in both treatments. Execution time includes model calls, tool execution, and session resumption, excluding container image builds.
While maintaining a 100% pass rate, 0.5.3 showed notable efficiency improvements: cumulative execution time dropped by 32.1%, tool calls decreased by 38.3%, and observed token volume was reduced by 12.1%. Model starts / resumptions and uncached input tokens rose slightly. The full report below details these metrics.
Interpretation limits
Because the two treatments ran across different time windows, model provider load fluctuations were not controlled; timing differences should not be attributed entirely to code adjustments. Conclusions apply strictly to this fixed task set.
Token statistics cover only observed calls from the primary Agent session, excluding independent sub-agents. Costs reported are CLI estimates and do not represent actual provider invoices.
Transient environment preparation issues before model launch were not counted as model errors; affected positions were cleanly rerun according to plan without duplicating valid scores. Historical MiMo baseline figures in the appendix serve as background reference only and are not pooled with the DeepSeek dataset.
Full HTML report
The report below presents the per-task result matrix, pairwise efficiency, peak context, execution batches, and benchmark limits. Source links open in a new tab.
Per-sample data · Execution record, in Chinese
Related experiment: Native 0.5.3 vs. native Goal. Last modified on November 13, 2026
