On this page
← Evaluation experiments
Native 0.5.3 vs. Native 0.4.0-rc.1: measured results
With identical tasks, repetitions, and DeepSeek configurations, both Native versions maintained a 100% pass rate; mean cumulative Agent execution time was 32.1% lower with 0.5.3 than 0.4.0-rc.1.
Across the same 16 benchmark tasks, both Native 0.5.3 and Native 0.4.0-rc.1 achieved 48/48 strict passes (100% pass rate). Native 0.5.3 recorded an average cumulative Agent execution time of 159.8 seconds, a 32.1% reduction compared to 235.2 seconds for 0.4.0-rc.1.Both benchmark cohorts ran under the Native workflow. Version 0.4.0-rc.1 used its frozen published package, adhering to the confirmation, verification, and Archive requirements of that release.
Experiment setup
Telemetry in both groups reported
deepseek-flash[1m] and deepseek-v4-flash labels. The report preserves both labels as recorded rather than merging them.Success rates and repeat-run stability
Strict pass requires satisfying two conditions simultaneously: the task functionality checks pass, and the execution satisfies the respective version’s Native workflow contract.pass@3 indicates at least one success across 3 attempts per task, representing problem-solving coverage; pass^3 indicates all 3 attempts succeeded, representing execution stability. Both versions achieved 100% on both metrics, tying in this benchmark.Execution overhead when both pass
The following averages reflect all 48 matched pairs that strictly passed in both treatments. Execution time includes model calls, tool execution, and session resumption, excluding container image builds.While maintaining a 100% pass rate, 0.5.3 showed notable efficiency improvements: cumulative execution time dropped by 32.1%, tool calls decreased by 38.3%, and observed token volume was reduced by 12.1%. Model starts / resumptions and uncached input tokens rose slightly. The full report below details these metrics.