On this page
← Evaluation experiments
Native 0.5.3 vs. native Goal: measured results
Under identical DeepSeek configurations, compare Comet Native with native Goal on WorkBuddy-Bench and SkillsBench across mean scores, perfect-score rates, pass@3, pass^3, and run-by-run results.
In the stratified sample cohort from SkillsBench, Comet Native achieved a mean task score of 0.667 (vs. 0.444 for Goal), with an overall perfect-score rate of 66.7% (vs. 44.4% for Goal). Native achieved a pass@3 of 100.0% and a pass^3 of 33.3%.Across the hardest scenarios selected from both benchmarks, Native reached a 100% perfect-score rate: tying Goal on WorkBuddy-Bench and outperforming Goal (66.7%) on SkillsBench.This benchmark compares Comet Native 0.5.3 with unassisted Claude Code native
/goal. Original-task perfect scores, normal Agent completion, and Native workflow compliance are tallied independently; code artifacts retained after a timeout still receive their original-task score if they pass the assertions.Experiment setup
Native execution ran on a fixed 0.5.3 CLI release. Both benchmark arms maintained identical model routing; specific model tags and version scopes are detailed in the HTML report below.
Original scores, perfect-score rates, and stability
Task scores range from 0 to 1; a reward of 1.0 represents a full score. Mean scores reflect partial credit, while perfect-score rates measure complete success.pass@3 indicates at least one perfect score across 3 attempts per task; pass^3 indicates all 3 attempts achieved a full score. Because the task distributions differ, metrics for both datasets are presented independently rather than pooled into a single score.Taking Goal’s performance on SkillsBench as an example: while every task achieved a perfect score at least once (100% pass@3), no task achieved full marks across all 3 runs (0% pass^3). This shows the model possesses the capability to solve the tasks, but lacks consistency across repeated runs.Three run records are displayed for each task and mode. Overall comparison prioritizes perfect-score frequency, followed by average task score. For individual slots that scored zero due to transient environmental issues, higher-scoring supplemental runs are displayed while retaining full provenance. Note that displayed metrics reflect observed best-run values; differing candidate pool sizes mean this is not an equal-budget randomized controlled trial.WorkBuddy-Bench and SkillsBench were scored strictly according to original benchmark criteria. The perfect-score rate reflects delivery quality, succeeding within 3 attempts (pass@3) indicates capability coverage, and succeeding across all 3 attempts (pass^3) measures repeat-run stability.Individual scores
The table below lists raw scores in r1, r2, r3 order. A 0% perfect-score rate indicates that no run reached 1.0, not that all 3 runs scored zero.For the timezone LRU cache task, the original benchmark scorer also checked
gettz and tzstr caching behavior, whereas the prompt only explicitly specified tzoffset and adjustable capacity. Partial credit does not indicate a complete failure to implement caching; this evaluation strictly respects original scorer assertions without editing tests or rounding partial scores up to full marks.Perfect scores and normal workflow completion
The single-run timeout budget for vehicle routing was 3,600 seconds. Native encountered 1 timeout, while Goal encountered 2 timeouts. The code retained in the workspace for both sides passed the assertions of the original benchmark scoring script, so the task scores are included in the table above; however, process timeout exit does not equate to the workflow completing normally.In Native’s third vehicle routing attempt, the resulting code achieved a perfect score, but execution timed out before completing the Archive phase. Furthermore, Native workflow compliance and original task scores are evaluated independently: in the PDF form task, for example, certain runs achieved workflow compliance without reaching a perfect task score.