Rank
1
Model
gemini-3.5-flashgoogle
Overall
79.7
Runs
1
Usable Fit
100.0%
Zero-Shim Fit
85.7%
Total Cost
$0.10
Avg Cost/Case
$0.01
Failures
FORMAT_INVALID:1
Agent runtime benchmark
OpenClaw Benchmark Leaderboard
The OpenClaw runtime fit benchmark tests whether models behave cleanly inside an agent-style persistent runtime loop. Usable Fit is the primary metric. Zero-Shim Fit is the stricter secondary metric and indicates cleaner operation without additional shim/orchestrator help.
Showing 36 of 36
| Model | Failures | |||||||
|---|---|---|---|---|---|---|---|---|
| 1 | gemini-3.5-flashgoogle | 79.7 | 1 | 100.0% | 85.7% | $0.10 | $0.01 | FORMAT_INVALID:1 |
| 2 | kimi-k3moonshot | 78.6 | 1 | 100.0% | 85.7% | $0.36 | $0.05 | TOOL_USAGE_FAIL:1 |
| 3 | claude-opus-5anthropic | 78.5 | 1 | 85.7% | 71.4% | $0.49 | $0.07 | STRUCTURE_FAIL:1, TOOL_USAGE_FAIL:1 |
| 4 | claude-fable-5anthropic | 78.4 | 2 | 100.0% | 85.7% | $1.55 | $0.11 | FORMAT_INVALID:2 |
| 5 | grok-4.5grok | 77.6 | 4 | 100.0% | 82.1% | $0.52 | $0.02 | TOOL_USAGE_FAIL:4, FORMAT_INVALID:1 |
| 6 | gemini-3.1-pro-previewgoogle | 74.9 | 5 | 100.0% | 75.9% | $0.33 | $0.01 | TOOL_USAGE_FAIL:7 |
| 7 | glm-5zai | 74.1 | 5 | 95.7% | 87.0% | $0.21 | <$0.01 | TOOL_USAGE_FAIL:3 |
| 8 | claude-opus-4-7anthropic | 73.4 | 3 | 95.2% | 81.0% | $0.93 | $0.04 | TOOL_USAGE_FAIL:3, TEST_FAIL:1 |
| 9 | kimi-k2.7-code-highspeedmoonshot | 72.3 | 1 | 85.7% | 57.1% | $0.15 | $0.02 | FORMAT_INVALID:2, STRUCTURE_FAIL:1 |
| 10 | qwen3.7-plusqwen | 65.9 | 1 | 100.0% | 57.1% | $0.09 | $0.01 | TOOL_USAGE_FAIL:3 |
| 11 | kimi-k2.6moonshot | 65.4 | 3 | 95.2% | 52.4% | $0.48 | $0.02 | FORMAT_INVALID:9, STRUCTURE_FAIL:1 |
| 12 | grok-4.3grok | 63.0 | 4 | 85.7% | 78.6% | $0.39 | $0.01 | FORMAT_INVALID:3, MEMORY_PERSISTENCE_FAIL:1, STRUCTURE_FAIL:1, TOOL_USAGE_FAIL:1 |
| 13 | kimi-k2.5moonshot | 60.1 | 4 | 92.9% | 57.1% | $0.40 | $0.01 | FORMAT_INVALID:7, TOOL_USAGE_FAIL:4, STRUCTURE_FAIL:1 |
| 14 | gpt-5.6-solopenai | 59.9 | 1 | 85.7% | 71.4% | $0.24 | $0.03 | STRUCTURE_FAIL:1, TOOL_USAGE_FAIL:1 |
| 15 | gpt-5.2openai | 58.0 | 405 | 73.4% | 65.2% | $2.06 | <$0.01 | FORMAT_INVALID:68, BUDGET_EXCEEDED:40, INTERRUPT_IGNORED:30, STRUCTURE_FAIL:13, TOOL_USAGE_FAIL:3, FORBIDDEN_TOOL_ATTEMPT:1, MEMORY_PERSISTENCE_FAIL:1, MEMORY_USE_FAIL:1 |
| 16 | glm-5-turbozai | 56.7 | 4 | 85.7% | 60.7% | $0.18 | <$0.01 | TOOL_USAGE_FAIL:9, TEST_FAIL:2 |
| 17 | deepseek-v4-prodeepseek | 56.4 | 4 | 82.1% | 75.0% | $0.31 | $0.01 | TOOL_USAGE_FAIL:3, FORBIDDEN_TOOL_ATTEMPT:1, FORMAT_INVALID:1, MEMORY_PERSISTENCE_FAIL:1, STRUCTURE_FAIL:1 |
| 18 | claude-sonnet-4-6anthropic | 55.7 | 21 | 78.0% | 76.8% | $2.60 | $0.03 | TEST_FAIL:11, STRUCTURE_FAIL:4, TOOL_USAGE_FAIL:3, MEMORY_USE_FAIL:1 |
| 19 | claude-opus-4-8anthropic | 54.3 | 3 | 90.5% | 38.1% | $0.94 | $0.04 | FORMAT_INVALID:12, STRUCTURE_FAIL:1 |
| 20 | grok-4.20-0309-reasoninggrok | 53.4 | 4 | 81.8% | 63.6% | $0.97 | $0.04 | TOOL_USAGE_FAIL:4, MEMORY_PERSISTENCE_FAIL:3, FORMAT_INVALID:1 |
| 21 | glm-5.1zai | 52.7 | 3 | 81.0% | 66.7% | $0.22 | $0.01 | TOOL_USAGE_FAIL:3, FORBIDDEN_FILE_ATTEMPT:2, FORMAT_INVALID:1, TEST_FAIL:1 |
| 22 | qwen3.5-plusqwen | 50.9 | 4 | 72.7% | 72.7% | $0.19 | <$0.01 | FORMAT_INVALID:2, STRUCTURE_FAIL:2, MEMORY_PERSISTENCE_FAIL:1, TEST_FAIL:1 |
| 23 | qwen3.6-plus-2026-04-02qwen | 47.9 | 3 | 76.2% | 57.1% | $0.38 | $0.02 | TOOL_USAGE_FAIL:5, MEMORY_PERSISTENCE_FAIL:3, TEST_FAIL:1 |
| 24 | claude-opus-4-6anthropic | 44.2 | 3 | 85.7% | 14.3% | $0.78 | $0.04 | FORMAT_INVALID:18 |
| 25 | glm-5.2zai | 43.4 | 1 | 71.4% | 57.1% | $0.14 | $0.02 | TOOL_USAGE_FAIL:2, STRUCTURE_FAIL:1 |
| 26 | GPT-5.5openai | 41.1 | 4 | 71.4% | 60.7% | $1.14 | $0.04 | FORMAT_INVALID:6, TOOL_USAGE_FAIL:4, STRUCTURE_FAIL:1 |
| 27 | gpt-5.6-terraopenai | 35.4 | 1 | 71.4% | 42.9% | $0.09 | $0.01 | FORMAT_INVALID:4 |
| 28 | qwen3.6-max-previewqwen | 33.9 | 3 | 66.7% | 57.1% | $0.36 | $0.02 | TOOL_USAGE_FAIL:4, STRUCTURE_FAIL:3, FORMAT_INVALID:1, TEST_FAIL:1 |
| 29 | claude-sonnet-5anthropic | 28.7 | 1 | 57.1% | 42.9% | $0.32 | $0.05 | STRUCTURE_FAIL:2, FORMAT_INVALID:1, TOOL_USAGE_FAIL:1 |
| 30 | gemma-4-31b-itgoogle | 28.6 | 3 | 57.1% | 0.0% | $0.00 | $0.00 | FORMAT_INVALID:21 |
| 31 | MiniMax-M2.7minimax | 25.2 | 4 | 50.0% | 0.0% | $0.07 | <$0.01 | FORMAT_INVALID:22 |
| 32 | gpt-5.4-xhighopenai | 24.9 | 4 | 46.4% | 28.6% | $2.25 | $0.08 | FORMAT_INVALID:16, STRUCTURE_FAIL:4 |
| 33 | gpt-5.4openai | 19.5 | 39 | 54.4% | 28.2% | $0.83 | <$0.01 | FORMAT_INVALID:67, STRUCTURE_FAIL:4, TOOL_USAGE_FAIL:2, MEMORY_USE_FAIL:1 |
| 34 | gemini-3.1-flash-litegoogle | 13.6 | 3 | 42.9% | 38.1% | $0.04 | <$0.01 | TOOL_USAGE_FAIL:6, ACTION_LOOP_LIMIT:3, STRUCTURE_FAIL:3, TEST_FAIL:1 |
| 35 | deepseek-chatdeepseek | 10.8 | 4 | 22.7% | 22.7% | $0.03 | <$0.01 | ACTION_LOOP_LIMIT:10, STRUCTURE_FAIL:3, FORBIDDEN_FILE_ATTEMPT:2, TEST_FAIL:2 |
| 36 | gpt-5.6-lunaopenai | 9.5 | 1 | 42.9% | 42.9% | $0.08 | $0.01 | FORMAT_INVALID:4 |
Rank
2
Model
kimi-k3moonshot
Overall
78.6
Runs
1
Usable Fit
100.0%
Zero-Shim Fit
85.7%
Total Cost
$0.36
Avg Cost/Case
$0.05
Failures
TOOL_USAGE_FAIL:1
Rank
3
Model
claude-opus-5anthropic
Overall
78.5
Runs
1
Usable Fit
85.7%
Zero-Shim Fit
71.4%
Total Cost
$0.49
Avg Cost/Case
$0.07
Failures
STRUCTURE_FAIL:1, TOOL_USAGE_FAIL:1
Rank
4
Model
claude-fable-5anthropic
Overall
78.4
Runs
2
Usable Fit
100.0%
Zero-Shim Fit
85.7%
Total Cost
$1.55
Avg Cost/Case
$0.11
Failures
FORMAT_INVALID:2
Rank
5
Model
grok-4.5grok
Overall
77.6
Runs
4
Usable Fit
100.0%
Zero-Shim Fit
82.1%
Total Cost
$0.52
Avg Cost/Case
$0.02
Failures
TOOL_USAGE_FAIL:4, FORMAT_INVALID:1
Rank
6
Model
gemini-3.1-pro-previewgoogle
Overall
74.9
Runs
5
Usable Fit
100.0%
Zero-Shim Fit
75.9%
Total Cost
$0.33
Avg Cost/Case
$0.01
Failures
TOOL_USAGE_FAIL:7
Rank
7
Model
glm-5zai
Overall
74.1
Runs
5
Usable Fit
95.7%
Zero-Shim Fit
87.0%
Total Cost
$0.21
Avg Cost/Case
<$0.01
Failures
TOOL_USAGE_FAIL:3
Rank
8
Model
claude-opus-4-7anthropic
Overall
73.4
Runs
3
Usable Fit
95.2%
Zero-Shim Fit
81.0%
Total Cost
$0.93
Avg Cost/Case
$0.04
Failures
TOOL_USAGE_FAIL:3, TEST_FAIL:1
Rank
9
Model
kimi-k2.7-code-highspeedmoonshot
Overall
72.3
Runs
1
Usable Fit
85.7%
Zero-Shim Fit
57.1%
Total Cost
$0.15
Avg Cost/Case
$0.02
Failures
FORMAT_INVALID:2, STRUCTURE_FAIL:1
Rank
10
Model
qwen3.7-plusqwen
Overall
65.9
Runs
1
Usable Fit
100.0%
Zero-Shim Fit
57.1%
Total Cost
$0.09
Avg Cost/Case
$0.01
Failures
TOOL_USAGE_FAIL:3
Rank
11
Model
kimi-k2.6moonshot
Overall
65.4
Runs
3
Usable Fit
95.2%
Zero-Shim Fit
52.4%
Total Cost
$0.48
Avg Cost/Case
$0.02
Failures
FORMAT_INVALID:9, STRUCTURE_FAIL:1
Rank
12
Model
grok-4.3grok
Overall
63.0
Runs
4
Usable Fit
85.7%
Zero-Shim Fit
78.6%
Total Cost
$0.39
Avg Cost/Case
$0.01
Failures
FORMAT_INVALID:3, MEMORY_PERSISTENCE_FAIL:1, STRUCTURE_FAIL:1, TOOL_USAGE_FAIL:1
Rank
13
Model
kimi-k2.5moonshot
Overall
60.1
Runs
4
Usable Fit
92.9%
Zero-Shim Fit
57.1%
Total Cost
$0.40
Avg Cost/Case
$0.01
Failures
FORMAT_INVALID:7, TOOL_USAGE_FAIL:4, STRUCTURE_FAIL:1
Rank
14
Model
gpt-5.6-solopenai
Overall
59.9
Runs
1
Usable Fit
85.7%
Zero-Shim Fit
71.4%
Total Cost
$0.24
Avg Cost/Case
$0.03
Failures
STRUCTURE_FAIL:1, TOOL_USAGE_FAIL:1
Rank
15
Model
gpt-5.2openai
Overall
58.0
Runs
405
Usable Fit
73.4%
Zero-Shim Fit
65.2%
Total Cost
$2.06
Avg Cost/Case
<$0.01
Failures
FORMAT_INVALID:68, BUDGET_EXCEEDED:40, INTERRUPT_IGNORED:30, STRUCTURE_FAIL:13, TOOL_USAGE_FAIL:3, FORBIDDEN_TOOL_ATTEMPT:1, MEMORY_PERSISTENCE_FAIL:1, MEMORY_USE_FAIL:1
Rank
16
Model
glm-5-turbozai
Overall
56.7
Runs
4
Usable Fit
85.7%
Zero-Shim Fit
60.7%
Total Cost
$0.18
Avg Cost/Case
<$0.01
Failures
TOOL_USAGE_FAIL:9, TEST_FAIL:2
Rank
17
Model
deepseek-v4-prodeepseek
Overall
56.4
Runs
4
Usable Fit
82.1%
Zero-Shim Fit
75.0%
Total Cost
$0.31
Avg Cost/Case
$0.01
Failures
TOOL_USAGE_FAIL:3, FORBIDDEN_TOOL_ATTEMPT:1, FORMAT_INVALID:1, MEMORY_PERSISTENCE_FAIL:1, STRUCTURE_FAIL:1
Rank
18
Model
claude-sonnet-4-6anthropic
Overall
55.7
Runs
21
Usable Fit
78.0%
Zero-Shim Fit
76.8%
Total Cost
$2.60
Avg Cost/Case
$0.03
Failures
TEST_FAIL:11, STRUCTURE_FAIL:4, TOOL_USAGE_FAIL:3, MEMORY_USE_FAIL:1
Rank
19
Model
claude-opus-4-8anthropic
Overall
54.3
Runs
3
Usable Fit
90.5%
Zero-Shim Fit
38.1%
Total Cost
$0.94
Avg Cost/Case
$0.04
Failures
FORMAT_INVALID:12, STRUCTURE_FAIL:1
Rank
20
Model
grok-4.20-0309-reasoninggrok
Overall
53.4
Runs
4
Usable Fit
81.8%
Zero-Shim Fit
63.6%
Total Cost
$0.97
Avg Cost/Case
$0.04
Failures
TOOL_USAGE_FAIL:4, MEMORY_PERSISTENCE_FAIL:3, FORMAT_INVALID:1
Rank
21
Model
glm-5.1zai
Overall
52.7
Runs
3
Usable Fit
81.0%
Zero-Shim Fit
66.7%
Total Cost
$0.22
Avg Cost/Case
$0.01
Failures
TOOL_USAGE_FAIL:3, FORBIDDEN_FILE_ATTEMPT:2, FORMAT_INVALID:1, TEST_FAIL:1
Rank
22
Model
qwen3.5-plusqwen
Overall
50.9
Runs
4
Usable Fit
72.7%
Zero-Shim Fit
72.7%
Total Cost
$0.19
Avg Cost/Case
<$0.01
Failures
FORMAT_INVALID:2, STRUCTURE_FAIL:2, MEMORY_PERSISTENCE_FAIL:1, TEST_FAIL:1
Rank
23
Model
qwen3.6-plus-2026-04-02qwen
Overall
47.9
Runs
3
Usable Fit
76.2%
Zero-Shim Fit
57.1%
Total Cost
$0.38
Avg Cost/Case
$0.02
Failures
TOOL_USAGE_FAIL:5, MEMORY_PERSISTENCE_FAIL:3, TEST_FAIL:1
Rank
24
Model
claude-opus-4-6anthropic
Overall
44.2
Runs
3
Usable Fit
85.7%
Zero-Shim Fit
14.3%
Total Cost
$0.78
Avg Cost/Case
$0.04
Failures
FORMAT_INVALID:18
Rank
25
Model
glm-5.2zai
Overall
43.4
Runs
1
Usable Fit
71.4%
Zero-Shim Fit
57.1%
Total Cost
$0.14
Avg Cost/Case
$0.02
Failures
TOOL_USAGE_FAIL:2, STRUCTURE_FAIL:1
Rank
26
Model
GPT-5.5openai
Overall
41.1
Runs
4
Usable Fit
71.4%
Zero-Shim Fit
60.7%
Total Cost
$1.14
Avg Cost/Case
$0.04
Failures
FORMAT_INVALID:6, TOOL_USAGE_FAIL:4, STRUCTURE_FAIL:1
Rank
27
Model
gpt-5.6-terraopenai
Overall
35.4
Runs
1
Usable Fit
71.4%
Zero-Shim Fit
42.9%
Total Cost
$0.09
Avg Cost/Case
$0.01
Failures
FORMAT_INVALID:4
Rank
28
Model
qwen3.6-max-previewqwen
Overall
33.9
Runs
3
Usable Fit
66.7%
Zero-Shim Fit
57.1%
Total Cost
$0.36
Avg Cost/Case
$0.02
Failures
TOOL_USAGE_FAIL:4, STRUCTURE_FAIL:3, FORMAT_INVALID:1, TEST_FAIL:1
Rank
29
Model
claude-sonnet-5anthropic
Overall
28.7
Runs
1
Usable Fit
57.1%
Zero-Shim Fit
42.9%
Total Cost
$0.32
Avg Cost/Case
$0.05
Failures
STRUCTURE_FAIL:2, FORMAT_INVALID:1, TOOL_USAGE_FAIL:1
Rank
30
Model
gemma-4-31b-itgoogle
Overall
28.6
Runs
3
Usable Fit
57.1%
Zero-Shim Fit
0.0%
Total Cost
$0.00
Avg Cost/Case
$0.00
Failures
FORMAT_INVALID:21
Rank
31
Model
MiniMax-M2.7minimax
Overall
25.2
Runs
4
Usable Fit
50.0%
Zero-Shim Fit
0.0%
Total Cost
$0.07
Avg Cost/Case
<$0.01
Failures
FORMAT_INVALID:22
Rank
32
Model
gpt-5.4-xhighopenai
Overall
24.9
Runs
4
Usable Fit
46.4%
Zero-Shim Fit
28.6%
Total Cost
$2.25
Avg Cost/Case
$0.08
Failures
FORMAT_INVALID:16, STRUCTURE_FAIL:4
Rank
33
Model
gpt-5.4openai
Overall
19.5
Runs
39
Usable Fit
54.4%
Zero-Shim Fit
28.2%
Total Cost
$0.83
Avg Cost/Case
<$0.01
Failures
FORMAT_INVALID:67, STRUCTURE_FAIL:4, TOOL_USAGE_FAIL:2, MEMORY_USE_FAIL:1
Rank
34
Model
gemini-3.1-flash-litegoogle
Overall
13.6
Runs
3
Usable Fit
42.9%
Zero-Shim Fit
38.1%
Total Cost
$0.04
Avg Cost/Case
<$0.01
Failures
TOOL_USAGE_FAIL:6, ACTION_LOOP_LIMIT:3, STRUCTURE_FAIL:3, TEST_FAIL:1
Rank
35
Model
deepseek-chatdeepseek
Overall
10.8
Runs
4
Usable Fit
22.7%
Zero-Shim Fit
22.7%
Total Cost
$0.03
Avg Cost/Case
<$0.01
Failures
ACTION_LOOP_LIMIT:10, STRUCTURE_FAIL:3, FORBIDDEN_FILE_ATTEMPT:2, TEST_FAIL:2
Rank
36
Model
gpt-5.6-lunaopenai
Overall
9.5
Runs
1
Usable Fit
42.9%
Zero-Shim Fit
42.9%
Total Cost
$0.08
Avg Cost/Case
$0.01
Failures
FORMAT_INVALID:4
Numeric columns are sortable. Currency is rounded to two decimals for readability.
Runtime fit run counts vary by model based on accumulated runs; Overall score and usable/zero-shim percentages are the normalized comparison basis.