Rank
1
Model
glm-5zai
Overall
82.5
Runs
3
Usable Fit
100.0%
Zero-Shim Fit
90.5%
Total Cost
$0.22
Avg Cost/Case
$0.01
Failures
FORMAT_INVALID:1, TOOL_USAGE_FAIL:1
Agent runtime benchmark
Hermes Benchmark Leaderboard
The Hermes runtime fit benchmark tests whether models behave cleanly inside a soul-style persistent runtime loop. Usable Fit is the primary metric. Zero-Shim Fit is the stricter secondary metric and indicates cleaner operation without additional shim/orchestrator help.
Showing 34 of 34
| Model | Failures | |||||||
|---|---|---|---|---|---|---|---|---|
| 1 | glm-5zai | 82.5 | 3 | 100.0% | 90.5% | $0.22 | $0.01 | FORMAT_INVALID:1, TOOL_USAGE_FAIL:1 |
| 2 | gemini-3.5-flashgoogle | 79.8 | 1 | 100.0% | 85.7% | $0.09 | $0.01 | FORMAT_INVALID:1 |
| 3 | grok-4.5grok | 79.5 | 4 | 100.0% | 85.7% | $0.45 | $0.02 | TOOL_USAGE_FAIL:4 |
| 4 | kimi-k3moonshot | 78.7 | 1 | 100.0% | 85.7% | $0.33 | $0.05 | TOOL_USAGE_FAIL:1 |
| 5 | claude-fable-5anthropic | 78.4 | 2 | 100.0% | 85.7% | $1.59 | $0.11 | TOOL_USAGE_FAIL:2 |
| 6 | glm-5.1zai | 77.8 | 3 | 100.0% | 81.0% | $0.23 | $0.01 | TOOL_USAGE_FAIL:4 |
| 7 | gemini-3.1-pro-previewgoogle | 76.5 | 4 | 100.0% | 78.6% | $0.34 | $0.01 | TOOL_USAGE_FAIL:6 |
| 8 | grok-4.20-0309-reasoninggrok | 71.3 | 3 | 95.2% | 76.2% | $0.88 | $0.04 | TOOL_USAGE_FAIL:3, FORMAT_INVALID:1, MEMORY_PERSISTENCE_FAIL:1 |
| 9 | kimi-k2.5moonshot | 70.4 | 3 | 100.0% | 66.7% | $0.31 | $0.01 | TOOL_USAGE_FAIL:4, FORMAT_INVALID:3 |
| 10 | qwen3.7-plusqwen | 66.0 | 1 | 100.0% | 57.1% | $0.09 | $0.01 | TOOL_USAGE_FAIL:3 |
| 11 | kimi-k2.6moonshot | 65.4 | 3 | 95.2% | 52.4% | $0.43 | $0.02 | FORMAT_INVALID:5, TOOL_USAGE_FAIL:4, STRUCTURE_FAIL:1 |
| 12 | kimi-k2.7-code-highspeedmoonshot | 65.4 | 1 | 85.7% | 42.9% | $0.15 | $0.02 | TOOL_USAGE_FAIL:2, FORMAT_INVALID:1, STRUCTURE_FAIL:1 |
| 13 | claude-opus-5anthropic | 64.8 | 1 | 85.7% | 42.9% | $0.49 | $0.07 | FORMAT_INVALID:2, INTERRUPT_IGNORED:1, TOOL_USAGE_FAIL:1 |
| 14 | deepseek-v4-prodeepseek | 62.9 | 5 | 88.6% | 74.3% | $0.48 | $0.01 | TOOL_USAGE_FAIL:6, ACTION_LOOP_LIMIT:1, FORMAT_INVALID:1, MEMORY_PERSISTENCE_FAIL:1 |
| 15 | qwen3.6-plus-2026-04-02qwen | 61.7 | 3 | 85.7% | 66.7% | $0.38 | $0.02 | TOOL_USAGE_FAIL:4, MEMORY_PERSISTENCE_FAIL:2, TEST_FAIL:1 |
| 16 | grok-4.3grok | 61.1 | 4 | 89.3% | 75.0% | $0.39 | $0.01 | TOOL_USAGE_FAIL:4, FORMAT_INVALID:3 |
| 17 | gpt-5.6-solopenai | 59.9 | 1 | 85.7% | 71.4% | $0.23 | $0.03 | STRUCTURE_FAIL:1, TOOL_USAGE_FAIL:1 |
| 18 | claude-opus-4-7anthropic | 58.3 | 3 | 90.5% | 61.9% | $1.02 | $0.05 | FORMAT_INVALID:5, TOOL_USAGE_FAIL:3 |
| 19 | claude-sonnet-4-6anthropic | 58.2 | 20 | 80.0% | 80.0% | $2.55 | $0.03 | TEST_FAIL:12, STRUCTURE_FAIL:2, MEMORY_USE_FAIL:1, TOOL_USAGE_FAIL:1 |
| 20 | qwen3.6-max-previewqwen | 52.0 | 3 | 85.7% | 42.9% | $0.81 | $0.04 | TOOL_USAGE_FAIL:9, STRUCTURE_FAIL:2, TEST_FAIL:1 |
| 21 | claude-opus-4-8anthropic | 48.4 | 3 | 71.4% | 28.6% | $0.90 | $0.04 | FORMAT_INVALID:11, STRUCTURE_FAIL:4 |
| 22 | GPT-5.5openai | 45.2 | 4 | 67.9% | 46.4% | $1.08 | $0.04 | TOOL_USAGE_FAIL:7, STRUCTURE_FAIL:5, FORMAT_INVALID:3 |
| 23 | claude-opus-4-6anthropic | 44.3 | 3 | 85.7% | 14.3% | $0.71 | $0.03 | FORMAT_INVALID:18 |
| 24 | claude-sonnet-5anthropic | 41.6 | 1 | 57.1% | 42.9% | $0.32 | $0.05 | ACTION_LOOP_LIMIT:1, FORMAT_INVALID:1, STRUCTURE_FAIL:1, TOOL_USAGE_FAIL:1 |
| 25 | glm-5-turbozai | 39.0 | 4 | 60.7% | 39.3% | $0.18 | <$0.01 | TOOL_USAGE_FAIL:7, FORMAT_INVALID:5, STRUCTURE_FAIL:3, MEMORY_PERSISTENCE_FAIL:1, TEST_FAIL:1 |
| 26 | qwen3.5-plusqwen | 38.5 | 3 | 66.7% | 66.7% | $0.18 | <$0.01 | FORMAT_INVALID:3, TOOL_USAGE_FAIL:2, STRUCTURE_FAIL:1, TEST_FAIL:1 |
| 27 | gpt-5.4-xhighopenai | 34.8 | 3 | 57.1% | 28.6% | $2.12 | $0.10 | FORMAT_INVALID:14, STRUCTURE_FAIL:1 |
| 28 | gpt-5.6-lunaopenai | 23.9 | 1 | 42.9% | 28.6% | $0.06 | <$0.01 | FORMAT_INVALID:4, STRUCTURE_FAIL:1 |
| 29 | gpt-5.6-terraopenai | 23.6 | 1 | 57.1% | 42.9% | $0.09 | $0.01 | FORMAT_INVALID:4 |
| 30 | MiniMax-M2.7minimax | 17.8 | 3 | 33.3% | 0.0% | $0.06 | <$0.01 | FORMAT_INVALID:21 |
| 31 | gpt-5.2openai | 15.5 | 18 | 44.4% | 44.4% | $0.60 | <$0.01 | FORMAT_INVALID:24, STRUCTURE_FAIL:7, TOOL_USAGE_FAIL:3, MEMORY_USE_FAIL:1 |
| 32 | gpt-5.4openai | 14.1 | 20 | 44.6% | 34.9% | $0.77 | <$0.01 | FORMAT_INVALID:49, STRUCTURE_FAIL:4, MEMORY_USE_FAIL:1 |
| 33 | deepseek-chatdeepseek | 10.9 | 3 | 28.6% | 28.6% | $0.03 | <$0.01 | ACTION_LOOP_LIMIT:10, STRUCTURE_FAIL:3, FORBIDDEN_FILE_ATTEMPT:2 |
| 34 | gemini-3.1-flash-litegoogle | 4.3 | 3 | 33.3% | 33.3% | $0.04 | <$0.01 | TOOL_USAGE_FAIL:6, TEST_FAIL:4, STRUCTURE_FAIL:3, ACTION_LOOP_LIMIT:1 |
Rank
2
Model
gemini-3.5-flashgoogle
Overall
79.8
Runs
1
Usable Fit
100.0%
Zero-Shim Fit
85.7%
Total Cost
$0.09
Avg Cost/Case
$0.01
Failures
FORMAT_INVALID:1
Rank
3
Model
grok-4.5grok
Overall
79.5
Runs
4
Usable Fit
100.0%
Zero-Shim Fit
85.7%
Total Cost
$0.45
Avg Cost/Case
$0.02
Failures
TOOL_USAGE_FAIL:4
Rank
4
Model
kimi-k3moonshot
Overall
78.7
Runs
1
Usable Fit
100.0%
Zero-Shim Fit
85.7%
Total Cost
$0.33
Avg Cost/Case
$0.05
Failures
TOOL_USAGE_FAIL:1
Rank
5
Model
claude-fable-5anthropic
Overall
78.4
Runs
2
Usable Fit
100.0%
Zero-Shim Fit
85.7%
Total Cost
$1.59
Avg Cost/Case
$0.11
Failures
TOOL_USAGE_FAIL:2
Rank
6
Model
glm-5.1zai
Overall
77.8
Runs
3
Usable Fit
100.0%
Zero-Shim Fit
81.0%
Total Cost
$0.23
Avg Cost/Case
$0.01
Failures
TOOL_USAGE_FAIL:4
Rank
7
Model
gemini-3.1-pro-previewgoogle
Overall
76.5
Runs
4
Usable Fit
100.0%
Zero-Shim Fit
78.6%
Total Cost
$0.34
Avg Cost/Case
$0.01
Failures
TOOL_USAGE_FAIL:6
Rank
8
Model
grok-4.20-0309-reasoninggrok
Overall
71.3
Runs
3
Usable Fit
95.2%
Zero-Shim Fit
76.2%
Total Cost
$0.88
Avg Cost/Case
$0.04
Failures
TOOL_USAGE_FAIL:3, FORMAT_INVALID:1, MEMORY_PERSISTENCE_FAIL:1
Rank
9
Model
kimi-k2.5moonshot
Overall
70.4
Runs
3
Usable Fit
100.0%
Zero-Shim Fit
66.7%
Total Cost
$0.31
Avg Cost/Case
$0.01
Failures
TOOL_USAGE_FAIL:4, FORMAT_INVALID:3
Rank
10
Model
qwen3.7-plusqwen
Overall
66.0
Runs
1
Usable Fit
100.0%
Zero-Shim Fit
57.1%
Total Cost
$0.09
Avg Cost/Case
$0.01
Failures
TOOL_USAGE_FAIL:3
Rank
11
Model
kimi-k2.6moonshot
Overall
65.4
Runs
3
Usable Fit
95.2%
Zero-Shim Fit
52.4%
Total Cost
$0.43
Avg Cost/Case
$0.02
Failures
FORMAT_INVALID:5, TOOL_USAGE_FAIL:4, STRUCTURE_FAIL:1
Rank
12
Model
kimi-k2.7-code-highspeedmoonshot
Overall
65.4
Runs
1
Usable Fit
85.7%
Zero-Shim Fit
42.9%
Total Cost
$0.15
Avg Cost/Case
$0.02
Failures
TOOL_USAGE_FAIL:2, FORMAT_INVALID:1, STRUCTURE_FAIL:1
Rank
13
Model
claude-opus-5anthropic
Overall
64.8
Runs
1
Usable Fit
85.7%
Zero-Shim Fit
42.9%
Total Cost
$0.49
Avg Cost/Case
$0.07
Failures
FORMAT_INVALID:2, INTERRUPT_IGNORED:1, TOOL_USAGE_FAIL:1
Rank
14
Model
deepseek-v4-prodeepseek
Overall
62.9
Runs
5
Usable Fit
88.6%
Zero-Shim Fit
74.3%
Total Cost
$0.48
Avg Cost/Case
$0.01
Failures
TOOL_USAGE_FAIL:6, ACTION_LOOP_LIMIT:1, FORMAT_INVALID:1, MEMORY_PERSISTENCE_FAIL:1
Rank
15
Model
qwen3.6-plus-2026-04-02qwen
Overall
61.7
Runs
3
Usable Fit
85.7%
Zero-Shim Fit
66.7%
Total Cost
$0.38
Avg Cost/Case
$0.02
Failures
TOOL_USAGE_FAIL:4, MEMORY_PERSISTENCE_FAIL:2, TEST_FAIL:1
Rank
16
Model
grok-4.3grok
Overall
61.1
Runs
4
Usable Fit
89.3%
Zero-Shim Fit
75.0%
Total Cost
$0.39
Avg Cost/Case
$0.01
Failures
TOOL_USAGE_FAIL:4, FORMAT_INVALID:3
Rank
17
Model
gpt-5.6-solopenai
Overall
59.9
Runs
1
Usable Fit
85.7%
Zero-Shim Fit
71.4%
Total Cost
$0.23
Avg Cost/Case
$0.03
Failures
STRUCTURE_FAIL:1, TOOL_USAGE_FAIL:1
Rank
18
Model
claude-opus-4-7anthropic
Overall
58.3
Runs
3
Usable Fit
90.5%
Zero-Shim Fit
61.9%
Total Cost
$1.02
Avg Cost/Case
$0.05
Failures
FORMAT_INVALID:5, TOOL_USAGE_FAIL:3
Rank
19
Model
claude-sonnet-4-6anthropic
Overall
58.2
Runs
20
Usable Fit
80.0%
Zero-Shim Fit
80.0%
Total Cost
$2.55
Avg Cost/Case
$0.03
Failures
TEST_FAIL:12, STRUCTURE_FAIL:2, MEMORY_USE_FAIL:1, TOOL_USAGE_FAIL:1
Rank
20
Model
qwen3.6-max-previewqwen
Overall
52.0
Runs
3
Usable Fit
85.7%
Zero-Shim Fit
42.9%
Total Cost
$0.81
Avg Cost/Case
$0.04
Failures
TOOL_USAGE_FAIL:9, STRUCTURE_FAIL:2, TEST_FAIL:1
Rank
21
Model
claude-opus-4-8anthropic
Overall
48.4
Runs
3
Usable Fit
71.4%
Zero-Shim Fit
28.6%
Total Cost
$0.90
Avg Cost/Case
$0.04
Failures
FORMAT_INVALID:11, STRUCTURE_FAIL:4
Rank
22
Model
GPT-5.5openai
Overall
45.2
Runs
4
Usable Fit
67.9%
Zero-Shim Fit
46.4%
Total Cost
$1.08
Avg Cost/Case
$0.04
Failures
TOOL_USAGE_FAIL:7, STRUCTURE_FAIL:5, FORMAT_INVALID:3
Rank
23
Model
claude-opus-4-6anthropic
Overall
44.3
Runs
3
Usable Fit
85.7%
Zero-Shim Fit
14.3%
Total Cost
$0.71
Avg Cost/Case
$0.03
Failures
FORMAT_INVALID:18
Rank
24
Model
claude-sonnet-5anthropic
Overall
41.6
Runs
1
Usable Fit
57.1%
Zero-Shim Fit
42.9%
Total Cost
$0.32
Avg Cost/Case
$0.05
Failures
ACTION_LOOP_LIMIT:1, FORMAT_INVALID:1, STRUCTURE_FAIL:1, TOOL_USAGE_FAIL:1
Rank
25
Model
glm-5-turbozai
Overall
39.0
Runs
4
Usable Fit
60.7%
Zero-Shim Fit
39.3%
Total Cost
$0.18
Avg Cost/Case
<$0.01
Failures
TOOL_USAGE_FAIL:7, FORMAT_INVALID:5, STRUCTURE_FAIL:3, MEMORY_PERSISTENCE_FAIL:1, TEST_FAIL:1
Rank
26
Model
qwen3.5-plusqwen
Overall
38.5
Runs
3
Usable Fit
66.7%
Zero-Shim Fit
66.7%
Total Cost
$0.18
Avg Cost/Case
<$0.01
Failures
FORMAT_INVALID:3, TOOL_USAGE_FAIL:2, STRUCTURE_FAIL:1, TEST_FAIL:1
Rank
27
Model
gpt-5.4-xhighopenai
Overall
34.8
Runs
3
Usable Fit
57.1%
Zero-Shim Fit
28.6%
Total Cost
$2.12
Avg Cost/Case
$0.10
Failures
FORMAT_INVALID:14, STRUCTURE_FAIL:1
Rank
28
Model
gpt-5.6-lunaopenai
Overall
23.9
Runs
1
Usable Fit
42.9%
Zero-Shim Fit
28.6%
Total Cost
$0.06
Avg Cost/Case
<$0.01
Failures
FORMAT_INVALID:4, STRUCTURE_FAIL:1
Rank
29
Model
gpt-5.6-terraopenai
Overall
23.6
Runs
1
Usable Fit
57.1%
Zero-Shim Fit
42.9%
Total Cost
$0.09
Avg Cost/Case
$0.01
Failures
FORMAT_INVALID:4
Rank
30
Model
MiniMax-M2.7minimax
Overall
17.8
Runs
3
Usable Fit
33.3%
Zero-Shim Fit
0.0%
Total Cost
$0.06
Avg Cost/Case
<$0.01
Failures
FORMAT_INVALID:21
Rank
31
Model
gpt-5.2openai
Overall
15.5
Runs
18
Usable Fit
44.4%
Zero-Shim Fit
44.4%
Total Cost
$0.60
Avg Cost/Case
<$0.01
Failures
FORMAT_INVALID:24, STRUCTURE_FAIL:7, TOOL_USAGE_FAIL:3, MEMORY_USE_FAIL:1
Rank
32
Model
gpt-5.4openai
Overall
14.1
Runs
20
Usable Fit
44.6%
Zero-Shim Fit
34.9%
Total Cost
$0.77
Avg Cost/Case
<$0.01
Failures
FORMAT_INVALID:49, STRUCTURE_FAIL:4, MEMORY_USE_FAIL:1
Rank
33
Model
deepseek-chatdeepseek
Overall
10.9
Runs
3
Usable Fit
28.6%
Zero-Shim Fit
28.6%
Total Cost
$0.03
Avg Cost/Case
<$0.01
Failures
ACTION_LOOP_LIMIT:10, STRUCTURE_FAIL:3, FORBIDDEN_FILE_ATTEMPT:2
Rank
34
Model
gemini-3.1-flash-litegoogle
Overall
4.3
Runs
3
Usable Fit
33.3%
Zero-Shim Fit
33.3%
Total Cost
$0.04
Avg Cost/Case
<$0.01
Failures
TOOL_USAGE_FAIL:6, TEST_FAIL:4, STRUCTURE_FAIL:3, ACTION_LOOP_LIMIT:1
Numeric columns are sortable. Currency is rounded to two decimals for readability.
Runtime fit run counts vary by model based on accumulated runs; Overall score and usable/zero-shim percentages are the normalized comparison basis.