Rank
1
Model
kimi-k2.7-codemoonshot
Result
3/3 Strict / 3/3 ScoredSoft 0/3 · Hard 0/3
Stages
6/6
Init
3/3
Repairs
0/3
Output
46685
Cost
$0.34
Wall
811.5
Cases
Bug FixPRefactorPMigrationP
Multi-turn model evaluation
Multi-Turn Benchmark Leaderboard
This multi-turn benchmark uses three tasks: bug-fix-02, refactor-02, and migration-02, each split across two staged turns for six total stages. Requirements are withheld until each stage is reached. Ranking order is: scored passes desc, strict passes desc, soft passes desc, completed stages desc, repairs used asc, total output tokens asc, total wall time asc, total cost asc, then stage-1 initial passes desc.
LegendPStrict passSSoft passF:*Failed case
Showing 39 of 39
| Model | Cases | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| 1 | kimi-k2.7-codemoonshot | 3/3 Strict / 3/3 ScoredSoft 0/3 · Hard 0/3 | 6/6 | 3/3 | 0/3 | 46685 | $0.34 | 811.5 | Bug FixPRefactorPMigrationP |
| 2 | gpt-5.6-terraopenai | 3/3 Strict / 3/3 ScoredSoft 0/3 · Hard 0/3 | 6/6 | 2/3 | 1/3 | 11151 | $0.60 | 94.4 | Bug FixPRefactorPMigrationP |
| 3 | gemini-3.5-flashgoogle | 3/3 Strict / 3/3 ScoredSoft 0/3 · Hard 0/3 | 6/6 | 2/3 | 1/3 | 13208 | $0.55 | 333.1 | Bug FixPRefactorPMigrationP |
| 4 | gpt-5.4openai | 3/3 Strict / 3/3 ScoredSoft 0/3 · Hard 0/3 | 6/6 | 2/3 | 1/3 | 13467 | $0.70 | 144.8 | Bug FixPRefactorPMigrationP |
| 5 | qwen3.5-plusqwen | 3/3 Strict / 3/3 ScoredSoft 0/3 · Hard 0/3 | 6/6 | 2/3 | 1/3 | 16719 | $0.12 | 186.5 | Bug FixPRefactorPMigrationP |
| 6 | claude-opus-5anthropic | 3/3 Strict / 3/3 ScoredSoft 0/3 · Hard 0/3 | 6/6 | 2/3 | 1/3 | 21267 | $1.82 | 203.9 | Bug FixPRefactorPMigrationP |
| 7 | gpt-5.3-codexopenai | 3/3 Strict / 3/3 ScoredSoft 0/3 · Hard 0/3 | 6/6 | 2/3 | 1/3 | 23459 | $0.63 | 367.0 | Bug FixPRefactorPMigrationP |
| 8 | GPT-5.5openai | 3/3 Strict / 3/3 ScoredSoft 0/3 · Hard 0/3 | 6/6 | 2/3 | 1/3 | 26611 | $1.65 | 328.2 | Bug FixPRefactorPMigrationP |
| 9 | deepseek-v4-prodeepseek | 3/3 Strict / 3/3 ScoredSoft 0/3 · Hard 0/3 | 6/6 | 2/3 | 1/3 | 30324 | $0.11 | 344.0 | Bug FixPRefactorPMigrationP |
| 10 | kimi-k2.5moonshot | 3/3 Strict / 3/3 ScoredSoft 0/3 · Hard 0/3 | 6/6 | 2/3 | 1/3 | 40544 | $0.23 | 935.6 | Bug FixPRefactorPMigrationP |
| 11 | kimi-k2.7-code-highspeedmoonshot | 3/3 Strict / 3/3 ScoredSoft 0/3 · Hard 0/3 | 6/6 | 2/3 | 1/3 | 41530 | $0.69 | 264.4 | Bug FixPRefactorPMigrationP |
| 12 | qwen3.6-plus-2026-04-02qwen | 3/3 Strict / 3/3 ScoredSoft 0/3 · Hard 0/3 | 6/6 | 2/3 | 1/3 | 53757 | $0.27 | 996.8 | Bug FixPRefactorPMigrationP |
| 13 | qwen3.7-maxqwen | 3/3 Strict / 3/3 ScoredSoft 0/3 · Hard 0/3 | 6/6 | 2/3 | 1/3 | 58931 | $0.82 | 1090.0 | Bug FixPRefactorPMigrationP |
| 14 | gemini-3.1-pro-previewgoogle | 3/3 Strict / 3/3 ScoredSoft 0/3 · Hard 0/3 | 6/6 | 2/3 | 2/3 | 13083 | $0.71 | 1253.1 | Bug FixPRefactorPMigrationP |
| 15 | grok-4.3grok | 3/3 Strict / 3/3 ScoredSoft 0/3 · Hard 0/3 | 6/6 | 1/3 | 2/3 | 24055 | $0.30 | 305.0 | Bug FixPRefactorPMigrationP |
| 16 | glm-5.1zai | 3/3 Strict / 3/3 ScoredSoft 0/3 · Hard 0/3 | 6/6 | 1/3 | 2/3 | 53999 | $0.59 | 1083.1 | Bug FixPRefactorPMigrationP |
| 17 | glm-5.2zai | 3/3 Strict / 3/3 ScoredSoft 0/3 · Hard 0/3 | 6/6 | 2/3 | 2/3 | 85545 | $0.68 | 1022.5 | Bug FixPRefactorPMigrationP |
| 18 | claude-opus-4-7anthropic | 3/3 Strict / 3/3 ScoredSoft 0/3 · Hard 0/3 | 6/6 | 1/3 | 3/3 | 15499 | $2.02 | 145.5 | Bug FixPRefactorPMigrationP |
| 19 | glm-5zai | 2/3 Strict / 3/3 ScoredSoft 1/3 · Hard 0/3 | 5/6 | 1/3 | 2/3 | 40943 | $0.40 | 785.8 | Bug FixPRefactorSMigrationP |
| 20 | claude-opus-4-8anthropic | 2/3 Strict / 2/3 ScoredSoft 0/3 · Hard 1/3 | 5/6 | 1/3 | 2/3 | 16099 | $2.21 | 148.8 | Bug FixPRefactorF:FORMATMigrationP |
| 21 | gpt-5.6-solopenai | 2/3 Strict / 2/3 ScoredSoft 0/3 · Hard 1/3 | 4/6 | 2/3 | 0/3 | 10079 | $0.96 | 129.2 | Bug FixPRefactorF:STRUCTURE_FAILMigrationP |
| 22 | gpt-5.6-lunaopenai | 2/3 Strict / 2/3 ScoredSoft 0/3 · Hard 1/3 | 4/6 | 2/3 | 0/3 | 10722 | $0.19 | 65.9 | Bug FixPRefactorF:STRUCTURE_FAILMigrationP |
| 23 | grok-4.5grok | 2/3 Strict / 2/3 ScoredSoft 0/3 · Hard 1/3 | 4/6 | 2/3 | 0/3 | 11891 | $0.23 | 76.7 | Bug FixPRefactorF:STRUCTURE_FAILMigrationP |
| 24 | claude-fable-5anthropic | 2/3 Strict / 2/3 ScoredSoft 0/3 · Hard 1/3 | 4/6 | 2/3 | 0/3 | 12334 | $2.38 | 136.2 | Bug FixPRefactorF:STRUCTURE_FAILMigrationP |
| 25 | qwen3.7-plusqwen | 2/3 Strict / 2/3 ScoredSoft 0/3 · Hard 1/3 | 4/6 | 2/3 | 0/3 | 28872 | $0.17 | 556.5 | Bug FixPRefactorF:STRUCTURE_FAILMigrationP |
| 26 | qwen3.6-max-previewqwen | 2/3 Strict / 2/3 ScoredSoft 0/3 · Hard 1/3 | 4/6 | 2/3 | 0/3 | 36347 | $0.54 | 893.9 | Bug FixPRefactorF:STRUCTURE_FAILMigrationP |
| 27 | kimi-k3moonshot | 2/3 Strict / 2/3 ScoredSoft 0/3 · Hard 1/3 | 4/6 | 2/3 | 0/3 | 42088 | $1.05 | 1431.2 | Bug FixPRefactorF:STRUCTURE_FAILMigrationP |
| 28 | kimi-k2.6moonshot | 2/3 Strict / 2/3 ScoredSoft 0/3 · Hard 1/3 | 4/6 | 2/3 | 0/3 | 69038 | $0.40 | 1942.9 | Bug FixPRefactorF:STRUCTURE_FAILMigrationP |
| 29 | deepseek-v4-flashdeepseek | 2/3 Strict / 2/3 ScoredSoft 0/3 · Hard 1/3 | 4/6 | 1/3 | 1/3 | 16890 | $0.03 | 162.3 | Bug FixPRefactorF:STRUCTURE_FAILMigrationP |
| 30 | glm-5-turbozai | 2/3 Strict / 2/3 ScoredSoft 0/3 · Hard 1/3 | 4/6 | 1/3 | 1/3 | 17196 | $0.27 | 289.7 | Bug FixPRefactorF:STRUCTURE_FAILMigrationP |
| 31 | claude-sonnet-5anthropic | 2/3 Strict / 2/3 ScoredSoft 0/3 · Hard 1/3 | 4/6 | 1/3 | 1/3 | 23304 | $1.04 | 175.4 | Bug FixPRefactorF:STRUCTURE_FAILMigrationP |
| 32 | grok-4.20-0309-reasoninggrok | 2/3 Strict / 2/3 ScoredSoft 0/3 · Hard 1/3 | 4/6 | 1/3 | 1/3 | 34710 | $0.52 | 170.5 | Bug FixPRefactorF:STRUCTURE_FAILMigrationP |
| 33 | grok-build-0.1grok | 2/3 Strict / 2/3 ScoredSoft 0/3 · Hard 1/3 | 4/6 | 1/3 | 1/3 | 47900 | $0.26 | 247.2 | Bug FixPRefactorF:STRUCTURE_FAILMigrationP |
| 34 | gemini-3.1-flash-litegoogle | 2/3 Strict / 2/3 ScoredSoft 0/3 · Hard 1/3 | 4/6 | 0/3 | 2/3 | 12158 | $0.08 | 41.8 | Bug FixPRefactorF:STRUCTURE_FAILMigrationP |
| 35 | claude-opus-4-6anthropic | 2/3 Strict / 2/3 ScoredSoft 0/3 · Hard 1/3 | 4/6 | 1/3 | 2/3 | 17122 | $1.47 | 167.3 | Bug FixPRefactorF:STRUCTURE_FAILMigrationP |
| 36 | deepseek-reasonerdeepseek | 1/3 Strict / 1/3 ScoredSoft 0/3 · Hard 2/3 | 3/6 | 1/3 | 2/3 | 21980 | $0.02 | 676.3 | Bug FixPRefactorF:STRUCTURE_FAILMigrationF:FORMAT |
| 37 | deepseek-chatdeepseek | 1/3 Strict / 1/3 ScoredSoft 0/3 · Hard 2/3 | 3/6 | 0/3 | 3/3 | 19806 | $0.11 | 537.1 | Bug FixF:TEST_FAILRefactorF:TEST_FAILMigrationP |
| 38 | grok-4.20-beta-latest-non-reasoninggrok | 1/3 Strict / 1/3 ScoredSoft 0/3 · Hard 2/3 | 2/6 | 1/3 | 1/3 | 9499 | $0.38 | 48.6 | Bug FixF:TEST_FAILRefactorF:STRUCTURE_FAILMigrationP |
| 39 | MiniMax-M2.7minimax | 0/3 Strict / 1/3 ScoredSoft 1/3 · Hard 2/3 | 0/6 | 0/3 | 3/3 | 43042 | $0.09 | 988.2 | Bug FixF:FORMATRefactorF:FORMATMigrationS |
Rank
2
Model
gpt-5.6-terraopenai
Result
3/3 Strict / 3/3 ScoredSoft 0/3 · Hard 0/3
Stages
6/6
Init
2/3
Repairs
1/3
Output
11151
Cost
$0.60
Wall
94.4
Cases
Bug FixPRefactorPMigrationP
Rank
3
Model
gemini-3.5-flashgoogle
Result
3/3 Strict / 3/3 ScoredSoft 0/3 · Hard 0/3
Stages
6/6
Init
2/3
Repairs
1/3
Output
13208
Cost
$0.55
Wall
333.1
Cases
Bug FixPRefactorPMigrationP
Rank
4
Model
gpt-5.4openai
Result
3/3 Strict / 3/3 ScoredSoft 0/3 · Hard 0/3
Stages
6/6
Init
2/3
Repairs
1/3
Output
13467
Cost
$0.70
Wall
144.8
Cases
Bug FixPRefactorPMigrationP
Rank
5
Model
qwen3.5-plusqwen
Result
3/3 Strict / 3/3 ScoredSoft 0/3 · Hard 0/3
Stages
6/6
Init
2/3
Repairs
1/3
Output
16719
Cost
$0.12
Wall
186.5
Cases
Bug FixPRefactorPMigrationP
Rank
6
Model
claude-opus-5anthropic
Result
3/3 Strict / 3/3 ScoredSoft 0/3 · Hard 0/3
Stages
6/6
Init
2/3
Repairs
1/3
Output
21267
Cost
$1.82
Wall
203.9
Cases
Bug FixPRefactorPMigrationP
Rank
7
Model
gpt-5.3-codexopenai
Result
3/3 Strict / 3/3 ScoredSoft 0/3 · Hard 0/3
Stages
6/6
Init
2/3
Repairs
1/3
Output
23459
Cost
$0.63
Wall
367.0
Cases
Bug FixPRefactorPMigrationP
Rank
8
Model
GPT-5.5openai
Result
3/3 Strict / 3/3 ScoredSoft 0/3 · Hard 0/3
Stages
6/6
Init
2/3
Repairs
1/3
Output
26611
Cost
$1.65
Wall
328.2
Cases
Bug FixPRefactorPMigrationP
Rank
9
Model
deepseek-v4-prodeepseek
Result
3/3 Strict / 3/3 ScoredSoft 0/3 · Hard 0/3
Stages
6/6
Init
2/3
Repairs
1/3
Output
30324
Cost
$0.11
Wall
344.0
Cases
Bug FixPRefactorPMigrationP
Rank
10
Model
kimi-k2.5moonshot
Result
3/3 Strict / 3/3 ScoredSoft 0/3 · Hard 0/3
Stages
6/6
Init
2/3
Repairs
1/3
Output
40544
Cost
$0.23
Wall
935.6
Cases
Bug FixPRefactorPMigrationP
Rank
11
Model
kimi-k2.7-code-highspeedmoonshot
Result
3/3 Strict / 3/3 ScoredSoft 0/3 · Hard 0/3
Stages
6/6
Init
2/3
Repairs
1/3
Output
41530
Cost
$0.69
Wall
264.4
Cases
Bug FixPRefactorPMigrationP
Rank
12
Model
qwen3.6-plus-2026-04-02qwen
Result
3/3 Strict / 3/3 ScoredSoft 0/3 · Hard 0/3
Stages
6/6
Init
2/3
Repairs
1/3
Output
53757
Cost
$0.27
Wall
996.8
Cases
Bug FixPRefactorPMigrationP
Rank
13
Model
qwen3.7-maxqwen
Result
3/3 Strict / 3/3 ScoredSoft 0/3 · Hard 0/3
Stages
6/6
Init
2/3
Repairs
1/3
Output
58931
Cost
$0.82
Wall
1090.0
Cases
Bug FixPRefactorPMigrationP
Rank
14
Model
gemini-3.1-pro-previewgoogle
Result
3/3 Strict / 3/3 ScoredSoft 0/3 · Hard 0/3
Stages
6/6
Init
2/3
Repairs
2/3
Output
13083
Cost
$0.71
Wall
1253.1
Cases
Bug FixPRefactorPMigrationP
Rank
15
Model
grok-4.3grok
Result
3/3 Strict / 3/3 ScoredSoft 0/3 · Hard 0/3
Stages
6/6
Init
1/3
Repairs
2/3
Output
24055
Cost
$0.30
Wall
305.0
Cases
Bug FixPRefactorPMigrationP
Rank
16
Model
glm-5.1zai
Result
3/3 Strict / 3/3 ScoredSoft 0/3 · Hard 0/3
Stages
6/6
Init
1/3
Repairs
2/3
Output
53999
Cost
$0.59
Wall
1083.1
Cases
Bug FixPRefactorPMigrationP
Rank
17
Model
glm-5.2zai
Result
3/3 Strict / 3/3 ScoredSoft 0/3 · Hard 0/3
Stages
6/6
Init
2/3
Repairs
2/3
Output
85545
Cost
$0.68
Wall
1022.5
Cases
Bug FixPRefactorPMigrationP
Rank
18
Model
claude-opus-4-7anthropic
Result
3/3 Strict / 3/3 ScoredSoft 0/3 · Hard 0/3
Stages
6/6
Init
1/3
Repairs
3/3
Output
15499
Cost
$2.02
Wall
145.5
Cases
Bug FixPRefactorPMigrationP
Rank
19
Model
glm-5zai
Result
2/3 Strict / 3/3 ScoredSoft 1/3 · Hard 0/3
Stages
5/6
Init
1/3
Repairs
2/3
Output
40943
Cost
$0.40
Wall
785.8
Cases
Bug FixPRefactorSMigrationP
Rank
20
Model
claude-opus-4-8anthropic
Result
2/3 Strict / 2/3 ScoredSoft 0/3 · Hard 1/3
Stages
5/6
Init
1/3
Repairs
2/3
Output
16099
Cost
$2.21
Wall
148.8
Cases
Bug FixPRefactorF:FORMATMigrationP
Rank
21
Model
gpt-5.6-solopenai
Result
2/3 Strict / 2/3 ScoredSoft 0/3 · Hard 1/3
Stages
4/6
Init
2/3
Repairs
0/3
Output
10079
Cost
$0.96
Wall
129.2
Cases
Bug FixPRefactorF:STRUCTURE_FAILMigrationP
Rank
22
Model
gpt-5.6-lunaopenai
Result
2/3 Strict / 2/3 ScoredSoft 0/3 · Hard 1/3
Stages
4/6
Init
2/3
Repairs
0/3
Output
10722
Cost
$0.19
Wall
65.9
Cases
Bug FixPRefactorF:STRUCTURE_FAILMigrationP
Rank
23
Model
grok-4.5grok
Result
2/3 Strict / 2/3 ScoredSoft 0/3 · Hard 1/3
Stages
4/6
Init
2/3
Repairs
0/3
Output
11891
Cost
$0.23
Wall
76.7
Cases
Bug FixPRefactorF:STRUCTURE_FAILMigrationP
Rank
24
Model
claude-fable-5anthropic
Result
2/3 Strict / 2/3 ScoredSoft 0/3 · Hard 1/3
Stages
4/6
Init
2/3
Repairs
0/3
Output
12334
Cost
$2.38
Wall
136.2
Cases
Bug FixPRefactorF:STRUCTURE_FAILMigrationP
Rank
25
Model
qwen3.7-plusqwen
Result
2/3 Strict / 2/3 ScoredSoft 0/3 · Hard 1/3
Stages
4/6
Init
2/3
Repairs
0/3
Output
28872
Cost
$0.17
Wall
556.5
Cases
Bug FixPRefactorF:STRUCTURE_FAILMigrationP
Rank
26
Model
qwen3.6-max-previewqwen
Result
2/3 Strict / 2/3 ScoredSoft 0/3 · Hard 1/3
Stages
4/6
Init
2/3
Repairs
0/3
Output
36347
Cost
$0.54
Wall
893.9
Cases
Bug FixPRefactorF:STRUCTURE_FAILMigrationP
Rank
27
Model
kimi-k3moonshot
Result
2/3 Strict / 2/3 ScoredSoft 0/3 · Hard 1/3
Stages
4/6
Init
2/3
Repairs
0/3
Output
42088
Cost
$1.05
Wall
1431.2
Cases
Bug FixPRefactorF:STRUCTURE_FAILMigrationP
Rank
28
Model
kimi-k2.6moonshot
Result
2/3 Strict / 2/3 ScoredSoft 0/3 · Hard 1/3
Stages
4/6
Init
2/3
Repairs
0/3
Output
69038
Cost
$0.40
Wall
1942.9
Cases
Bug FixPRefactorF:STRUCTURE_FAILMigrationP
Rank
29
Model
deepseek-v4-flashdeepseek
Result
2/3 Strict / 2/3 ScoredSoft 0/3 · Hard 1/3
Stages
4/6
Init
1/3
Repairs
1/3
Output
16890
Cost
$0.03
Wall
162.3
Cases
Bug FixPRefactorF:STRUCTURE_FAILMigrationP
Rank
30
Model
glm-5-turbozai
Result
2/3 Strict / 2/3 ScoredSoft 0/3 · Hard 1/3
Stages
4/6
Init
1/3
Repairs
1/3
Output
17196
Cost
$0.27
Wall
289.7
Cases
Bug FixPRefactorF:STRUCTURE_FAILMigrationP
Rank
31
Model
claude-sonnet-5anthropic
Result
2/3 Strict / 2/3 ScoredSoft 0/3 · Hard 1/3
Stages
4/6
Init
1/3
Repairs
1/3
Output
23304
Cost
$1.04
Wall
175.4
Cases
Bug FixPRefactorF:STRUCTURE_FAILMigrationP
Rank
32
Model
grok-4.20-0309-reasoninggrok
Result
2/3 Strict / 2/3 ScoredSoft 0/3 · Hard 1/3
Stages
4/6
Init
1/3
Repairs
1/3
Output
34710
Cost
$0.52
Wall
170.5
Cases
Bug FixPRefactorF:STRUCTURE_FAILMigrationP
Rank
33
Model
grok-build-0.1grok
Result
2/3 Strict / 2/3 ScoredSoft 0/3 · Hard 1/3
Stages
4/6
Init
1/3
Repairs
1/3
Output
47900
Cost
$0.26
Wall
247.2
Cases
Bug FixPRefactorF:STRUCTURE_FAILMigrationP
Rank
34
Model
gemini-3.1-flash-litegoogle
Result
2/3 Strict / 2/3 ScoredSoft 0/3 · Hard 1/3
Stages
4/6
Init
0/3
Repairs
2/3
Output
12158
Cost
$0.08
Wall
41.8
Cases
Bug FixPRefactorF:STRUCTURE_FAILMigrationP
Rank
35
Model
claude-opus-4-6anthropic
Result
2/3 Strict / 2/3 ScoredSoft 0/3 · Hard 1/3
Stages
4/6
Init
1/3
Repairs
2/3
Output
17122
Cost
$1.47
Wall
167.3
Cases
Bug FixPRefactorF:STRUCTURE_FAILMigrationP
Rank
36
Model
deepseek-reasonerdeepseek
Result
1/3 Strict / 1/3 ScoredSoft 0/3 · Hard 2/3
Stages
3/6
Init
1/3
Repairs
2/3
Output
21980
Cost
$0.02
Wall
676.3
Cases
Bug FixPRefactorF:STRUCTURE_FAILMigrationF:FORMAT
Rank
37
Model
deepseek-chatdeepseek
Result
1/3 Strict / 1/3 ScoredSoft 0/3 · Hard 2/3
Stages
3/6
Init
0/3
Repairs
3/3
Output
19806
Cost
$0.11
Wall
537.1
Cases
Bug FixF:TEST_FAILRefactorF:TEST_FAILMigrationP
Rank
38
Model
grok-4.20-beta-latest-non-reasoninggrok
Result
1/3 Strict / 1/3 ScoredSoft 0/3 · Hard 2/3
Stages
2/6
Init
1/3
Repairs
1/3
Output
9499
Cost
$0.38
Wall
48.6
Cases
Bug FixF:TEST_FAILRefactorF:STRUCTURE_FAILMigrationP
Rank
39
Model
MiniMax-M2.7minimax
Result
0/3 Strict / 1/3 ScoredSoft 1/3 · Hard 2/3
Stages
0/6
Init
0/3
Repairs
3/3
Output
43042
Cost
$0.09
Wall
988.2
Cases
Bug FixF:FORMATRefactorF:FORMATMigrationS
Numeric columns are sortable. Currency is rounded to two decimals for readability.