500 Questions x 6 Models | 15 Categories | Reasoning, Debug, Taiwan-localized | 3 Difficulty Levels
| Model | Params | Hardware | Accuracy | EN | ZH | |
|---|---|---|---|---|---|---|
| Mistral-Small-4 | 119B MoE |
2x H100 |
91.5%
|
96% | 68% | |
| 2 | Nemotron-3-Ultra | 550B MoE (A55B) |
4x B200 |
91.3%
|
95% | 72% |
| 3 | Gemma-4-31B | 31B QAT (w4a16) |
1x L40S |
90.2%
|
95% | 64% |
| 4 | DiffusionGemma-26B | 26B MoE (A4B) |
1x H100 NVL |
88.8%
|
94% | 61% |
| 5 | Nemotron-3-Super | 120B MoE (A12B) |
2x H100 |
88.7%
|
93% | 65% |
| 6 | Llama-4-Scout | 109B MoE (17B act) |
1x L40S |
88.5%
|
94% | 61% |
Accuracy purely reflects answer correctness, independent of speed.
| Model | Easy (143) | Medium (185) | Hard (172) | Easy-Hard Gap |
|---|---|---|---|---|
| Mistral-Small-4 | 91.6% | 90.8% | 92% | -0.4% |
| Nemotron-3-Ultra | 91.1% | 91.6% | 91% | +0.1% |
| Gemma-4-31B | 90.6% | 88.5% | 91.7% | -1.1% |
| DiffusionGemma-26B | 89.2% | 88.4% | 88.9% | +0.3% |
| Nemotron-3-Super | 91.8% | 88.8% | 86.1% | +5.7% |
| Llama-4-Scout | 86.9% | 88.4% | 90% | -3.1% |
Gap = Easy accuracy - Hard accuracy. Larger positive = more drop on hard questions. Negative = better on hard (rare).
| Model | Throughput (tok/s) | tok/s/GPU | EN | ZH | ZH/EN | C/P |
|---|---|---|---|---|---|---|
| DiffusionGemma-26B | 807 | 807.1 | 825.3 tok/s | 545.8 tok/s | 0.66 | 716.6 |
| Nemotron-3-Ultra | 284 | 71.1 | 312 tok/s | 242.8 tok/s | 0.78 | 64.8 |
| Mistral-Small-4 | 167 | 83.4 | 163.5 tok/s | 163.2 tok/s | 1 | 76.4 |
| Nemotron-3-Super | 117 | 58.3 | 115.8 tok/s | 115.8 tok/s | 1 | 51.9 |
| Llama-4-Scout | 73 | 72.9 | 71.8 tok/s | 69.1 tok/s | 0.96 | 64.6 |
| Gemma-4-31B | 30 | 30.4 | 31.3 tok/s | 29.5 tok/s | 0.94 | 27.1 |
C/P = Accuracy x tok/s / GPU count. ZH/EN ratio >= 0.9 means similar speed for both languages.