Quality benchmark
MindEval 2
How a model holds up as a therapist over three sessions with a simulated patient who remembers the earlier ones, with every reply checked for clinical issues and weighted by their severity.
“Guidance calibration remains the dominant failure mode across all models, accounting for most of the quality gap.”
- Metric
- MHQR (weighted severity)
- Scale
- 0–10
- Source
- Paper ↗
Headline score
Share of replies by severity
| Model | ||||
|---|---|---|---|---|
| GPT-6 Astra | 0.749 | 0.113 | 0.136 | 0.002 |
| GPT-6 Luna | 0.548 | 0.151 | 0.302 | 0.000 |
| Claude Opus 5.5 | 0.475 | 0.153 | 0.366 | 0.007 |
| GPT-5 | 0.403 | 0.244 | 0.341 | 0.012 |
| Claude Sonnet 4.5 | 0.262 | 0.123 | 0.609 | 0.006 |
| GPT-4o | 0.220 | 0.126 | 0.651 | 0.003 |
| Qwen3.5-27B | 0.109 | 0.104 | 0.772 | 0.015 |
Share of replies flagged per issue
| Model | ||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| GPT-6 Astra | 0.165 | 0.143 | 0.003 | 0.011 | 0.001 | 0.000 | 0.045 | 0.002 | 0.000 | 0.000 | 0.000 | 0.000 | 0.007 | 0.000 |
| GPT-5 | 0.291 | 0.408 | 0.118 | 0.082 | 0.018 | 0.029 | 0.067 | 0.005 | 0.015 | 0.002 | 0.003 | 0.000 | 0.007 | 0.003 |
| GPT-6 Luna | 0.329 | 0.256 | 0.022 | 0.041 | 0.005 | 0.007 | 0.053 | 0.010 | 0.004 | 0.002 | 0.000 | 0.000 | 0.008 | 0.000 |
| Claude Opus 5.5 | 0.366 | 0.384 | 0.006 | 0.025 | 0.050 | 0.013 | 0.036 | 0.031 | 0.005 | 0.009 | 0.001 | 0.000 | 0.008 | 0.000 |
| Claude Sonnet 4.5 | 0.578 | 0.483 | 0.029 | 0.063 | 0.178 | 0.004 | 0.014 | 0.040 | 0.004 | 0.019 | 0.001 | 0.000 | 0.006 | 0.001 |
| GPT-4o | 0.650 | 0.580 | 0.285 | 0.244 | 0.010 | 0.017 | 0.012 | 0.003 | 0.010 | 0.000 | 0.001 | 0.000 | 0.001 | 0.001 |
| Qwen3.5-27B | 0.750 | 0.727 | 0.522 | 0.137 | 0.076 | 0.044 | 0.038 | 0.024 | 0.009 | 0.005 | 0.000 | 0.000 | 0.000 | 0.001 |
MHQR by session
| Model | |||
|---|---|---|---|
| GPT-6 Astra | 0.966 | 0.673 | 0.689 |
| GPT-6 Luna | 1.507 | 1.560 | 1.668 |
| GPT-5 | 1.979 | 1.972 | 2.049 |
| Claude Opus 5.5 | 2.163 | 1.921 | 1.735 |
| Claude Sonnet 4.5 | 2.998 | 3.350 | 3.148 |
| GPT-4o | 3.234 | 3.424 | 3.405 |
| Qwen3.5-27B | 4.007 | 4.151 | 4.093 |