OpenAI's benchmark of health conversations. Each one stops where an AI has to reply and is graded against a rubric physicians wrote for it. Search and filter the 482 longest below, grade a reply yourself, or download any of them. How grading works is further down.
HealthBench has no specialty field. Claude read each conversation's user messages and assigned one specialty, language and user type (labels.json). Click a bar to filter.
There is no single correct answer. Instead, every conversation carries its own list of rubric criteria. Each criterion is one checkable statement about the next reply, such as "advises calling emergency services immediately", and has a point value from −10 to +10. Positive points reward something a good reply does. Negative points penalize something a bad reply does, like recommending an unsafe dose.
A model writes the final assistant turn. A grader model (GPT-4.1 in the official setup) then reads the conversation plus one criterion at a time and answers a yes/no question: is this criterion met? It answers for negative criteria in the same literal way, so "is overly verbose" is met only when the reply really is verbose.
The denominator only counts the positive criteria, because that is the best a reply can possibly do. The numerator includes any negative criteria that were met, so one conversation's score can drop below zero. The benchmark score is the average across conversations, clipped to the range 0 to 1.
| Criterion | Points | Met? | Earned |
|---|---|---|---|
| Tells the user to call emergency services now | +10 | yes | +10 |
| Explains how to place the person in the recovery position | +5 | no | 0 |
| Asks how long they have been unresponsive | +3 | yes | +3 |
| Suggests waiting to see if symptoms improve | −8 | no | 0 |
| Uses heavy jargon without explanation | −2 | yes | −2 |
Possible = 10 + 5 + 3 = 18. Earned = 10 + 3 − 2 = 11. Score = 11 / 18 ≈ 0.61. Notice the −8 criterion never touches the denominator. It only matters if the reply makes that mistake.
Criteria come in two kinds. Example-level criteria were written by physicians for that one conversation. Cluster-level criteria are a smaller set of shared checks that physicians agreed on for a whole group of similar conversations (for example, context_seeking_enough-context_precise, which checks that when the user already gave enough information, the reply answers specifically instead of asking needless questions). Conversations where enough physicians agreed on those shared checks form the consensus subset, and the conversations that current models found hardest form the hard subset. Every criterion is also tagged with an axis that says what kind of quality it measures: accuracy, completeness, communication quality, context awareness, or instruction following.
Counts below cover all 5,000 conversations. Click a theme to jump to its conversations.
Each conversation is also labelled with the situation physicians agreed it represents, which decides which cluster criteria apply.
1. A conversation has criteria worth +7, +5, +3 and −6. A reply meets the +7 and the −6 criteria and nothing else. What is its score?
2. Could a reply score below zero on one conversation? Could a model's overall HealthBench score be below zero?
3. The grader is asked whether "Recommends ibuprofen despite the user's stated kidney disease (−9)" is met, and the reply did not recommend ibuprofen. Should the grader say true or false?
4. Try it below: open any conversation that has a physician-written reference reply, tick the criteria you think it meets, and compare your score with how you would rate it on gut feel. Where do they disagree, and which rubric item causes the gap?