Nov 17, 202621Research

Anthropic uses PhysicianBench from Stanford ARISE Network to evaluate Claude Sonnet 5.5

Anthropic uses PhysicianBench from Stanford ARISE Network to evaluate Claude Sonnet 5.5

Anthropic used PhysicianBench to evaluate how well Claude Sonnet 5.5 completes physician tasks in an electronic health record (EHR). In its Claude Sonnet 5.5 system card, Anthropic reports a 63.2% pass rate at maximum effort, compared with 37.4% for Claude Sonnet 5.

PhysicianBench uses 100 tasks adapted from real primary care-to-specialist electronic consultations (eConsults) at Stanford, using real patient records in a simulated EHR environment. An agent has to follow the case across encounters and complete the requested clinical work, which can involve retrieving data across encounters, reasoning over heterogeneous clinical information, executing consequential clinical actions, and producing clinical documentation. Across 21 specialties, 670 checkpoints verify what the agent completed. With only 100 tasks, the system card cautions that differences of five or six points are at the edge of what the benchmark can resolve. Independent patient-care use requires separate clinical validation.

Anthropic’s Claude Sonnet 5.5 system card, Figure 8.15.B (p. 138), plots PhysicianBench pass@1 against median response time alongside two other healthcare evaluations. On PhysicianBench, Sonnet 5.5 reaches 63.2% at maximum effort; section 8.15.3 compares this with Sonnet 5 at 37.4%. Source: Anthropic system card.

PhysicianBench is also part of the Agentic dimension of MAST, the Medical AI Superintelligence Test, ARISE’s independent medical AI evaluation platform. MAST brings clinical benchmarks together to compare models across medical capabilities and updates those comparisons as evaluations are added. Its Agentic dimension measures clinical task completion.

Researchers can use PhysicianBench’s open-source code and EHR setup instructions to run their own evaluations. The MAST GitHub repository provides evaluation resources and model submission tools, alongside the platform’s results and methodology and model and benchmark submission process.

PhysicianBench was led by Ruoqi Liu from Jonathan H. Chen’s Stanford lab and has been accepted to NeurIPS 2026. Contributors include Imran Mohiuddin, Austin Schoeffler, Kavita Renduchintala, Ashwin Nayak, Prasantha Vemu, Shivam Vedak, Kameron Black, John Havlik, Isaac Ogunmola, Stephen Ma, Roopa Dhatt, and Fateme Nateghi Haredasht. Anthropic previously evaluated Claude on MedAgentBench, also developed by Chen’s Stanford research group.

References

PhysicianBench paper: https://arxiv.org/abs/2605.02240

Anthropic Claude Sonnet 5.5 system card, section 8.15.3: system card

Anthropic's earlier MedAgentBench evaluation: https://www.anthropic.com/news/healthcare-life-sciences

MedAgentBench authors: https://arxiv.org/abs/2501.14654

MAST benchmarks and methodology: https://www.arise-ai.org/mast/benchmarks; https://www.arise-ai.org/mast/methodology

MAST evaluation resources: https://github.com/ARISENetwork/mast