Anthropic uses PhysicianBench from Stanford ARISE Network to evaluate Claude Sonnet 5.5

Anthropic used PhysicianBench to evaluate how well Claude Sonnet 5.5 completes physician tasks in an electronic health record (EHR). In its Claude Sonnet 5.5 system card, Anthropic reports a 63.2% pass rate at maximum effort, compared with 37.4% for Claude Sonnet 5.
PhysicianBench uses 100 tasks adapted from real primary care-to-specialist electronic consultations (eConsults) at Stanford, using real patient records in a simulated EHR environment. An agent has to follow the case across encounters and complete the requested clinical work, which can involve retrieving data across encounters, reasoning over heterogeneous clinical information, executing consequential clinical actions, and producing clinical documentation. Across 21 specialties, 670 checkpoints verify what the agent completed. With only 100 tasks, the system card cautions that differences of five or six points are at the edge of what the benchmark can resolve. Independent patient-care use requires separate clinical validation.

Anthropic’s Claude Sonnet 5.5 system card, Figure 8.15.B (p. 138), plots PhysicianBench pass@1 against median response time alongside two other healthcare evaluations. On PhysicianBench, Sonnet 5.5 reaches 63.2% at maximum effort; section 8.15.3 compares this with Sonnet 5 at 37.4%. Source: Anthropic system card.
PhysicianBench is also part of the Agentic dimension of MAST, the Medical AI Superintelligence Test, ARISE’s independent medical AI evaluation platform. MAST brings clinical benchmarks together to compare models across medical capabilities and updates those comparisons as evaluations are added. Its Agentic dimension measures clinical task completion.
Researchers can use PhysicianBench’s open-source code and EHR setup instructions to run their own evaluations. The MAST GitHub repository provides evaluation resources and model submission tools, alongside the platform’s results and methodology and model and benchmark submission process.
PhysicianBench was led by Ruoqi Liu from Jonathan H. Chen’s Stanford lab and has been accepted to NeurIPS 2026. Contributors include Imran Mohiuddin, Austin Schoeffler, Kavita Renduchintala, Ashwin Nayak, Prasantha Vemu, Shivam Vedak, Kameron Black, John Havlik, Isaac Ogunmola, Stephen Ma, Roopa Dhatt, and Fateme Nateghi Haredasht. Anthropic previously evaluated Claude on MedAgentBench, also developed by Chen’s Stanford research group.
References
PhysicianBench paper: https://arxiv.org/abs/2605.02240
Anthropic Claude Sonnet 5.5 system card, section 8.15.3: system card
Anthropic's earlier MedAgentBench evaluation: https://www.anthropic.com/news/healthcare-life-sciences
MedAgentBench authors: https://arxiv.org/abs/2501.14654
MAST benchmarks and methodology: https://www.arise-ai.org/mast/benchmarks; https://www.arise-ai.org/mast/methodology
MAST evaluation resources: https://github.com/ARISENetwork/mast


