Protege runs independent, double-blind evaluations of healthcare AI on uncontaminated real-world patient data. Measure efficacy, cost, latency, and safety in workflows, rather than toy tasks.
Verification is the bottleneck and moat.
When models start continually updating, much of AI evaluations and alignment goes out the window.
Trust and alignment are quickly becoming the most important capabilities that will differentiate agents and models.
Context tokens each public benchmark sends the model per case (log scale). Across 504,185 real patients, the median patient has 13 notes; the median single note is 153 tokens.
RED = Less context than median single clinical note
ORANGE = less context than median single patient
Public leaderboards measure preference, not outcome. Deployment replays miss the 1-in-10,000,000 failure, and treat dynamic clinical workflows as linear steps
Tasks are not the atomic unit of study. Workflows are. Benchmarks should measure entire workflows.
How humans and AI work together matters more than AI on its own. We should measure this interaction.
Once deployed, indirect effects may outweigh the direct ones. Both should be measured.
Measurement must happen alongside the work, continuously.
Failure modes are presented so that model makers can improve their models.
Always measure the safety of the model alongside the efficacy.
Call it by API, host it with us, or use frontier models through our zero-data-retention keys.
De-identified records on ~40M patients. The data never leaves your private sandbox.
Shape the test to your workflow — optionally to a specific health system or payer.
At most two runs on the held-out test set. No tuning to the answers.
Publish it as a double-blind evaluation by Protege — or keep it private.
Access to data you can later use for fine-tuning — never from the test set.
Expand CPT codes, win health systems and payers, prove your position.
A clear view against frontier models, on data no model has seen.
Watch the replay: Isaac Kohane (NEJM AI, Harvard Medical School) and Protege co-founder Engy Ziedan, moderated by Second Opinion's Christina Farr, on verification, trust, and real-world readiness in medical AI.
Is this model a good doctor, or was it good at the test? This is the big hairy question for AI adoption in healthcare -…
Two peer-reviewed journals, three weeks apart. Opposite answers on whether general or clinical AI performs better. Both contained flawed benchmarks...and there's no referee. This editorial…
Two of the biggest drivers are clinical documentation (turning an encounter into a usable clinical note) and medical coding (translating care into billable codes that…
Summary Healthcare Data for Real World Use Cases: A leading healthcare AI company partnered with Protege to connect its patient-level data to Loopback Health’s EHR…
Summary Millions of Images in 30 Days: A leading AI model builder sourced millions of de-identified imaging studies in a month of contract execution via…
Robust healthcare AI evaluations require benchmark-specific real world data that does not overlap with training data, while also reflecting the full multimodal patient journey. Current…
