Disease Odyssey

What is a disease odyssey, and why study it?

A disease odyssey is the search for an explanation for perplexing symptoms: moving from unresolved findings to a diagnosis and, where possible, to treatment that alleviates symptoms or addresses their cause. Not every patient experiences a prolonged search, but disease-specific studies show that it can last months or years. In a U.S. claims study of 538 patients with pulmonary arterial hypertension, the median interval from the first recorded symptom claim to the first pulmonary-hypertension-related diagnosis was 8.3 months. In another U.S. study, 35.3% of 11,793 patients with endometriosis had an interval of three to five years between their first recorded symptom and diagnosis. These estimates describe recorded encounters in selected populations, rather than the full journey from symptom onset.^{[1, 2]}

This period of search consumes clinical time and resources as clinicians order tests, consider competing explanations, and revise their judgments. In a U.S. study of alpha-1 antitrypsin deficiency, adjusted annual all-cause healthcare costs were $41,441 in the year before diagnosis, rose to $51,785 in the first year afterward, and fell to $36,937 and $28,558 in years two and three. These costs include care beyond the diagnostic workup; the pattern does not establish how much earlier diagnosis would save.^[3]

AI could help clinicians connect evidence across the electronic medical record (EMR), reconsider unresolved findings, and prioritize further investigation. If it identifies the correct disease earlier and guides appropriate confirmation, it could reduce avoidable testing and release clinical time and capacity for other patients. These gains depend on whether the resources saved exceed those required for additional investigations, false alerts, and earlier treatment. Disease Odyssey examines an initial capability needed for this approach: recognizing a disease from the record before its diagnosis is documented.

An EMR accumulates information over time and across encounters. The relevant evidence may be distributed among clinical notes, laboratory results, medications, and specialist assessments. Disease Odyssey asks whether frontier models can bring that evidence together and identify a condition that has not yet been explicitly documented in the record available to them.

The literature comparison below distinguishes record-based prediction from diagnostic dialogue, adaptive testing, and clinician assistance. These studies address complementary parts of the diagnostic journey.

The current benchmark is a retrospective test of that capability. It uses records that stop before the target diagnosis is first documented, while later information supplies the reference diagnosis. The purpose is to learn where models can recognize a developing disease from the available history and where they fail, rather than to infer clinical usefulness from a ranking alone.

The longer-term direction is disease surveillance: reviewing patient histories as new information becomes available to identify cases that may warrant further clinical investigation. A model could potentially revisit unresolved findings and compare explanations over time. The current evaluation does not yet test that repeated-review workflow; it provides an initial setting for studying it.

Previous literature: what does Disease Odyssey add?

This is not an exhaustive list. A human literature review and a meta-review using Elicit AI identified these studies as the closest to our work.

Prior work already evaluates real patient records, diagnostic dialogue, clinician assistance, and sequential investigation. Disease Odyssey focuses on comparing frontier models using longitudinal records cut off before the target diagnosis is first documented, with physician review of case validity and checks on information timing. This combination defines its scope; the table does not establish that each component is new.

Table A. Authors, publication year, data, and sample sizes.
* Year of the cited preprint, not a journal publication. Other years refer to the cited journal version. Sample sizes describe the named evaluation or component, not necessarily every experiment in a paper. Cases, notes, patients, clinicians, and repeated assessments are different units and should not be added together. “Real” describes the source data; it does not imply prospective clinical validation. Simulated encounters and adapted vignettes are identified separately from generated synthetic training data. Disease Odyssey’s count is cases, not a verified count of unique patients.

Table B. Evaluation features, by author and year.
Category key. Clinical records: clinical notes, records, or intake summaries used as inputs, excluding published case reports, exam-style vignettes, and patient dialogue alone. Multiple LLMs: the study compares at least two distinct large language models on an evaluation task; repeated runs or prompts for one model do not count. Specialized and general-purpose models both qualify. Models used only as graders, simulated patients, or data generators do not count. Sequential testing: the clinician or AI requests a test, receives its result, and uses that result to choose the next investigation. Reading a fixed record or listing recommended tests does not qualify. Clinician + AI deployed live: clinicians receive and use AI output during ongoing care of actual patients. Retrospective clinician review, adjudication, and assistance trials using historical cases or vignettes do not qualify. Live patients: prospective interaction with actual patients; a clinician trial using retrospective cases does not qualify.

The setting: how the task is constructed

The patient record and prediction date. Each case begins with a diagnosis documented in a patient’s record. The prediction cutoff limits model input to information available before that diagnosis. In the illustrated case, the cutoff is November 21, 2021, eight days before the documented primary biliary cholangitis (PBC) diagnosis. A retained item must have been available before the cutoff. For a laboratory result, this means checking both when the specimen was collected and when the result became available. Later information establishes the reference answer and is withheld from the evaluated model.

The question asked of the model. The model is asked to identify the new condition developing in the record. Each case is presented in two formats: free response, where the model names the diagnosis, and multiple choice, where it selects from six options. Both formats use the same history and reference diagnosis. The comparison helps distinguish generating a diagnostic hypothesis from recognizing it among supplied alternatives.

Clinician review and adjudication are retrospective; the benchmark does not involve live patients or clinician–AI deployment during ongoing care.

The current evaluation. Disease Odyssey contains 256 physician-accepted cases across 129 ICD-10 categories. Four frontier models were evaluated three times per case in each format. The median input was approximately 6,900 tokens, and no case exceeded 16,000 tokens. These are selected cases, not a representative sample of all patients or all diagnostic encounters.

The human annotators’ role. At least two physicians must accept a case. They assess whether the target is a diagnosis, whether the available record supports it without disclosing it, and whether the alternatives are distinct. They also review model errors to judge whether an alternative answer is defensible given what was knowable at the cutoff. Their role is to establish the clinical validity and answerability of the task, not simply to confirm that a later diagnosis exists.

Scoring. Multiple-choice answers are checked with a letter parser. Free-response answers are assessed by an LLM judge using a fixed rubric, with repeated judgments used to examine stability. The task prompts and the judge prompt are public at github.com/Protege-AI/disease-odyssey.

Example 1: earlier identification of PBC

A disease odyssey solved by the models. In this example, most frontier models identified primary biliary cholangitis (PBC) using records available eight days before the physicians’ documented diagnosis. The cause of this gap is unclear.

Model input: four GI visit notes (48 sections) and 63 lab results before November 21, 2021. PBC was identified in 11/12 free responses (91.7%) and 12/12 multiple-choice responses (100%); the diagnosis was documented on November 29. These are repeated runs of one case. The eight-day lead is relative to documentation, not necessarily to the physicians’ first suspicion.

What explains the eight-day lead?

The gap could reflect scheduling delays, the order in which tests were requested, or findings that were difficult to locate or act on in a large medical record. This case does not establish the cause. Understanding why AI reaches an answer earlier in some workflows should be part of the research.

A benchmark could stop at reporting a win; we want to go further and investigate three questions:

1. Was the earlier answer valid? Did the model infer the disease from information available at the cutoff, or exploit leaked information or shortcuts in the task?
2. Which shortcuts are useful? Some unexpected strategies may reveal an efficient way to recognize disease; others may exploit artifacts that would not be available in care. A correct answer alone does not tell us whether the method is clinically useful.
3. Where can AI help people work faster? Was the eight-day gap due to diagnostic reasoning, scheduling, test sequencing, or delays in reviewing the record? Identifying the constraint would help determine where to introduce AI tools in adjacent workflows. We then need to study whether improving one step increases capacity across the healthcare system or moves the bottleneck elsewhere.

Uncertain failures and verification issues in healthcare benchmarks

What the model never saw.

1. All 352 lab results before the cutoff. Every lab row had an empty specimen date, which the extraction used to place labs on the timeline, so none were included.
2. Notes from the cutoff day itself, including: “Patient wore CPAP 4 hours; reports headache from CPAP; oxygen saturation dropped into low 80’s when sleeping without CPAP.”
3. The 2018–2021 ED visits, which contain codes and labs but no notes.

Diagnosis date. “Sleep apnea, unspecified” was first coded in March 2023 (day 1,836). The benchmark’s confirmation cites the November 2023 OSA code and CPAP refusal after the sleep study (day 2,062).

Counts. There were 11 encounters before the OSA code: 7 ED visits (including one in which the patient left before being seen), 3 admissions, and 1 outpatient lab visit. Before the cutoff, there were 8 encounters and 42 distinct diagnosis codes, none related to sleep.

We regard this as a case in which the benchmark may handicap the models. Physician reviewers pointed to evidence of apnea before its recorded diagnosis, but they also considered the models’ alternative diagnoses plausible. The case is therefore imperfect: more than one condition may explain the patient’s symptoms, while the benchmark rewards a single reference answer.

This raises the possibility of models acting as “oracles”: predicting what might only be recognized years later. The case does not establish that this happened. It does show why fairness to the model can be difficult to verify. Physicians may be unable to identify a disease whose symptoms are still subtle, and a model might identify relationships between symptoms and diseases that are not yet recognized in medicine. It might also be wrong. Distinguishing these possibilities requires follow-up evidence.

Whose capabilities, then, are we comparing the model against? Agreement with today’s physician judgment may not capture every useful prediction, but disagreement alone is not evidence that a model has surpassed that judgment. To make the benchmark operational, we reward models for identifying the disease that resolves the diagnostic odyssey. Where physician disagreement about that target remains unresolved after adjudication, we exclude the case.

Below, we present types of failures in expert annotation that we studied while constructing this evaluation.

Verification in healthcare benchmarks is the bottleneck

We tested two parallel workflows for verifying grader fairness across 256 cases with more than 15 physicians. In one, reviewers saw the reference diagnosis from the EMR before assessing the case. In the other, they first reviewed the record without seeing that diagnosis, committed to their own answer, and then saw the reference answer.

Each approach has a tradeoff. When shown the reference diagnosis, physicians were much more likely to agree with it. When it was hidden, they could anchor on an incorrect initial diagnosis. Increasing review to four physicians per case did not reduce inter-rater disagreement. More reviewers alone did not resolve the uncertainty about which answers should count as correct.

We chose to keep the reference diagnosis visible. Our reasoning was that the treating physician may have observed patient-specific details that a retrospective reviewer could not fully recover from the EMR. We considered the recorded diagnosis a more useful reference than asking reviewers to choose independently from a range of possible diseases. It can still be fair to ask a model to predict that diagnosis, provided the available record supports it; a model should not be penalized for lacking essential context that was never supplied.

This choice reflects a broader difficulty: medical care does not always produce a single, binary answer. A plausible diagnosis, a documented diagnosis, and the only defensible diagnosis are not necessarily the same thing. Healthcare is therefore difficult to evaluate under a verification framework that assumes task outcomes can be checked against one unambiguous answer.

In the final review, both physicians accepted 84% of cases, agreeing that the diagnosis was real, the answer options were clinically distinct, and the reference diagnosis was hidden in the model-visible record.

The workflows below separate two decisions: whether the case is suitable for the benchmark, and whether the grader was fair to the model. A physician can accept the reference diagnosis while also finding a model’s alternative answer defensible.

Two workflows for verifying grader fairness

Study: 256 cases; more than 15 physicians.

Model performance

Disease Odyssey currently has 256 physician-accepted cases. Each model made three attempts per case in each format, for 768 attempts per model per format. All four models scored higher when choosing among six options than when naming the diagnosis without options. The cases we offer here can be accessed via Protege, and we welcome comments.

What is next: population disease surveillance

We believe population disease surveillance is a high-impact, system-wide workflow that would use a significant amount of inference compute.

As inference-time compute rises, we expect to be asked to evaluate healthcare AI capabilities that are more difficult and more futuristic. We should approach this from first principles. Disease Odyssey is intended as an exercise in understanding what it takes to reach difficult, economically useful, and verifiable answers. Improving a benchmark score is not the end goal. Even a carefully constructed evaluation requires substantial care in choosing the question, assembling the evidence, and deciding what counts as success.

What if an AI model predicted an entire disease outbreak rather than a single diagnosis? What if it identified a medical procedure that was consistently failing across settings, or a decline in the quality of care across an entire ward? How would we construct evaluations that account for these broader effects?

In our view, these should be treated as workflows. The evaluation must follow how a prediction leads to investigation, decisions, and changes in care, including how those changes affect other patients and services. The question is whether AI helps the healthcare system reach better outcomes, and how we can verify that it does.

Sources

1. DuBrock HM, Germack HD, Gauthier-Loiselle M, et al. Economic burden of delayed diagnosis in patients with pulmonary arterial hypertension (PAH). PharmacoEconomics Open. 2024;8:133–146. doi:10.1007/s41669-023-00453-8.
2. Surrey E, Soliman AM, Trenz H, Blauer-Peterson C, Sluis A. Impact of endometriosis diagnostic delays on healthcare resource utilization and costs. Advances in Therapy. 2020;37:1087–1099. doi:10.1007/s12325-019-01215-x.
3. Blanchette CM, Whitmire S, Oh J, et al. Higher healthcare cost and utilization before and after diagnosis of AATD in the United States. BMC Pulmonary Medicine. 2024;24:570. doi:10.1186/s12890-024-03396-w.
4. Porter R, et al. LLMD: a large language model for interpreting longitudinal medical records. arXiv preprint. 2024. doi:10.48550/arXiv.2410.12860.
5. Chen H, et al. A specialized reasoning large language model for accelerating rare disease diagnosis: a randomized AI physician assistance trial. arXiv preprint. 2026. doi:10.48550/arXiv.2606.24510.
6. McDuff D, et al. Towards accurate differential diagnosis with large language models. Nature. 2025. doi:10.1038/s41586-025-08869-4.
7. O’Sullivan JW, et al. A large language model for complex cardiology care. Nature Medicine. 2026. doi:10.1038/s41591-025-04190-9.
8. Brodeur P, et al. Conversational diagnostic artificial intelligence in ambulatory primary care: a prospective feasibility study. The Lancet. 2026. doi:10.1016/S0140-6736(26)01535-7.
9. Jaech A, et al. LLM-assisted reanalysis of unsolved rare disease genomes increases diagnostic yield. NEJM AI. 2026. doi:10.1056/aics2501343.
10. Everett S, et al. From tool to teammate in a randomized controlled trial of clinician-AI collaborative workflows for diagnosis. npj Digital Medicine. 2026. doi:10.1038/s41746-026-02545-1.
11. Shyr C, et al. An artificial intelligence framework for end-to-end rare disease phenotyping from clinical notes using large language models. arXiv preprint. 2026. doi:10.48550/arXiv.2602.20324.
12. Shyr C, et al. Large language models for rare disease diagnosis at the Undiagnosed Diseases Network. JAMA Network Open. 2025. doi:10.1001/jamanetworkopen.2025.28538.
13. Tu T, et al. Towards conversational diagnostic artificial intelligence. Nature. 2025. doi:10.1038/s41586-025-08866-7.
14. Nori H, et al. Sequential diagnosis with language models. arXiv preprint. 2025. doi:10.48550/arXiv.2506.22405.
15. Goh E, et al. Large language model influence on diagnostic reasoning. JAMA Network Open. 2024. doi:10.1001/jamanetworkopen.2024.40969.
16. Qazi IA, et al. Large language model diagnostic assistance for physicians in a lower-middle-income country: a randomized controlled trial. Nature Health. 2026. doi:10.1038/s44360-025-00007-8.
17. Kremer P, et al. Certified large language model-based diagnostic decision support in rheumatology: the ALLIANCE multicentre randomised controlled trial. medRxiv preprint. 2026. doi:10.64898/2026.08.29.26361715.
18. Saban M, et al. ChatGPT-4 versus emergency physicians for walk-in ED patients: history, differential diagnosis, testing, and disposition—a prospective feasibility study. Discover Artificial Intelligence. 2026. doi:10.1007/s44163-025-00667-1.