Protege Evaluation of A Disease Risk Prediction Model by Xlue AI

Key findings

• HealthSAW catches the most future cases at screening-grade specificity. When only 1 in 20 patients who stay healthy is flagged (95% specificity), HealthSAW flags 37.0% of future cases. XGBoost flags 30.0% and GPT-6 Astra 26.5%, so HealthSAW finds about a quarter more cases at the same false-alarm rate. It is best on 14 of 16 tasks and in every family: 33.7% vs. 29.4% for cancer, 34.4% vs. 25.7% for cardiovascular disease and 46.3% vs. 39.8% for neurodegenerative disease, each against the best baseline.
• Preventing Label Leakage. Not only do we remove label leakage in coded data looking all the way back to 2015, we also scan every clinical note for un-coded documentation of the target disease, removing 22% of candidate positives. Removing them lowered every model’s AUROC by 0.01–0.02 without changing the ranking.
• HealthSAW is small and cheap to run. At 1B parameters, it scored each task (about 1,000–1,800 patients) in 10–18 seconds on a single NVIDIA L40S GPU. Based on public pricing and best estimates of cost, 1,000 predictions cost about $0.04 with HealthSAW and about $32 with GPT-6 Astra, roughly 900 times more (Figure 3).

HealthSAW and its intended uses

A video world model predicts the next frame of a scene from the frames so far and the action taken, such as a steering command or a robot’s grasp. To do that well, it has to implicitly learn the laws of physics: objects fall, stay solid, persist when hidden and move when pushed. HealthSAW (Health State–Action World Model) does the same for patient health. It predicts a patient’s next health state from the previous health states and the actions taken: the interventions of doctors, such as treatments, medications and procedures. Whereas physical world models apply frame by frame, health records are delimited by visits to a health provider. For example, a rising creatinine precedes kidney failure, memory complaints can precede a dementia diagnosis by years, and potassium rises after spironolactone.

Each visit is split into a health state (diagnoses, labs, vital signs, imaging, diagnostic tests) and a health action (medications, procedures, referrals). Local encoders compress each into one vector. A Transformer reads the sequence of visits and the gaps between them, and predicts the next state, the next action and when the next visit happens. Every forecast passes through these compact vectors, so the model cannot score well by copying codes that recur from visit to visit; the vectors have to capture the patient’s underlying condition. Given a chosen treatment, the model can be rolled forward to compare what-if paths for one patient. Its forecasts have to be checked against what really happened, and this benchmark does that.

Clinical workflows and intended use

Earlier detection, better outcomes. For many diseases, when the disease is found shapes the outcome. Most cancers are far more treatable at a localized stage than after they have spread. For Alzheimer’s disease, the newest treatments are approved only for its early stages. The Disease Risk Prediction benchmark measures how well any AI model, not only HealthSAW, does early screening and future-risk modeling for cancer, cardiovascular and neurodegenerative disease: reading each patient’s record and flagging them before a diagnosis. The horizons (Figure 2) are 3–15 months ahead for acute coronary syndrome, stroke and cerebral aneurysm; 1–2 years for the cancers, coronary artery disease and heart failure; and 3–4 years for neurodegenerative disease. At 95% specificity, HealthSAW flags 1 in 20 patients who stay healthy and 37% of those who will be diagnosed. A health system or payer could use the score to decide who goes first for an existing confirmatory test (low-dose CT, colonoscopy, echocardiography, cognitive assessment), who gets outreach, and who is pre-screened for a trial. Moving those tests earlier for the people most likely to need them is how a risk model could turn into better patient outcomes.

Independent evaluation and embargoed data

Eyes-off. A test set that developers can see eventually leaks into training, tuning or prompts. Every patient in the benchmark comes from Protege’s list of records that have never been provisioned for AI training by Protege or the upstream data partners. A patient is also excluded if any other chart of the same person, matched by person ID or Datavant token, was ever released. Xlue never sees the test patients or their labels, and Protege independently re-ran HealthSAW and reproduced its scores. The test set was frozen before scoring, and every system is scored on the same rows.

Embargoed data. The 2,500 Disease Risk Prediction patients, with their full records, will never be sold, released or shared outside Protege. They stay a held-out test for future models.

Evaluation architecture

Labels come from the record, not from a model. No LLM acts as a judge. A label is a confirmed diagnosis code in the patient’s record, and scoring is deterministic. LLMs are used only to remove candidates whose notes reveal prior disease. They never create, change or grade a label.

Leakage removal (Figure 1). A prior diagnosis in the family, or a treatment that reveals one, excludes a patient from every task in that family. Such treatments include chemotherapy or radiation, coronary revascularization, and dementia or Parkinson’s drugs. Because codes begin in 2015, LLMs then read every clinical note of every candidate: over a million notes for the 2,745 candidate positives and their negatives. GPT-6 Luna read all of them. GPT-6 Sol re-read every flagged note and made the final call, and GPT-6 Astra reviewed 222 ambiguous patients. On 183 gold-labeled notes, this setup caught 78 of 79 leaking notes with no false alarm on 92 clean ones. Only firm evidence removes a positive: a stated diagnosis, a disease-specific treatment, or a later note that dates the disease before the anchor. Suspicion or work-up never does, because work-up is how new disease is found. The review removed 603 of 2,745 candidate positives (22%, and 31% in cardiovascular tasks).

Methods

Task methods

There are 16 binary tasks. Seven are cancers (bladder, breast, colorectal, kidney, liver, lung, pancreatic), five are cardiovascular conditions (acute coronary syndrome, coronary artery disease, cerebral aneurysm, heart failure, stroke) and four are neurodegenerative diseases (Alzheimer’s, any dementia, Parkinson’s, vascular dementia). Each task has 115 positives. Heart failure has 100, the most that passed the note review. The positives come from 1,789 patients, and with the 711 shared negatives the set totals 2,500 patients (Appendix A).

System specifications and adaptation

HealthSAW (1B) is fine-tuned per disease on Xlue’s development data. Its training negatives are matched to positives on anchor calendar year, so the recency of a record cannot stand in for risk. It uses its own task definitions, such as a 180-day rather than 365-day washout for the cancers, with no confirming code. No benchmark patient or label was used to train it. The baselines read the same rows: the most recent events before the anchor that fit 3,072 record tokens, with the anchor last. LightGBM 4.6.0 and XGBoost 3.2.0 are trained per disease on the same development data, using the count of each code in that history plus age and sex; order and timing are discarded. Both use up to 200 boosting rounds with early stopping on validation AUROC. Six frontier LLMs are prompted zero-shot with the same history written out as text, using each code’s description, and return a probability in a fixed JSON format. They are GPT-6 Astra, Sol and Luna, and Claude Opus 5, Sonnet 5 and Haiku 4.5, all at default reasoning effort; Opus 5 and Sonnet 5 were also run at low effort.

Task design and prediction timing

Anchor (prediction date). The day the prediction is made. The model sees the record up to and including the anchor, and nothing after it.

Washout window. The days right after the anchor: 1–90, 1–365 or 1–1,095, depending on the task. The model never sees them, and a positive’s diagnosis must come after them. The washout exists because much of the record just before a diagnosis is the diagnosis being made. In the weeks before a cancer is coded there may be labs, imaging and work-ups ordered for that specific cancer, or a primary-care note recording a strong suspicion. An anchor the day before diagnosis would almost always give the answer away. The washout puts a gap between the last record the model sees and the diagnosis, so the task measures advance prediction rather than recognition of a work-up already under way. A patient with a same-family diagnosis inside the washout is excluded.

Outcome window. The days in which a new diagnosis counts, such as days 366–730 for the cancers (one to two years ahead).

Label (reference). The ground truth, read from the record. It is 1 for a confirmed first diagnosis in the outcome window. It is 0 if the disease does not appear and the record continues, with recorded activity, to the end of the window.

Confirmed diagnosis. The first target ICD-10 code counts only if something confirms it: a second code with the same three-character prefix 1–60 days later, a family-specific treatment within 90 days, or (for acute coronary syndrome, cerebral aneurysm and stroke) death within 30 days.

Disease family; shared negatives. The families are cancer, cardiovascular and neurodegenerative. A positive must have no prior disease of its family, whether coded, treated or noted (Appendix B). Shared negatives develop none of the 16 diseases and serve as negatives in every task. The official setting scores each task’s positives against them (374–640 per task), as when screening a general population.

Leakage. Anything before the anchor that gives away the answer. An example is a 2022 note that reads “history of breast cancer, treated 2012” when coded data only begin in 2015.

Scoring and analysis

The primary metric is sensitivity at 95% specificity: the share of future cases flagged when only 5% of patients who stay disease-free are flagged (1 false alarm in 20). That is how a screening model would run on a population, where false alarms drive the cost. We also report sensitivity at 99% specificity and AUROC. Each metric is averaged over tasks, with 95% intervals from 500 bootstrap resamples.

Reported comparative results

HealthSAW shows higher performance across every column of Table 1. At 95% specificity it is best on 14 of 16 tasks. The two exceptions are within noise: Alzheimer’s disease, where LightGBM leads 45.2% to 40.9%, and Parkinson’s disease, where GPT-6 Astra leads 33.9% to 31.3%. The margin is widest for acute events 3–15 months ahead. For acute coronary syndrome, stroke and cerebral aneurysm, HealthSAW catches 28–33% of future events, against at most 24% for any baseline. Neurodegenerative disease is the most predictable, even 3–4 years ahead. HealthSAW also has the highest AUROC on 15 of 16 tasks; the exception is pancreatic cancer, where GPT-6 Astra leads 0.792 to 0.782. Appendix C has every task.

Table 1. Disease Risk Prediction, official setting (each task’s positives vs. the shared negatives). Sensitivity at 95% specificity is the primary metric; each value is the mean over tasks. Bold marks the best in each column. HealthSAW’s 95% bootstrap intervals: Sens@95 0.340–0.401, AUROC 0.818–0.837, Sens@99 0.132–0.183.

Cost and turnaround

Reported execution measures

Figure 3 plots each system’s sensitivity at 95% specificity against its cost. Every frontier LLM costs about 11 to 900 times more than HealthSAW and catches fewer cases. The tree baselines, not shown, cost less but also catch fewer cases (29–30%).

HealthSAW scored each task (about 1,000–1,800 patients) in 10–18 seconds on a single NVIDIA L40S GPU. The cost comparison uses public input/output token pricing for the frontier models. HealthSAW’s price is estimated at one-seventh of the public price of deepseek-7B because it is one-seventh the size. This estimate is distinct from the measured execution time.

Assessment

Implications for adoption

HealthSAW is a research model. These results are retrospective and measure how well the model ranks held-out records. They do not show that acting on the scores improves outcomes. Clinical use would be decision support with a clinician in the loop, meaning a prompt to test sooner and never a diagnosis. It would also need prospective validation, calibration to the deployment population and regulatory review.

Limitations

Each task has 115 positives, so per-task differences under about 0.03 AUROC or 0.05 sensitivity are within noise. Sensitivity at 99% specificity rests on about 6 negatives per task and is reliable only as the 16-task average. Positives are 15–24% of each task’s rows, far above real prevalence. AUROC and sensitivity do not depend on that share, but precision does, so we do not report it. Labels from codes cannot count disease that was never diagnosed.

The test measures retrospective prediction from the available records. It does not establish performance in a new healthcare system or show that using the scores leads to earlier diagnosis, better outcomes, or improved organizational economic performance.

HealthSAW’s reported cost is an estimate based on model size and a reference price, rather than a measurement of total deployment expense. The cost comparison does not establish the full costs of integration, operations, or follow-up care. Those costs, and the value of gains in prediction performance, would need to be assessed in the intended workflow.

Appendix A. Disease Risk Prediction tasks: definitions and counts

Table 2. Per-task definitions and scored rows. Windows are days after the anchor. “Removed” counts candidate positives removed by the clinical-note review, out of those scanned. Positives were required to have no prior disease of their family (Appendix B). Breast cancer includes women only.

History and anchor rules. For cancer and cardiovascular tasks, the anchor falls on or after 2020-01-01. The patient needs at least 5 years of record before it, a recorded event in every calendar year from five years before the anchor year through the anchor year, and clinical notes in at least 4 of those 6 years. For neurodegenerative tasks, the anchor falls on or after 2018-01-01, with at least 3 years of record, a recorded event every year and notes in at least 75% of years. Negatives need follow-up to the end of the outcome window and at least one recorded event inside it. Shared negatives (711 patients) are never released, have no positive in any task and have no linked released chart. A shared negative’s row for a task is dropped if a note shows that task’s disease. Seven of the 25,341 rows are left out for every system: five have no recorded event on the anchor day, and HealthSAW’s input reader rejects two. All systems are therefore scored on the same 25,334 rows.

Appendix B. Prior-disease exclusions by family

Prior history and washout are both checked at the family level, so every task in a family uses the same two code sets. The washout set is narrower: it uses diagnosis codes only.

Table 3. Code sets that exclude a patient from every task in a disease family.

Appendix C. Disease Risk Prediction results per task

Table 4. HealthSAW against the best of the 10 baselines on each task, official setting (shared negatives). “Best” names the top baseline (LGBM = LightGBM, XGB = XGBoost; Astra, Sol, Luna = GPT-6 models; Opus = Claude Opus 5). HS = HealthSAW. Bold marks the higher value; ties are unmarked. Per-task Sens@99 rests on about 6 negatives and is shown for completeness only.

Secondary setting (all negatives). If each task’s negatives also include the positives of the other 15 tasks, who are patients with other serious disease, every system scores lower. HealthSAW’s macro AUROC is 0.732, against 0.712 (LightGBM), 0.707 (XGBoost) and 0.706 (GPT-6 Astra). Its sensitivity is 0.237 at 95% specificity and 0.098 at 99%, against best baselines of 0.206 (GPT-6 Astra) and 0.086. We treat shared negatives as the official setting because a screening model runs on a general population, not on people already known to have another disease on the list.

Earlier HealthSAW checkpoints. An earlier set of HealthSAW classifiers, fine-tuned without matching negatives to positives on anchor year, scored 0.831 AUROC, 0.344 Sens@95 and 0.139 Sens@99 on the same rows. That is a tie on AUROC with the checkpoints reported here (paired difference −0.004, 95% CI −0.012 to +0.004).