The model is not the hard part

01The model is the small box

These are the notes from my talk at the 2nd MedAI Summit, given to medical and pharmacy students, most of whom have never written a line of code. I’ve written them up for anyone who wants the argument without the slides.

When I teach medical students about AI, I see the same pattern. They massively overestimate the model, the algorithm itself, and underestimate everything around it: where the data came from, how the labels were defined, and how the tool was evaluated. I think that’s good news for them, because those underestimated parts are exactly where a physician has an edge over a computer-science graduate.

Two ideas are all the AI you need

First, a model learns from examples that someone labeled. Someone decided the right answer for every training image or record, and that label became the model’s reference standard. If the labels are wrong, the model learns the wrong thing.

Second, a model gives a score, and someone picks a cut-off. It works like troponin: the number is continuous, but the decision isn’t. Move the cut-off and you trade missed cases for false alarms.

Everything else in this post is clinical.

I find it helps to judge an AI tool the way you’d judge a diagnostic test and a drug at the same time. It performs like a test, so you ask about sensitivity, specificity and predictive value. It lives like a drug, so you ask about its ingredients, its indication, its trials and its safety monitoring. That gives four questions.

02Data provenance: where did the data come from?

In 2019 a team in Heidelberg tested a commercial melanoma classifier, one already approved as a medical device in Europe, on 130 skin lesions: 107 benign moles and 23 melanomas. Each lesion was shown to the model twice, once clean and once marked with the purple ink surgeons draw around a lesion before excision. Nothing about the moles changed. The model’s specificity on benign moles fell from 84% to 46%, and its rate of false alarms more than tripled. Heat maps showed it was looking at the ink, and when the researchers cropped the ink out, performance came back.

The explanation is clinical. Surgeons mark the lesions that worry them, so the authors suspected that ink was over-represented among the melanomas in the training data. If so, the model learned the surgeon’s worry, not the disease. The people who suspected the ink were dermatologists, who know why lesions get marked. Without that context, ink is just pixels.

This isn’t a one-off. A model learns whatever separates its positives from its negatives. Pneumonia models have learned which hospital an X-ray came from, because one hospital had far more pneumonia than the others. Dermatology models have performed worse on darker skin, and the images used to train them have mostly shown lighter skin. So ask three things about the ingredients. Could it be right for the wrong reason? Who is in the data, and do they look like my patients, the way you’d read a trial’s inclusion criteria? And how were the labels made: biopsy and follow-up, or software skimming the report text?

In my own work, most of the effort goes into the data, not the model, and I’m not alone. A study of AI practitioners in high-stakes fields was titled, literally, “Everyone wants to do the model work, not the data work.” The authors described data cascades, problems that compound quietly downstream, and 92% of the practitioners they interviewed had hit at least one.

03Label definition: what exactly does it predict?

A widely used commercial algorithm helped US health systems choose patients for an extra-care program. The goal was to find the patients who needed help most. But need isn’t a column in a database, so the developers predicted something that is: each patient’s healthcare cost over the next year. In the US, less is spent on Black patients than on white patients at the same level of illness, partly because of unequal access to care. So at the same risk score, Black patients were considerably sicker. The algorithm predicted cost well. It was answering the wrong question.

The authors, led by Ziad Obermeyer, an emergency physician, estimated that removing the bias would raise Black patients’ share of the extra help from 17.7% to 46.5%. Working with the manufacturer, they then changed only the label, to a mix of health and cost, and one measure of the bias fell by 84%. The fix was a better clinical question.

Pharmacy students will recognize the pattern. In the Cardiac Arrhythmia Suppression Trial, drugs that suppressed the extra beats on the ECG looked excellent on the surrogate, and more patients died. A proxy label is a surrogate endpoint, and surrogates can fail.

A label can also be wrong in a quieter way: it predicts something that doesn’t change what the team does. When Michigan Medicine independently tested the Epic Sepsis Model across 38,455 hospitalizations, it measured an AUC of 0.63, against the 0.76 to 0.83 the vendor had reported. The model missed two-thirds of sepsis cases and alerted on 18% of all hospitalizations. Of 2,552 patients with sepsis, it flagged only 183 whom clinicians hadn’t already treated in time. An alert that mostly repeats what the team already knows, or stays silent, changes very little.

My fix is to rewrite any vague claim, such as “AI detects sepsis”, as a clinical question. Which patients? Predicting what, exactly? Who sees the output? At what moment? And what will they do differently? It’s PICO, plus the user and the moment. If nothing changes at the bedside, the model has no indication. I believe the decisions about the label and the moment of use need a clinician in the room, and across the field, many teams still make them without one. That empty seat is where clinicians belong.

04Evaluation: what evidence is there beyond accuracy?

Here’s a question from the talk. An AI flags inpatients at high risk of a serious adverse drug reaction. It’s 95% sensitive and 95% specific, and 1 in 100 patients will have the reaction. When it flags a patient, what’s the chance that patient actually has the reaction?

Take 10,000 inpatients. A hundred will have the reaction, and the model catches 95 of them. Of the 9,900 who won’t, 5% are flagged anyway: 495 false alarms. So of 590 alerts, 95 are real, about 16%. Five of every six alerts are false. Put the same tool on a ward where 1 in 10 patients react, and the figure rises to about 68%. Same tool, different value.

If your gut said 95%, you’re in good company. In a 2014 survey of 61 doctors and students at a Boston-area hospital, only about a quarter got a version of this question right, and the most common answer was 95%. Many AI papers still lead with accuracy or AUC. Clinical training teaches us to ask for the positive predictive value at our own prevalence, even if few of us can do the arithmetic on the spot, and I believe that habit matters more than any detail of how the model is built.

Regulatory clearance doesn’t settle the question either. A drug must show in trials that it works before approval. Most AI devices reach the US market through 510(k) clearance, which shows they are similar to an existing device, not that they help patients. A review of 130 FDA-cleared AI devices found that 126 had been evaluated only on past data, and none of the 54 high-risk devices had a prospective evaluation.

Drug development already has a language for this gap, and an older hierarchy from diagnostic imaging maps onto it well. Accuracy on past data is preclinical. Silent mode, where the model runs live but nobody sees its output, is an early-phase study. A supervised pilot asks whether it changes decisions. A randomized trial asks whether it changes outcomes. Monitoring after go-live is Phase IV. Most AI tools you’ll meet are still on the first rung.

The best example I know of a tool that climbed is MASAI, a randomized trial of AI-supported mammography screening in Sweden, where two radiologists normally read every exam. In the AI arm, the exams the AI scored as highest risk got two readers and the rest got one. Among the first 80,033 women, the AI arm found 244 cancers against 203, with a similar number of recalls, and radiologists did 44% fewer reads. The final results, for about 106,000 women, showed AI-supported screening was non-inferior on interval cancers, not proven better, and interval cancers are still a surrogate for deaths. To me, that’s what honest evidence looks like.

05After go-live: who watches it once it’s in use?

Strong results in testing don’t guarantee that a tool will work in a real clinic. Google’s diabetic retinopathy AI did well in earlier testing, but when it went into 11 clinics in Thailand, more than one in five images were rejected as too low in quality, often because of poor lighting, and slow internet in some clinics delayed results, because the images were sent to the cloud. That’s efficacy versus effectiveness, the same gap you know from drug trials.

Then the world changes. In 24 hospitals using the Epic sepsis model, the share of hospitalized patients triggering a sepsis alert each day rose from 9% in the three weeks before each hospital’s first COVID-19 case to 21% in the three weeks after, and Michigan paused its sepsis alerts. The study measured alert volume, not accuracy, but the authors suggest dataset shift as a likely cause. Nothing in the code changed; the patients did. Noticing that the patients have changed is a clinical skill too, and I believe clinicians often notice it before any dashboard does.

Drugs come with a package insert, and so should models. A team at Duke proposed a “Model Facts” label and published an example for their own sepsis model. It states what the model predicts (sepsis in the next four hours), for whom (adults in their emergency department), and how well it performed, with “TBD” where the evidence doesn’t exist yet. Most usefully, it says when not to use it without further testing: in the ICU, after a first episode of sepsis, or at another hospital. And it says plainly that it is not a diagnosis.

06Four questions for any medical AI tool

1. Data provenance. Where did the data come from, and do those patients look like mine?Like a drug’s ingredients and trial population
2. Label definition. What exactly does it predict, and who acts on it, when?Like a drug’s indication and endpoint
3. Evaluation. What evidence is there beyond accuracy, at my prevalence?Like a drug’s trial phases
4. After go-live. Who watches it once it’s in use?Like a package insert and pharmacovigilance

None of the four is about the model. A computer-science graduate can build the model. I believe the room where the label is chosen needs someone who knows why the image exists, which outcome matters, and what a positive predictive value means at the bedside. If you’re a medical or pharmacy student, that empty seat is yours.

ASources

1. Sculley D, et al. Hidden technical debt in machine learning systems. NeurIPS 2015.
2. Winkler JK, et al. Association between surgical skin markings in dermoscopic images and diagnostic performance of a deep learning convolutional neural network for melanoma recognition. JAMA Dermatol 2019;155:1135-41.
3. Zech JR, et al. Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs. PLoS Med 2018;15:e1002683.
4. Daneshjou R, et al. Disparities in dermatology AI performance on a diverse, curated clinical image set. Sci Adv 2022;8:eabq6147.
5. Sambasivan N, et al. "Everyone wants to do the model work, not the data work": data cascades in high-stakes AI. CHI 2021.
6. Obermeyer Z, Powers B, Vogeli C, Mullainathan S. Dissecting racial bias in an algorithm used to manage the health of populations. Science 2019;366:447-53.
7. Cardiac Arrhythmia Suppression Trial (CAST) Investigators. Preliminary report: effect of encainide and flecainide on mortality in a randomized trial of arrhythmia suppression after myocardial infarction. N Engl J Med 1989;321:406-12.
8. Wong A, et al. External validation of a widely implemented proprietary sepsis prediction model in hospitalized patients. JAMA Intern Med 2021;181:1065-70.
9. Manrai AK, et al. Medicine's uncomfortable relationship with math: calculating positive predictive value. JAMA Intern Med 2014;174:991-3.
10. Wu E, et al. How medical AI devices are evaluated: limitations and recommendations from an analysis of FDA approvals. Nat Med 2021;27:582-4.
11. Fryback DG, Thornbury JR. The efficacy of diagnostic imaging. Med Decis Making 1991;11:88-94.
12. Lång K, et al. Artificial intelligence-supported screen reading versus standard double reading in the Mammography Screening with Artificial Intelligence trial (MASAI): a clinical safety analysis of a randomised, controlled, non-inferiority, single-blinded, screening accuracy study. Lancet Oncol 2023;24:936-44.
13. Gommers J, et al. Interval cancer, sensitivity, and specificity comparing AI-supported mammography screening with standard double reading without AI in the MASAI study: a randomised, controlled, non-inferiority, single-blinded, population-based, screening-accuracy trial. Lancet 2026;407:505-14.
14. Beede E, et al. A human-centered evaluation of a deep learning system deployed in clinics for the detection of diabetic retinopathy. CHI 2020.
15. Wong A, et al. Quantification of sepsis model alerts in 24 US hospitals before and during the COVID-19 pandemic. JAMA Netw Open 2021;4:e2135286.
16. Sendak MP, Gao M, Brajer N, Balu S. Presenting machine learning model information to clinical end users with model facts labels. npj Digit Med 2020;3:41.