← AI stack

Essay 9 min read

What a neural network actually sees when it reads a radiograph

The model does not see a tooth. It sees numbers, and the gap between those two things is where all the interesting failures live.

When I show a colleague a heatmap from one of my models, the red blob sitting over the lesion, the reaction is almost always the same. They nod and say something like, so it found it. And it did, in the sense that the number came out right. But the word “found” smuggles in a whole theory of what happened inside the network, and that theory is usually wrong. The model did not find a lesion. It does not have a lesion in its head to find. It has pixels, and a function it learned that turns those pixels into a score, and the score happened to be high.

That sounds like a technicality. It is not. Almost everything that goes wrong with medical imaging AI, and almost everything that makes it worth using anyway, comes from taking that sentence seriously.

What the model is doing

A convolutional network starts with the image as a grid of numbers. The first thing it does is slide small filters across that grid, a few pixels at a time, and at each position it multiplies the filter against the patch under it and adds up the result. Stanford’s CS231n notes describe this plainly: each filter is small in width and height but runs through the full depth of the input, and you convolve it across the image computing dot products. A five by five filter over a color image is just seventy five numbers the network gets to tune.

Slide one filter over the whole image and you get a new grid called an activation map, bright where the filter’s pattern was present and dark where it was not. Stack the maps from all the filters and you have the layer’s output, which the next layer treats as its input. The early filters learn tiny, generic things. Zeiler and Fergus, who built a way to see what each layer responds to, found the second layer keying on corners and edge and color junctions, the third layer on textures like mesh and text, the fourth on class-specific parts like a dog’s face, and the fifth on whole objects. Edges become textures become parts become things. Nobody programs that hierarchy. It falls out of training.

Two details matter for anyone who reads scans. The first is pooling, which throws away three quarters of the activations at a stroke by keeping only the strongest response in each little neighborhood. It buys efficiency and a bit of tolerance to small shifts, and it is one reason the output is coarse. The second is the receptive field, the region of the original image that can influence a given unit. You would think a deep network sees the whole image at every layer. It does not. Luo and colleagues showed that the part of the field that actually drives a unit is a Gaussian blob near the center, and it grows only with roughly the square root of the depth. The network’s real window on the image is smaller than the architecture suggests, and it is looking hardest at the middle.

So the honest description is this. The model learns a stack of pattern detectors, tuned by gradient descent to whatever features happen to lower its error on the training set. It is very good at that. It has no idea what a tooth is, what an infection is, or that a patient is attached to the image.

That is not how I read

When I open a scan I do not start with pixels. The perception research is fairly settled on this. Kundel and Nodine flashed chest films at radiologists for two tenths of a second, too short for a single eye movement, and readers still hit around seventy percent accuracy on normal versus abnormal. Something global and nearly instant happens before the eyes even move. Kundel’s later gaze tracking found that most cancers were fixated inside the first second of viewing. You get a gestalt, a sense that something is off, and only then does the slow, deliberate search begin to confirm or dismiss it.

That global read is powered by everything the model does not have. I know the patient’s age and history, so I walk in with a prior. Ground glass in a young patient with a painless expansile mandible points one way; the same texture somewhere else points another. Radiologists are running a rough Bayesian update, and a systematic review found clinical history improved interpretation in fifteen of twenty two studies. The scan is one input among several.

Human reading has its own failures, and they are humbling. Satisfaction of search is the classic one, where you find the first abnormality and stop seeing the second, and it accounts for up to a third of certain reading errors. Drew and Wolfe famously pasted a gorilla into a chest CT, forty eight times the size of a nodule, and eighty three percent of radiologists never reported it, many of them looking straight at it. Expertise did not protect against it. So the two readers, the model and me, fail in different directions. I anchor, I tire, I stop searching once satisfied. The model never tires and never anchors, and it also never knows what it is looking at.

Right for the wrong reason

Here is the failure mode that should keep anyone honest. Because the model chases whatever features lower its error, it will happily learn something that has nothing to do with disease, as long as that something is correlated with the label in the data you gave it. Geirhos and colleagues call these shortcuts, and the literature on them in medical imaging is genuinely alarming.

Zech’s group trained a pneumonia detector that worked well on its home hospitals and fell apart across town. It turned out the network could tell which hospital a film came from, and different hospitals had different disease rates, so it partly learned to read the scanner instead of the lungs. Oakden-Rayner found a pneumothorax model scoring far better on images that contained a chest drain, which is a treatment for pneumothorax, not a cause. A model leaning on the drain is dangerous precisely because the drain-free cases, the ones nobody has treated yet, are the ones you need it to catch. A melanoma classifier’s confidence jumped when a clinician had drawn a purple ink mark next to the lesion. During the pandemic, a wave of COVID chest x-ray models turned out to be reading dataset artifacts and patient positioning rather than lung disease. Every one of these models scored beautifully on its own test set. That is the trap. A good number on data that looks like your training data tells you almost nothing about whether the model learned the thing you wanted.

The heatmap does not save you

The natural response is to look at where the model attended and check that it looks reasonable. That is what my colleague was doing with the red blob. The most common tool for it, Grad-CAM, takes the gradients flowing into the last convolutional layer and turns them into a coarse map of which regions pushed the score up. It is useful. It is also weaker than it looks.

Adebayo and colleagues ran a simple sanity check: randomize the model’s weights and see if the explanation changes. For some popular methods it barely did, which means the pretty map was reflecting the image, not what the model learned, behaving more like an edge detector than an explanation. In medical imaging specifically the picture is worse. Saporta’s group built the first radiologist benchmark for chest x-ray localization and tested seven saliency methods; Grad-CAM did best among them and still fell well short of the human readers, worst on small and oddly shaped findings. A follow-up showed the maps could stay nearly identical while the model’s accuracy collapsed from an AUC of 0.88 to 0.01, and two radiologists caught the manipulated images less than half the time. A heatmap tells you where, at best, and coarsely. It does not tell you why, and it can look perfectly convincing while the model underneath is broken. Rudin’s argument, that for high stakes decisions we should build models that are interpretable to begin with rather than explaining black boxes after the fact, lands harder the more of this you read.

The coyote

The first time I got scared about any of this was a line I found while looking for reassurance, back when I had just committed to this specialty. Geoffrey Hinton, 2016: people should stop training radiologists now, because within five years, maybe ten, deep learning will do it better. He compared radiologists to the coyote in the cartoon, already over the edge of the cliff, just not yet looking down.

The five years came and went. By 2026 the reporting had flipped the other way. The radiologist workforce grew rather than shrank, roughly ten percent over the decade, into what has been called the largest shortage in the field’s history, with average compensation around five hundred seventy one thousand dollars and imaging volumes climbing faster than the people to read them. Hinton himself walked the timeline back and clarified he had meant image analysis specifically. Curtis Langlotz’s version has aged much better than the coyote: AI will not replace radiologists, but radiologists who use AI will replace those who do not.

I think the reason the prediction missed is the same sentence I started with. The model sees numbers. It is a genuinely powerful second reader, one that does not get tired or anchor or quit searching, and it is worth building and worth using. It is also a reader with no priors, no history, no idea what it is looking at, and a documented habit of learning the wrong thing and hiding it behind a plausible heatmap. That is not a machine you handoff the read to. It is a machine you read alongside, knowing exactly which of you is prone to which mistake. Which is why I still read the scan myself.

Sources

Still have a question?

Ask me directly.

This page answers the version I get asked most. If your case is the exception, the inbox is faster than the website.

Get in touch →