When I was in high school in the early 2000s, I spent a week of my summer vacation shadowing a pathologist at the local hospital. Every day in his basement office was basically the same; he’d focus his microscope on a slide of tissue, squinting for minutes at a time, methodically making notes about the shape of the cells, their size, their surroundings. When he had enough data points he’d make the phone call: “Squamous cell carcinoma.” “Serrated adenocarcinoma.” “Benign.”

For decades, doctors have relied on the well-trained eyes of human pathologists to give their patients a cancer diagnosis. Now, researchers are teaching machines to do that time-intensive work in as little as a few seconds.

In new research published today in Nature Medicine, scientists at New York University re-trained an off-the-shelf Google deep learning algorithm to distinguish between two of the most common types of lung cancers with 97 percent accuracy. This type of AI—the same tech that identifies faces, animals, and objects in pictures uploaded to Google’s online services—has proven adept at diagnosing disease before, including diabetic blindness and heart conditions. But NYU’s neural network learned how to do something no pathologist has ever done: identify the genetic mutations teeming inside each tumor from just a picture.

“I thought the real novelty would be not just to show the AI is as good as humans, but to have it provide insights a human expert would not be able to,” says Aristotelis Tsirigos, a pathologist at the NYU School of Medicine and a lead author on the new study.

To do so, Tsirigos’ team started with Google’s Inception v3—an open-source algorithm that Google trained to identify 1000 different classes of objects. To teach the algorithm to distinguish between images of cancerous and healthy tissue, the researchers showed it hundreds of thousands of images taken from The Cancer Genome Atlas, a public library of patient tissue samples.

Once Inception figured out how to pick out cancerous cells with 99 percent accuracy, the next step was teaching it to tell two kinds of lung cancers apart—adenocarcinoma from squamous cell carcinoma. Together, they represent the most prevalent forms of the disease, which kills more than 150,000 people a year. While they appear frustratingly similar under the microscope, the two cancer types are treated very differently. Getting it right can mean the difference between life and death for patients.

When the researchers tested Inception on independent samples taken from cancer patients at NYU, its accuracy went down a bit, but not much. It still correctly diagnosed the images between 83 and 97 percent of the time. That’s not surprising, says Tsirigos, given that the hospital’s samples carried much more noise—inflammation, dead tissue, and white blood cells—and were often processed differently than the frozen TCGA samples. Improving the accuracy will just be a matter of having pathologists annotate slides with more of those additional features, so the algorithm can learn to pick those out too.

But it wasn’t a helping human hand that taught Inception to ‘see’ genetic mutations in those histology slides. That trick the algorithm learned all on its own.

Again working with data from the TCGA, Tsirigos’ team fed Inception genetic profiles for each tumor, along with the slide images. When they tested their system on new images, it was able to not only identify which ones showed cancerous tissue, but the genetic mutations of that particular tissue sample. The neural network had learned to notice extremely subtle changes to a tumor sample’s appearance, which pathologists cannot see. “These cancer-driving mutations appear to have microscopic effects that the algorithm can detect,” says Tsirigos. What those subtle changes are, however, “we don’t know. They’re buried [in the algorithm] and nobody really knows how to extract them.”