Library of Congress An 1853 print by Japanese artist Toyokuni Utagawa shows a ninja sneaking up behind a musician.

Ninja are everywhere in popular culture — they slice up mobsters in movies and fruit on your iPhone. We seem to think it makes sense to compare them to pirates and then think of them fondly as adolescent mutant turtles living in a New York City sewer with a kimono-clad rat.

But who were the ninja, really? That’s what John Man, a British travel writer and historian, set out to explore in his new book, Ninja: 1,000 Years of the Shadow Warrior. He spoke to TIME about the fighters behind the legend.

LIST: Happy Ninja Day: Top 5 Ninja in Pop Culture

Next 1. The Smartest Ninja Didn’t Wear Black