It is a truth universally acknowledged that messing with a guy who has facial tattoos is a really bad idea.

Getting dirty words tattooed on your eyelids—a popular choice, judging from the mug shots available online—is a serious commitment. It is, as social scientists say, a "signal that is costly to fake." The bearer of a facial tattoo announces to the world: I expect to be in prison for most of my life, or to hang out with people who consider prison experience a character reference.

Those of us who are not a part of the criminal underworld have a much cheaper system: Asked for a reference, we happily provide our colleagues' phone numbers and email addresses. But for crooks, broadcasting signals about their professional pasts and current social networks is a good way to wind up with a new pair of concrete shoes. In Codes of the Underworld: How Criminals Communicate (Princeton), the Oxford sociologist Diego Gambetta uses colorful stories and a minimum of jargon in his quest to analyze how people advertise when their business happens to be illegal.

Unlike a legal trademark, an underworld brand can't be defended with little more than an expensive attorney. If another gang steps into your turf, you can opt for a violent defense of your signal of choice. But gangsters who previously relied on large gaudy tattoos to get a message across can hardly go around roughing up every 17-year-old with a tramp stamp on her tailbone.

As tattoos go mainstream, criminals have to adapt. These days, even art on your neck, collarbone, and wrists is barely enough to signal your commitment to subcultures that are totally legal.

But there are still some kinds of tattoos—including those inky eyelid admonitions and the homespun variety created with a shard of a ballpoint pen during long hours behind bars—that retain their signaling power, demonstrating a commitment to the criminal way of life. A guy with extensive Aryan Brotherhood facial tattoos is unlikely to snitch on his buddies. The only thing worse than getting an eyelid tattoo is having one removed. What's he going to do, go into witness protection and start a new life as a kindergarten teacher in Ohio?

In Japan, members of the yakuza have long favored tattoos covering the entire upper body to signal their mafia status. They also amputate all or part of a pinky finger. One study estimated that between 40 percent and 70 percent of the yakuza had sacrificed a digit, generally making the cut themselves.

Even with tattoos and auto-amputations, this traditional mafia once operated almost completely in the open. Members had business cards, embossed with the emblems of their gangs and legends indicating their rank. There's an entire genre of Japanese movies devoted to their exploits. By 1974, Gambetta writes in an excellent chapter on how "art imitates (low) life," the Japanese B-movie industry was producing 100 yakuza movies a year.

But in 1992, as the Japanese government began a crackdown on gangsters under the Violent Groups Control Law, filmmaker Juzo Itami released a yakuza film different from the traditional B-movie fare. Content, even proud, to be immortalized in film as brutal killers—a useful reputation for anyone who can't take their complaints to a court of law—the yakuza drew the line at the bumbling idiots depicted in The Gentle Art of Japanese Extortion. Late one night, three young men jumped Itami and slowly carved up his face.

By scarring Itami for life, the yakuza gangsters provided evidence not only of their toughness but of his. Itami wore the scars with pride. This too has precedent. German duelers in the 19th century prized Rennomierschmiss, or bragging scars. It was a great disappointment if a hit didn't scar properly, and young men were known to tamper with the healing process, purposely patching themselves together with rough stitches. Gambetta calls the result "an upscale tattoo, borne by a generation of doctors and jurists and professors and officials, certifying the proprietor's claim to both manly stature and cultivated rank." The scars, says Gambetta, might be properly classified as "the joint production of a signal of courage—I slash your face, you slash mine."

If you aren't a properly tattooed or otherwise credentialed tough guy, scars can act as more than mere symbols of courage. They can also be signs of madness—a useful and understudied tool in the symbolic criminal conversation. Here Gambetta borrows from the work of Marek Kaminski, a sociology student imprisoned in Poland in the 1980s for "anti-Communist activities." Kaminski lays out a jailhouse hierarchy in which 70 percent are grypsmen, or initiated gang members, 1 percent to 2 percent are "fags," and the rest are "suckers," outsiders vulnerable to all manner of prison predation. Then he tells a story about a man named Prince.

Thrust into a new Polish prison cell—which, as usual, was open-plan and almost completely unsupervised, as the guards were afraid to enter—Prince is immediately asked if he is a grypsman. He says no, then pulls a razor from his cuff and shouts, "I am a sucker, sucker-madman!" He threatens the assembled grypsmen and then cuts himself until he bleeds on the floor. The grypsmen back off: "OK, you are a sucker but you are a tough-boy. You are kind, we are kind. Now clean it up." Kaminski later relates his discovery that Prince is crazy like a fox, wielding his razor only when he knows he is entering a tough room. (Gambetta dryly notes that many of these strategies work in another environment with a stable population and forced attendance: school.)

The (acknowledged) ghost inside Codes of the Underworld is the Nobel-winning economist Thomas Schelling, who has for decades been applying this kind of economic analysis to the coordination and signaling problems inherent in bank robberies, blackmail, and negotiations. Schelling's 1960 book about the nuclear arms race, The Strategy of Conflict, centers around those paradoxical situations when it makes sense to impose limits on yourself in order to improve your negotiating position. His famous "madman theory" holds that a "careless or even self-destructive attitude toward injury…can be a genuine strategic advantage."

Guided by those words, Gambetta sets out to illuminate the world inhabited by these face-tattooed, duel-scarred, razor-brandishing inmates. The result is a book that explains the hidden logic of their behavior in language intelligible to those of us who make it a point to steer clear of both well-armed dictators and well-decorated mafiosi.

Katherine Mangu-Ward is a senior editor at reason.