Brett Martin's Difficult Men does for the outstanding American TV dramas of recent years what Peter Biskind's Easy Riders, Raging Bulls did for the great US movies of the 1970s: it's an entertaining and insightful history of how they came to be made. The title refers not just to the new breed of televisual anti-hero that emerged from 1999 onwards, when Tony Soprano debuted on HBO, but also to the "showrunners", the "all-powerful" writer-producers behind them: The Wire's David Simon, Deadwood's David Milch, Mad Men's Matthew Weiner, and most of all, The Sopranos' David Chase. Martin argues that the open-ended 12- or 13-episode serialised drama became "the signature American art form of the first decade of the 21st century", the equivalent of the novels of Roth, Updike and Mailer in the 1960s, or the films of Scorsese, Altman, Coppola and others in the 70s. And he presents the people who made them: a group of grumpy, stubborn, obsessive middle-aged men – mostly veterans of a hard and demoralising (if well-paid) industry, who have spent years writing formulaic cop shows and tired sitcoms. "You're here for two things," Chase once told a junior colleague on an earlier show, who had dared to use the word "art": "selling Buicks and making Americans feel cosy."

In these days of meth-dealing school teachers, it's easy to forget how shocking the idea of a criminal protagonist used to be. The conventional TV wisdom, enforced by advertisers, had always been that Americans would watch films with morally compromised heroes, but that they would never allow them into their living rooms. However, the proliferation of cable TV channels – and later, the introduction of DVD box sets – led to the creation of new, lucrative niche markets. HBO, a pay-TV channel that doesn't take advertising, produced the first wave of these shows as part of a deliberate corporate strategy: it was seeking out a sophisticated, affluent audience, and "adult themes", as they say, were a distinctive selling point. (HBO is of course famous for its use of "sexposition", a speciality of Game of Thrones: spicing up boring exposition with a gratuitous sex scene.) Its executives were famously hands-off and sympathetic to writers. Even so, when the time came to give the go ahead to The Sopranos, HBO's two pioneering bosses, Chris Albrecht and Carolyn Strauss, asked each other: "Should we do this? We should do this! Can we do this?" After shooting the pilot, Chase told his cast and crew: "You've been great. It's been lots of fun. Unfortunately, nobody is going to watch this."

But people did watch it: soon The Sopranos was racking up 10 million viewers per week, unheard of in cable TV, and became a nationwide cultural event. Industry lore has it that HBO delivered only two notes – registered only two serious reservations – about the show. The first was about the title, which they worried was misleading; they offered the lame alternative of Family Man. The second was about the fifth episode of the first series, "College", in which Tony encounters a Mafia informer while taking his daughter Meadow to an open day at a liberal arts college in Maine, and strangles him with a length of wire. Albrecht felt that this would "lose the audience". Chase prevailed on both points, and once this Rubicon had successfully been crossed, the floodgates opened: in the wake of the suburban mob boss came a slew of very bad cops (The Shield), dysfunctional undertakers (Six Feet Under), drug king-pins (The Wire), and murderous pimps (Deadwood). Writers vied to break all kinds of TV-show taboos: they had policemen murdering policemen; they killed off children; and they indulged in the frequent slaughter of regular characters that made new scripts so nerve-racking for cast members.

The main pleasure of this book is its detailed descriptions of the creation of each show, particularly The Sopranos and The Wire. The former was rooted firmly in Chase's own New Jersey Italian childhood; the family name was originally "DeCesare". His "passive aggressive, fearful, domineering" mother provided the model for Tony's mother Livia, "one of the more idiosyncratically terrifying and funny characters" in American TV; while his father, a disappointed store owner, "radiated frustration in way that would later find expression in Tony Soprano". When Chase discovered Fellini as a student, he thought: "This is the kind of nonsense that goes on in my house. The melodrama and the self-pity and the obsessiveness and the craziness. This is my DNA."

After film school, Chase worked in television for years, on the detective show The Rockford Files and on Northern Exposure, despising himself for not making avant garde films, while absorbing the lessons of the medium. Under Stephen J Cannell, the producer of 21 Jump Street and The A-Team, he learned that "your hero can do a lot of bad things, he can make all kinds of mistakes, can be lazy and look like a fool, as long as he's the smartest guy in the room and he's good at his job. That's what we ask of our heroes."

Chase also learned how to resist outside interference, skills born of a working lifetime's struggles with executives. The book is full of episodes in which he is asked to mildly tone down some bleak plot development, and says: "No, I just can't see it that way." One collaborator remembers: "And sometimes you'd say, 'Can we just move this comma over here?' And he'd say, 'I thought about it and I just can't do it.'" When he finally got his own show, he ran it in a Tony Soprano-esque fashion. The writers' room – an institution of US TV drama, in which a group of highly-strung writers is assembled in a room "for eight hours a day, mostly to face rejection and all in service of another person's vision" – sounds like a terrifying place. Chase sacked one of his writers, Todd Kessler, minutes after Kessler received an Emmy nomination for writing the terrific finale to season two ("I guess the timing isn't great …" Chase said). Kessler went on to create Damages, in which Glenn Close plays a terrible boss – "brilliant but manipulative, vain, imperious, unpredictable" – who sucks in and corrupts a young protege.

The only Sopranos writer who seems to have emerged unscathed was Weiner, later the showrunner of Mad Men, an abrasive egotist whose character is probably best summed up by his rousing speech after his show won a Golden Globe: "This is what you wait for," he said, "so you can tell all those people who ever said anything bad to you to go fuck themselves!" But even Weiner sounds like an ideal colleague compared to Milch, of NYPD Blue and Deadwood. A drug‑addled gambler, Milch gave up on scripts altogether and extemporaneously dictated lines to bemused actors, when not randomly handing out hundred-dollar bills, urinating out of windows, or assembling teams of young, attractive women – "vestal virgins" – to assist his creative processes. Breaking Bad's showrunner Vince Gilligan is, in stark contrast, is a "relaxed southern gentleman" who prides himself on running the happiest writers' room in the business.

The genesis of The Wire is a more familiar story, because of its background in Simon's work as a Baltimore police reporter, and Homicide, the now-famous book that came out of it. But Martin's account is still interesting. If you've ever wondered why the final series is so much worse than the other four, it's all here. Simon's writing partner Ed Burns, a brilliant, arrogant former policeman whose surveillance operations inspired the titular wire, usually shot down his bad ideas and came up with good ones by the truck-load; but he was off working on the Iraq war mini-series Generation Kill. Simon, meanwhile, became carried away by a long-running feud with his former employer, the Baltimore Sun. The show's other main writers, particularly George Pelecanos, had little interest in the satirical newspaper storyline, and felt, surely correctly, that the subplot in which Dominic West's McNulty simulates a serial killing spree was poorly judged.

There's not a great deal in the way of criticism in this book, which is perhaps a pity, because Martin does it well. His essential argument is that most of these dramas are concerned with middle-aged masculinity and its discontents; he's good on the balance of domesticity and fantasy, identification and wish-fulfilment in shows such as The Sopranos and Breaking Bad; he contextualises them well in the Bush years, and the economic slump. But what the reader mostly takes away from Difficult Men is its memorable description of the industry: the writers' room, desperately searching for the episode's 18 "beats"; stressed-out character actors inhabiting their difficult parts for up to a decade (James Gandolfini used to work himself up for his scenes by, say, smashing a stereo in his trailer or smacking the back of his own head). Most of all, the book leaves the image of the showrunner: the possibly unstable writer, in charge of every detail of a massive artistic-commercial enterprise. As one TV veteran remarks: "This isn't like publishing some lunatic's novel or letting him direct a movie. This is handing a lunatic a division of General Motors.