In recent years “true crime”, in the form of television documentaries and podcasts, has become very fashionable. But there has always been a small niche for true crime books, sometimes tucked – rather guiltily – below the much larger crime fiction sections in bookshops and libraries. I have about 400 such volumes – the memoirs of criminals, detectives, crime reporters and the explorations and investigations carried out by authors and academics over the last 150 or so years – many of which I have been using while working on Underworld, which is a history of the last century and a half of organised – and disorganised – crime in Britain.

What is striking is how much things have changed in the last couple of decades, in that many of those involved in territorial gang warfare are now so young. Researching my book, I attended five gang murder trials at the Old Bailey and almost all of the accused and their victims were either teenagers or in their very early 20s. And at the more organised end of the underworld are now men – they are mostly men – who are almost indistinguishable from the bankers and brokers through whom they launder their loot in what is an increasingly international and tech-savvy game.

Of true crime books, the most renowned of the genre are probably Truman Capote’s In Cold Blood and the late Gordon Burn’s Somebody’s Husband, Somebody’s Son, his account of the crimes of Peter Sutcliffe, the Yorkshire Ripper. And putting everything in context is Judith Flanders’s wonderful book, The Invention of Murder, so I’ve omitted those from the list on the grounds that they need no introduction.

I’ll add a plea to future true crime authors: please, no more Jack the Ripper books!

1. A Sense of Freedom by Jimmy Boyle and Redeemable by Erwin James

I’m cheating here and naming two books because they both deal with redemption. The former is a very powerful account by the Glasgow-hardman-turned-acclaimed-artist of his time in Barlinnie prison. Erwin James, who was jailed for life for two murders in 1984, is a writer who needs no introduction to Guardian readers, who will recall the articles on prison life he wrote for the paper during his 20 years inside. When Redeemable came out, I wrote that it was “one of the most powerful and touching books on crime and punishment I have ever read … It should be on the bedside table of every Home Office minister and anyone involved in the criminal justice system.” I stick by that.

2. The Profession of Violence by John Pearson

The best book of the more than 50 volumes so far on the Kray twins. Pearson was granted remarkable access to Ronnie and Reggie at the same time as they were busy posing for David Bailey portraits. Hard to imagine that any modern-day gangsters will be quite as open with any would-be biographer.

Tricky customer … Shirley Pitts

3. Gone Shopping by Lorraine Gamman

Shirley Pitts was a shoplifter who operated in the 1950s and 60s and, when she died in 1992, was buried in a £5,000 Zandra Rhodes dress that she did not buy over the counter. Above her grave was a floral tribute in the shape of a Harrods shopping bag and the legend “Gone Shopping”, which has given the title to this perceptive and pithy book about her and the criminal background from which she came.

4. East End Underworld by Raphael Samuel

Published in 1981, this is the story of Arthur Harding, who was born in 1896 and became known as “the slipperiest character in Brick Lane”. Samuel interviewed him over a period of six years and chronicled his tales of the old East End racecourse gangs and protection rackets and Harding’s role in a bloody pub fight in 1911 that became known as “the Vendetta Affair”.

5. Good Cop, Bad War by Neil Woods, with JS Rafaeli

Police memoirs – like criminal memoirs – have tended to be on the vainglorious side, with little self-reflection. This, along with Graham Satchwell’s equally frank An Inspector Recalls, is an exception. It is the inside story of an undercover drugs squad cop told in great detail and with remarkable frankness. It is also an attack on the current drugs laws, which Woods now believes mainly benefit professional criminals and cause enormous social damage: “Fighting to end the War on Drugs will do more to harm the gangsters than anything I ever accomplished as a cop.”

6. Strange Piece of Paradise by Terri Jentz

In 1977, Jentz and a fellow undergraduate were cycling across the US together when they were attacked randomly at night in an Oregon campsite by a stranger with an axe. Both women survived but no one was arrested for the crime, so 15 years later Jentz went back to try and track down the man who tried to kill them. She succeeded and her work led to a change in the law in Oregon. Jentz is a friend, but I hope I would have chosen this remarkable book even if I didn’t know her.

‘Master idiot’ … Peter Scott seen in the 2004 documentary The Heist. Photograph: Channel 4 Picture Publicity

7. Gentleman Thief by Peter Scott

He was the “king of the cat burglars” who recited from the WE Henley poem Invictus when he was arrested for his last crime, which involved the theft of a Picasso. At his funeral the gospel song Steal Away was played. This is a very candid account by the man who was described as a “master criminal” but who said that “master idiot” would have been a more accurate title. Like his contemporary, the Great Train Robber Bruce Reynolds, author of The Autobiography of a Thief, he writes with wit and self-knowledge.

8. The Lost Girl by Caroline Roberts

There were many fine books about the serial killers Fred and Rose West, including Gordon Burn’s Happy Like Murderers, Andrew O’Hagan’s The Missing, Howard Sounes’s Fred & Rose and Brian Masters’s She Must Have Known. But this is a book by a victim of the Wests who lived to tell her tale. Caroline Owens, as she then was, was 16 when she was savagely attacked by the Wests but she managed to escape and would give evidence many years later in Rose’s trial, on behalf of “all those girls who didn’t make it”. Sadly, she died of cancer three years ago but her account, both of the Wests and of her depressing experiences with some sections of the press, is very poignant.

9. Murder in Notting Hill by Mark Olden

The story of the 1959 murder of Antiguan carpenter Kelso Cochrane by a white gang who stabbed him to death at a time of great racial tension in this area of west London. Told succinctly by Olden, who tracked down elderly witnesses and went through the archives on the case, this is a very timely reminder not only of a grim murder but of the ways in which investigations were carried out and reported in those days. Some of that has changed, but not all.

10. A Few Kind Words and a Loaded Gun by Noel “Razor” Smith

In this memoir, the former bank robber demystifies much of the “glamour” surrounding professional crime. He recounts one occasion when he tried to hold up a newsagent’s with a Luger pistol and was told by its Ugandan-Asian proprietor: “ Your gun is unloaded – you are minus the magazine. And you swear far too much for such a young man.” Smith bought a Mars Bar instead and told him to keep the change.

• Underworld: The Definitive History of Britain’s Organised Crime by Duncan Campbell is published by Ebury Press.