In 1999 Sergio De La Pava, then a New York defence lawyer in his late 20s, began writing a book. He tried to avoid rigid schedules and instead wrote on the train in the mornings or while waiting around in court. A Naked Singularity was to be a blistering attack on the American criminal justice system in which De La Pava still works. "If you learn only one thing from the ensuing maybe let it be this," he tells his reader. "The police were not merely interested observers who occasionally witnessed criminality and were then basically compelled to make an arrest, rather the police had the special ability to in effect create crime by making an arrest almost whenever they wished."

De La Pava represented poor, desperate, ill and addicted people, accused mainly of drug crimes but also of other offences. In his manuscript he raged against a system he viewed as ruled by cruelty and stupidity, in a plot that took its rookie lawyer and hero Casi on a journey to the most violent extremities of American crime and punishment.

In 2004 De La Pava, who by then had two small boys with his wife Susanna, a supervising attorney in a legal aid office whom he met at law school, finished the book. He had written 270,000 words, around three times the length of the average literary novel. Nobody wanted to publish it. "I knew no one," he says, meaning he knew no publishers, editors or authors."What you're supposed to do is send these letters describing the book to agents, and if they're interested you send them the manuscript. So I did that for three years. It was a really grim process.

"Replies varied. Some said 'not interested', others said 'sounds great, send it to me'. I think what I found most dispiriting was that quite a few people were into the concept of a book about criminal justice, but when confronted with something that was complicated and not easily quantifiable, that interest disappeared. It was humiliating. It was horrifying."

So he gave up, and began thinking about another book. But his wife did not give up. Instead she set about convincing him they should self-publish the novel, a much newer concept in 2008 than it is now. They paid a few thousand dollars for a print-on-demand deal, ordered 100 copies, had a party and sold the book to their friends. And after that, De La Pava says, "I forgot about it, the experience had been a bad one in my opinion, even though it was nice to see it in book form. I couldn't spend any more time thinking about publishing or editing. It was like an hour I spent thinking about it was an hour that I hated. I hated every aspect of it."

Susanna had other plans. She began sending review copies to every literary journal and website she could think of, and eventually her persistence paid off. Posts about the book began to pop up. "A blog, what the hell's that?" is how De La Pava describes his reaction to his wife's updates. "OK keep at it, do whatever you're doing and leave me alone."

Finally, more than 10 years after De La Pava began writing his novel, a publishing insider got in touch. Levi Stahl was publicity manager at University of Chicago Press. In 2010 he was also, in his spare time, poetry reviews editor at the Quarterly Conversation, an online literary magazine. Stahl got hold of A Naked Singularity after it was flagged up by a contributor, and was impressed.

He took it to an editor who agreed that although, as an academic publisher, University of Chicago does not publish new fiction, it should make an offer for this book. In 2012 De La Pava's slow-burn, word-of-mouth novel won the PEN/Robert W Bingham prize for the best debut novel of the year.

In 2013 University of Chicago published De La Pava's follow-up, the fragmentary and experimental Personae, much of which is made up of the imaginative writings of a character who has died alone, aged 111, in his Manhattan flat. Last year publisher Christopher MacLehose (who introduced the English-speaking world to Stieg Larsson) also brought out a British edition of A Naked Singularity following a tip from a retired editor who read about it on Twitter – and not before several well-known London publishing houses had turned it down. Then, this February, A Naked Singularity made the shortlist of the Folio prize, the newly launched £40,000 literary award for English-language fiction (the eventual winner was George Saunders, for his short stories, but the judges singled out De La Pava for praise; also shortlisted was Eimear McBride, another first-time writer who struggled for years to find a publisher, and who has since won the £30,000 Baileys prize).

The son of Colombian immigrants to the US, De La Pava grew up in New Jersey speaking Spanish (a Spanish translation of A Naked Singularity is shortly to appear). His father drove a cab, among other jobs, his mother worked in offices. They sent him to a Catholic high school from where he went to a state university and law school.

He is full of gratitude to MacLehose ("an outsize figure literally and figuratively – that's an individual who has devoted his life to literature"). But success has not removed the taste of failure, and De la Pava is not alone in feeling unnerved. The Folio prize judges called his novel a "messy masterpiece"– it is not only full of insight into judicial processes, but philosophy, science (the naked singularity of the title comes from particle physics), acerbic commentary on mass media, the life of Puerto Rican boxer Wilfred Benitez, Latino family life – still a rarity in English-language fiction – and pages of absorbing dialogue. His fictional world is a big, funny, information-packed place, yet initially no one wanted to know.

Through it all, his faith in his book was unshaken,"not that it's perfect, but it's what I set out to do," he says."I wanted it to have an angry, propulsive core but at the same time to be full of heart. It's a difficult combination to pull off. You can do one or the other, and there are dangers with either, but I wanted both and when I finished I read it and I was like 'Oh yeah'."

De La Pava began reading in Spanish, "but the reading in English kind of took off". He cites Melville, Woolf, Dostoevsky and Dickinson as influences and singles out Moby-Dick and To the Lighthouse. "I've always been reading, to the point of obsession. A novel to me is a magical thing so it just felt natural as a writer to sit down and say, 'I want to write a novel'."

He intended for his first book to be unlike anything else. "I wanted it to be funny, first of all because comedy is enjoyable and keeps you reading. I wanted it to be achingly accurate."

As a teenager his interest in law had been piqued by watching LA Law. As a public defender, paid by the city to represent the poorest people, he decided every TV show and film he had ever watched about law and order was wrong. "My experience of depictions of the criminal justice system was that they were grimly inaccurate, either purposefully because considerations of entertainment were taking over, or out of ignorance because the author didn't really understand. So I thought since I'm in this unique position where I actually do know what happens, intimately and accurately, let's make sure my depiction doesn't have the silly gloss that's usually applied to these things."

De La Pava's critique of US criminal justice is fierce and unrelenting, the clients in his novel are victims as well as law-breakers. The system is both mindless and soulless in his portrayal, fixated on definitions and processes that make no sense, lacking even the most basic humanity. Casi's typical defendant is a penniless addict involved in low-level dealing at worst, risking years behind bars for the few dollars he needs to feed his addiction.

As a law student on a placement in a public defender's office, De La Pava fell "head over heels" for the thrill of representing real people with problems rather than the state. Over two decades in the job, he came to the view that the whole system – judges, police, penal code – is "truly toxic". "I've probably handled 15,000 cases in my life," he says.

"You go into the system, let's say you're good at trying cases for whatever reason, so you will have success in the sense you will be able to help people who are in trouble. But at some point you will reach the limit of that and realise there's only so much you can do, and only so much that your being good at the job really means for the people you're representing."

This is the realisation faced by Casi in A Naked Singularity. The novel's second half, in which he involves himself in both a death row appeal on behalf of a man with profound learning difficulties and a heist, dramatises his spectacular response. Did De La Pava once face a similar crisis?

"I think I was naive when I started," he says. "Doing the work, even though I find it gratifying, breeds in me a certain dissatisfaction, a certain anger." Most public defenders believe the "common person's thoughts" on law and policing are mainly wrong, he says. "Our view generally speaking is that what is happening is mass injustice. Indigent people and racial minorities are being rounded up in huge numbers. I mean we [the US] incarcerate a higher percentage of our population than pretty much any country in the world."

But for a long time De La Pava didn't see it as his role to try to communicate this perspective beyond the circle of his colleagues. More recently, inspired by Michelle Alexander's book The New Jim Crow, he has chosen to speak out. His next novel, which he aims to finish this year, is about prisons. Writing A Naked Singularity, the tension between his artistic and social aims was "terrifying and thrilling. How didactic can you get while still hopefully having it function as a work of art? I think I was keenly aware of that tension the whole time. It's there on every page."

The tension suits him, and he wouldn't give up the day job and legal activism that goes with it even were he able to, feeling a sense of obligation to go on using his legal training "in a way that's useful to actual human beings", as well as uneasy about the prospect of writing full-time.

He thinks most American fiction these days is shaped by postgraduate courses in creative writing, and knows his own background sets him apart. The blurb for Personae make a joke of his apartness from the literary smart set: "Sergio De La Pava still does not live in Brooklyn."

The morning after I meet him, De La Pava will be back in court. "Whether I feel up to it or not I will have to get my head clear and start thinking about some legal things that are happening, and that will slap me back into reality and get me out of my own head, which is a scary place sometimes," he says.

• A Naked Singularity is published in paperback by MacLehose Press on 3 July. Personae is published by University of Chicago Press.