Colm Tóibín is one of the most highly regarded Irish writers of his generation, loved by his readers and admired by his peers, but when it comes to major book prizes he is something of a bridesmaid. He so often nearly wins them but doesn't – until, that is, tonight when he was named winner of the Costa novel of the year award.

It was an achievement all the more notable because Tóibín was up against the literary sensation of last year: Hilary Mantel's Booker prize-winning tale of Tudor intrigue, Wolf Hall. "It's just great but I'm very surprised," said Tóibín. "Wolf Hall was a wonderful book."

Brooklyn, a sparely written account of a young woman's emigration from 1950s Ireland to New York, was one of five category winners announced tonight which will now compete for the overall Costa book prize.

Other winners were Christopher Reid in the poetry category for A Scattering; Graham Farmelo in the biography section for his account of the life of quantum physicist Paul Dirac; Patrick Ness, the children's book award for The Ask and the Answer; and Raphael Selbourne, the first novel award for Beauty.

Tóibín was on the Booker longlist but to widespread surprise was not shortlisted. Previously, The Master was just pipped to the Booker by The Line of Beauty in 2004 while The Blackwater Lightship was shortlisted in 1999, the year JM Coetzee won for Disgrace. Tóibín also has form in the Costas – or Whitbreads as they were formerly known – with a shortlisting in 1990 for his first novel, The South.

Tóibín said he was delighted to win and that book awards did matter. "It does make a great difference to what publishers call sales and what I call readers." He described Brooklyn as quite low key, about somebody very ordinary and not a book that would be considered as an automatic prize winner.

Brooklyn was straight away installed by William Hill as 6-4 favourite for the overall prize – odds too short for Tóibín.

"I won quite a lot of money when I bet on Hilary Mantel when she was 12-1 for the Booker. I don't think I'll be betting on me," he said.

The novelist made headlines last year when he suggested in an interview that he did not really enjoy writing and the best thing about it was the money. There was, though, probably a twinkle in his eye. Asked yesterday if he enjoyed writing he said: "Look I'm working at the moment and it has been a great Christmas and everybody has been out drinking and I've been locked in here since December 27 with these characters and sentences trying to get out. So, no. I want to finish this book."

Second favourite, at 3-1, for the overall prize is a debut biography by Farmelo – a five-year labour of love telling the story of one of the least-known yet most important scientists of the last century. "It is absolutely appalling that most people in this country have not heard of the name Paul Dirac," said Farmelo. "He was the greatest scientist Britain produced in the 20th century."

Part of the reason for the lack of knowledge is that Dirac loathed publicity. Farmelo gained access to a previously unmined family archive in Florida hopes the book will bring the genius of Dirac – and his importance to science – to a wider public. "He was a publicly educated boy from a Bristol terrace and what he achieved was immense. He conceived half the universe in his head, he conceived antimatter."

The winner of the first novel award, Selbourne's Beauty, is the story of a young Bangladeshi woman on the run from her family. "I'm very pleased. It's great to be recognised and of course it's invaluable in terms of getting the book out there and being seen by people."

Like most novelists Selbourne has had a range of jobs over the years, including as a scooter salesman, but all have been attempts to get out of teaching, he said. It was while teaching in Italy that Selbourne made the arguably unusual decision to move to Wolverhampton, attracted by a job teaching unemployed adults with basic skills needs.

The city, he said, gets a bad press. "The people for one thing are incredibly friendly. I'm from Oxford so I'm a southerner used to unfriendly people. I find Wolverhampton very stimulating."

For Reid, winning the poetry award is third time lucky – he has been nominated twice before. His winning collection is a tribute to his wife, who died in 2005.

The 6-1 outsider for the overall award is Ness for the second book in his trilogy Chaos Walking, which the judges called "a major achievement in the making."

A judging panel chaired by novelist Josephine Hart – including Marie Helvin, Caroline Quentin, Dervla Kirwan, Gary Kemp and Tom Bradby – will now decide the overall winner from the all-male list and announce it on 26 January.