The Irish writer Eimear McBride has won the Baileys book prize for her first novel, A Girl is a Half-formed Thing, beating bookie's favourite Donna Tartt's gigantic third novel, The Goldfinch.

Although Tartt was tipped from the start for her 771-pager, it was McBride's dark family tale, written in the hurtling, comma-free voice of an unnamed narrator, that won over the judges of the women's prize for fiction, formerly the Orange prize.

The chair of the judging panel, Helen Fraser, a former managing director of Penguin Books, said: "This has been a fantastic year for women's fiction, as the quality of both the long and shortlist made clear, and I think what has emerged as the worthy winner is a really original new voice."

McBride was hailed as "that old-fashioned thing, a genius" by fellow Irish novelist Anne Enright. Her story of a girl's life in the shadow of sexual abuse and the brain tumour of a beloved brother took six months to write and many years to get published.

McBride had accumulated a hefty pile of rejection slips, and the manuscript had gone into the back of a drawer, when a conversation between her theatre director husband and a bookshop owner in their adopted home city, Norwich, led to its becoming one of the launch titles of the small independent press Galley Beggar.

By the time Faber & Faber took up the paperback rights, A Girl was already scooping up rave reviews and awards, including the €15,000 (£12,000) Irish novel of the year award and the first Goldsmiths prize for "boldly original fiction". It is shortlisted for the Folio prize.

Fraser said several publishers would be kicking themselves. "It is an amazing book. Publishers do make mistakes, and I can only imagine that some may have read the first two or three pages and thought, 'No, this is too difficult.' But once you get beyond that, we found it utterly engaging, readable, unputdownable. The style is no obstacle."

She said the final judging session quickly came down to two books, the second of which she discreetly declined to name.

"Very early on Eimear stood out from the crowd. We all put ourselves into purdah to reread the shortlisted books but it was only when we started cautiously exchanging emails in the past week that we realised what a strong contender it was. It took us one hour to get the shortlist down to two books, and the remaining three hours to decide between them – but this is a truly worthy winner."

It was perhaps a hint to the hall of what was coming that while all the other authors turned up, Tartt was represented at the ceremony by her UK agent.

Snuffling a little if not quite weeping, McBride said of her triumph: "I hope it will serve as an incentive to publishers everywhere to take a look at difficult books and think again. We are all writers but we are all readers first. There is a contract between publishers and readers which must be honoured, readers can not be underestimated."

Sam Jordison, co-founder of the tiny Galley Beggar press (and of the wildly successful Crap Towns books), was elated. "It's not an easy read but it is an incredible one. I find it hard to believe so many other publishers didn't see that."

McBride was a popular winner: there were whoops and cheers when Fraser announced that hers was the book that got the judges "talking and thinking and arguing".

At the traditional eve-of-prize public reading by the authors, Fraser was struck by the audience response to McBride. "There was a great roar when she stood up, and it was immediately apparent how many people, men and women, had read the book and really loved it. She will be an immensely popular winner."

McBride is already well into her next book, and unlikely to have any problem at all getting it published. "I am quite sure this is the beginning of a major career," Fraser said.

Born in Liverpool in 1976, McBride was brought up in the west of Ireland, trained as an actor, and now lives in Norwich with her daughter and husband, William Galinksy, who had the fortuitous conversation with Henry Layte, owner of the Book Hive shop and co-founder of Galley Beggar.

McBride's was one of two Irish first novels to make the shortlist, with Audrey Magee's second world war story, The Undertaking, the other. Several big names fell out at the longlist stage, including the veteran Canadian author Margaret Atwood. Tartt, who seemed to lead the pack from the start, won the then Orange prize in 2003 for her second novel, The Little Friend, and has the considerable consolation prize for her latest that it has just won the Pulitzer prize.

Although A Girl is a Half-formed Thing won almost unanimous praise from literary critics, it has divided readers, and has the lowest Amazon star rating of any of the shortlisted books. In a comma-free snarl, one reader wrote: "Bad bad bad book."