On a sultry evening in Manhattan last month, Joseph O'Neill was to be found in the cool downtown atmosphere of the independent SoHo bookstore McNally Jackson. The occasion was a party thrown for the store's renaming (it had previously been called McNally Robinson), a champagne affair with a guest list and a doorman. A slew of hip young writers was present, but O'Neill, a trim, saturnine figure with the welcoming smile of someone familiar with anonymity, was the main attraction. In T-shirt, shorts and untied pumps, he looked like that most uncommon of urban creatures: a relaxed author. In front of him was a queue of readers - mostly young women - eager to gain his signature on their copies of his book, and all around him was the pleasing and unmistakable buzz of success.

This is the celebrity version of the literary life: the fashionable event, the limelight, the approbation. But O'Neill, an Irishman of Turkish descent who grew up in Holland and went to university in England, is far more intimate with the other version: the long days and dark nights that form the basis of the creative process, and the quiet, frustrating obscurity of the unread author. His two previous novels, and a memoir, had slipped almost unnoticed into the bottomless void in which nearly all books sooner or later - and mostly sooner - come to reside.

'I'd never met a disinterested third party who'd read my work,' he told me. But here, as at various readings he had attended in recent weeks, there was one big party of unknown readers lining up to tell him how much they loved his book. New York magazine had already declared O'Neill 'King of New York', and he was to be seen on video screens in the back of yellow cabs being interviewed about the cause of all the fanfare, his third novel, Netherland, the current favourite to win the Man Booker prize.

Courtesy of his education at an international school in the Hague and Cambridge University (where he read law), O'Neill sounds like an impeccable Englishman. As if to complete the effect, he also worked until 1998 as a barrister in that most English of establishments, the Temple. Now 44, he has been living in New York for the past decade. Seven of those years he spent working on Netherland, ostensibly a slim novel concerning the small world of New York cricket. When eventually the book was finished, he was turned down by a number of agents and then, having secured an agent, he was rejected by every major New York publisher, except for one, Pantheon books. The imprint is part of the Random House empire overseen by Sonny Mehta, the great panjandrum of New York publishing, and it was O'Neill's good fortune that Mehta, a keen cricket fan, read the book and wrote a personal recommendation to booksellers.

In any case, Netherland was greeted with extravagant praise by some of America's most influential critics. Writing in the New Yorker, James Wood called it 'one of the most remarkable post-colonial books I have ever read'. The New York Times hailed it as 'the wittiest, angriest, most exacting and most desolate work of fiction we've yet had about life in New York and London after the World Trade Center fell.'

These were the kind of notices authors spend their bathtimes fantasising about. In Britain, critical acclaim tends to have a limited impact on sales, unless it comes from Richard and Judy. But in America, when Michiko Kakutani, of the New York Times, and James Wood rave about a book, it often provides a helpful nudge towards the bestseller list. 'I knew that I was at the tipping point,' O'Neill says. 'I knew that strangers would buy my book. That for me was a huge breakthrough. Members of your extended family are not the only ones who will read it.'

In many respects, Netherland is an unlikely hit. First of all, any novel that describes the beauty of the square cut and cover drive does not scream out 'read me' to an American audience. As one of the characters says in the novel, 'There's a limit to what Americans understand. The limit is cricket.' Then there is the plot, which centres around a Dutch banker named Hans, his abandonment by his English wife Rachel, and his friendship with a shady Trinidadian entrepreneur called Chuck Ramkissoon. Nothing much happens in the story. The big event - Chuck's death, or rather, murder - takes place at the beginning, and thereafter the narrative meanders through Hans's troubled thoughts and his outsider's descriptions of New York.

Yet it is a thoroughly absorbing book, not least because it grapples with the dislocated reality of 21st-century urban life, steeped in comfort and beset by fear, in which the internationally wealthy remain largely ignorant of, or indifferent to, the immigrant poor. As Hans ventures into Chuck's netherworld in the outer boroughs of New York, and learns of his scheme to introduce professional cricket to America, the Dutchman begins to see people and situations about which he previously knew nothing. Inevitably, Hans's attempt to grasp common bonds amid his emotional desolation is resonant of the wider search for community and identity that marked the anxious aftermath of the attacks on the World Trade Center. Hence, on top of its literary merits, the novel has been lauded for capturing the temper of the post-9/11 times. O'Neill acknowledges that his book arrived at just the right moment. 'It may be in tune with the zeitgeist,' he says. 'But you don't know that seven years before. One year earlier or later? If this book comes out after Obama, probably nobody cares.'

O'Neill always wanted to be a writer. Such was his love of books that he decided not to read English at university, because literature was 'too precious to study - you want it to remain a hobby'. By the same token, he became a barrister, though not before taking a year off to write his first novel, This is the Life, about the travails of a young lawyer. 'Novel-writing is a bit like deception,' O'Neill has said. 'You lie as little as you possibly can. That's the way I do it, anyway.'

In Netherland, Hans and his wife move into the Chelsea Hotel, the legendary bohemian refuge on West 23rd street, when they have to evacuate their Tribeca apartment after 9/11. With its cramped space and community of exiles and eccentrics, the hotel acts as a kind of idealised microcosm of Manhattan, an island unto itself. In reality, O'Neill also lives in the hotel with his wife, Sally Singer, a talented writer who is a senior editor at American Vogue, and their three young boys. The lobby, hall and staircase of the Chelsea is decorated with work by artists who live, or have lived, in the hotel. On the door of O'Neill's three-bedroom suite is an Obama sticker. Inside on the wall is a silkscreen multiple image of Sid Vicious, a former resident of the hotel who killed his girlfriend Nancy Spungen in a room downstairs.

O'Neill met Singer back in 1991 when she was his American editor at Farrar, Straus and Giroux. She turned down his second novel, though that did not prevent them from setting up home in Soho in London. Singer worked for the London Review of Books and then British Vogue, but she was tempted back to the States in 1998 by an offer from American Elle. In Manhattan's heated medialand she is a name to be reckoned with. It can't have been easy, being her seemingly non-productive other half.

'Joe was never really interested in any of that stuff,' says Singer, a sharp-looking woman whose fascination with fashion in no way disguises her formidable intellect. 'He never wanted to go to book parties. In fact, he much preferred to hang out with his cricket friends.'

Even so, O'Neill discovered that people in New York were more interested in his work than they had been in London. 'I thought it was normal if you're a writer to get absolutely zero encouragement from anyone you met. I thought it was normal for even your friends to express no real interest in what you were doing and to take the piss as and when the opportunity arose. Here, much to my amazement, people would say, "Oh you're a writer, you must meet so and so. Come round for dinner." Whereas if you meet someone in London, and you say come round for dinner, they think you're weird. "Don't you have friends of your own?" "Who's this loser who wants us to come round for dinner? Why can't he have dinner with his own pals, like everyone else?"'

Notwithstanding the dinner invitations, he was initially homesick for London, his friends and family. He had enjoyed being a barrister and drew a great deal of professional satisfaction from the job. Although he continued to work for a year or so on 'run-off' cases, a crunch point was waiting for him. Originally, O'Neill and Singer had planned to stay for two years and then return to London. But as that deadline loomed, he was faced with the choice of going back to the law or becoming a full-time writer. If he returned to the bar, he felt that he would have to give up writing because, as a parent, he wouldn't have time for the law, his kids and literature, too.

'It was really hard. Everyone I spoke to said, "That's a tough decision." Then my wife said: "Why don't you go and see the shrink," who's an expert on the question of the crossroads. So I went to see this guy, who knows all about life's forks, and I said, "I have this really difficult problem. On the one hand, there's the bar, years of experience, my income and my family, maybe I should support my family. Then there's the writing..."

'He said, "Yes I see that is a difficult problem. Tell me, what do you want to do?"

'I said, "Well, of course I want to be a writer." And he said: "Well, there you are."

'Nobody had authorised me, because I was, and I am, a sort of conventional soul. I felt it was self-indulgent, but as he well understood - it was a one-session problem - the price you pay for not doing what you want to do is incalculable.'

The resolution of one problem, however, opened up another: the business of writing a novel. O'Neill's first two novels, This is the Life and The Breezes, were suffused with intelligence and good writing, but essentially comic in tone and limited in scope. He knew it was time to stretch himself.

'My ambition right at the beginning was to write the kind of book that I like to read, which is a "voice". You don't really care what the preoccupations are, you just want to be with this voice. I remember talking to my friend Albert Weinstein [a fellow Chelsea Hotel inhabitant], who's now dead, and he said that's a really hard thing to write.'

James Wood suggested O'Neill should have cried 'Eureka!' when he had the idea of writing about cricket. 'It didn't seem like that,' the Irishman recalls. 'I had no other ideas. It's not like I had War and Peace up my sleeve. I just had another tacky novel about blokes as my fallback novel, and no one would actually want to read or buy that. And there was no point in delivering a novel preparatory to something else. So I went for broke, and that's in fact the only way to write.'

After five years, though, it was the author who was broke, not financially so much as creatively. In a log cabin in the wilds of Canada, where he had gone to focus on writing, he realised he couldn't write the book he wanted to write. He phoned Singer and told her that he was giving up.

'I was stuck in Canada and my plane ticket didn't take me back for another couple of days, so I read for a day and I read a book that really helped me called Housekeeping, by Marilynne Robinson. She is the sort of person who spends 20 years writing a novel. It was so slow. Nothing really happened and it was so attentive just to sentences. And I suddenly thought, why don't I write exactly what I want to write and to hell with the plot points.'

He junked the second half of the book and started again from scratch. The result is a gorgeous, ruminative prose in which every sentence feels written, not typed. Comparisons have been made to F Scott Fitzgerald's masterpiece The Great Gatsby, and the poignant echo of that book can be clearly heard in a number of passages. The elegy is not commonplace in the modern English novel. There are examples, like Alan Hollinghurst's exquisite The Line of Beauty, which is also reminiscent of Gatsby, but on the whole it's an American form, inextricably tied to what Mehta, in relation to Netherland, called 'the compromised beauty of the American dream'.

'I think Gatsby's mourning something much broader than the American dream,' says O'Neill. 'That last sentence ["So we beat on, boats against the current, borne back ceaselessly into the past"], he's thinking more about the tragedy of time.'

By contrast, it could be said that in Netherland the tragedy is more concerned with space, or dislocation. Time washes back and forth like the tide, but the real struggle is for a sense of place. Chuck's dream is to build a cricket ground where no one wants it. Hans's cricket team have to fight for their small slice of park land against competing and more popular sports. Hidden in the suburbs are the forgotten foundations of New York's history. Hans spends his sleepless nights looking at Google Earth images of the house in London to which his wife and son have moved. He lives in the suspended reality of the Chelsea Hotel, a kind of urban Neverland. And Ground Zero is the empty hole that fills the city's nightmares.

Netherland: rarely has a one-word title worked on so many levels. Indeed it's so perfect that I wondered if he had it right from the start or whether he had a 'Trimalchio moment' - Fitzgerald at one time had planned to call The Great Gatsby Trimalchio in West Egg.

'I certainly did have a Trimalchio moment,' O'Neill confesses. 'For years the book was called The Brooklyn Dream Game, till my friend, the poet Paul Muldoon, raised a friendly eyebrow... and I thought I had a terrific title. Netherland was suggested by Sally.'

As a teenager, O'Neill played cricket for the Netherlands' under-19 side. He kept up the sport in England and, when he came to New York, he set about looking for a game. Eventually he called up the Staten Island cricket club. 'Joe,' asked the club official, 'are you a white man?' 'Yes,' O'Neill replied. 'Well,' said the official, 'you better pack a helmet.'

As with Hans, O'Neill's teammates were largely Asian and West Indian. Though it was a 'recreational decision' to take up cricket, he soon realised that he'd stumbled across a promising milieu for a novel. 'But I knew that I would have to spend another year or two penetrating the closed world of New York cricket.'

Did he think he'd also located a suitable fictional arena in which to investigate the 'post-colonial' tensions to which Wood had referred?

'I think you sense the metaphorical resonance of what you're writing without analysing it too carefully,' says O'Neill. 'That leads you down dead ends. You stop imagining things and start writing towards these themes. I think if you're writing about cricket you're obviously writing about power, because cricket is such a loaded sport, much more so than soccer. And in this country it's a sport of powerlessness.'

One of O'Neill's fellow cricketers at the Staten Island cricket club is Habib Rehman, a 43-year-old taxi driver originally from Pakistan. 'He's my brother,' Habib told me outside the Chelsea Hotel. 'I call myself half-Irish and he calls himself half-Pakistani.' Habib hadn't read Netherland. He said he was waiting for an Urdu translation.

Habib took us out to Brooklyn in his cab to see some of the places O'Neill describes in his book. It was a Friday and he had come straight from the mosque. 'I love Brooklyn,' he told me. He reeled off the street names and districts as if they were magical legends - Flatbush, Prospect Park, Brooklyn Heights - and sung the praises of the rich mix of people on the streets. 'Here,' he said, pointing out one parade of shops, 'they're West Indian. Those women,' he said of an approaching group of hijabis, 'they're Punjabi, like me. And those there,' gesturing towards some women in brightly coloured saris, 'they're from Bangladesh.'

He took obvious pride in the multicultural make-up of his adopted home in a way, it struck me, that is unusual in Britain. In one sense, he occupied the margins of American life, working the nightshift in a taxi, playing cricket beyond the boundaries of fashionable Manhattan, and yet he seemed enthusiastic about his life prospects. Chuck Ramkissoon, too, is a man who knows that he is estranged from conventional American success and yet he retains an engaging, if perhaps misplaced, optimism about the way forward.

O'Neill showed me a large wooden house in the classic American suburban mould in one of the more leafy districts of Brooklyn. It was an attractive home, full of space and light and horse chestnut trees. O'Neill and his family had lived here for a while during a hiatus away from the Chelsea Hotel. But they sold up and moved back to the hotel because Singer felt cut off from the action of Manhattan.

Perhaps it's too melodramatic to call a change of neighbourhood a crisis of identity, but if Netherland has a point it's that our surroundings shape who we are and how we feel. And the more detached we are from our environment, the more disengaged we become from ourselves. For Hans, this realisation hits home during a bizarre date with a bewitching young woman who wants to be thrashed. He glimpses a reflection of himself wielding his belt and suddenly he feels a sadness 'produced when the mirroring world no longer offers a surface in which one may recognise one's true likeness'.

Later, he's taken to the graveyard of the 18th-century Dutch reform church in Brooklyn. Chuck expects Hans to experience some kind of tribal connection among his dead countrymen. But he feels nothing. Looking at the headstones, with names like Jansen, van Dam and de Jong, Hans asks: 'What was one supposed to do with this information?'

It's an interesting question because O'Neill knows how ghosts from the past can weigh on the living. In Blood-Dark Track, the memoir he published shortly before 9/11, he tells the story of his Turkish grandfather who was imprisoned by the British during the Second World War, and that of his Irish grandfather who was jailed at the same time for being a member of the IRA. Among other things, it's a fascinating exploration of familial and national identity. Towards the end of the book, during an argument with his Republican uncle, O'Neill becomes 'furious' that his patriotism and Irish nationality is brought into question.

Nowadays, he seems to have come to terms with his rootless status. Indeed, he recognises its advantages. 'You don't have a functioning substantial identity as a writer,' he says, as we follow in Hans's footsteps in the graveyard of the Dutch church. 'You have a notional identity... I used to be quite exercised by nationality, but really I was an early member of the global flotsam. And if you stop thinking in terms of countries, you're left with cities.'

If he thinks of another city, it is London, which he describes as a 'fantastic place'. But he also harbours reservations about the narrowness of vision it can impose. Netherland makes a number of references to the temporal currents that Fitzgerald writes of in Gatsby, so it is surely worth noting when Hans observes: 'Londoners remain in the business of rowing their boats gently down the stream.'

'It's a nihilistic thing,' says O'Neill, when I ask about this sentence. 'It's about shrinking the significance of their achievements. People find satisfaction in shrinking their lives. It's an English recipe for living. Whereas here [in New York] there is in the air an almost inexhaustible sense of possibility.'

Perhaps it says something about the endlessly expanding nature of American horizons that just recently O'Neill has begun to think Chuck's doomed scheme to popularise cricket in the States, which represents futility in the novel, may yet come to fruition in the real world. 'I mean,' says O'Neill, 'it would surprise me still, but it's now within the bounds of imagination.'

Americans falling for cricket? That sounds a little far-fetched to my ears. Almost as incredible, in fact, as a novel about cricket becoming an American bestseller.