The guy at the checkout at a Brooklyn branch of Trader Joe's, a wildly popular speciality grocery chain, had a spray of blond hair, a masters in literature and a California drawl on the words: "Hey, man." He was packing the soy chorizo into a brown bag when he glanced at the book I was carrying – a copy of Brooklyn Is, a 1939 essay by the author and journalist James Agee – and asked if I'd read it. I shook my head. "Man, it's amazing," he said. "Everyone in Brooklyn should read it."

Reading Agee's essay later that day, I kept thinking of that young man and what Agee would have made of him, and how Agee's odyssey of Brooklyn might look if he were to undertake the assignment today. Trader Joe's itself stands in a section of Atlantic Avenue that Agee described as "vacant lots, the ghosts of floors against their walls: and the dark hard bars at street corners". Few of those dark hard bars exist now, but a few blocks west of Trader Joe's stands Montero's, which opened in 1945. Its neon sign and nautical paraphernalia embody something of the era Agee was writing in, but the bar no longer opens at 8am to serve longshoremen coming off the midnight shift. Artists and writers (in search of "authenticity") have replaced the Baltic seafarers. The same is true of Sunny's, a 120-year old bar in Red Hook – a waterfront neighbourhood of warehouses and old single-row homes that served as the inspiration for On the Waterfront.

Today Sunny's is popular for bluegrass sessions and literary salons that attract aficionados from across the borough. There is not a night of the week when you can't attend a reading in Brooklyn, or several. Many take place at the independent bookstores that have proliferated in the last few years, or – like BookCourt in Cobble Hill, where I remember waiting in a long line of young tattooed men and women to hear Bret Easton Ellis read – doubled in size. And writers aren't just coming here to read; they are flocking here to live. Some, such as Paul Auster, have been here for decades; others, like Martin Amis (a stone's throw from BookCourt), are fresh off the boat. On Saturdays you can go Pulitzer spotting at Fort Greene's farmers' market, where both Jhumpa Lahiri and Jennifer Egan may be found perusing the vegetables. When Jonathan Safran Foer and his wife Nicole Krauss, author of The History of Love, brought a Park Slope townhouse in 2005, bloggers gasped at the $3.5m [£2.26m] price tag.

These are just the boldface names. For every Safran Foer and Lahiri there are a hundred more toiling away in relative anonymity, supplementing their incomes by teaching or securing the occasional magazine commission. The association between Brooklyn and writing has become so axiomatic that when Sergio De La Pava published his debut novel A Naked Singularity in May, he quipped in his author bio: "Sergio De La Pava is a writer who does not live in Brooklyn." Native son Jonathan Lethem, who upped and left for California last year, was more pointed, grumbling to the Los Angeles Times: "Brooklyn is repulsive with novelists; it's cancerous with novelists." Elsewhere he complained that the borough he so tenderly limned in his novel Motherless Brooklyn had "been made blander, a little more accessible, and it's taken over the world".

You could ascribe this shift to the rampant gentrification that has swept across much of Brooklyn, or at least the part of it that lies closest to Manhattan (a recent survey indicated that Brooklyn contained four of the nation's top 25 most rapidly gentrifying neighbourhoods). The response to it has been conflicted. On the one hand no one wants to return to the 1980s, when a neighbourhood such as Red Hook, now home to Ikea, was dubbed by Life magazine the crack capital of America. On the other hand, rising rents and property prices change the character. If he were able to return, Agee would still find Syrians along Atlantic Avenue and Sicilian-speaking Italians in Carroll Gardens. The Polish atmosphere of Greenpoint persists, as do the descendants of the "grizzling skull-capped Jew" Agee encountered on DeKalb Avenue. But here's what else he would find: spiralling rents; waiting-list only restaurants specialising in complex pork dishes; immaculately rendered bars posing as prohibition-era watering holes; and scores of cafés crowded all day long with young men and women staring intently at their laptops while the expensive cappuccino makers hiss and sigh.

Of course writers can and do live anywhere, but the fact that Brooklyn has become, in the words of the New York Observer, "a zone of infestation, not only of novelists but reporters, pundits, poets, and those often closeted scribblers who call themselves editors and agents", is a peculiarity it shares with no other major metropolitan area. When Martin Amis moved to Cobble Hill last year it was widely viewed by the press as an official imprimatur on Brooklyn's status as the writing factory of America. Not that Brooklyn was ever short of writers – Walt Whitman used to edit the Brooklyn Eagle, and Norman Mailer held court in Brooklyn Heights for much of his life, alongside Truman Capote – but the phenomenon is now so pronounced that you could say, without exaggeration, that there are two principal avenues for would-be writers in America. The first is to swallow the exorbitant price tag for one of the country's multiplying creative-writing courses (usually Masters of Fine Arts, or MFAs); the second is to move to Brooklyn.

Chad Harbach did both and then wrote a witty and thoughtful essay about it. "MFA vs NYC" was published in 2010 in n+1, the earnest and erudite literary journal that he co-founded – itself a cornerstone of Brooklyn's literary scene. The essay's premise was provocative: MFA courses have devolved into little more than recruitment centres for the next generation of MFA writing instructors. If you want to write novels, you move to Brooklyn – more specifically "a small area of west-central Brooklyn bounded by Dumbo [short for Down Under the Manhattan Bridge Overpass] and Prospect Heights". True, most books are "released to a shudder of silence, if at all," wrote Harbach, but there are those that defy the odds, mainly by accepting that a successful literary novel, like a pop song, needs a hook, comprehensive prose and plots "tied neat as a bow". It also helps, he suggested, to write them long.

That was in 2010. Within a year Harbach became the poster boy for his own semi-satirical thesis, scoring a staggering advance of $650,000 [£419,000] for his debut novel The Art of Fielding, which clocked in at a hefty 512 pages. It made the New York Times bestseller list and was picked up by ace producer Scott Rudin for HBO, the acme of quality television. The reviews were uniformly handsome. The Art of Fielding was "triumphant" (Wall Street Journal), "unusually charming" (New Yorker) and one of the 10 best books of 2011 (New York Times). Bloomberg News simply ran with the numbers: "Unemployed Harvard Man Auctions Baseball Novel for $650,000". As with Jonathan Franzen's The Corrections, this lavish success also served to ameliorate the suspicion that percolates through those same publications that the literary bestseller is all but dead.

Harbach now lives in Virginia, having decamped from Brooklyn last year, though he returns to New York frequently. On a bright, perfectly dry spring morning he could be found in Brooklyn's Prospect Park, where he and a group of friends meet regularly on Saturdays to play touch football – essentially American football minus the tackling. With Prospect Park awash with joggers and dog walkers, Frisbee throwers and sunbathers, he stakes out a patch of space on the pastoral sweep of the Long Meadow and divides the players into teams. Among those playing are three editors from New York magazine, another from the culture and politics website Slate, a literary agent, a copy editor, the editor of the Queens Ledger/Brooklyn Star and an editor for Picador Books. "It's a little embarrassingly literary at times," acknowledges James Morris, an art director for the CBS News website who lives nearby.

It turns out to be a slow game. There are so many players they have to keep switching in and out to ensure everyone gets to play. In between turns they sit on the rise of a bank beneath the unfurling oaks and beech trees. There's a consensus that Harbach's success is a great thing – for him but also for them. The typical elements of his experience – the years of revising, the near penury (his friend Keith Gessen recalled the phone in their shared Brooklyn apartment going unanswered to avoid collection agencies), the countless rejections – makes his atypical success all the sweeter. It reinforces a collective faith that this is a place that nourishes its writers.

"I identify with it – it's almost as if it was me," said Gessen, a Russian émigré with a slick of thick black hair whose own novel, All the Sad Young Literary Men, was published in 2008 to considerably less fanfare; Gessen also wrote a 10,000 word essay for Vanity Fair on the writing and publication of The Art of Fielding that later became an e-book. Ben Mathis-Lilley, an editor at New York magazine who moved from Michigan, said: "Coming from a place where it was only a few weird kids from high school who were into books, I love the fact that every third house contains a writer, stereotype though it might be."

This is a theme that surfaces again and again. "There is an atmosphere here where if you say you've written a book people will say: 'Congratulations'," Rosie Dastgir, who moved to Brooklyn from London in 2005, told me one warm afternoon as we sat in her garden against a mellifluous backdrop of birdsong. A swing hung from a crabapple tree, roses sprawled along a garden fence. It was hard to believe that Manhattan was a few subway stops away. Dastgir recalled feeling very differently during her first few months in Brooklyn, but a lot of walking – "for miles and miles and miles" – changed that. Like James Agee, she began to develop a broader picture of the borough's diversity. "I remember discovering Fifth Avenue in Brooklyn, walking the whole length, and I'd take a bus out to the far reach of Coney Island Avenue and do lot of walking there, in the Pakistani area," she says. Often she'd spot other writers – "Paul Auster in the Second Street Café, or Siri Hustvedt in the Café Moutarde" – and that was a thrill, too. "Jhumpa Lahiri – I'm a big fan of hers. I've seen her pushing her buggy in Fort Greene," she says. She and her husband, a journalist for the Financial Times, were introduced to the novelist Amitav Ghosh and his wife, the biographer Deborah Baker, who in turn helped Dastgir find an agent. Her debut novel A Small Fortune, about Pakistanis in England, was published in the UK in February and has just been published in the US to good reviews.

"My head was down and I was working, and when I lifted it all of a sudden it seemed a new requirement for writers to move to Brooklyn," says Amanda Stern, a novelist who has lived for the past 10 years in Fort Greene, a fast-gentrifying grid of 19th-century brownstones bordering a park that might be considered the heart of Brooklyn's writing factory. Stern had suggested we meet at Bittersweet, one of those artfully bohemian cafés with a coffee menu – Cortado, Misto, Quad Americano – that requires a degree in romance languages. Stern is perched at the end of a long wooden bar, where she is able to bypass the queue and ask passing baristas for free refills, in return for which she provides writing lessons to the staff.

She is bright, eager and immediately likable. A fifth-generation Manhattanite who came to Brooklyn on the heels of her then-boyfriend, Stern has had a writing career that has followed a more typical trajectory than Harbach's. When I ask how much she has made from writing she thinks for a moment, and then replies: "Hang on, I have a $5 royalty check that I've never cashed – it's hanging on my wall." That paltry check (Stern later emails me a photo) was for her first novel, The Long Haul, published in 2005. "My advance was $600, which really allowed me the freedom to quit all my other jobs and live the life of luxury I'd always desired," she says. "In some New York stores it might buy me two vests." (This is something of a trope of the Brooklyn writing factory – almost no one makes anything from writing. The founders of n+1 had pooled $8,000 in order to publish the first issue – money they didn't earn back for another six months. "We thought we were business geniuses until we realised we'd worked for six months with no pay to make our $8,000 back," says Gessen.)

Stern does a little better from writing children's fiction, but like Dastgir, like Harbach and his squad of football-playing wordsmiths, much of what sustains her is proximity to other writers. Eight years ago, Stern organised the first of what would become a weekly series of literary cabaret shows, Happy Ending, at which writers would read from their work before undertaking a dare of their choice. This might be, as in the case of AM Homes, speed dating four members of the audience, or – to take the reckless contribution of Israeli author Etgar Keret – smoking an entire joint onstage. This kind of performance art might sound like the antithesis of the writer's natural inclination, but Stern has scheduled more than 200 Happy Endings without ever running short of writers willing to risk humiliation. Accidentally or by design she has become one of those all-important facilitators in the network of writers on both sides of the East River. "It's a shit ton of work for very, very, very little financial reward, but I'm sick in the head because I love it anyway," she says as we walk to the farmers' market in Fort Greene, running into various writer-friends of Stern's along the way. "This is actually a great place to run into writers," she says, picking through some Chinese broccoli. "I've bumped into Jhumpa Lahiri here, and Julie Orringer."

"There were points where I thought it would be a nightmare to be surrounded by other writers," admits Keith Gessen when we meet in Dumbo, near the office of n+1, the magazine he launched with Benjamin Kunkel, Mark Greif, Marco Roth and Harbach in 2004. Since then all five have published books (Roth's, a memoir called The Scientists, comes out this autumn) to varying degrees of success while continuing to put out their scrappy, pugnacious and unabashedly sincere journal. Their philosophical enemies include Dave Eggers, author of A Heartbreaking Work of Staggering Genius – accused, alongside his cohorts at n+1's rival cultural journal The Believer, of engaging in an orgy of postmodern infantilism and whimsy – and the New Yorker critic James Wood. On the other side stand Jonathan Franzen, Don DeLillo and Michel Houellebecq, writers Gessen describes as "spiritual travellers". "One thing we've said over and over is that we've felt like there is a backward-looking culture," Gessen explained. "We, too, look back, but we also think it is important to look forward, and to the side."

Unlike Gessen – the son of Russian intellectuals who took pleasure in reading together – Harbach grew up in Racine in Wisconsin, the kind of hardscrabble industrial town that did not place a high premium on the life of the mind. Although he was a voracious reader there was little opportunity to develop a boy's bookishness in Racine, and Harbach's attention shifted to baseball and basketball. In high school he held down a job as a janitor in a metalwork factory. "I was going to these Catholic schools where tuition wasn't very steep, but they still paid tuition, so it was just expected that when I reached working age, which in Wisconsin is 14, I would go and work."

Instead Harbach wound up in Massachusetts, at Harvard, where he met Gessen. After graduating, the two would mail stories to each other for feedback, creating a mutual support system that helped push them forward. Many writers find their way to Brooklyn for similar reasons. "What's amazing about Brooklyn is that there really is this harmony where people are helping each other out," Julia Fierro told me one Friday. We were sitting at Café Pedlar, a Cobble Hill coffee shop, and Fierro, who earned an MFA in creative writing at the prestigious Iowa Writers' Workshop (it has 17 Pulitzer-winning graduates), was explaining the genesis of her hugely successful Sackett Street Writers' Workshop. In 2002 she was teaching as an adjunct professor at Hofstra University on Long Island, making about $10,000 a year. Her first novel, Roseland, was with an agent and in the process of being submitted to – and rejected by – one publishing house after another.

"It was tough," she admits. "I'd gotten so much praise at Iowa that I think I expected to be published." Dispirited and dejected, she put an ad on Craig's List recruiting students for a writing workshop in her apartment. Within months she was adding more nights as the classes grew in size. Today it's a full-time business with a cadre of tutors and nightly workshops in a swathe of neighbourhoods across Brooklyn and Manhattan. "A lot of people ask how I started this, and I reply that you couldn't do it anywhere except Brooklyn because there are so many writers here," she says, before adding: "Now it's hard to find someone who will teach in Manhattan, because everyone lives in Brooklyn."

Like so many writers, Fierro was forced by circumstance to find other ways to earn money, but her industriousness, like Stern's literary cabaret, or the creation of n+1, has helped to nurture a more vibrant society of writers who feel enhanced, not diminished by proximity to one another. "One person's success does not mean your failure, so having a group of writers championing one another is rare but refreshing," Stern said as we meandered through the market at Fort Greene. "We're not all the same, and we're not all at the same place, and that is the key to having a writing community."

These days it's not only writers who are pouring into Brooklyn. It sometimes seems as if New York's centre of gravity is shifting inexorably eastwards from Manhattan, across the river, where there is more space and prices are generally less prohibitive. In September, after years of controversy and construction, Brooklyn will inaugurate a new stadium, the Barclays Center, with a concert by Jay-Z, part-owner of the New Jersey Nets bastketball team, whose move to Brooklyn – henceforth they will be known as the Brooklyn Nets – represents the borough's first major league team in any sport since in 1957.

An area that long represented stagnation and neglect is now experiencing a real- estate boom that may force many writers further afield. The change has been swift. Gessen had recalled the area as "desolate" when he threw the launch party for n+1 a mere eight years ago. That party was also a moment of crystallisation. "I remember having to get more alcohol and seeing a long line for the elevator outside," he said. "I couldn't believe the interest – right from the start it turned out there were a lot of people who were really excited by this." I asked if he ever thought of himself in relation to the writers who had found their muse in Brooklyn – Joseph Heller, Norman Mailer, Paul Auster. Gessen shook his head. "Nope, I never think of those people." He paused for a second, and reconsidered. "Well, Whitman, of course – he addresses writers of the future, saying something like: 'As you stand here now, so I stood…'" He frowned as he searched his memory. "That's not quite right," he said finally, "but it's good stuff."



The Art of Fielding by Chad Harbach is out now in paperback (Fourth Estate, £8.99); A Small Fortune by Rosie Dastgir is published by Quercus (£12.99)