Poet and painter Lawrence Ferlinghetti came to San Francisco in 1951 because he heard it was a great place to be a bohemian. He settled in the Italian working-class neighbourhood of North Beach with its cheap rents and European ambience. And before long he put the city on the world's counter-cultural map by publishing the work of Beat poets such as Allen Ginsberg and Jack Kerouac. But despite his status as world and local literary legend, the 94-year-old co-owner of the renowned City Lights bookshop and publishing house doesn't feel so at home in the City by the Bay anymore.

He complains of a "soulless group of people", a "new breed" of men and women too busy with iPhones to "be here" in the moment, and shiny new Mercedes-Benzs on his street. The major art galley in central San Francisco that has shown Ferlinghetti's work for two decades is closing because it can't afford the new rent. It, along with several other galleries, will make way for a cloud computing startup called MuleSoft said to have offered to triple the rent. "It is totally shocking to see Silicon Valley take over the city," says Ferlinghetti, who still rents in North Beach. "San Francisco is radically changing and we don't know where it is going to end up."

Until recently, San Francisco, California – a small city of around 825,000 poised on the tip of a peninsular on America's western edge that sprang up during the 1840s gold rush – wasn't thought of as a centre for business. Rather, it was famed as an artistic, bohemian place with a history of flowering counter-cultures that spilled over and changed America and the world, from the beats in North Beach to the hippies in the hilly region of Haight- Ashbury to the gay rights movement in the Castro neighbourhood. Jefferson Airplane's Paul Kantner called it "49 square miles surrounded by reality".

But times have changed in Ferlinghetti's city. San Francisco has become the hype- and capital-fuelled epicentre of America's technology industry, which has traditionally centred on the string of suburban cities known as Silicon Valley 40 miles to the south. In 2011, Mayor Ed Lee introduced tax breaks for Twitter and several other tech companies to encourage them to settle in and revitalise the downtown San Francisco neighbourhood South of Market, or Soma, and help the city climb out of the recession. Soma has become home to some of the most important companies in the new economy, such as Twitter and Dropbox, and many small startups hoping to challenge them. AngelList, a networking site for investors, now lists 5,249 tech startups in San Francisco, each worth $4.6m (£2.8m) on average and offering an average salary of $105,000 (£64,000).

Moving on: Zeph Fishlyn, who was forced out of her studio in the Mission District. Photograph: Barry J Holmes for the Observer

At the same time, San Francisco has become a bedroom city for people who work in Silicon Valley and prefer vibrant urban neighbourhoods to sleepy suburban towns. Facebook, Google, Apple and other companies lay on shiny luxury buses to ferry their employees on the approximately 90-minute trip. San Francisco's Municipal Transportation Authority estimates about 35,000 ride the air-conditioned, Wi-Fi-provisioned buses each day.

In one sense, San Francisco is thriving. The unemployment rate is just 4.8%, compared to 8.3% for California as a whole. In 2013 job growth in San Francisco County led all others in the nation. But the influx of so many young, rich tech workers has caused significant tensions. Starting in mid-2011, rents and house prices began to soar. Eviction rates soon followed as property speculators sought to cash in by flipping rent-controlled apartment buildings into flats to sell. Evicted residents have found themselves unable to afford to live in their city anymore and many businesses and non-profits have been squeezed. "There is only a handful of cities in the world that have such an extreme problem of gentrification," says Richard Walker, an urban geographer at the University of California, Berkeley.

The facts are stark. The median household income of the San Francisco Bay Area is now higher than anywhere else in America, and San Francisco has twice as many billionaires per capita as London (financial analysts PrivCo estimated that Twitter's stock market launch in November 2013 created more than 1,600 new millionaires in a single day, mostly employees). The median monthly rent is already the highest in the country and is still increasing at a rate three times the national average. Based on official figures from the San Francisco Rent Board, the San Francisco Tenants Union estimates that no-fault evictions displaced nearly 1,400 renters in 2013. About a third of those evictions were under California's Ellis Act, which allows landlords to evict tenants and sell their apartments. A City study from October 2013 says Ellis Act evictions increased by 170% from 2010 to 2013. There are also untold numbers who have left the area after accepting buyouts.

It isn't as if San Francisco hasn't seen a tech boom before. Silicon Valley's dotcom boom of 1998 to 2001 also led to significant displacement in San Francisco. But this latest one is focused on the city and visibly changing it faster. Many long-time San Francisco residents worry not only about being forced out of the city they love, but also that their city is being changed for the worse. Critics say that San Francisco's communities of alternative culture, ethnic or otherwise – the soil of its creative mojo and legendary social movements – are being turned into playgrounds for rich people. If San Francisco's soul is its social and economic diversity and status as a refuge for those outside the mainstream, then it is being lost.

Street life: Anthony Krumeich in his mobile home on the back of a truck. Photograph: Barry J Holmes for the Observer

Emerging in its place is the mostly white, male-dominated, monied monoculture of the tech industry and there appears no end in sight. "It is not like it is over, but the tide is going out on San Francisco," says Chris Carlsson, a resident since 1978 and co-founder of the Critical Mass bicycle activism movement, whose offshoots regularly take over the streets of London and other cities worldwide. Writer Rebecca Solnit, whose book Hollow City documented the effect of the first dotcom boom, fears the world is about to lose one of its most radical outposts. "I am not arguing for a city frozen in amber," she says, "but this particular iteration of change is eliminating a lot of what the city's identity has been for the past 150 years."

Artist Zeph Fishlyn, aged 47, came to San Francisco in 1988 and settled in the working-class Hispanic Mission District, drawn by the large lesbian community there. In late 2012 she and 16 other artists who were part of the Million Fishes Art Collective were kicked out of the studio space they had lived and worked in for almost a decade. Rents have soared in the Mission, which is conveniently located for the freeways to Silicon Valley and has become a fashionable place to live. A new landlord had bought the building and, citing non-compliance with zoning laws, kicked them out.

Unable to afford to stay in San Francisco, Fishlyn moved east across the bay to Oakland, where the burgeoning art and activism scene is buoyed by a steady flow of economic refugees. "Anybody spending their time doing something that doesn't come with a big pay cheque is having to move," says Fishlyn, "and that includes the creative sector and any kind of social justice work." So many creative types have relocated to Oakland that Oakland's mayor, Jean Quan, recently likened the city to Brooklyn (San Francisco was Manhattan); San Francisco-based street artist Eclair Bandersnatch, whose stencil portrait of Edward Snowden recently featured in the Guardian, says so many of her friends have moved away that she feels like an anomaly. The irony, she notes, is that "with money you get people who are more into the arts".

The Mission and Ferlinghetti's North Beach are "ground zero" for gentrification, says Ted Gullickson, director of the San Francisco Tenants Union. Others have already been subsumed including, he says, the Castro district, the world famous "gayborhood" synonymous with progressive hero Harvey Milk. The area was hit badly by evictions in the first dotcom boom, he says, and has been finished off by the latest tech surge. "It is more homeowner and much straighter, much whiter and much more conservative." In late 2012, the elected representative of the Castro introduced a measure to ban public nudity outside of festivals in San Francisco – despite, or perhaps because of, male nudity being commonplace in the Castro for decades.

Old school: Lawrence Ferlinghetti of City Lights book shop. Photograph: Barry J Holmes for the Observer

Gullickson believes the future of the city's character now rests on the Mission, North Beach and the wider Haight-Ashbury, iconic neighbourhoods which people associate with San Francisco. "I think ultimately if they become gentrified, we are talking about San Francisco as a whole."

There are others who see what is happening in San Francisco in a different light. Fred Turner, an American cultural history professor at Stanford University, argues that gentrification driven by white, middle-class newcomers to the city is nothing new, and has even underpinned its famous counter-culture movements. The arrival of the bohemians in North Beach began the displacement of the working-class Italians; the arrival of the hippies in Haight-Ashbury displaced some of the long-standing working-class residents; and the Castro had a large working-class Irish population before it became a gay mecca. The latest incarnation – digital workers displacing working-class Latinos and artists from the Mission District who themselves were already gentrifying it – is not radically different. "Nearly everything that is said about them – the taking of public resources, the pushing out of poor folks and different ethnic minorities – was said about the hippies of the 1960s and not without good reason," says Turner.

In his book From Counterculture to Cyberculture, he even argues that today's tech culture is a direct descendant of the hippy movement. The techies are far richer and aren't a counter-culture, but like some hippies they have the same sense of social mission to transform the world for the better with technology. Likewise the way that tech culture mixes work and play and emphasises personal growth has echoes of hippy life. "The same logic that was driving the counter-culture – and that continues to drive much of San Francisco today – is the very logic that drives Google," says Turner. "In a limited sense, the 1960s are turning around to bite San Francisco."

Stewart Brand, who personified the link between San Francisco's 60s flower power and the emerging technological hub of Silicon Valley, lives on a houseboat in Sausalito, just across the Golden Gate Bridge. He is watching with pleasure as the tech boom enfolds San Francisco. Now 75, Brand came to the Bay Area in 1956 and became famous for publishing the counter-cultural bible the Whole Earth Catalog which recommended the tools, technology and attitudes hippies would need to advance themselves and society as a whole.

As Brand sees it, history is being made again in the city. There is the suburban version of Bay Area cyber-business and there is a new urban version being created in San Francisco. "Market Street has been this sleepy dead street for a long time," says Brand, referring to the thoroughfare that bounds Soma. "Well, it is lively and exciting again now, thanks to the tech guys… A creative form is a creative form." Brand is convinced that the injection of so many young people with technical skills, money to play with and no family ties will spawn new ideas in San Francisco, a well-heeled, much needed creative renaissance.

He has little sympathy for those displaced along the way. San Francisco is a small corner of the Bay Area, he points out, and the rest still has significant economic diversity. Even if San Francisco becomes a Manhattan-like redoubt of the rich, the area as a whole will see benefits. "One side effect of this may well be that Oakland, which is pretty damn interesting, becomes even more interesting."

Curiosity drew Zeph Fishlyn back to Million Fishes's old building last year. She found it occupied by a startup called Bloodhound that had moved in mid-2013 and was paying two and a half times the old rent. The company designs apps to make exchanging contact details with people easier in work situations. Its founder and CEO is Anthony Krumeich, a 27-year-old dropout from Stanford University's Symbolic Systems course, which has produced senior executives for companies such as Google, Facebook and LinkedIn. On Twitter, he describes himself as: "Inventor, dog owner, free and present thinker, entrepreneur, drop-out, sailor."

Originally from suburban New York, Krumeich has curly hair, thick-rimmed glasses and wears a plaid shirt – standard urban hipster uniform. He arrived in San Francisco in late 2010, after a couple of years trying to get Bloodhound going in Silicon Valley. The company now has 15 employees and nearly $5m in investment funding including from Peter Thiel, a co-founder of PayPal and the first investor in Facebook. Bloodhound has revenues but not profits and Krumeich moved his company to the Mission from Soma in search of lower rents and some soul. The office's aesthetic is white space, wood and large Apple computers. It overflows with signs of a start-up culture – there are also soft furnishings, a table-tennis table and a copy of the tech entrepreneurs' bible The Lean Start-up. Employees who commit to not driving get a custom-made bike from a local bike shop and three days a week a chef cooks the office a wheat- and dairy-free lunch.

Krumeich and I walk the one block to Lower 24th Street, San Francisco's most vibrant centre of Hispanic culture and commerce. It has the highest concentration of Latino businesses in the city, an eclectic mixture of speciality stores, Mexican bakeries, grocers and butchers. But 24th Street is in transition. High-end coffee shops and restaurants are poking in, along with a fashionable Jewish deli selling $13 sandwiches. Facebook's Mark Zuckerberg, who has a second home in the Mission, has been spotted there.

Written on stone: Eclair Bandersnatch, an SF native street artist, standing above one of her stencils. Photograph: Barry J Holmes for the Observer

Krumeich is busy with the kind of project that Brand claims will define San Francisco's future. Inspired by the housing crisis, he and his companion – a Boxer dog – have just moved into what he calls an "alternative living situation", a shipping container on a flatbed truck. Krumeich is converting it into an off-grid, mobile living space that he says will be self-sustaining in its finished form. Solar panels provide electricity and a system of plants is to be used to recycle grey water. He thinks more mobile living might be the future.

But unlike Brand, Krumeich believes new San Francisco doesn't have to eradicate the old. The big ground-floor windows of his office are currently exhibiting canvases painted by an Oakland-based artist. A couple of weeks ago he held his first "artists' showcase", where he opened the doors to passers-by and had various local artists show their work – and he is thinking about a new kind of app to connect artists with potential buyers.

All this was inspired by finding out that Million Fishes had been in the space before, courtesy of Fishlyn. "They had an interesting place," he says, and while he is determined not to suffer from "tech guilt" he is thinking about the role he can play in his community. "I don't have a lot of preconceived notions about how this should work, but I am just going to start from: I care about other people; I am trying to do interesting and good things; I would like my presence here to be a contribution."

That kind of tech-led mission might be possible, but perhaps first an endangered species needs to be saved from extinction. Since late 2013, neighbourhood marches and blockades against Google's commuter buses have captured local, national and international attention. Tenant and neighbourhood organisations are working on proposals to be taken to San Francisco voters in November – suggestions include a moratorium on no-fault evictions. In January, the mayor responded to the growing pressure, urging people to stop demonising tech workers while announcing a seven-point housing plan which includes a target of 30,000 new homes by 2020, at least a third of which will be affordable. More immediately, he plans to try to reform the state's Ellis Act.

The San Francisco peninsular is where the world's new dominant industry – information technology – is most concentrated. Its tensions between highly paid tech workers and the communities that came before them may be a preview for other places. "What happens here may well happen in similar ways elsewhere, such as the emerging tech zones in London and Berlin," says Turner. "San Francisco is a canary in a coal mine." What reinvented San Francisco will look like when the dust settles is difficult to predict. But the nature of urbanity is that people packed in together do encounter each other and discover history and traditions. "Cities are more resilient than you might think," says Richard Walker. It's unlikely to be all doom for old San Francisco.

Lawrence Ferlinghetti, for one, is still convinced that the San Francisco he knows ultimately can't be engulfed by Silicon Valley, seeing too strong a connection to its geography, with water on three sides. "It still has an island mentality," he says. "At nearly 95, all I can say is good luck!"