The most desirable career of the 21st century, with numerous advantages over other fast-growing occupations such as hospice carer and rickshaw driver, is being a billionaire. Prior to the incorporation of US Steel in 1901, the world didn’t have a single billion-dollar company, much less a billion-dollar individual. Today, more people than ever are becoming billionaires – 2,000 and counting have made the great leap upward, according to the “global wealth team” at Forbes. And the US’s hottest billionaire factory is located in the most hyped yet least understood swath of suburban sprawl in the world: Silicon Valley.

Despite what you may have heard, hard work in your chosen trade is absolutely the stupidest way to join the billionaires club. In Silicon Valley, the world’s most brilliant MBAs and IT professionals discovered a shortcut to fabulous riches. Ambitious Ivy Leaguers who once flocked to Wall Street are now packing up and heading west. The Valley’s startup founders, investors, equity-holding executives and fee-taking middlemen have thrived above all. Inspired by their success, my idea was to move to Silicon Valley, pitch a startup and become obscenely rich. I left home with some homemade business cards showing my new email address, futurebillionaire@aol.com, and a bunch of half-baked ideas.

The first thing I needed was a place to stay. The best deal I could find on short notice was a place I called Hacker Condo. Like most Bay Area newcomers, I was relying on the short-term apartment rental app Airbnb. At $85 (£59) per night, the place cost less than the market average, but was still more than I could afford. On the upside, it was in what the real estate hucksters called SoMa – a trendy San Francisco neighbourhood well suited to my journalistic and entrepreneurial purposes. Once a low-rent manufacturing district, the south of Market Street area had become the go-to place for startups seeking industrial-chic open-plan offices, although the poor and homeless had not yet been fully purged.

The ad for Hacker Condo stated an express preference for techies: “We would like to welcome motivated and serious entrepreneurs who are looking to expand their network,” it said. Perfect. The best part: “No bunk beds.” I told the hosts that I was an “embryo-stage” startup founder and author. The hosts didn’t own the place. I looked it up: the mortgage was held by some European guy who seemed to spend most of his time surfing at a resort and dabbled in the tech business as a hobby. The legal status of this rental arrangement was, let’s say, unclear.

I rang the buzzer for a unit labelled TENANT. A man answered right away. He had been waiting. After a moment, the door opened, and I met my new roommate, a gangly Kiwi. We took the elevator three floors up and entered a silent, beige-carpeted hallway. Our unit was No 16. The first thing I noticed inside was a small mountain of men’s shoes. Hacker Condo was modern and more spacious than seemed possible from the outside. The unit was spread over three floors. The furniture consisted of a picnic bench and a sectional sofa spanning the width of the living room. I counted five other short-term tenants. The Kiwi told me that soon, some Norwegian guys – a whole startup team – would be moving in. We calculated that Hacker Condo would soon have three more guests than it had beds.

“What’s the key situation?” I asked.

“There’s one key,” the Kiwi said.

“One key?” I said. “For everybody?”

There were more tricks to learn, as a consequence of the possibly illicit nature of this type of rental arrangement and the evident stinginess of our Airbnb hosts. The Condo Hackers never came in through the front door. It was too conspicuous. I followed the Kiwi down to the ground-floor garage, then outside to the rear of the building. He showed me how to slide my hand along a grate to locate the tiny combination safe that contained the exterior door key. It was best to do this when no one was looking.

I knew not to spend too much time getting to know my flatmates, for we were all rootless high-tech transients, our relationships temporary, our status revocable.

The room I had booked was available for only two weeks. As soon as I connected to the wifi network, I would need to start looking for another place. “My” room had five beds in it. I thought I had paid for a private space. I double-checked. The listing clearly stated “no bunk beds”, but down in the fine print I finally found the words “shared room”.

Two weeks was not enough time to find an apartment in San Francisco. Not on my budget. Rents were higher than in New York or London. One-beds were running at about $3,000 per month; studios, about $2,500; shares, $1,500; and illegal crap shares, $1,000. It was the same deal across the bay to the east in Oakland and Berkeley, as well as to the south in the Silicon suburbs of Redwood City, Palo Alto and Mountain View. Whatever I might save in rent by living on the periphery I would lose in transportation costs and time.

These “hacker houses” were the products of disruptive innovation in the urban property market. The city was once riddled with small apartments and single-family homes that sheltered trifling handfuls of obsolete labourers and their unproductive children, often for decades at a stretch. But the tech boom let such so-called family homes reach their full potential as investment properties. Some hacker houses were attached to startup investment incubators or shared workspaces. Others amounted to little more than flimsy bunks in a windowless room. A number of trend-savvy investors purchased or leased dozens of residential properties around the Bay Area to rent out in this fashion.

Although I envied them from my dark and squalid quarters, the San Francisco long-timers who lived in rent-controlled apartments were in situations nearly as precarious as my own. I met a musician who lived in a $600 rent-controlled apartment in the Mission. When I met her, she was terrified that her landlord would evict her and sell the building so that it could be rented out at six times the price to white techie colonisers such as myself.

With landlords eager to cash in, formal evictions had increased 55% in five years. More often, though, landlords simply bullied their tenants into packing up. “Tenants are getting evicted for having cups in their cupboards. The landlords say it’s clutter. They’ll say anything. Eventually the tenants just give up,” a lawyer for a tenants’ rights organisation told me. His employer, the Eviction Defense Collaborative, was itself getting evicted from its offices so that the landlord could rent the space to a tech startup.

My earnings potential had plummeted when I stopped writing software and started writing for newspapers. I now looked with envy at the techies, the winners, the pioneers. They had ideas. They had momentum. Most important, they had money. Why not me?

I wasn’t just changing careers and jumping on the “learn to code” bandwagon. I was being steadily indoctrinated in a specious ideology. As proud as I was of having learned new skills, I didn’t understand that the only way to turn those skills into a livelihood was to embrace the economy of the digital world, where giant corporations wrote the rules.

My idea was to pitch a tech startup and get obscenely rich while writing a book about how to pitch a tech startup and get obscenely rich – the Silicon Valley way.

To save money, I took to cooking my own meals most of the time. This was when I discovered that it was much easier to launch a tech startup if you could afford to always have food delivered and never had to deal with mundane chores such as doing laundry, washing dishes or buying groceries. As one Twitter wag observed, San Francisco’s “tech culture is focused on solving one problem: what is my mother no longer doing for me?”

I never felt older nor crankier than when watching these “digital natives” stumble through the daily rituals of adulthood. One of the kids, an overachieving Ivy Leaguer whose Google internship demanded an advanced understanding of high-level mathematics, was completely baffled when it came to using a simple rice cooker. I explained the process: put in rice, add water, press the button labelled “cook”. He grew increasingly flustered, and I suspected he wanted me to make the rice for him. He managed to sauté a boneless, skinless chicken breast, but only by following the instructions on the package to the letter.

“How did it turn out?” I asked.

“It’s terrible. Bland,” he said. “I’m full, that’s all that matters. I don’t care how it tastes.”

When I first heard about Soylent, the startup selling a gooey “meal replacement beverage” powder with a determinedly “neutral” flavour, I wondered what sort of miserable insensates would choose to subsist on such glop. Now I knew.

It may have been better for everyone when the overpaid nerds stayed home. “They’re importing children to destroy the culture,” one bar owner told me.

Airbnb employees at work in San Francisco. Photograph: Alamy Stock Photo

Indeed, to overhear the baby-faced billionaire wannabes exchanging boastful inanities in public could be enraging. Their inevitable first question was: “What’s your space?” Not “How’s it going?” Not “Where are you from?” But: “What’s your space?”

This was perhaps the most insufferable bit of tech jargon I heard. “What’s your space?” meant “What does your company do?” This was not quite the same as asking: “What do you do for a living?” because one’s company may well produce no living at all. A “space” had an aspirational quality a day job never would. If you were a writer, you would never say “I’m a writer”. You would say “I’m in the content space”, or, if you were more ambitious, “I’m in the media space”. But if you were really ambitious you would know that “media” was out and “platforms” were in, and that the measure – excuse me, the “metric” – that investors used to judge platform companies was attention, because this ephemeral thing, attention, could be sold to advertisers for cash. So if someone asked “What’s your space?” and you had a deeply unfashionable job like, say, writer, it behooved you to say “I deliver eyeballs like a fucking ninja”.

In my former life I would have sooner gouged out my own eyeballs than describe myself in such a way, but in post-recession, post-boom, post-work, post-shame San Francisco, we all did what we had to do to survive.

I was beginning to become acquainted with the infinite solipsism of my new milieu. We were grown men who lived like captive gerbils, pressing one lever to make food appear and another for some fleeting entertainment – everything on demand. Airbnb and Foodpanda served the flesh, Netflix and Lifehacker nourished the soul.

I relied on sites such as EventBrite and Meetup to keep my social calendar full and my expenses down. I went to a party at the Yelp office – like most of the freebies around town, it was advertised online. The venue was a forbidding art deco tower – the old PacBell building, constructed for the California branch of the national telephone monopoly in its heyday. Now the tower’s largest tenant was a website that allows anonymous semi-literates to post critiques of local establishments. Most of the crowd seemed to work at Yelp, and felt obliged to stick around for the event. But there was something else keeping these people here – an overriding anxiety about unfamiliar spaces.

Life outside the startup bubble was frightening and unpredictable. Inside, it was safe. “Fun” was mandatory in the Bay Area tech world, and inebriation strongly encouraged. The bar at Yelp, for instance, featured three kegs of high-end craft beer and an array of wines and spirits. This was not a temporary selection for the benefit of us honoured guests, but a permanent fixture of the commissary. Normally open only to employees, the Yelp Cafe had a perfect five-star rating ... on Yelp. “Well, looks like I’m never leaving my office compound!” one reviewer wrote.

A corporate recruiter explained to me the forces driving the “perks war”, an escalating tit-for-tat of such freebies as steak dinners delivered to employees’ desks, free laundry service, free bikes and bike repair, free concierge service and, of course, free drinks.

“They might get a $20 steak, but with the extra time they’ve stayed at work, they’ve provided an extra $200 in value to their employer,” the recruiter said. Thus the seemingly lavish enticements were a way to attract profit-producing programmers, who were in exceedingly high demand, without offering higher salaries. The perks also provided effective cover for the companies’ slave-driving work schedules.

My flatmates seemed happy with the arrangement, at least at first. “Everything they say about Google is true,” one intern told me after his orientation at the Googleplex. “There are 20 cafeterias, a gym – everything.” Early each weekday morning, he and the other Googlers in his neighbourhood swiped their ID cards to board a chartered bus parked near the Bart station, then rode 35 miles to Mountain View. They started working onboard the bus, which was equipped with wifi, and didn’t leave the campus until about 8pm, when another bus ferried them home after they ate at the company cafeteria. This was a pretty standard deal at the big Silicon Valley companies. Even rinky-dink startups in SoMa warehouses offered free catering. “The perks, man!” another roommate, a non-Googler, raved after arriving home at 10pm from his first day on the job.

“I worked until 9pm because dinner is free if you work that late ... And they’ll pay for your cab home,” he went on. That became his routine, and he never questioned it. Come to think of it, like a lot of his contemporaries, he never questioned anything.

In this milieu, a certain tolerance for phoniness was prerequisite. It was not enough to have the right skills, put in your time and get the job done – you had to be fucking pumped about your job. Certain specialities were in more demand than others. Any chump with a humanities degree could talk his or her way into a marketing job, but programmers were harder to come by. One sunny day, I followed the waterfront to the event center at Pier 27 and signed in to the DeveloperWeek conference.

DevWeek, as everyone called it, was basically a week-long recruitment fair sprinkled with slideshows and panel talks. It was jarring to see employers desperate to hire, not the other way around. In 2010s America, the only place that was always hiring, apart from Silicon Valley, was the local US army recruiting centre. Hundreds upon hundreds of people had flocked here to look for a better job and still there were not enough applicants to fill all the openings for “Java Legends, Python Badasses, Hadoop Heroes”, and other gratingly childish classifications describing various programming specialities. Techies would call themselves just about anything to avoid the stigmatising label of “worker”. They could only face themselves in the mirror if their business card proved that they were rock stars or ninjas or something romantic and brave and individualistic – anything but the truth, anything but a drone.

I had an important realisation at DevWeek: I wasn’t the only one bluffing my way through the tech scene. Everyone was doing it, even the much-sought-after engineering talent. I was struck by how many developers were, like myself, not really programmers, but rather this, that and the other. A great number of tech ninjas were not exactly black belts when it came to the actual onerous work of computer programming. So many of the complex, discrete tasks involved in the creation of a website or an app had been automated that it was no longer necessary to possess knowledge of software mechanics. The coder’s work was rarely a craft. The apps ran on an assembly line, built with “open-source”, off-the-shelf components. The most important computer commands for the ninja to master were copy and paste.

Employees at the Square Inc headquarters in San Francisco. Photograph: Bloomberg via Getty Images

Barack Obama’s White House had endorsed Silicon Valley’s “learn to code” campaign – it was an official government job-creation programme. With the traditional US job market still a smouldering charcoal pit after the 2008 crash, computer programming skills were promoted as one sure way to attain the sort of prosperity and stability Americans had over many decades come to expect.

And yet, many programmers who had “made it” in Silicon Valley were scrambling to promote themselves from coder to “founder”. There wasn’t necessarily more money to be had running a startup, and the increase in status was marginal unless one’s startup attracted major investment and the right kind of press coverage. It’s because the programmers knew that their own ladder to prosperity was on fire and disintegrating fast. They knew that well-paid programming jobs would also soon turn to smoke and ash, as the proliferation of learn-to-code courses around the world lowered the market value of their skills, and as advances in artificial intelligence allowed for computers to take over more of the mundane work of producing software. The programmers also knew that the fastest way to win that promotion to founder was to find some new domain that hadn’t yet been automated. Every tech industry campaign designed to spur investment in the Next Big Thing – at that time, it was the “sharing economy” – concealed a larger programme for the transformation of society, always in a direction that favoured the investor and executive classes.

In the first seven years after the 2008 crash, 16 million people left the US labour force. And in that same period, thanks to Silicon Valley’s timely opportunism, the country gained an endless bounty of gigs. Tech startups, backed by Wall Street, swept in to offer displaced workers countless push-button moneymaking schemes – what Bloomberg News called “entrepreneurialism-in-a-box”. Need fast cash? Take out a “peer-to-peer” loan, or start a crowdfunding campaign. Need a career? Take on odd jobs as a TaskRabbit or pitch corporate swag as a YouTube “vlogger”. Nine-to-five jobs with benefits and overtime may be in the process of getting disrupted out of existence, but in their place we have the internet, with endless gigs and freelance opportunities, where survival becomes something like a video game – a matter of pressing the right buttons to attain instant gratification and meagre rewards.

More than a third of American workers now qualify as “freelancers” or “contingent workers” – that is, their livelihoods are contingent upon the whims of their managers. That’s because the choice to become entrepreneurs has been made for them. The destruction of social welfare, public education and organised labour has created what might be called the 50 Cent economy, a system structured to offer only two options: “Get rich or die trying.” George W Bush called it the “ownership society”. Obama, smitten with his Silicon Valley donors, gave us “Startup America”. And Donald Trump, history’s luckiest winner, reigned over a nation of “losers”. Under the latest iteration of the American Dream, if you aren’t a billionaire yet, you haven’t tried hard enough.

The contemporary equivalent of an entry-level job in the corporate mailroom was a work-from-home service called Mechanical Turk, operated by Amazon, the $136bn online retailer controlled by Jeff Bezos. The idea with Mechanical Turk was to create a digitised assembly line featuring thousands of separate “human intelligence tasks”, designed to be completed within seconds and paying pennies. Academic surveys found that many Turkers worked more than 30 hours per week for average wages of under $2 per hour. Yet these workers were considered self-employed small business owners. Their work was commissioned by social scientists seeking to cut costs on large-sample surveys, but also by profit-minded companies that hired hundreds of Turkers as needed, instead of a full- or part-time employee.

Another sharing economy upstart called Fiverr was a catalogue of freelance “gigs”, from illustration to translation, all sold at a fixed cost of $5. Launched in 2010 by two Israelis, Fiverr raised more than $50m in investment within five years, on annual revenue of $15m. Silicon Valley investors praised the founders’ “incredible vision” and swooned over the “liquidity, velocity and engagement” the company brought to the global marketplace.

It was remarkable what people were willing to do for $5, or more like $3.92 after service fees. A lot of ads promised custom website development. Others offered quick-and-dirty logos, proofreading, or résumé writing. I hoped to forge my place in the strange niche of bargain basement flat-fee consulting. Thousands of people were paying $5 to strangers for direction on matters they found too difficult, too stressful or too trivial to face alone. Fiverr’s terms of service forbade “nonsense” and “uncool stuff” but the service seemed to tolerate ads like one for an Amazon “Kindle ghostwriting machine”; or another for tools designed “to cheat likes on social networks”; and still another for “a profitable forex cheating strategy” – an obvious scam that Fiverr marked for a while as “recommended”. I had entered a murky ethical realm. I scanned gigs methodically. I learned that it paid to over-promise. No matter was too momentous:

“I will teach you to make Life and Death Decisions for $5.”

This gig was listed by a Fiverr-certified “top-rated seller” who claimed experience as a broker of precious metals.

“I will help you Survive the Fatal Ebola Virus Epidemic for $5.”

As far as I knew, there was no cure for Ebola. But who was I to argue with a five-star-rated seller? Could 2,679 customers be wrong?

On the site’s discussion boards, sellers swapped stories of unfair competition from scammers, insufficient payments from Fiverr, capricious rules, meagre sales and endless hours. Some sounded genuinely desperate. Fiverr even sent its workers emails about increasing productivity by avoiding depression. Full-time Fiverring took a physical toll, as well, with many slavish gig-peddlers reporting rapid weight gain. “I know what you mean! I bought some jeggings this weekend,” one woman wrote. Another commenter saw opportunity. “If anyone is interested,” he wrote, “I’m putting together a Fiverr gig where I will be offering online fitness coaching.”

Fiverr offered a glimpse at the new model worker: a fat, depressed con artist forever scheming against his comrades, egged on by the distant architects of the virtual marketplace– the only real winners. The company eventually embraced this image and celebrated it with a subway ad campaign featuring a fatigued-looking model with frizzy hair and circles under her eyes. “You eat a coffee for lunch. You follow through on your follow through. Sleep deprivation is your drug of choice,” the ad said. “You might be a doer,” it concluded. When busy-ness became a status symbol, the glamorisation of exhaustion was inevitable.

A sign supporting Proposition F to restrict short-term rentals via companies such as Airbnb in San Francisco. Photograph: Josh Edelson/AFP/Getty Images

I found Corey Ferreira through his website, makefiverrmoney.com, which was a marketing vehicle for his ebook, Fiverr Success: $4,000 a Month. 8 Hours of Work a Week. Having made a decent amount on Fiverr, Ferreira had found rates of pay had halved. Faced with slowing business, he had adopted a new approach: he could “sell the method”. He got the idea from a book called The Laptop Millionaire, which describes “a guy’s journey from being basically homeless to making money online. One of the things he talks about is making ‘information products’.” Hence Fiverr Success by Corey Ferreira was born, selling “hundreds” of copies at $17.

The book marked a transition for Ferreira, as he spent less time doing labour-intensive web design and more time searching for the cold fusion of internet marketing: “passive income.” “I remember when eBay started,” he told me. “I was kinda young. Everybody was talking about how to make money on eBay. I remember somebody telling me, ‘During a gold rush, you should sell shovels’.”

I felt he had let me in on some oracular wisdom. Don’t dig for gold: sell shovels to all the suckers who think they’ll get rich digging for gold. To post an ad on Fiverr was to announce one’s status as an easy mark. To hawk get-rich-quick manuals to all those eager Fiverrers, however, was to join the exalted ranks of the shovel merchants.

My Airbnb landlord, I realised, was a shovel merchant. As was the company that rented me server space for website hosting. As were the “startup community organisers” selling tickets to conferences and networking parties. As were the startup awards shows and Hacker News and the whole Silicon Valley economic apparatus promoting the ideal of individual achievement. We startup wannabes were not entrepreneurs. We were suckers for the shovel merchants, who were much cleverer than the thick-skulled “innovators” who did all the work while trading away the rewards.

For a business incompetent such as myself, this concept of selling a method, rather than a straightforward product or service, was revelatory. I understood this lesson as an extension of that old saying about teaching a man to fish instead of just giving him a fish. Now the idea was: you made him pay for fishing lessons, offering student loans if necessary, and failed to mention that you had already depleted the pool. In a late capitalist society with dwindling opportunities for cash-poor workers and few checks on entrepreneurial conduct, what could be better to sell than false hope? This was a smart business.

Unfortunately, the techie hustlers can be a little too clever for their own good – and ours. With decades of unwavering support from the military-industrial complex, Congress and Wall Street, the pallid princelings of Silicon Valley rewrote the rules of the global economy in their favour. The public, fooled as it was by the tech industry’s slick marketing and lulled by the novelty and convenience of its gadgetry, might be forgiven for missing some early warning signs. (Remember when the Google guys used to rhapsodise about beaming the internet – with the attendant targeted advertising – directly into people’s brains? It doesn’t sound so far-fetched and quirky now, does it?)

If we are feeling generous, the same retrospective clemency could even be shown to politicians who mistook Silicon Valley for just another well-heeled lobby looking for favours, and to the reporters who were suckered by the rapid rise of “revolutionary” companies such as Theranos and Uber. But the builders of our digital dystopia – the tech titans themselves, and their armies of engineers – have no such excuses. They will talk about the mistakes they have made. They will express regret for their oversights and make a show of contrition. Don’t be fooled.

The dark side of Big Tech, which many consumers are only beginning to come to grips with, is not some byproduct of California-style “conscious capitalism” – an unfortunate misstep in an otherwise heroic effort to “change the world”. Profit-hunger, philistinism and misanthropy are and always have been at the core of the enterprise. The new breed of Silicon Valley billionaires knew exactly what they were doing. The plan was to take all the money and run – to Mars, if necessary.

Adapted from Live Work Work Work Die: A Journey into the Savage Heart of Silicon Valley, published by Metropolitan Books in the US, and forthcoming from Scribe in the UK

• Follow the Long Read on Twitter at @gdnlongread, or sign up to the long read weekly email here.