Have you heard the story about the tip from the shoeshine boy, a Brit called James Pallot asks me on my last day at TechCrunch Disrupt. I have, I say, though later I Google it to get the facts straight.

It’s attributed to Joseph Kennedy, paterfamilias of the Kennedy clan who, in 1929, was getting his shoes shined by a young boy who was also making confident predictions about which stocks would rise. For Kennedy, it was a moment of revelation. He sold his portfolio. Not long afterwards, Wall Street crashed and the world was plunged into the greatest depression ever seen. So a tip from the shoeshine boy is a sign that the bubble is about to burst. That the wave of confidence will finally crash upon the shore. That the jig is up.

Pallot used to be the digital editorial director of Condé Nast in New York and now he has a startup. But then, we’re at the world’s biggest startup conference in San Francisco, a few miles down the road from Silicon Valley where the world’s greatest concentration of technology startups first started up.

His company is in the booming field of VR, or virtual reality, which is to 2015 roughly what Rubik’s Cubes were to 1982, though with rather bigger potential consequences. Pallot claims it’s the logical next step for journalistic content. In 20 years’ time, you won’t be reading this on the page, I’ll probably be leading you by the hand through a 3D rendering of a virtual TechCrunch conference floor. Or, more likely, you’ll be leading yourself and I’ll be claiming jobseeker’s allowance.

But anyway. In the meantime, Pallot asks me if I’ve heard of the tip from the shoeshine boy. I have, I say, and tell him it’s been on my mind. Because for three days, I’ve been hearing about “unicorns” – a Silicon Valley term for companies that have been valued at more than $1bn. When this usage was first coined, less than two years ago, there were 39 of them. Today, there are 147. Or as Matthew Wong, a senior analyst at CB Insights, tells me: “The funding is at levels that we haven’t seen since 2000.”

As those with longer memories will recall, that was the year the dotcom bubble burst. It needs explaining because there are an awful lot of people at TechCrunch whose memories simply don’t go back that far: the typical startup founder is male and in his 20s. Back in 2000, Google was less than 18 months old and Facebook wasn’t even a glimmer in Mark Zuckerberg’s eye – he was still at high school. (At 31, he’s now practically Silicon Valley’s elder statesman.)

Everything has changed. And is changing at an ever-faster pace. Eight years ago, TechCrunch launched its Disrupt conference with 45 startups. This year, there are 5,000 of them. Over three days I talk to founders of companies from San Francisco and Texas and Uruguay and Beirut and Stockholm and Tel Aviv and Warsaw. There are apps for crowdfunded mortgages and cheaper divorces and better sex. There’s “Expedia for golf” and “Facebook for cars” and “Nest for water” and “Tinder for dogs”. There’s a virtual reality teddy bear, a device that claims it will be able to read your emotions via a contact lens in your eye and another that will automate your home cannabis farm (marijuana is a big deal in Silicon Valley right now). I miss the panel on nuclear fusion startups but they’re around.

They’ve all paid upwards of $3,000 (£1,900) to be here and they’re all trying to attract the attention of Silicon Valley’s biggest beasts. The VCs – venture capitalists to you and me. The money guys.

“How do you spot them?” I ask Peter Becronis, the founder of a real estate startup called Owner’s Vault. “Oh, it’s easy,” he says. “They’re all men, older guys who are in jeans and brown boots and perhaps a blue jacket. Oh, and a good watch. They’re the ones who shuffle past you trying not to catch your eye.”

It’s a long shot for the likes of Becronis to be here, but not a total pipe dream. Because hundreds of startups are being funded each month. Vast sums of money are changing hands. Crunchbase, TechCrunch’s sister site, lists the deals that are being done on a daily basis. On the day I write this, I check it and find 24 companies that have just received funding, including Kreditech, which got $92m (it uses “big data and complex machine-learning algorithms to credit score everyone worldwide”) and Medium, which received $57m (it’s a platform that has found another new business model that seems to involve not paying journalists).

Every month the amount of money being invested in early-stage startups goes up. And every month, more and more people are starting to use the B-word. Bubble. The last time this amount of money was swilling around, we know how it ended. “Back then, a lot of websites launched but that’s all they were, websites,” Mike Butcher, TechCrunch’s editor-at-large, tells me. “Now in 2015, all those technologies that were predicted – AI, drones, VR – have all turned up. The innovation is real. And it just continues to get bigger and bigger. There are more VC firms here than you can poke a stick at.

“Is it a bubble?” he asks and then answers the question himself, vividly, if not entirely clearly. “It depends. How many unicorns can you fit through the eye of a needle? Anyway, unicorns are over. It’s all about decacorns now. Companies that are worth tens of billions of dollars.”

AOL’s takeover of Time Warner in 2000 was the largest merger in US corporate history at the time. Photograph: Suzanne Plunkett/AP

In 2000 the bubble was in publicly listed companies – organisations like the then upstart AOL, which bought Time Warner for $164bn, the largest merger in America business history, and then most spectacular blow-up. Or in Britain, Lastminute.com, whose share price peaked at 511p before crashing to 80p a month later. Both companies survived, unlike many, but it was a long struggle back up for both of them. (In a neat bit of circularity, AOL bought TechCrunch along the way.) In 2015, it’s private money flowing into companies that may or may not go public one day.

The shoeshine boy wouldn’t be tipping stocks in 2015, but what would he be doing? I ask Ned Desmond, the chief operating officer of TechCrunch. He thinks for a moment. “He would probably be an Uber driver who has his own angel investment line,” he says.

But James Pallot tops that. He’s flown in from JFK and had his shoes shined in the airport. “And the guy had a startup. I literally got a tip from the shoeshine boy! He was trying to find an investor for his national shoeshine franchise.” But then, in many ways, there has never been a better time to be a startup. Niko Bonatsos, a VC with General Catalyst Partners, tells me that the sheer number of companies at TechCrunch “speaks volumes about how the barriers to entry have been removed. It’s really easy to start a company. And lots of companies from other parts of the world see this as a lottery ticket. And for some of them, it will be. It’s the survival of the fittest. And the luckiest.”

Pallot and his co-founder are currently “bootstrapping” their company, Emblematic Group, which is creating virtual reality news content. “Bootstrapping” is Silicon Valley jargon. It means getting by with what you’ve got. It’s how people have set up companies since the dawn of capitalism. You start a business with a bit of money you already have and you try to attract customers and build it from there.

“Bootstrapping” is how you figure out if there’s a market and, if so, how you reach it. It’s also, like, totally 20th century. The reason 5,000 companies pay $3,000-plus to come to TechCrunch is because Silicon Valley has another model. People – strangers – will give you vast sums of cash to build your company into a global brand overnight. If you can deliver the killer pitch. The pitch that convinces the valley’s top VCs that you are the next Facebook, the next Uber, the next Airbnb.

“It doesn’t work like this in the rest of the world,” Ned Desmond tells me. “In Indonesia or Turkey or wherever, normal business culture demands collateral and security. Venture investing has none of that. You are investing in potential.” You’re gambling, basically. Silicon Valley, in 2015, is a giant casino. And the bets are so large because the potential payoffs are so huge. The next Google has to start somewhere.

So is it a bubble? “Everything is cyclical,” says Desmond. Does he remember the last crash? “I was there! I was in it. It was terrible. We had just launched a magazine, Business 2.0. Even the name sounds so cringeworthy now. We launched in May 2000 with a record number of advertisements. We had 150 ad pages. A year on, we had 15.”

This is not exactly an answer, so I try again. Is it a bubble? “We published a graph showing the unicorns. It’s a hockey stick. It’s near vertical growth.”

So, is it a bubble? “Some people say we are living through a period of history that is unprecedented. They say that everything has changed. That it’s different now.”

So is it a bubble? “Look,” he says finally. “You can answer that question for yourself. Look at the graph. It’s just about common sense.”

*****

Out on the exhibition floor, it’s mayhem. Everyone is pitching everyone else. And it’s all being conducted at volume 11. Because if you have a big idea and you want to find a VC, it’s just possible that you might find one out there. I lived through the 2000 bubble and subsequent crash vicariously via a flatmate who got a job with a thrusting new website, and it’s probably my imagination, but there’s something of the atmosphere that I remember of the time: loud, overexcited twentysomethings who tended to leave a mess on the floor.

A dozen or more countries have bought into the Silicon Valley dream and sponsored their companies to be here: there are pavilions representing Brazil, the Middle East, Uruguay, Taiwan, Norway, the Czech Republic. And the posters above the stands look like they’ve been produced by an automatic startup buzzword generator. “Digital innovation platform!” reads one. “Accelerate growth disruption” another. A third offers to “quantify your coding”.

The first person I talk to – and that’s because he stops me in my tracks by holding an iPhone with a picture of an anchor on it right in my face – is Lev Moravtchik. He’s a 33-year-old from Israel who has a dating app – Diggidi – that he tries to convince me is going to be bigger than Tinder. Is this the dream? To start your own company? “No, never! Nothing is forcing me to ruin my life.”

Is that the fear? That you’re ruining your life? “Of course! Do you know the figures? More than 90% of all startups fail.” So why are you here? “Israel is a startup nation. And Diggidi is different from anything else. It’s disruptive.” This is TechCrunch Disrupt. Of course it is. But how exactly? “Because it’s a dating revolution. Because it can’t be ignored.”

The process of ‘disruption’ is supposed to describe how established firms such as IBM were overtaken by the likes of Microsoft. Photograph: Fabian Bimmer/AP

But then, if there’s one thing that I learn, it’s that disrupting is to now what dropping out and tuning in was to the 1960s. Everyone’s disrupting. Or at least everyone thinks they are. The usage of the term was coined by Clayton Christensen, a professor at Harvard Business School, to describe how even massive, seemingly invincible companies like IBM can be displaced by scrappy little upstarts like (once upon a time) Microsoft. He called it the “technology mudslide hypothesis”. Or how some small new technological breakthrough can have huge world-changing consequences.

And if I’m not convinced that Diggidi is the revolution the world has been waiting for – it takes me a good five minutes to understand how the anchor even comes into it – I also have no idea what revolution the world is waiting for. But then, as Tod Francis, a Silicon Valley VC with the firm Shasta Ventures, tells me, often the world has no idea either. Until it arrives. “It’s why you’ve got to keep an open mind. Think about what billion-dollar companies looked like at this stage. Most of the really big ideas were easy to dismiss. Nobody thought sleeping on people’s couches was some amazing business plan, but look at Airbnb. Look at FitBit – did anyone really need that? No, but people bought it. We see crazy ideas in action every day. Often they are the ones that are accepted.”

I meet him on the way into the startup “battlefield”. It’s the heart of TechCrunch, where 25 companies take to the stage and pitch their hearts out to a panel of VCs who make the judges on Dragons’ Den look like kindergarten teachers. It’s a big deal. Dropbox was a battlefield contender, as was Yammer (sold to Microsoft for $1.2bn), and Mint.

Everyone knows it’s a bubble. That doesn’t mean a lot of people won’t make a lot of money. It’s like musical chairs

And it’s incredibly competitive. More than 950 startups applied and yet, to the outsider, it’s hard to quite see how the ones that got through, got through. There’s a company that is looking to change the world through its robotic nail-art machine. There’s a “personalised content relevancy platform”. And an app that will deliver food for your kids’ lunchboxes. (You can’t move in Silicon Valley for delivery apps these days. Or, as one tech editor summarised the sector to me, “twentysomething men who have set up companies to provide things their mother used to do for them.”) The odds are that one of them will be the next unicorn, but I’m having a hard time spotting which, not least because the man sitting next to me is offering his own running commentary.

“EasyPaint?” he says, as we watch a presentation for an app that’s going to revolutionise the painting and decorating industry. “EasyDud more like. People will use it once then take the guy’s number and call him directly.” Michael, who doesn’t want his full name in this article, works for a Japanese technology company and is at TechCrunch to make recommendations for early-stage startups that he thinks his company should acquire. He practically guffaws when I ask him if it’s a bubble.

“Of course it’s a bubble. Everyone knows it’s a bubble. Which doesn’t mean that a lot of people won’t make a lot of money. It’s like musical chairs. The question is not if it’s a bubble, but will it remain long enough for us to get in on it? The question is: am I too late?

“Look at how many companies are cash-negative. Twitter is still cash-negative. Even Uber is cash-negative. Silicon Valley is a hype machine. That’s how it works. And sometimes if there’s enough hype, it pays off. And sometimes it doesn’t. It’s like when you go house-hunting and the real estate agent doesn’t say, ‘Look at how great the space is. Look at how great the area is.’ They say, ‘If you don’t buy now, it’ll be twice the price in six months’ time.’ It doesn’t reflect true value. It’s just a sort of herd mentality.”

And he gives me the example of San Francisco’s first gold rush, the huge influx of money in 1849 that created large swaths of the Victorian city that still stands today. “People thought prices would go up and up and up and that eventually they would fill in the bay and build there. People were buying plots of land beneath the ocean because they thought that in the future prices would rise. It’s exactly the same as we’re seeing today.”

Rocketing property prices have caused tension in San Francisco between wealthy tech staff – who are often driven by bus from downtown areas to their places of work – and longer-established residents. Photograph: John Orvis/Demotix/Corbis

It’s an apposite example because San Francisco is in the grip of a property boom that makes London’s look almost sane and reasonable by comparison. The average rent on a one-bed apartment has just hit $3,500 (£2,300) a month, and that’s if you can find one. The entire city is pinched. I can’t find a single hotel room for under $350 and so rent a room in someone’s house via Airbnb, and even that is $150 a night.

The founders of a Parisian startup, Aircall, tell me they’ve come to look for funding. “We are four guys so we thought, OK, we will share an apartment and it is $10,000 a month! We were like, wow!” They still came though. “But we left the tech team in France. Europe is like the third world to Silicon Valley now.” Mike Butcher tells me that “Silicon Valley isn’t a place any more. It’s a state of mind. You can be in Ukraine or Berlin or wherever.”

You can. But if you want to be where the money is, you still want to be here. And it’s here in San Francisco that the tectonics of the tech industry are smashing up against those of the real world, a San Andreas-sized faultline rumbling deep beneath the ground. “Silicon Valley” used to be a chain of dormitory towns a commuter train ride away from San Francisco where the likes of Google and Apple have their campuses. But increasingly, it’s moving to San Francisco itself. Everyone wants to be downtown, where the bars and hipster cafes are. But the more attractive the city becomes to startups, the more cripplingly expensive it is for everyone else.

The city that gave birth to the counterculture, that stuck it to the Man, that is still home to organic co-ops and vegan co-living spaces, is right in the path of the Great Disruption – a storm whipped up and driven by the most relentless form of capitalism ever witnessed.

Tim Fernando, 31-year-old co-founder of an Oxford-based startup, Esplorio, explains why they’re staying in Oxford. “A software engineer will [cost us] $120,000 here, whereas in Oxford, it’s more like £26,000.” And yet, it’s hard to ignore the benefits that just being here confers. “We’d literally just arrived and were sitting in a coffee shop and it turned out that the woman sitting opposite us was a director of marketing at Apple. That kind of thing just doesn’t happen in Britain.”

I rather like Esplorio. It uses data from your phone to create an automated travelogue of where you’ve been, and they tell me they’ve had $300,000 in early-stage funding.

“Well done, I say. “That’s sounds like a lot.” “Does it?” says Fernando and he sounds slightly mournful. “Not in Silicon Valley terms it’s not.”

And it’s true, it’s not. And it’s as useful a measure as any of how Silicon Valley has parted ways with reality. Or at least the reality that most of us know; a world where $300,000 remains quite a lot of money. Silicon Valley, on the other hand, is a place where billion-dollar companies are being created almost overnight. Six years ago, Uber didn’t exist. In its last round of funding, it raised $1bn in capital from private investors, giving it a supposed value of $50bn. It’s a penta-decacorn. Less a mythical creature, some might argue, more a Frankenstein monster.

But Silicon Valley’s primacy remains indisputable. “Look at how many world-changing technologies have been created within a few miles of here,” Sam Altman, the president of Y Combinator, tells me. “You’re using an iPhone, which was invented 25 minutes away. You’re staying in an Airbnb, which was invented five minutes from here. You’re probably using Twitter, which was dreamed up around the corner. It’s incredible how concentrated the real innovation has been. How much has come out of this one area.”

Y Combinator is probably the most famous of Silicon Valley’s seed funds. It runs a three-month programme that nurtures startups and eventually helps launch them into the world. Dropbox, Airbnb and Reddit all began life there. “Eight thousand companies applied for the last batch,” says Altman. “And we have two batches a year.”

There’s so much money in Silicon Valley at the moment, he says, because interest rates are so low and “startups have been one of the few asset classes that have generated real growth”.

And the reason there are so many unicorns is that VCs bet big or they go home. He says it doesn’t matter if there’s a bubble or not, good companies will still make it whatever, an argument that echoes Bill Gates’s remarks from last year. He claimed that half the companies being created in Silicon Valley were “silly” and two thirds of them would fail. “But the dozen or so ideas that emerge out of that are going to be really important.”

Already “unicorpses” have started to appear. Groupon’s value is dropping like a stone. It’s being predicted that Dropbox, once valued at $10bn, will be “the first dead decacorn”. On the TechCrunch stage, people try not to say the “bubble” word. They say there is “frothiness”. That there is “a lot of capital that is seeking returns”. That “valuations are a bit robust”.

Elsewhere, people are not so restrained. Earlier this year, Bill Gurley, a partner at Benchmark, a big Silicon Valley venture firm, sent out a series of tweets warning of a bubble, including this one: “Arguing we aren’t in a bubble because it’s not as bad as 1999 is like saying that Kim Jong-Un is fine because he’s not as bad as Hitler.”

When Airbnb first pitched their idea at me, that you’d stay on an airbed in a stranger’s house, I was not excited Marcus Hawkins

Out on the exhibition floor, I meet Marcus Hawkins, another Brit. He’s from Norfolk and has successfully run his own software company for a decade or so but he recently set up a new company, Patrolo, a business-to-business enterprise offering a website mistake-correction service. What are you disrupting, I ask him. He thinks for a moment. “King’s Lynn.”

We swap other bits of jargon we’ve picked up. Have you pivoted, I say, a piece of startup-ese I’d learned five minutes earlier. It means to change your business strategy rapidly. “I’ve pivoted so many times I’m practically facing backwards,” he says. “But it’s OK. I have a lot of runway left.” Which is the amount of time a startup has before the money runs out, he explains.

With the flights and so on, it was a £6,000 gamble to come, he says, “but I know everyone says this. But what we’re offering is unique.”

And unique is what Sam Altman tells me is what investors want. They want unique. Quirky. Crazy. And he cites the Airbnb example that everyone cites, though in fairness he was actually there at the time. “When Airbnb first pitched their idea at me, that you’d stay on an airbed in a stranger’s house, I was not excited at all by that. Only halfway through did they develop the idea of renting the whole apartment and then I got excited. If the ideas aren’t crazy enough, the industry is not taking enough risk.”

It’s something you’d be hard-pushed to say of some of the startups here. I inspect a smart dog bowl aimed at preventing your dog from overeating, a wristband that will tell a woman precisely when she’s at her most fertile, and a robotic cooking machine that has a lot of parts that look like they’d need washing up.

Even rapper Snoop Dogg is attempting to establish a tech startup – with the vital ingredient of cannabis. Photograph: Tim Mosenfelder/Getty Images

And any number of cannabis-related businesses. Since marijuana was made legal in four states, the tech industry has rushed into the space. “There’s going to be a marijuana unicorn,” a man from a website called Investing in Cannabis tells me. “It’s hotter than hot right now. Everyone’s placing their bets.” A point that is proven when Snoop Dogg takes to the stage to announce his cannabis startup, Merry Jane (a lifestyle media site aimed at cannabis users).

The inventiveness of the hardware and ingenuity of the software and chutzpah of everyone is quite mind-boggling. There are so many ideas and so much energy and hard work and, more often than not, hard-earned cash being invested in them. If you have come from Bratislava, you’ll have put your own money into the business, a Slovak startup founder called Lukas Alner tells me. “Whereas here, they all graduate from Stanford and Harvard and then go and work in the same places. If they want to make a contact or raise money, they just pick up the phone. It’s very hard to break into that as an outsider. They are pack animals.”

We all love to talk about disruption. But if you are doing something disruptive, people are getting hurt Vinod Khosla

And then there’s the small matter of what it may all mean one day not long from now. Where all this innovation will lead – a question I contemplate when a founder from Hong Kong called Henry Hu shows me Cafe X, a robotic coffee machine which, he claims, makes perfect artisanal cappuccino.

So you’re disrupting baristas? “No,” he says. “A barista will design the programme which the machine will copy. So, we are extending the skills of talented baristas to a lot more people.”

Yadda, yadda, yadda. A Dublin-based startup shows me impressive software that uses “machine creativity” to create graphic art. “Sixty one per cent of the cost of video games is the artwork,” the co-founder, Eric Risser, tells me. “We’re disrupting that.”

So you’re putting artists out of work? “No, we’re enabling video-games companies to create more and better games.”

*****

The truth is, it’s a dog-eat-dog world out there. Disrupt or be disrupted. As Vinod Khosla, a somewhat irascible billionaire and head of a leading venture capital firm reminds the audience: “We all love to talk about disruption. But if you are doing something disruptive, people are getting hurt. Revolutions are hard on people. People get hurt.”

It’s what is forgotten in all of this. If it is a bubble, and if it bursts, the VCs will be OK, of course they will. They’re already rich. It’ll be the little people who will suffer. The 1,100 employees of Groupon who just lost their jobs will be a tiny taste of the pain to come. The ripple of potential consequences vast and as yet unknowable.

And if it isn’t a bubble, if the world really has changed, and economic laws too, and everything keeps going up and up and up, it’s still the little people who’ll suffer. The great swaths of us whose services will no longer be required in a coming era that is just around the corner. In the past 18 months, all of us who live in cities around the world have watched as taxi drivers have protested and lost their livelihoods and Uber has swept in and cleaned up, taking 20% of every fare along the way. Don’t think this is going to stop at taxis – where Uber is going to end up is more or less anyone’s guess. Last month I met the head of Carnegie Mellon University’s robotics faculty, whose entire department has just been poached by the firm. Including him.

First they came for the booksellers and I did not speak out because I was not a bookseller. Next they came for the taxi drivers etc etc. Then baristas, divorce lawyers, artists, journalists… there are not going to be an awful lot of jobs of any description left. How any of us are going to be earning a living in 20 years’ time, in 10 years’ time, is something that most of us aren’t thinking about. In this light, building a startup that has a 90% chance of failure looks like a pretty smart option.

“I think there is going to be massive job destruction in the next 10 to 20 years,” says Sam Altman. “I think we are already seeing it. Technology increases wealth but it also concentrates it. There’s a huge coming threat to all of us.”

Companies in Silicon Valley think differently from the rest of us. Their ambition is of a different order. Their rate of growth is like nothing we’ve ever seen before. And it’s in San Francisco, where technology meets the real world, that we’re starting to see the beginnings of what this clash of civilisations will look like. Or as Altman puts it: “If founders in San Francisco can build a new $50bn company in five years but the city can’t approve a single new housing development in that time, that’s a mismatch.”

It’s all a mismatch. We and the great tech world we’re building. But then I probably would say that. My industry is just one of hundreds in the throes of the Great Disruption. Yours too, possibly. Even if you haven’t realised it yet. It might not just be startups that need to learn to pivot.