If you had asked me, when I was 10, to forecast my life as an adult, I would probably have sketched out something like this: I would be driving a Trans Am, a Corvette, or some other muscle car. My house would boast a mammoth collection of pinball machines. I would sip sophisticated drinks (like Baileys Irish Cream), read Robert Ludlum novels, and blast Van Halen while sitting in an easy chair wearing sunglasses. Now that I am at a point to actually be able to realise every one of these feverishly envisioned tastes, they hold zero interest (well, perhaps the pinball machines in a weak moment).

It was not just that my 10-year-old self could not predict whom I would become but that I was incapable of imagining that my tastes could undergo such wholesale change. How could I know what I would want if I did not know who I would be?

One problem is that we do not anticipate the effect of experiencing things. We may instinctively realise we will tire of our favourite food if we eat too much of it, but we might underestimate how much more we could like something if only we ate it more often. Another issue is psychological “salience”, or the things we pay attention to. In the moment we buy a consumer good that offers cashback, the offer is claiming our attention; it might even have influenced the purchase. By the time we get home, the salience fades; the cashback goes unclaimed. When I was 10, what mattered in a car to me was that it be “cool” and fast. What did not matter to me were monthly payments, side-impact crash protection, being able to fit a stroller in the back, and wanting to avoid the appearance of being in a midlife crisis.

Even when we look back and see how much our tastes have changed, the idea that we will change equally in the future seems to confound us. It is what keeps tattoo removal practitioners in business. The psychologist Timothy Wilson and colleagues have identified the illusion that for many, the present is a “watershed moment at which they have finally become the person they will be for the rest of their lives”.

In one experiment, they found that people were willing to pay more money to see their favourite band perform 10 years from now than they were willing to pay to see their favourite band from 10 years ago play now. It is reminiscent of the moment, looking through an old photo album, when you see an earlier picture of yourself and exclaim, “Oh my God, that hair!” Or “Those corduroys!” Just as pictures of ourselves can look jarring because we do not normally see ourselves as others see us, our previous tastes, viewed from “outside”, from the perspective of what looks good now, come as a surprise. Your hairstyle per se was probably not good or bad, simply a reflection of contemporary taste. We say, with condescension, “I can’t believe people actually dressed like that,” without realising we ourselves are currently wearing what will be considered bad taste in the future.

One of the reasons we cannot predict our future preferences is one of the things that makes those very preferences change: novelty. In the science of taste and preferences, novelty is a rather elusive phenomenon. On the one hand, we crave novelty, which defines a field such as fashion (“a field of ugliness so absolutely unbearable,” quipped Oscar Wilde, “that we have to alter it every six months”). As Ronald Frasch, the dapper president of Saks Fifth Avenue, once told me, on the women’s designer floor of the flagship store: “The first thing the customer asks when they come into the store is, ‘What’s new?’ They don’t want to know what was; they want to know what is.” How strong is this impulse? “We will sell 60% of what we’re going to sell the first four weeks the goods are on the floor.”

A lot of times, people don’t know what they want until you show it to them Steve Jobs

But we also adore familiarity. There are many who believe we like what we are used to. And yet if this were strictly true, nothing would ever change. There would be no new art styles, no new musical genres, no new products. The economist Joseph Schumpeter argued that capitalism’s role was in teaching people to want (and buy) new things. Producers drive economic change, he wrote, and consumers “are taught to want new things, or things which differ in some respect or other from those which they have been in the habit of using”.

“A lot of times, people don’t know what they want until you show it to them,” as Steve Jobs put it. And even then, they still might not want it. Apple’s ill-fated Newton PDA device, as quaint as it now looks in this age of smartphone as human prosthesis, was arguably too new at the time of its release, anticipating needs and behaviours that were not yet fully realised. As Wired described it, it was “a completely new category of device running an entirely new architecture housed in a form factor that represented a completely new and bold design language”.

So, novelty or familiarity? As is often the case, the answer lies somewhere in between, on the midway point of some optimal U-shaped curve plotting the new and the known. The noted industrial designer Raymond Loewy sensed this optimum in what he termed the “MAYA stage”, for “most advanced, yet acceptable”. This was the moment in a product design cycle when, Loewy argued, “resistance to the unfamiliar reaches the threshold of a shock-zone and resistance to buying sets in”. We like the new as long as it reminds us in some way of the old.

Anticipating how much our tastes will change is hard because we cannot see past our inherent resistance to the unfamiliar. Or how much we will change when we do and how each change will open the door to another change. We forget just how fleeting even the most jarring novelty can be. When you had your first sip of beer (or whisky), you probably did not slap your knee and exclaim, “Where has this been all my life?” It was, “People like this?”

We come to like beer, but it is arguably wrong to call beer an “acquired taste”, as the philosopher Daniel Dennett argues, because it is not that first taste that people are coming to like. “If beer went on tasting to me the way the first sip tasted,” he writes, “I would never have gone on drinking beer.” Part of the problem is that alcohol is a shock to the system: it tastes like nothing that has come before, or at least nothing pleasant. New music or art can have the same effect. In a New Yorker profile, the music producer Rick Rubin recounted that when he first heard Pretty Hate Machine, the album by Nine Inch Nails, he did not care for it. But it soon became his favourite. Faced with something discordantly novel, “we don’t always have the reference points to absorb and digest it,” Rubin said. “It’s a bit like learning a new language.” The album, like the beer, was not an acquired taste, because he was not hearing the same album.

The architect's role is not to give the client exactly what he asked for, but to project tastes no one knew they had

Looking back, we can find it hard to believe we did not like something we now do. Current popularity gets projected backwards: we forget that a now ubiquitous song such as the Romantics’ What I Like About You was never a hit or that recently in vogue “antique” baby names such as Isabella or Chloe, which seem to speak to some once-flourishing tradition, were never popular.

It now seems impossible to imagine, a few decades ago, the scandal provoked by the now widely cherished Sydney Opera House. The Danish architect, Jørn Utzon, was practically driven from the country, his name went unuttered at the opening ceremony, the sense of national scandal was palpable towards this harbourside monstrosity. Not only did the building not fit the traditional form of an opera house; it did not fit the traditional form of a building. It was as foreign as its architect.

The truth is, most people probably did not know what to make of it, and our default setting, faced with an insecure unknown, is disliking. Frank Gehry, talking about his iconic, widely admired Guggenheim Museum in Bilbao, admitted that “it took a couple of years for me to start to like it, actually.” The architect Mark Wigley suggests that “maybe we only ever learn something when some form we think of as foreign provokes us – and we resist. But sometimes, many times, in the middle of the resistance, we end up loving this thing that has provoked us”.

Fluency begets liking. When shown images of buildings, architects have rated them as “less complex” than laypersons did; in other words, they “read” them more fluently, and the buildings seem less “foreign”. The role of the architect, suggests Wigley, is not to “give the client exactly what he was asking for” – in other words, to cater to current taste – but to “change the idea of what one can ask for”, or to project future tastes no one knew they had. No one said an opera house could look like the Sydney Opera House until Utzon, taking his idea from a peeled orange, said it could. The world changed around the building, in response to it, which is why, in the curious words of one architecture critic, “Utzon’s breathtaking building looks better today than ever.”

A few decades from now, someone will inevitably look with dread upon a new building and say, “The Sydney Opera House, now there’s a building. Why can’t we build things like that any more?”

This argument – for example, “Why isn’t music as good as it used to be?” – reflects a historical selection bias, one colourfully described by the designer Frank Chimero. “Let me let you in on a little secret,” he writes. “If you are hearing about something old, it is almost certainly good. Why? Because nobody wants to talk about shitty old stuff, but lots of people still talk about shitty new stuff, because they are still trying to figure out if it is shitty or not. The past wasn’t better, we just forgot about all the shitty shit.”

The only guarantee we have of taste is that it will change.

In a 2011 sketch on the show Portlandia, the obsessive satirical catalogue of the hipster mores of the Oregon city, an exaggeratedly posturing character known as Spyke – with “chin beard”, lobe-stretching disk earrings, and a fixed-gear bike – is shown walking past a bar. He sees some people inside, equally adorned with the trappings of a certain kind of cool, and gives an affirming nod. A few days later, he spies a clean-shaven guy wearing khakis and a dress shirt at the bar. “Aw, c’mon!” he hollers. “Guy like that is hanging out here? That bar is so over!” It only gets worse: he sees his straight-man nemesis astride a fixed-gear bicycle, partaking in “shell art”, and wearing a chin beard – all of which, he churlishly admonishes, is “over”. A year later, we see Spyke, freshly shorn of beard, wearing business casual, and having a banal conversation, perched in the very same bar that led off the whole cycle. The nemesis? He loiters outside, scornfully declaring the bar to be “over”.

The sketch wonderfully encapsulates the idea of taste as a kind of perpetual motion machine. This machine is driven in part by the oscillations of novelty and familiarity, of hunger and satiation, that curious internal calculus that causes us to tire of food, music, the colour orange. But it is also driven in part by the subtle movements of people trying to be like each other and people trying to be different from each other. There is a second-guessing kind of struggle here, not unknown to strategists of cold war–era game theory (in which players are rarely acting on “perfect information”). Or, indeed, to readers familiar with Dr Seuss’s Sneetches, the mythical star-adorned creatures who suddenly ditch their decorations when they discover their rival plain-bellied counterparts “have stars upon thars”.

That taste might move in the kind of never-ending cycle that Portlandia hypothesised is not so far-fetched. A French mathematician named Jonathan Touboul identified a phenomenon of “looking alike trying to look different”, or what he called the “hipster effect”. Unlike “cooperative systems”, in which everyone might agree in a coordinated fashion on what decisions to make, the hipster effect occurs, he suggests, when people try to make decisions in opposition to the majority.

Because no one knows exactly what other people are going to do next, and information can be noisy or delayed, there can also be periods of brief synchronisation, in which non-conformists are accidentally aligned with the majority. Spyke, in reality, might have had to see several people doing shell art – maybe it even suddenly appeared at a store in the mall – before quickly packing it in. And because there are varying degrees of hipness, one person may choose to wade into a trend later than another, that person is followed by another, and so on, until, like an astronomical explorer chasing a dead star, there is nothing really there any more. The quest for distinctiveness can also generate conformity.

The Portlandia sketch actually goes well beyond taste and illuminates two central, if seemingly contradictory, strands of human behaviour. The first is that we want to be like other people. “The social being, in the degree that he is social, is essentially imitative,” wrote the French sociologist Gabriel Tarde, in his 1890 book The Laws of Imitation. Imitating others, what is known as “social learning”, is an evolutionary adaptive strategy; that is, it helps you survive, even prosper. While it is seen in other species, there are no better social learners than humans, none that take that knowledge and continue to build upon it, through successive generations.

The sum of this social learning – culture – is what makes humans so unique, and so uniquely successful. As the anthropologist Joseph Henrich notes, humans have foraged in the Arctic, harvested crops in the tropics, and lived pastorally in deserts. This is not because we were meant to, but because we learned to.

In their book Not by Genes Alone, the anthropologists Robert Boyd and Peter Richerson use the example of a bitter plant that turns out to have medicinal value. Our sensory system would interpret the bitter as potentially harmful and thus inedible. Instinctively, there is no reason we should want to eat it. But someone eats it anyway and sees some curiously beneficial result. Someone else sees this and gives it a try. “We take our medicine in spite of its bitter taste,” they write, “not because our sensory psychology has evolved to make it less bitter, but because the idea that it has therapeutical value has spread through the population.”

People imitate, and culture becomes adaptive, they argue, because learning from others is more efficient than trying everything out on your own through costly and time-consuming trial and error. The same is as true for people now reading Netflix or TripAdvisor reviews as it was for primitive foragers trying to figure out which foods were poisonous or where to find water. When there are too many choices, or the answer does not seem obvious, it seems better to go with the flow; after all, you might miss out on something good.

But if social learning is so easy and efficient, it raises the question of why anyone does anything different to begin with. Or indeed why someone might abandon an innovation. It is a question asked of evolution itself: why is there so much stuff for natural selection to sift through? The artist or innovator who was attacked in his day seems like some kind of genetic altruist, sacrificing his own immediate fitness for some future payoff at the level of the group.

Boyd and Richerson suggest there is an optimal balance between social and individual learning in any group. Too many social learners, and the ability to innovate is lost: people know how to catch that one fish because they learned it, but what happens when that fish dies out? Too few social learners, and people might be so busy trying to learn things on their own that the society does not thrive; while people were busily inventing their own better bow and arrow, someone forgot to actually get food.

Perhaps some ingrained sense of the evolutionary utility of this differentiation explains why humans are so torn between wanting to belong to a group and wanting to be distinct individuals. People want to feel that their tastes are not unique, yet they feel anxiety when told they are exactly like another person. Think of the giddy discomfort you feel when a co-worker shows up wearing a similar outfit. We seek some happy medium, like the Miss America contestant in Woody Allen’s Bananas who responds to a reporter’s question, “Differences of opinion should be tolerated, but not when they’re too different.”

If all we did was conform, there would be no taste; nor would there be taste if no one conformed. We try to select the right-sized group or, if the group is too large, we choose a subgroup. Be not just a Democrat but a centrist Democrat. Do not just like the Beatles; be a fan of John’s.

Facebook Twitter Pinterest Illustration by Aart-Jan Venema

When distinguishing yourself from the mainstream becomes too exhausting, you can always ape some version of the mainstream. This was the premise behind the normcore anti-fashion trend, in which once energetically fashionable people were said to be downshifting, out of sheer fatigue, into humdrum New Balance sneakers and unremarkable denim. Normcore was more conceptual art project than business case study, but one whose premise – “the most different thing to do is to reject being different altogether,” ran the manifesto – seemed so plausible it was practically wish fulfilled into existence by a media that feasts upon novelty. As new as normcore seemed, Georg Simmel was talking about it a century ago: “If obedience to fashion consists in imitation of an example, conscious neglect of fashion represents similar imitation, but under an inverse sign.”

And so back to Spyke. When he felt his drive for individuality (which he shared with others who were like him) threatened by someone from outside the group, he moved on. But all the things he felt were threatened – the chin beard, the shell art – and that he was willing to walk away from, were not practical. We signal our identity only in certain domains: Spyke is not likely to change his brand of toilet paper or toothbrush just because he learns it is shared by his nemesis. When everyone listened to records on vinyl, they were a commodity material that allowed one to listen to music; it was not until they were nearly driven to extinction as a technology that they became a way to signal one’s identity – and as I write, there are stirrings of a cassette revival.

It was not until vinyl records were nearly driven to extinction that they became a way to signal one’s identity

In a revealing experiment conducted at Stanford University, Berger and Heath sold Lance Armstrong Foundation Livestrong wristbands (at a time when they were becoming increasingly popular) in a “target” dorm. The next week, they sold them in a dorm known for being somewhat “geeky”. A week later, the number of target dorm band wearers dropped by 32%. It was not that people from the target dorm disliked the geeks – or so they said – it was that they thought they were not like them. And so the yellow piece of rubber, worn for a good cause, became a means of signalling identity, or taste. The only way the target group could avoid being symbolically linked with the geeks was to abandon the taste and move on to something else. As much a search for novelty, new tastes can be a conscious rejection of what has come before – and a distancing from those now enjoying that taste. “I liked that band before they got big,” goes the common refrain.

What our tastes say about us is mostly that we want to be like other people whom we like and who have those tastes – up to a point – and unlike others who have other tastes. This is where the idea of simply socially learning what everyone else is doing, gets complicated. Sometimes we learn what others are doing and then stop doing that thing ourselves.

Then there is the question of whether we are conscious of picking up a behaviour from someone else. When someone knows he is being influenced by another and that other person knows it too, that is persuasion; when someone is unaware he is being influenced, and the influencer is unaware of his influence, that is contagion. In taste, we are rarely presumed to be picking up things randomly. Through “prestige bias”, for example, we learn from people who are deemed socially significant. The classic explanation in sociology was always trickle-down: upper-class people embraced some taste, people lower down followed, then upper-class people rejected the taste and embraced some new taste.

Tastes can change when people aspire to be different from other people; they can change when we are trying to be like other people. Groups transmit tastes to other groups, but tastes themselves can help create groups. Small, seemingly trivial differences – what sort of coffee one drinks – become real points of cultural contention. Witness the varieties of distinction now available in things that were once rather homogeneous commodities, like coffee and blue jeans; who had even heard of “single origin” or “selvage” a few decades ago?

There is an almost paradoxical cycle: an individual, such as Spyke in Portland, wants to be different. But in wanting to express that difference, he seeks out others who share those differences. He conforms to the group, but the conformists of that group, in being alike, increase their sense of difference from other groups, just as the Livestrong bracelet wearers took them off when they saw another group wearing them. The adoption of tastes is driven in part by this social jockeying. But this is not the whole picture.

In a famous 2006 experiment, groups of people were given the chance to download songs for free from a website after they had listened to and ranked the songs. When the participants could see what previous downloaders had chosen, they were more likely to follow that behaviour – so “popular” songs became more popular, less popular songs became less so.

When people made choices on their own, the choices were more predictable; people were more likely to simply choose the songs they said were best. Knowing what other listeners did was not enough to completely reorder people’s musical taste. As the scientist Duncan Watts and his co-author Matthew Salganik wrote: “The ‘best’ songs never do very badly, and the ‘worst’ songs never do extremely well.” But when others’ choices were visible, there was greater chance for the less good to do better, and vice versa. The pop chart, like taste itself, does not operate in a vacuum.

The route to the top of the charts has in theory got more democratic, less top-down, more unpredictable: it took a viral video to help make Pharrell’s Happy a hit a year after its release. But the hierarchy of popularity at the top, once established, is steeper than ever. In 2013, it was estimated that the top 1% of music acts took home 77% of all music income.

While record companies still try to engineer popularity, Chris Molanphy, a music critic and obsessive analyst of the pop charts, argues it is “the general public infecting each other who now decide if something is a hit”. The viral sensation Gangnam Style, he notes, was virtually forced on to radio. “Nobody manipulated that into being; that was clearly the general public being charmed by this goofy video and telling each other, ‘You’ve got to watch this video’.”

Today’s ever-sharper, real-time data about people’s actual listening behaviour strongly reinforces the feedback loop. “We always knew that people liked the familiar,” Molanphy says. “Now we know exactly when they flip the station and, wow, if they don’t already know a song, they really flip the station.” For the industry, there is an almost desperate attempt to convert, as fast as possible, the new into the familiar.

Simply to live in a large city is to dwell among a maelstrom of options: there are said to be – by many orders of magnitude – more choices of things to buy in New York than there are recorded species on the planet. R Alexander Bentley is an anthropologist at the University of Durham in the UK. As he put it to me: “By my recent count there were 3,500 different laptops on the market. How does anyone make a ‘utility-maximising’ choice among all those?” The cost of learning which one is truly best is almost beyond the individual; there may, in fact, actually be little that separates them in terms of quality, so any one purchase over another might simply reflect random copying.

For the Spanish philosopher José Ortega y Gasset, author of the 1930 tract The Revolt of the Masses, journalistic dispatches from explorers seemed to thrust one into a vertiginous global gyre. What would he make of the current situation, where a flurry of tweets comes even before the breaking news announcements, which then turn into wall-to-wall coverage, followed by a think piece in the next day’s newspaper? He would have to factor in social media, one has a peripheral, real-time awareness of any number of people’s whereabouts, achievements, status updates, via any number of platforms.

Ortega called this “the increase of life”. If media (large broadcasters creating audiences) helped define an age of mass society, social media (audiences creating ever more audiences) help define our age of mass individualism. The internet is exponential social learning: you have ever more ways to learn what other people are doing; how many of the more than 13,000 reviews of the Bellagio hotel in Las Vegas do you need to read on TripAdvisor before making a decision? There are ever more ways to learn that what you are doing is not good enough or was already done last week by someone else, that what you like or even who you like is also liked by some random person you have never met. This is social learning by proxy.

People have always wanted to be around other people and to learn from them. Cities have long been dynamos of social possibility, foundries of art, music, and fashion. Slang has always started in cities – an outgrowth of all those different, densely packed people so frequently exposed to one another. Cities drive taste change because they offer the greatest exposure to other people, who not surprisingly are often the creative people cities seem to attract.

With the internet, we have a kind of city of the mind, a medium that people do not just consume but inhabit, even if it often seems to replicate and extend existing cities (New Yorkers, already physically exposed to so many other people, use Twitter the most). As Bentley has argued, “Living and working online, people have perhaps never copied each other so profusely (since it usually costs nothing), so accurately, and so indiscriminately.”

But how do we know what to copy and from whom? The old ways of knowing what we should like – everything from radio station programmers to restaurant guides to book critics to brands themselves – have been supplanted by a mass of individuals, connected but apart, unified but disparate.

Whom to follow? What to choose? Whom can you trust? In an infinite realm of choice, our choices often seem to cluster towards those we can see others making (but away from those we sense too many are choosing). When there is too much social influence, people start to think more like one another. They take less information into account to make their decisions, yet are more confident that what they are thinking is the truth – because more people seem to think that way.

Social imitation has got easier, faster, and more volatile; all those micro-motives of trying to be like others and yet different can intensify into explosive bursts of macro-behaviour. The big waves have got bigger, and we know that they will come, but it is harder to tell from where, in the vast and random ocean surface, they will swell.

This is an edited extract from You May Also Like, published on 30 June by Simon & Schuster (£12.99). To order a copy for £10.39, go to bookshop.theguardian.com or call 0330 333 6846. Free UK p&p over £15, online orders only.

• Follow the Long Read on Twitter at @gdnlongread, or sign up to the long read weekly email here.