It was browsing in a bookshop that got me started. I was confronted by a bank of bestsellers on the brain: how it works and how we think. There were the books which have attracted huge attention, such as Nudge and Blink, but there were others popularising the new insights of a range of academic disciplines – social sciences such as evolutionary psychology as well as neuroscience – which are radically challenging the most fundamental assumptions on which human beings operate.

Perhaps that sounds a little overblown, but it's not. Who, dear reader, do you think you are? Do you think your mind is capable of independent judgment and largely directs the course of your life? Do you think that most of your decisions in life have been the product of your rational, conscious self? Do you believe you are in control of your life? Do you cherish ideas such as self-expression, a sense of autonomy and a distinct, self-authored identity? The chances are that, albeit with a few qualifications, most of your answers are yes. Indeed, given a pervasive culture which reinforces all these ideas, it would be a bit odd if you didn't.

But the point about this new explosion of interest in research into our brains is that it exposes as illusions much of these guiding principles of what it is to be a mature adult. They are a profound misunderstanding of how we think, and how our brains work. They are fairytales, about as fanciful and as implausible as goblins.

This is such dramatic stuff that Matthew Taylor at the Royal Society of Arts, which has pioneered public engagement with this new research, argues that we are on the verge of a new Enlightenment. He argues that the 18th-century concept of the individual self has run its course and that a new paradigm of human nature is emerging. Given that assumptions of an autonomous individual underpin every aspect of how we order society and our political economy, educate and tackle social issues, this kind of Big Idea tends to make you feel a tad dizzy.

It's not an accident that many of the biggest bestsellers in this territory are about decision-making – Blink, Nudge and The Decisive Moment. The image which comes to mind is that they are all sticks of dynamite dug in to explode the great sacred mythology of our time: namely that individual freedom is about having choices, and that progress is about the constant expansion of those choices.

Read these books and you discover that people are useless at making choices. We are lazy, imitative, over-optimistic, myopic, and much of our decision-making is made by unconscious habits of the mind which are largely socially primed. We are "not exactly lemmings, but we are easily influenced by the statements and deeds of others", according to Nudge's bleak view of human nature.

The thesis of Nudge which has attracted such keen interest from the Conservatives is that this information can be used to prime better decisions without compromising freedom of choice. Nudge has appeared to offer a neat alternative to state intervention for all those intractable areas of private behaviour – from obesity and smoking to energy use and recycling – which have such damaging consequences. It's intriguing how much attention the thesis has attracted from many parts of the political establishment, such as policymakers in pensions, health and the environment, because often the gains from nudging seem pretty small – it is fanciful to think it can solve the environmental crisis.

This humbling evidence of our hopeless decision-making exposes consumer capitalism as not being about millions of independent decisions of individuals expressing unique identities, but about how social norms can be manipulated to create eager shoppers. Or take the idea of introducing choice into public services; some bizarre consequences will result, such as the popularity of a hospital being determined by whether it has a car park, not the skill of medical staff.

There are two other areas of this new brain research which are arguably more important. First, we have much underestimated the social nature of the brain: how primed it is to recognise, interpret and respond all the time to the input of others and how that lays down patterns which govern our behaviour. We are herd-like animals who show a strong tendency to conform with group norms; what makes our brains so much bigger than other primates is this remarkable capacity for social skills such as empathy, co-operation and fairness. Instead of the old metaphor of individuals as discrete entities like billiard balls, we need to think instead of them as nodes in a relationship network.

The second area of astonishing discoveries is in the plasticity of the brain. We talk of "hardwiring" (computers have generated many misleading metaphors for the brain) but in fact, the brain can be changed. Parts of the brain can learn entirely new tricks. Neural pathways are not fixed, and even much of the damage done by deprivation in childhood can be repaired with the right circumstances of example, support and determination. We can shape our own brains to create new habits that we might have thought we were not capable of – it's a long, hard process but it is possible.

This all may seem remote from politics, but it's not. Jon Cruddas has a habit of startling audiences by arguing that the regeneration of the left requires a convincing new account of what it is to be human. Are human beings self-interested creatures or are they collaborative? The right's argument for market capitalism is rooted in the former but the research on the social brain supports the latter. Put crudely, we are social creatures with an inbuilt tendency to co-operate and seek out each other's approval and that is probably more important in determining day-to-day behaviours than narrowly conceived self-interest.

In a thought-provoking pamphlet on the implications for politics to be published early next month, Matt Grist, who runs the RSA's Social Brain project, concludes that both the right and the left have lessons to learn. The rightwing emphasis on the individual's capacity to triumph over their environment through willpower is undermined by the research which shows how childhood deprivation leaves such scarring on the brain. While the challenge to the left is to recognise that the myopic tendencies of the brain to privilege the short term has been held in check by institutions and traditions which can safeguard longer-term interests. Perhaps that requires greater understanding on the left of how such institutions operate and a revision of assumptions about why they restrict individual autonomy.

To add one more element to this potent brew of extraordinary ideas: what has been left out of the UK debate so far is how much of this new research maps on to Buddhism. In the US, a group of researchers has been involved in an ongoing dialogue with the Dalai Lama to deepen understanding of the correlations between the new research and Buddhism. Here is a system of thought which has maintained for several thousand years that the idea of a separate individual self is an illusion, which urges a set of practices to increase awareness of the processes of the mind in order to transform them and cultivate ethical habits such as compassion or courage.

Apologies if by now you are feeling giddy. This is the kind of stuff which challenges almost everything you're used to thinking about yourself.