I am fascinated by the group David Cameron has set up in No.10, called The Behavioural Insights Unit. I think it is evidence of a massive shift that is just beginning in British politics which will change the way politicians govern and manage the rest of us.

Tony Blair believed in a consumerist idea of democracy. He used focus groups to try and find out what people wanted as a way of shaping policy (except, of course, over Iraq). Like Mrs Thatcher, he believed that the people knew best. They expressed their desires and wants clearly through the market. And politics, he believed, should imitate this.

The Behavioural Insights Team believe the opposite. That in many cases you can't trust the people. That if you let them just follow their desires they will often do things that are bad both for themselves and for society.

This doesn't mean you get rid of the market. Instead governments should find ways to manipulate ordinary peoples' feelings and desires so they "choose" to do the right thing.

Behind this are the ideas of what is called Behavioural Economics. They were popularised by a book called "Nudge" written by Richard Thaler and Cass Sunstein.

The idea of "nudging" citizens to do the right thing sounds cute. But in reality it marks the return of a powerful psycho-political theory that rose up in the mid-20th century. It was called Behaviourism. And it was hated by both the right and the left.

Behaviourism's most famous exponent was an American psychologist called B. F. Skinner who was an idealist and a utopian. He believed that his techniques of behaviour modification could be used to create a completely new kind of world.

In the 1960s and 70s Skinner became a controversial figure. Students in America and Britain protested wherever he spoke.

The reason was that Skinner showed just how easy it was to manipulate and change human behaviour. He called it "operant conditioning". Skinner used pigeons to demonstrate how you simply "reinforced" the behaviour you wanted with rewards.

And humans, Skinner said, are just like pigeons.

Here is film of Skinner with his pigeons showing how it is done - and explaining how operant conditioning can be used in human society. It was shot in 1968.

In order to see this content you need to have both Javascript enabled and Flash installed. Visit BBC Webwise for full instructions. If you're reading via RSS, you'll need to visit the blog to access this content.

The Downing Street unit uses a lot of language from contemporary brain science but their fundamental ideas come from Skinner's pigeons.

The key figure is Richard Thaler, one of the authors of Nudge. He has been powerfuly influenced by behaviourist theories including the work of a psychiatrist called George Ainslie.

Ainslie did operant conditioning experiments in Skinner's pigeon lab in the 1970s. He showed that the pigeons could be trained to choose deferred gratification - by making them realise they could get even more food if they waited.

Drawing on these and other behaviourist ideas Thaler wrote a paper in 1981 with a great title - An Economic Theory of Self-Control.

This is what lies behind the Downing Street unit's plans to find mechanisms to manipulate people so they will do "good" things - like save more for retirement or eat less bad food.

Skinner himself was acutely aware that modifying human behaviour in these ways raises serious political questions. Not just about individual freedom, but about who decides what is "good" behaviour, and what happens when others decide it is bad.

These are questions that the Nudge enthusiasts seem to be blithely unaware of.

But, despite the potential dangers, Skinner was convinced that his operant conditioning should be used by politicians to completely reshape societies in a good way. It would be a giant experiment in behaviour modification - "a gigantic effort of self-control".

Skinner ran experiments with humans to show this. Here is film of one experiment in a mental hospital where patients' behaviour is being modified. Followed by Skinner talking about the political implications.

In order to see this content you need to have both Javascript enabled and Flash installed. Visit BBC Webwise for full instructions. If you're reading via RSS, you'll need to visit the blog to access this content.

Skinner's guiding belief was that you completely ignored what went on inside the minds of human beings. The thoughts and feelings that went on inside the "black box" - as he called it - were unmeasurable and ultimately unknowable.

Instead you observed human behaviour from the outside. That was the only reality that could be scientifically described.

In this way Skinner was part of a much greater tradition that has been forgotten in our age of individualism. An age where our feelings are the most important thing.

But from the 1930s through to the end of the 1950s Sociology and Social and Market research were dominated by a behaviourist approach that observed, quantified and categorised human behaviour scientifically into groups whose behaviour could then be predicted.

Two powerful men helped shape this movement in the late 30s in America. One was a psychologist called Frank Stanton, who later ran CBS television. The other was a social scientist called Paul Lazarsfeld.

Stanton was fascinated by behaviourist psychology - and the two men developed all kinds of ways to measure and categorise human behaviour as mass consumerism developed.

Here is a lovely photograph of the two men with the machine they invented called The Program Analyzer.

The machine was given out to thousands of television viewers. The viewers sat with a green button in their right hand and a red one in their left. They pressed the green if they liked the programme and the red if they disliked it. The buttons connected to a pen that drew out their responses on a moving paper tape.

It measured their deviation either side of what was called "The Apathy Line", the straight line down the middle that resulted if no buttons were pressed.

Pigeons watching TV.

Getty Images/CBS Photo Archive

Lazarsfeld had an enormous influence on a British market researcher who I am fascinated by.

He was called Mark Abrams.

In the 1950s Abrams did as much as any politician to shape the way British society thought about itself. This is because he invented what is called the Social Grades system. It divides people into six categories - A, B, C1, C2, D and E.

Abrams originally developed the Social Grades for the National Readership Survey - so publishers of newspapers and magazines could objectively classify their readers.

But the six types quickly became far more than that. They worked their way deep into the national psyche, shaping the way people looked at themselves and their relationship to others.

Here is a great bit from a programme made in the mid-1960s called This Is Marketing - Know Your Consumer. The presenter uses three-dimensional environments to explain the different categories.

In order to see this content you need to have both Javascript enabled and Flash installed. Visit BBC Webwise for full instructions. If you're reading via RSS, you'll need to visit the blog to access this content.

The A, B, C1, C2, D and E categories were created by Abrams as scientific, objective ways of dividing the population into groups.

But that is not the way television-makers saw them.

Here are two sections from a documentary made in 1966 called Not in My Class, Dear which is all about the social grades. It is fantastically patronising and snobbish, and automatically links the categories with social class.

It shows how, as so often with the social sciences, what appear to be objective categories of behaviour and measurement seem to fit very neatly with the pre-existing power structure. And reinforces it.

The Nudge enthusiasts may not realise it - but this may be the reason why they have become so popular in Downing Street.

Here is a wonderful bit of two women market researchers judging what category people are as they come down the steps from St Pancras station.

And, although they insist "class is never mentioned", the researchers are just as bad underneath as the film-makers – "ghastly plastic flowers".

In order to see this content you need to have both Javascript enabled and Flash installed. Visit BBC Webwise for full instructions. If you're reading via RSS, you'll need to visit the blog to access this content.

The programme-makers then went and found people who fitted into the different social grades and asked them what they thought.

Here are four of them.

Mr Ryan – a tunneller on the Victoria Line – a C2

Mr Langrish – a prep-school master – a C1

Mr Tennant – a Lieutenant in the Royal Horse Artillery – a C1 on the way to becoming a B

The Duffels – a "with-it" couple in Islington – "A's beyond all question"

As you watch the people in the film, you realise that the social categories are also far from neutral in another way.

In all the interviews the people have been told they have been selected as representatives of a particular category. And, as they talk, you can see them helpfully describing their behaviour in ways so it fits with that category

What you are watching is another example of B F Skinner's "operant conditioning" at work. By defining people as members of social classes they become more like that. More like each other.

In order to see this content you need to have both Javascript enabled and Flash installed. Visit BBC Webwise for full instructions. If you're reading via RSS, you'll need to visit the blog to access this content.

All this seems now to be a lost world.

What replaced it was a different kind of theory of how to analyse behaviour in society. Its roots lay in Freudian psychoanalysis and radical psycho-therapy of the 1960s and its practicioners wanted to go inside the heads of consumers and voters and find out what they wanted - and give it to them. (It is the story I told in the Century of the Self)

Out of this came Lifestyle Marketing. This still grouped and segmented people - but according to their inner desires, wants and feelings. Our inner states of mind became the central focus of the age.

Both Paul Lazarsfeld and Mark Abrams were deeply suspicious of all this. They thought much of it was psycho-analytic charlatanism. But it was a losing battle. In the 1960s both the counter culture on the political left, and the promoters of a market based democracy on the right, said that the individual and their inner feelings were the motor of society.

And behaviourism was dead.

But now it is back - and not just in Downing Street. I have a suspicion that the politicians' revival of the old behaviourist ideas and techniques will be helped and reinforced by a powerful ally - the machines we have built.

The computers.

In our age of individualism we see computers as ways through which we can express our individuality. But the truth is that the computers are really good at spotting the very opposite.

The computers can see how similar we are, and they then have the ability to agglomerate us together into groups that have the same behaviours. And from that they can predict what choices and decisions we will make.

And they do it solely through our observed behaviour.

In 1964 B F Skinner wrote a utopian vision of the future called Beyond Freedom and Dignity. It argued that the idea of individualism was actually a terrible prison.

By following only our own inner desires and feelings human beings limit themselves, Skinner said, to a narrow, dessicated existence. Your feelings are not good, in reality they are horrible little demons that live inside you and possess you.

Skinner said that his techniques of "operant conditioning" could free human beings from those demons. And their behaviour would be modified so as to engineer a completely new kind of society. It was not fascism. It was a new kind of liberation.

And it is only our belief in our individual authenticity that is holding us back.

I leave the last word to the 1960s techno-theorist, Lewis Mumford, interviewed in 1968 about B F Skinner's utopian vision.