The most significant revolution of the 21st century so far is not political. It is the information technology revolution. Its transformative effects are everywhere. In many places, rapid technological change stands in stark contrast to the lack of political change. Take the United States. Its political system has hardly changed at all in the past 25 years. Even the moments of apparent transformation – such as the election of Obama in 2008 – have only reinforced how entrenched the established order is: once the excitement died away, Obama was left facing the same constrained political choices. American politics is stuck in a rut. But the lives of American citizens have been revolutionised over the same period. The birth of the web and the development of cheap and efficient devices through which to access it have completely altered the way people connect with each other. Networks of people with shared interests, tastes, concerns, fetishes, prejudices and fears have sprung up in limitless varieties. The information technology revolution has changed the way human beings befriend each other, how they meet, date, communicate, medicate, investigate, negotiate and decide who they want to be and what they want to do. Many aspects of our online world would be unrecognisable to someone who was transplanted here from any point in the 20th century. But the infighting and gridlock in Washington would be all too familiar.

This isn't just an American story. China hasn't changed much politically since 4 June 1989, when the massacre in Tiananmen Square snuffed out a would-be revolution and secured the current regime's hold on power. But China itself has been totally altered since then. Economic growth is a large part of the difference. But so is the revolution in technology. A country of more than a billion people, nearly half of whom still live in the countryside, has been transformed by the mobile phone. There are currently over a billion phones in use in China. Ten years ago, fewer than one in 10 Chinese had access to one; today there is nearly one per person. Individuals whose horizons were until very recently constrained by physical geography – to live and die within a radius of a few miles from your birthplace was not unusual for Chinese peasants even into this century – now have access to the wider world. For the present, though maybe not for much longer, the spread of new technology has helped to stifle the call for greater political change. Who needs a political revolution when you've got a technological one?

Technology has the power to make politics seem obsolete. The speed of change leaves government looking slow, cumbersome, unwieldy and often irrelevant. It can also make political thinking look tame by comparison with the big ideas coming out of the tech industry. This doesn't just apply to far‑out ideas about what will soon be technologically possible: intelligent robots, computer implants in the human brain, virtual reality that is indistinguishable from "real" reality (all things that Ray Kurzweil, co-founder of the Google-sponsored Singularity University, thinks are coming by 2030). In this post-ideological age some of the most exotic political visions are the ones that emerge from discussions about tech. You'll find more radical libertarians and outright communists among computer scientists than among political scientists. Advances in computing have thrown up fresh ways to think about what it means to own something, what it means to share something and what it means to have a private life at all. These are among the basic questions of modern politics. However, the new answers rarely get expressed in political terms (with the exception of occasional debates about civil rights for robots). More often they are expressions of frustration with politics and sometimes of outright contempt for it. Technology isn't seen as a way of doing politics better. It's seen as a way of bypassing politics altogether.

In some circumstances, technology can and should bypass politics. The advent of widespread mobile phone ownership has allowed some of the world's poorest citizens to wriggle free from the trap of failed government. In countries that lack basic infrastructure – an accessible transport network, a reliable legal system, a usable banking sector – phones enable people to create their own networks of ownership and exchange. In Africa, a grassroots, phone-based banking system has sprung up that for the first time permits money transfers without the physical exchange of cash. This makes it possible for the inhabitants of desperately poor and isolated rural areas to do business outside of their local communities. Technology caused this to happen; government didn't. For many Africans, phones are an escape route from the constrained existence that bad politics has for so long mired them in.

But it would be a mistake to overstate what phones can do. They won't rescue anyone from civil war. Africans can use their phones to tell the wider world of the horrors that are still taking place in some parts of the continent – in South Sudan, in Eritrea, in the Niger Delta, in the Central African Republic, in Somalia. Unfortunately the world does not often listen, and nor do the soldiers who are doing the killing. Phones have not changed the basic equation of political security: the people with the guns need a compelling reason not to use them. Technology by itself doesn't give them that reason. Equally, technology by itself won't provide the basic infrastructure whose lack it has provided a way around. If there are no functioning roads to get you to market, a phone is a godsend when you have something to sell. But in the long run, you still need the roads. In the end, only politics can rescue you from bad politics.

Reading on mobile? Click here

In the developed world, impatience with politics takes another form. We don't look to technology to rescue us from failed states. We look to it to rescue us from overbearing ones. Politics in the west can appear bloated and stale. By contrast, the tech world looks dynamic, flexible and exciting. It invents stuff all the time. It is relentless in its search for what works, unencumbered by sterile political mindsets. When did a government last create anything as beneficial for the public welfare as Wikipedia? When did a bureaucracy ever invent anything as life-enhancing as Google?

It can be painful watching democratic politicians attempt to play catch-up with the new technology. They know they need to try, but often they don't know how. A few politicians have worked out how to use Twitter effectively, but most only get the public's attention when they discover new ways to make fools of themselves. The track record of many government bureaucracies in making best use of the new technology is lamentable. More taxpayers' money in Britain has been wasted on mismanaged IT projects in recent years than on anything else, including mismanaged wars (and the cost of these is driven inexorably higher by the wasteful expenditure of the military on hi-tech systems that turn out to be less efficient than the ones they replaced). There have been countless local experiments around the world in how to use the internet to promote more accountable and efficient government: online town hall meetings, interactive consultation exercises, micro-referendums. The trouble comes in knowing how to filter the results, learn from them and then scale them up. Government is not much good at any of this: it fails to pick up on what works in time to take advantage of it.

These failures help breed contempt for politicians not only among citizens but from the tech industry, which often assumes that government is simply an obstacle to be overcome, an analogue annoyance in a digital world. But there are some things the tech industry doesn't understand very well. Its blind spots include the story of its own origins. There would be no tech industry on the scale we know it today without government. This is not simply because every industry needs stable and reliable political institutions to uphold the property rights on which its dynamism depends. (Tech giants are hardly less litigious than previous industrialists, and some of them, in their voracious appetite to buy up and protect patents, are as litigious as anyone in history.) It is because government investment is what made the information technology revolution possible in the first place. The historical evidence shows that really big technological change requires vast amounts of waste. Someone has to be willing to throw huge sums of money around, knowing that most of it will be money down the drain. The foundations of the information technology revolution were laid during the cold war. It has its roots in the massive US government research and development programmes of the 1950s and 1960s. During this period most of the money spent on scientific research in the US came out of the military budget. That spending was fuelled by cold war paranoia – we've got to out-invent those crazy, ruthless Russians! – and it was enormously wasteful. But it was what made the difference. The internet began life as a military project; so too did text messaging. Of course, government didn't know what to do with these things it had created. (The US military assumed that texting would have only very limited, exclusively military, uses.) The whiz-kids of the tech industry had to step in and turn scientific innovation into marketable products: from Milnet and Arpanet to Google and Twitter. Private companies do that sort of thing much better than government does. But private companies can only do it because of the heavy lifting that's already been done by government, spending the public's money like there's no tomorrow.

The same story can be told about fracking, another transformative technology that is enriching plenty of private individuals who can claim to be supplying a public good (cheaper fuel). Whether or not it is a public good remains debatable: the environmental costs have yet to be reckoned. What is much harder to claim is that the people making the money are the ones responsible for the technology itself. The crucial innovations were a consequence of large-scale government spending on new means of energy extraction during the 1970s. The driving force behind that spending was the decade's oil crisis, starting with the Arab-Israeli war of 1973, which triggered a quadrupling of oil prices and a worldwide recession. Politicians were terrified of the possible consequences of oil scarcity: civil unrest, military weakness, social breakdown. Frightened politicians promote technological revolutions to forestall political ones. Further down the line, private investors reap the rewards.

Unfortunately, fracking doesn't herald the advent of a green technological revolution. It is simply a more efficient way of extracting hard-to-get-at fossil fuels. A big shift to green technology would take something extra: a fresh set of acute political threats to get the politicians spending our money to the extent needed to spark a fresh round of innovation. For now, the politicians are more scared of other things, including the risk of running out of the public's money. Anyone who thinks that technological innovation driven by market forces alone will solve a problem on the scale of climate change is deluding themself. Market players aren't willing to take big enough risks to effect the genuinely transformative changes. Only governments do that.

At the moment, the one government that is investing on a significant scale in green technology is China. This spending is driven by fears of popular unrest in response to the very high current levels of pollution. In many parts of China ordinary citizens have been living with filthy water and toxic air for a long time, and there are signs that they are not prepared to put up with it for much longer. The Chinese government is worried about what some future environmental disaster might do to its hold on power. But Chinese government spending will not be enough to make the difference on its own, and for now western governments do not face the same kind of fears. So they are holding off. Things will probably have to get worse before they have a chance to get better. As yet, climate change hasn't got politically scary enough: there needs to be a greater threat of violence. That's the truly scary thought.

States can do plenty of things that business organisations can't. States fight wars; Google doesn't, and not just because the company motto is "Don't be evil". Google lacks the organisational capacity and the coercive authority for war. It couldn't fight one even if it wanted to. A state – the US – put a man on the moon, another massively costly enterprise that had all sorts of unexpected technological spin-offs. Google might like to do something as ambitious, but it wouldn't dare be so reckless with its cash. (The Apollo programme cost well over $100bn in today's money; the space shuttle programme cost twice as much, or more than half the current net worth of Google.) States – thanks to their tax-raising powers – are able to pool resources to a degree that not even the biggest businesses could match.

Illustration: Luke Best at heartagency.com Photograph: Luke Best

But businesses can do plenty of things that states can't. Google has just come up with a self-driving car that actually works. It has married its mapping technology to its super-smart computers to produce a machine that performs a complex task far more safely than any human being could manage. Google's self-driving cars don't crash (so far). It is hard to imagine a government programme resulting in a self-driving car that didn't crash. Governments tend to screw up complex, open-ended tasks like that. (The mission to put a man on the moon was complex, but it wasn't open-ended: it had a straightforward, hard-to-miss target in the moon itself.) Governments don't build good cars. The hopelessly inefficient and unreliable bangers turned out by the communist states of eastern Europe – their puttering Ladas, their tin-box Trabants – are enduring testimony to that.

Most resources work best when they aren't pooled. Competition encourages diversification as well as innovation. There are limits to what markets can do, however. Champions of the free market have a tendency to extrapolate from its creative power an unjustified faith in its ability to solve any problem. Yes, private enterprise has given us the self-driving car, which may one day have the power to change the way we live. (Sit in the back, read a book, sleep, work out, make out and suddenly your daily commute becomes the best part of your day.) But that car still needs roads to drive on and rules to govern what happens there. What about the people who don't want a self-driving car, or can't afford one, or simply enjoy being behind the wheel? Who is going to manage the transition from a driven to a driverless world? Google won't do it. Government will have to.

If the self-driving car is going to become the industry norm, it will take time and it will be messy. The transport network will have to adapt, the insurance industry will have to adapt and the legal system will have to adapt (not least to decide what to do with all those people who still insist on their right to have crashes). The market may be able to take care of some of these things over time, but it won't be able to take care of all of them, certainly not all at the same time. Change on that scale is too fractious: as Hobbes said, people have an inbuilt tendency to collide. Government needs Google to build a car that really works. Google needs government if its car is ever really going to work.

'Government needs Google to build a car that really works. Google needs government if its car is ever really going to work' … Google's self-driving car. Photograph: Glenn Chapman/AFP/Getty Images

New technology has made it much easier for government to oversee what people and institutions are up to in order to check that they are not posing an unacceptable risk. Government can now spy on us in all sorts of exciting new ways: read our emails, listen to our phone calls, track our text messages, access our bank accounts. Government being government, it often does this inefficiently and cack-handedly, which only makes it more frightening. The revelation that the US government has been routinely conducting electronic surveillance on its own citizens has caused deep disquiet. Among the people who have been most disturbed are members of the tech industry. Their unease is twofold: first, techies don't like being spied on; second, they don't like not being able to prevent it. After all, it's their technology that's being abused. This puts the giants of the tech industry in a bind. They have to admit their complicity – we could have stopped it, but we didn't – or they have to admit their powerlessness – we couldn't have stopped it even if we had wanted to. Either way, it makes them look like pawns of the state.

No one likes to see politicians using technology as an instrument of control, least of all the people who invented the technology. But we have to remember the alternative to politicians controlling the tech industry: it's the tech industry controlling the politicians. Government using its monopoly power to manipulate Google is bad. But Google using its monopoly power to manipulate government would be worse. Who would you rather controlled your government: a techie or a politician? I'm afraid we're stuck with politics.

One question that has yet to be answered in China is whether the country's technocrats have the political skills to cope with all the small failures that are bound to accompany its shift to a consumer economy and world power. For now, China's technocrats remain in control (the Chinese political elite since 1989 has been dominated by engineers). They have even managed to corral the internet as an instrument of managerial politics. The Chinese state employs tens of thousands of online snoopers, whose job is to oversee the way China's citizens use the new technology and make sure it doesn't get out of hand. This is not simply an exercise in censorship and suppression (though there is plenty of that). It is also a way for the state to find out what irks its people, so as to head off those grievances before they become unmanageable. In the absence of elections, this is a valuable service. At the dawn of the information technology revolution in the 1990s there was a widespread hope that it spelt doom for authoritarians because they would not be able to control it. That is not what has happened. The internet has not democratised the Chinese state. Instead the Chinese state has used it to bypass democracy. Of course, it's not easy to control something as complex and multifarious as the web: it is a cumbersome and time-consuming business; it requires lots of money and lots of coercive power. That's why the only people who can do it are politicians.

Illustration: Andrew Park/Cognitive Media

Can they keep doing it? The information technology revolution is a long way from being over; in many respects it is only just beginning. At some point China's politicians will fail in their attempts to manage it: something will get away from them. Already China's citizens are proving adept at finding ways through and around the firewalls created by the Chinese state. They are also finding novel and subversive ways to voice their anger at the politicians, often using symbols and gestures the censors are too slow-witted to pick up on. Technology moves faster than politics. When technology escapes from political control, politicians face a choice: do they adapt to the change, or do they insist that it adapts to them? In a democracy, politicians have no choice but to adapt. In an autocratic regime like contemporary China, however pragmatic and technically adept its rulers, there is always the temptation to double down on the use of force for fear of losing control altogether. The Chinese Communist party has learned from its past failures. Nonetheless, from its own perspective, it is still too big to fail. What happens when its determination to hold on to power collides with the unruliness of the modern information technology age remains to be seen. It is the great unknown of 21st-century politics.

In the west, technocracy doesn't mean rule by engineers. It means rule by economists and financiers. But since 2008, it's been increasingly tough for financiers to assert their legitimacy as political decision-makers. Might the wizards of the tech industry have a better chance? There are signs that the disdain of Silicon Valley for the squalid world of politics is starting to dissipate. Tech giants who once thought they were above the miserable business of law-making have begun to take an interest in how laws get made. They've noticed that laws are going to get made anyway, so they might as well spend some of their vast resources trying to get the laws they want.

However, there is little sign that the people who built the new technology want to actually do politics themselves, rather than paying someone else to do it for them. They'll lobby, they'll fund, they'll campaign. Some of them will dabble with old-fashioned tools of political influence: Amazon founder Jeff Bezos last year bought the venerable but impoverished Washington Post. But, for now, few have shown any desire to stand for office or offer their personal services to the government as agents of the state.

This isn't just a story about tech. Many people retain an interest in politics – we all would like laws made to suit us – but fewer and fewer people seem interested in being politicians. It's simply not a very attractive job. In a world of myriad possibilities, especially for those who have the technical abilities that bring lavish rewards in the private sector, politics looks like a real grind. True, successful politicians get to exercise real power now and then, which must be a thrill. But most politicians are not successful: they labour away, scrabbling for votes, striving for influence, only to find that someone has beaten them to it.

Illustration: Andrew Park/Cognitive Media

The result is that contemporary politics demands an appetite for that gruelling way of life. I don't have it. Do you? The class of people keen to be involved in politics is shrinking. This is good news if you do happen to have an appetite for it. The competition is not what it was, so that a desire to get into politics is often all it takes to be given that chance. In Britain, the current crop of leading politicians is drawn from a remarkably narrow set of political careerists, most of whom have been doing politics since they were at university. Many of them were at university together.

The present British prime minister, foreign secretary, chancellor of the exchequer, education secretary, leader of the opposition, shadow chancellor and shadow home secretary were all part of the same generation of Oxford politics students. I didn't go to Oxford, but I did go to the same school as David Cameron – Eton – at the same time he did. When we were there, he was pointed out to me as someone who wanted to be prime minister. We were 16. Eton is an absurdly privileged school full of well-connected and ambitious boys, but few had an interest in politics: most wanted to be bankers or film stars. I only heard of one other who wanted to be prime minister. His name was Boris Johnson. Watching these two rise effortlessly to the top of British politics makes it hard to believe that the greasy pole is as greasy as it used to be.

However, this is not simply a Tory phenomenon. The leadership of the Labour party is also made up of individuals who are connected to each other by ties of family and education. Labour politics has been dominated in recent years by the rivalry between two brothers – David and Ed Miliband – who were brought up surrounded by politics by their Marxist academic father, Ralph Miliband. The Marxism didn't stick, but the connection with the world of politics did (both boys were mingling with the high-ups in the Labour party from their teenage years). In the future, Labour politics may be dominated by another family: the husband-and-wife team of Ed Balls and Yvette Cooper, currently Ed Miliband's nearest rivals (now that his brother has gone off to New York in a sulk). Nor is this simply a British phenomenon. Family ties play an increasingly prominent role in American politics. The next presidential election could be between the two clans who continue to dominate American political life: Hillary Clinton v Jeb Bush, the wife of one president v the son and brother of two more. People are already speculating about some future contest between Chelsea Clinton, daughter of Hillary, and George P Bush, son of Jeb. At the state level, many families have a strong hold on the top jobs, from the Cuomos in New York (the current governor is the son of a former governor) to the Browns in California (ditto). The children of politicians are as likely to become politicians as they have ever been.

The narrowing of the political class through family ties is a function of the professionalisation of politics and the increasingly high barriers to entry. Politics has become a specialised business, and the best way to get good at it is to do a lot of it. It helps to start early. It also helps to have connections with anyone who can give you a head start. At the same time, politics has become a widely despised profession. (A recent survey suggested that most American parents would rather their children did almost anything else.) So it helps to have parents or siblings who can encourage you to give it a go, notwithstanding what the rest of the world thinks. Politics is hardly unique in this respect. Lots of children end up doing what their parents did, simply for reasons of familiarity. My father is an academic sociologist, and here I am writing this piece. I don't really know why it turned out like that. I don't think it's especially sinister, though it's not very imaginative on my part. I suspect that for many politicians the situation is not much different.

Does it matter that the political class is shrinking? In one sense, no. It is a sign of broad satisfaction with the political system that most people don't want to have anything to do with politics if they can help it. However, there are real dangers to the narrowing of the political class. It opens up a gap between politicians and the rest of us, which can breed contempt both ways. If we think that professional politics is only for the peculiar people who have a interest in politics, we will start to look down on them as weirdos. Meanwhile, the politicians will start to look down on us as fools, because we don't understand the business they are in. The disdain many voters feel for professional politicians is matched by the disdain many professional politicians feel for the voters. Each thinks the other lot don't get it. As the gap grows, it becomes easier for politicians to gravitate towards their fellow elites, who at least have a respect for insider knowledge. The political network intermeshes with networks of financial, technological and military expertise, which lock the public out.

Ordinary citizens haven't given up on politics entirely. In some respects there has been a proliferation of political activity beyond the traditional outlets. As the membership of mainstream political parties has fallen away and voter turnout has declined across the western world, irregular political campaigning has expanded. Concerned individuals often coalesce around issues that reflect their own particular interests. The new information technology has been an enormous help in this regard, enabling ad hoc pressure groups to form and allowing like-minded individuals to find each other and share their concerns. But this too creates an imbalance between the political class and the rest. Professional politics is becoming more concentrated at the same time that citizen politics is becoming more fragmented. The new technology brings people together, but it also separates them out by hiving them off into online silos of political concern. The connections that are made through the new technology can be speedy, but for that reason they may also be superficial. Malcolm Gladwell wrote in 2010 that "the revolution will not be tweeted". He is probably right. Political change requires more lasting and durable connections. There is a danger that a proliferation of unconventional campaigns leaves the political class to sew up the lasting connections among themselves.

Above all there is the danger that Benjamin Constant warned against 200 years ago: we need to keep paying attention to politics however much we would prefer to be doing something else. Vigilance is the price we pay for our freedom. If we leave routine politics in the hands of a narrow group of specialists, we won't know how to take it back from them when we need it. The multiple scandals of the last few years – the banking scandal, the MPs' expenses scandal, the phone-hacking scandal, the GCHQ/NSA scandal – have tended to generate a widespread sense of powerlessness alongside the occasional spasms of fury. The political elite have been exploiting our inattention to shore up their own position. We would like to hold them to account for their temerity, but we lack the tools to do it: their superior knowledge of how politics works leaves us feeling impotent. People who think they can pick up politics when they need it often find that when they really need it they don't know where to find it. The professionals run rings round them. The only way to learn how to do politics is to keep on doing it, in good times as well as bad. We need more politics and we need more politicians.

• Politics by David Runciman is due from Profile in both print and enhanced eBook edition (eBook £4.99; paperback £8.99). It is the first in a series of "Ideas in Profile".