For more than a decade, we have been told that the pharmaceutical industry faces a crisis: it finds it more and more difficult to develop new drugs. The returns on research and development, company executives plead, are dismal. In the US particularly they have therefore lobbied for longer periods of patent protection, more government subsidies and less regulation of new drugs. The growing costs of research, they argue, justify the high prices. But this week a widely reported paper for the British Medical Journal, by two North American professors, Donald Light and Joel Lexchin, points out that the annual licensing of new drugs is much the same as it has been for 50 years. Unfortunately, 85-90% are minor variations on existing drugs (often introduced when patents are near expiry) and provide "few or no clinical advantages for patients".

The US drugs industry, say Light and Lexchin, spends only 1.3% of revenues (excluding taxpayer subsidies) on basic research to discover molecules that could lead to genuinely new medicines. It spends far more on maintaining profits – among the highest of any industry, after tax – and on PR, marketing and lobbying. There is an innovation crisis, but largely of the companies' own making.

For years nearly all original drugs brought to market have been based on research either at taxpayer-funded institutions, mainly universities, or in small biotechnology companies. Big companies, such as Pfizer and GlaxoSmithKline (recently fined $3bn by US regulators for aggressive and misleading marketing), are essentially rent-seekers. They do not create wealth and add social benefit, but enrich themselves through control of resources, as landowners have done for generations. And what has happened in "big pharma" – long marked down by the left, and some on the right, as an unacceptable face of capitalism – mirrors what has happened across the British and US economies. The innovation crisis is not confined to the drugs industry.

Max Levchin and Peter Thiel, co-founders of PayPal, said last year that innovation in the US was "somewhere between dire straits and dead". In his book Rise of the Creative Class (2002), Richard Florida of Toronto University argued that, while a time traveller from 1900 arriving in 1950 would be astonished by phones, planes, cars, electricity, fridges, radio, TV, penicillin and so on, a traveller from 1950 to the present would find little to amaze beyond the internet, PCs, mobile phones and, perhaps, how old technologies had become infinitely more reliable. The US economist Tyler Cowen made a similar point last year in The Great Stagnation: innovation slowed after the 1970s, he argued, and failed to create jobs. No development of the past 50 years, he could have added, bestowed benefits comparable to what washing machines and vacuum cleaners did to liberate women from drudgery.

Green energy technologies have not developed for large-scale use as was widely expected. Nor have electric cars, still less flying cars. The promise of gene therapy is unfulfilled. The development of MEMs (miniaturised pumps, levers or sensors on silicon chips) and nanotechnology – predicted as long ago as 1959 – has been agonisingly slow. In some respects, we have gone backwards: since the retirement of Concorde, the speed of air travel has slowed.

The desirability of some of these technologies is contested, but their spread has not been inhibited on social or ethical grounds. So what stops their development? Excessive government regulation and high taxes, stifling animal spirits among innovators and entrepreneurs, are commonly blamed. I see it differently. Invention and development of genuinely new, beneficial products are being stifled, as in the pharmaceutical industry, by big, established companies, not government. Just as drug companies tweak existing products, and deploy large marketing budgets to present them as new, so do other companies tweak and sometimes incrementally improve technologies that were familiar to our grandparents.

Supermarkets are full of things that claim to be "new and improved". Technologists tweak vegetables and fruits to make them last longer, look better and travel more easily, without regard to flavour. Bankers develop new trading "products" that, however you cut it, are still about borrowing and lending. We have digital radio and high-definition TV, though not everybody thinks either improves on what existed before. For many companies, skilful marketing of products that aren't significantly different from what preceded them has replaced innovation. It's cheaper and less risky to convince customers that something is ground-breaking, even when it isn't, than develop something truly innovatory.

In short, rent-seeking is now far more lucrative than innovation that delivers social benefits. The big rewards go to directors and executives of large companies – and financial traders, the ultimate rent-seekers who impose an unproductive tax on invention, investment and hard work across the world. Like the inventors of lasers and transistors, Alan Turing – without whom the modern computer wouldn't exist – and Tim Berners-Lee, inventor of the worldwide web, did not become rich. Bill Gates and the late Steve Jobs did.

Gates and Jobs recognised the possibilities of what others invented, risked money to develop them, and through marketing and design made them accessible to millions. Nobody suggests they should go unrewarded. But as Joseph Stiglitz points out in The Price of Inequality, Microsoft "did not develop the first widely used word processor, the first spreadsheet, the first browser, the first media player, or the first dominant search engine". Gates became rich because he established a near-monopoly in PC operating systems. When innovations such as the web browser appeared, he saw off their initial developers and other potential competitors by launching his own versions at below-cost prices or none at all.

Innovation always entails risk. But the bar is set higher because those who innovate or support innovation must overcome the entrenched interests of monopolistic rent-seekers, who usually have the ear of governments and regulators. Knowing this, many scientists and mathematicians put their innovatory talents at the well-rewarded service of financial companies rather than struggle in research labs. Light and Lexchin argue for better regulation of the drugs industry. We need better regulation of rent-seekers across the board if the British and US economies are to be steered towards creating genuinely new, marketable technologies that are both job-creating and socially beneficial.