When Justin Bieber collapsed last week at the O2 arena in London and was taken to a private clinic feeling "short of breath" and needing oxygen, the rumours started flying that he had had an asthma attack. They were denied by his management, but it would have been understandable if he had. Most of last week, London's air was heavily polluted, with many of the capital's pollution monitors recording "high" nitrogen dioxide (NO2) levels as an acute photochemical smog of fumes and microscopic particles (PM) of acids, chemicals, metals and dust drifted in from the continent, mixed with London diesel exhaust and then became trapped in the still, dry air.

Only a mile or so from the O2, Rosalind Dalton had also been feeling short of breath and needed her Symbicort 200 steroid inhaler. She, too, is a singer, who has been in operatic societies since she was 15, but she says she can't hold the long phrases these days. She lives near the Woolwich flyover, where a grey, 3ft-high air pollution monitoring box on a slip road to a busy road regularly shows pollution regularly well over the legal limit. Recently she was diagnosed with a long-term lung condition, even though neither she nor her family have ever smoked. "The air pollution has been bad in the last few weeks. On one occasion I set off to walk to Sainsbury's and turned back because I was having symptoms," she says.

Meanwhile, Malachi Chadwick found himself wheezing just months after he moved in 2009 from York to London to work with climate change group 10:10. He bikes around 40 miles a week in the city and his doctor has diagnosed asthma – almost certainly aggravated by air pollution. "The air quality of the two cities is noticeably different. When you bike you get [air pollution] full in the face," she said.

The last few weeks have been stressful for many of the 5.4 million people, including 1.1 million children, who are receiving treatment for asthma and for the tens of thousands of others with respiratory diseases. Since Christmas, there have been four major air pollution episodes, stretching from London to Nottingham, Birmingham, Leeds, Dundee and Glasgow. A pollution monitor in Downpatrick in Northern Ireland registered 10, the highest possible level of NO2. On 3 March, the department of the environment advised people to reduce or avoid strenuous activity and Matthew Pencharz, the mayor of London's environment adviser, said it would be "sensible" for children to be kept away from playgrounds during smog episodes.

Dr Ian Mudway, a lecturer in respiratory toxicology with the environmental research group at King's College London university, has spent several years walking the routes that children take to school in the London Borough of Tower Hamlets, measuring the pollutants in the air they breathe and determining their impacts on their respiratory health. He is shocked at the levels of pollutants these children are exposed to on a daily basis and fears for the permanent damage being done by to their lungs by the ultra-fine particles and gases emitted by diesel engines.

East London has long been heavily polluted by industry but Tower Hamlets has some of the busiest roads in Britain passing close to large high-density housing estates. Nowhere in the borough is further than 500 metres from a busy road and new housing developments targeted at young families are popping up right by main roads.

Air pollution, especially from diesel engines, is a "neglected, hidden killer" and children and old people are especially at risk, says Mudway. "There's strong evidence that if you live near main roads you will have smaller lungs," he says. "They will not reach capacity and will be stunted. When, or if, people move to a cleaner environment they still do not recover the function they lost. We have good evidence that every child born in Tower Hamlets will have a reduction in the volume of their lungs by the age of eight. The point is, people die of lung disease later on. You store up a problem that will affect you later," he says.

He lists some of the effects of polluted air. In the short term, it leads to irritation to the eyes, nose and throat, headaches, nausea, bronchitis and pneumonia. Over a longer period it can result in heart attacks and lung diseases, cancers, even damage to the brain, nerves, liver, and kidneys.

"The [people who die] are only the very end of a spectrum of health effects," he told a group of Tower Hamlet residents at a public meeting organised last month by Friends of the Earth on the extra air pollution which would be caused by a proposed new four-lane road tunnel below the Thames.

"For everyone who dies there are many more who are hospitalised or who have impaired health. Prolonged exposure to elevated [particulate pollution levels] is associated with significant life-shortening and poor respiratory health. Acute episodes can precipitate death in sensitive subjects."

The more researchers like Mudway look at the health effects of air pollution, the worse it seems to get. The latest figures suggest 29,000 people die prematurely from it every year in Britain, twice as many as from road traffic, obesity and alcohol combined, and that air pollution is now second only to smoking as a cause of death.

Its seriousness is confirmed by Asthma UK polls: "Two thirds of people with asthma have told us that traffic fumes make it worse and one third say a reduction in air pollution would make the most difference to their lives," says a spokeswoman.

After years of focusing on climate change, government and environment groups are only now slowly waking up to the public health crisis. In 2011, the House of Commons Environmental Audit Committee calculated that living in an air pollution hot spot could shave nine years off the lives of the most vulnerable people. It concluded that it cost Britain £6-19bn a year, or up to 17% of the total NHS budget, and that 15-20% more people died prematurely from it in cities with high levels of pollution than those in relatively cleaner ones.

London, with 4,300 deaths a year, is one of the worst in Europe and the pollution monitor on Marylebone Road shows the fourth highest levels of NO2 of over 2,000 monitoring stations in Europe. The city has 2,500 schools and 180,000 children within 150m of roads carrying 10,000 or more vehicles a day.

"Fighting change are a few people in government who have either failed to understand that long-term exposure to air pollution is the biggest public health risk after smoking or they simply don't care and want to cover-up the issue for as long as possible. It is much worse than most of us have realised. It is one of the biggest public health failings for decades," says Simon Birkett, a former banker who set up the campaigning group Clean Air in London (CAL) in 2009. Joan Walley MP, chair of the Environmental Audit Committee, despairs: "It's a scandal that the same number of people are dying of air pollution in London now as back in the 1950s. The Government needs to step in."

Faced with massive health costs, threatened with large fines for not complying with EU laws passed 13 years ago, and warned last week by the UN World Health Organisation that exposure to NO2 is harmful at far lower levels than the limits currently set by Europe, you might think the government would act. But Britain has spent nearly 15 years ignoring the problem, lobbying to extend timetables, working with other countries to weaken the rules and giving financial incentives for people to switch to the most polluting technologies.

Ministers admit they are breaking the laws but claim it is not possible to meet the EU limits. Mayor Boris Johnson has tried small-scale techno-fixes like living walls of plants and dust suppressants but these measures have been shown to be not nearly enough. Last week he proposed an "ultra low emission zone", which would ban all but the very lowest emission vehicles from central London during working hours. But the measure would not come into force until 2020 and was widely dismissed as PR.

The result of official inaction is that air pollution has barely improved in 20 years and legal limits for NO2 are being regularly breached in most urban areas. Government does not expect EU targets to be met until 2025 in London and 2020 in the West Midlands, Greater Manchester, Glasgow, West Yorkshire, Teesside, the Potteries, Kingston Upon Hull, Southampton and seven other conurbations. "It's a disgrace the UK is failing so badly on air pollution – tens of thousands of people die every year. Action by the government to clean up our dirty air is too little too late – and road-building plans will simply make the situation worse," said Friends of the Earth air pollution campaigner Jenny Bates.

One reason that it has been able to dodge the law is that modern air pollution is mostly invisible, colourless, odourless, and tasteless, or comes in particles so small they can pas through masks. Sixty years ago you could practically cut the coal smoke belching from chimneys. It turned buildings and clothes black, damaged crops and gave people lasting diseases. But when coal declined, the problem was assumed to have gone.

"We see the health impact today but it's difficult to take seriously because you cannot see it. The solutions involve closing roads and reducing traffic, so it's very hard for most political parties to even imagine acting," said Jenny Jones, London Green party assembly member.

These days air pollution comes largely from diesel engines. It can be best seen when fumes get trapped and a dull orange-grey smog develops. Technically, it is produced by sunlight reacting with nitrogen oxides and volatile organic compounds (VOC) in the atmosphere. When sunlight hits these chemicals, they form airborne particles and the result is ground-level ozone or smog. Overall, diesel cars emit less hydrocarbons, carbon monoxide and lead pollution than petrol cars, but produce more noxious gases and significantly more minute particles. A 2011 test by government to measure emissions from vehicles in everyday use concluded that, while petrol emissions had improved by 96%, "emissions of NOx [nitrogen oxide] from diesel cars and light goods vehicles have not decreased for the past 15-20 years.

"The pollution mix has changed over time as traffic has emerged as the predominate source. It's not only the small, nanosize of the particles, but also their changed composition and their interaction with gaseous co-pollutants that give us cause for concern. The lower levels of these particles in today's air in no way suggests they are any less harmful than the historic pollutant episodes." says Ian Mudway.

Meanwhile, there are many more diesels than before. They have increased across Europe by 35% since 1990 and, says the Society of Motor Manufacturers, over 50% of all cars registered in Britain are now diesel, up from 23% in 2002. One reason is that cities and government have offered tax incentives for diesels.

"Air pollution remains one of the most under-addressed public health problems, comparable to obesity and alcohol, but some government policies such as encouraging diesel vehicles in cities, are making the problem even worse. It is crucial that perverse incentives that encourage polluting vehicles and technologies are removed," says Conservative thinktank Policy Exchange.

Last week, ClientEarth, an organisation of activist environmental lawyers took the government to the highest court in the land over its failure to meet European laws on nitrogen pollution. The five supreme court judges, who only hear cases "of the greatest public or constitutional importance affecting the whole population", must decide whose responsibility it is to enforce European laws.

"The case raises a fundamental question about the rule of law. If the supreme court is unable to give an effective remedy to a clear and admitted breach of EU environmental law, there are grave constitutional consequences. There is now the distinct possibility that this will be referred to the European court of justice," says ClientEarth lawyer Alan Andrews.

If it is ruled that Europe should have no say in whether its laws are implemented, then the government need do nothing more and pollution will go on unchecked. If ClientEarth win, it may take Europe years to act. Either way, Malachi Chadwick, Rosalind Dalton and 5.4 million people with asthma will have to wait for respite.