Charles Smith always enjoyed visiting US troops aboard. Though a civilian, he had worked for the army for decades, helping to run logistical operations from the Rock Island arsenal near Davenport, Iowa.

He helped keep troops supplied, and on trips to Iraq made a point of sitting down with soldiers in mess halls. "I would always ask them: what are we doing for you?" Smith told the Guardian.

Smith eventually got oversight of a multibillion-dollar contract the military had struck with private firm KBR, then part of the Halliburton empire, to supply US soldiers in Iraq. But, by 2004, he noticed problems: KBR could not account for a staggering $1bn (£620m) of spending.

So Smith took a stand. He made sure a letter was hand-delivered to KBR officials, telling them that some future payments would be blocked. According to Smith, one KBR official reacted by saying: "This is going to get turned around."

A few days later, Smith was abruptly transferred. The payments he suspended were resumed. "The emphasis had shifted. It was not about the troops. It was all about taking care of KBR," he said. Eventually, Smith left the army. When he told his story to the New York Times, the paper ran an editorial. "In the annals of Iraq war profiteering, put Charles Smith down as one of the casualties," it wrote.

What Smith had blundered into is one of the most disturbing developments of the post-9/11 world: the growth of a national security industrial complex that melds together government and big business and is fuelled by an unstoppable flow of money. It takes many forms. In the military, it has seen the explosive growth of the contracting industry with firms such as Xe, formerly known as Blackwater, or DynCorp increasingly doing the jobs of professional soldiers. In the world of intelligence, private contractors are hired to do the jobs of America's spies. A shadowy world of domestic security has grown up, milking billions from the government and establishing a presence in every state. From border fences that don't work to dubious airport scanners, spending has been lavished on security projects as lobbyists cash in on behalf of corporate clients.

Meanwhile, generals, government officials and intelligence chiefs flock to private industry and embark on new careers selling services back to government.

"The creation of this whole industry is a disaster. But no one is talking about it," said John Mueller, a professor at Ohio State University and author of Overblown: How Politicians and the Terrorism Industry Inflate National Security Threats and Why We Believe Them.

Contractors form huge parts of the lines of supply for American troops. But they also fly planes, provide security and take on big infrastructure projects. Next year, as US combat troops draw down from Iraq, an estimated 5,000 private contractors will provide security on behalf of the US state department. That's a deployment roughly the size of an army brigade.

Worldwide, the ratio of contractors to US soldiers in uniform is about one-to-one. During Vietnam it was one-to-eight. It has speeded up since 9/11. "In the last 10 years, spending at the Pentagon has shifted enormously to contractors," said Pratap Chatterjee, a fellow at the Centre for American Progress and industry expert.

That shift has a cost. Incidents of malpractice and fraud by contracting firms – whom critics call modern-day mercenaries – are legion. One of the most infamous was the Baghdad battle involving Blackwater that saw 17 Iraqi civilians killed in 2007. But that was just one event. An AP investigation in 2010 looked at incidents involving more than 200 contractors worldwide that ranged from drinking to sexual misconduct to a gunfight outside a nightclub in Haiti.

Oversight of contracting is weak or opaque – and is often contracted out, too. One recent investigation found $4.5bn of contracts awarded to firms with a history of problems or which had violated laws. A federal audit found an oil firm had overcharged the Pentagon by $204m for fuel in Iraq.

A similar process has hit the intelligence world. An industry has sprung up of recruitment firms that service the intelligence community. A search on website IntelligenceCareers.com – headed by former army intelligence officer William Golden – found highly paid jobs in Iraq and Japan as well as all over the US. Many required "top secret" security clearance. Of those private firms with a top secret clearance, more than a quarter came into being since 9/11.

The benefits are obvious to employees. Pay is higher and some companies have offered sign-on bonuses or free cars. It is estimated that contractors from more than 100 firms make up a third of the CIA.

And as the rest of America suffers recession, this is an economic boom. The US intelligence budget last year was $80bn, more than twice 2001 levels.

When CIA agent Raymond Davis was arrested in Pakistan after shooting dead two men in Lahore it caused a huge diplomatic spat. But what went mostly unrealised was that Davis, engaged to work in one of the world's most dangerous places, was a contractor.

Those cashing in on the international "war on terror" pale beside the security boom that is taking place in the US itself. Across America, new organisations sprang up in the wake of 9/11 as the flow of money was turned on. Nine days after the tragedy, Congress committed $40bn to fortify America's domestic anti-terror defences. In 2002, the figure was a further $36.5bn. In 2003 it was $44bn. More than 260 new government organisations have been created since 2001. The biggest of all is the Department of Homeland Security, whose workforce is 230,000-strong and awaiting new headquarters in Washington, which will be the biggest new federal building since the Pentagon. It is rising up on the grounds of a former asylum.

Since 2003, the DHS has distributed more than $30bn to state and local governments to spend on security and counter-terrorism. The Washington Post last year produced an exhaustive survey called Top Secret America. It revealed there are now 1,271 government organisations and 1,931 private firms related to counter-terrorism, intelligence or homeland security in some 10,000 locations around the US. In the Washington DC area, they have built enough new office space to add up to three Pentagons. A startling 854,000 Americans now hold top secret security clearances, around 250,000 of them in the private sector.

Among the things created are Fusion Centres. There are around 70 of them across the US, set up to allow local law enforcement to collect data on "suspicious activity". Critics claim there is little for them to do, and point to mistakes. The North Texas Fusion Centre, for example, once issued a warning of a "significant" threat posed by a harmless local Muslim civil rights group. Civil rights activists have also complained they threaten data privacy laws and bring in military and corporate interests into policing civilians. "It is odd to create a whole intelligence apparatus in a decade. One that exists only because of bureaucratic inefficiency and not national security needs," said David Rittgers of the Cato Institute. What security impact does he think they have had? "None. We could abolish Fusion Centres tomorrow. Any intelligent analysis would show we don't need these things."

Airport scanners are another big business that has sprung up since 9/11. Tech firm L-3 has won nearly $900m in business from the Transportation Security Administration. L-3's full-body scanners, which cost around $200,000 each, are being rolled out across the US, boosted by the unsuccessful terror attack in 2009 by a would-be plane bomber who put a device in his underwear. Yet a report by the Government Accountability Office found that it was "unclear" if scanners could have detected the bomb. "They have vastly overbuilt the airport security complex," said Chris Calabrese, a lawyer with the American Civil Liberties Union.

There is precedent though. Prior to the body scanners, so-called "puffers" were tried. They used a gust of air to detect particles on clothing. After spending $20m, the TSA abandoned the scheme in 2007. Though that pales in comparison with the $1bn given to Boeing to build a hi-tech wall on the border with Mexico.

That scheme, created in part out of a fear of terrorists crossing over, was scrapped with just 53 miles of it built.

One of the main reasons for the ongoing gold rush in national security lies on "K Street" in Washington, the busy downtown thoroughfare is where many lobbying firms have their headquarters, clustered around the fountain of national security funding like bees around a hive. "Wherever there is government money in that amount, there is going to be a swarm of lobbyists. They are very active players," said Michael Beckel, a researcher at the Centre for Responsive Politics watchdog.

Spending on lobbying by the scanning industry has doubled in the past five years. Lobbyists for the industry pull out all the stops, too: they once rolled scanners into the Capitol in a bid to persuade lawmakers of their usefulness.

But more important than the millions spent by businesses involved in national security is who is doing the lobbying. The border between lobbying firms and the government departments they seduce is called simply the "revolving door". A startling eight out of 10 lobbyists for the scanner industry come from a previous career in government or congress.

Among firms specialising in intelligence work, the picture is the same. Booz Allen once hired James Woolsey, a former head of the CIA. Northrop Grumman hired former NSA director William Studeman. CACI hired Barbara McNamara, an NSA deputy director. It is the same in the military, where defence firms snap up retiring generals as advisers and lobbyists. Between 2004 and 2008, a startling 80% of retiring three- and four-star officers went to work in the private sector.

All of this could perhaps be understood on grounds of cost. After all, much of the original justification for contracting and bringing the private sector into national security was that it was cheaper. But this is not true. A 2008 survey of the DHS found that contractors made up 29% of staff, but some 49% of the budget. "By all means spend money on national security, but spend it wisely," said Mueller.