Charlie Bruce was a Texas police chief of the old school. In more than four decades on the force he gave homegrown criminals good reason to steer clear of Del Rio, his small town on the United States's southern border, but held no grudge against the steady flow of Mexicans across the frontier in search of opportunity. He admired them for their hard work and the chances they took to better themselves. Besides, some of them built his house.

What happened on the other side of the border, in Mexico, was another matter. There, Bruce unashamedly admits that for years he used his authority as a Texan police officer to run a lucrative smuggling racket. Mostly he dealt in duty-free whisky and cigarettes shipped in to Mexico, bribing officials with tens of thousands of dollars a time to avoid taxes, and then promptly selling the contraband on to Americans who brought it back across the border.

Occasionally Bruce branched out. He laughs when he recalls the handsome profit made from exploiting a sugar shortage in the 70s by paying off an official to illegally sell him a stock of subsidised sugar sitting in a Mexican government warehouse, which he shipped to a pie-maker in Philadelphia.

Now 75 and retired to a new house a stone's throw from the border, he recounts his years as a smuggler with undisguised pride and admits that it was all made possible by being a police officer. "That's exactly why I got by with it, because I was well known over there. My shield was law enforcement. I got by with murder more than other people," he says. "Other people may think it's wrong but the border's its own world."

Bruce laughs derisively at Washington's grand scheme to change that world. In the coming weeks, the US department of homeland security expects to complete the final parts of a nearly 700 mile (1,100km) fence and wall along the Mexican border intended to curb the perpetual flow of Latin Americans in search of work, and to block the ceaseless caravan of drugs feeding a very demanding American habit. The spur, though, was 9/11 and the ever-present fear of terrorist infiltrators.

The barrier covers one-third of the US's entire southern frontier with Mexico. In parts it is a fence about 5 metres (17ft) high built of a strong steel mesh and painted the same rust colour as the surrounding earth. In some places it is topped by coils of barbed wire; in others it is a solid steel wall. The fence cuts through towns and divides the desert. Its length is patrolled by thousands of armed border, drug enforcement and FBI agents. In Arizona they are complemented by an armed vigilante militia, the Minutemen.

The remaining 1,300 miles of border will be protected by a "virtual fence" – a network of electronic sensors, cameras, towers and high-flying drones that can see for more than 300 miles – that's already in place along parts of the frontier, setting off border patrols in pursuit of figures seen scurrying across screens or picked up by the motion detectors. The whole project is costing more than $4bn (£2.6bn), with the border fence alone working out at about $5m a mile.

The barrier's supporters say it is good value for money in the face of what they portray as an onslaught of illegal immigrants – increasingly scapegoats for economic blight and unemployment as they are accused of "stealing our jobs" – drug traffickers and the threat of terrorism. Others back the fence as a means to discourage what they describe as a flood of Mexican women pouring in to the US to have "anchor babies" – children who automatically gain American citizenship by being born inside the country.

Arizona's governor, Jan Brewer, has backed the fence because, she says, her state has become "the gateway to America for drug trafficking, extortion, kidnapping and crime". Fear in the state was stoked by the death in March last year of an Arizona rancher who authorities believe was shot on his farm by a drug-smuggling scout. In December, a border patrol agent was murdered by smugglers.

A Mexican leaves the border after being deported from the US for entering illegally. Photograph: John Moore

The barrier is a popular backdrop for political campaign adverts: John McCain was pictured driving along it demanding that the government "finish the dang fence". But for many who live on the border, particularly in Texas, alarm at the prospect of a fence dividing communities has turned to derision at what is shaping up to be a spectacular – and expensive – failure.

"The only thing it's deterred is a few wetbacks [Mexicans illegal immigrants] coming in and out," said Bruce. "And it's only slowed them down because the economy's so bad there's no jobs on this side of the border. You get back to a hot economy, here they come. Goddamn you'd better get your horses out because there are going to be lots of them coming and no fence is going to stop them or anyone else."

Eagle Pass was the first US settlement on the Rio Grande. A narrow stretch of the river divides it from Piedras Negras in Mexico, whose singular claim to recognition is as the birthplace of the nacho. Residents of the two towns mostly regard them as one. For years, Eagle Pass had almost no restaurants because its population strolled across the border to eat in the cheaper establishments on the other side. Almost everyone in the town is of Mexican origin and families straddle the border, which was what made estate agent Chad Foster so unusual when he was elected the first non-Hispanic mayor of Eagle Pass in more than four decades.

"They couldn't find anyone else so they came to me," he jokes. He proved a shrewd choice in 2004 when there was a Texan in the White House, George Bush, and Eagle Pass had caught Washington's eye as a gaping hole in border security.

Foster is the kind of Texan – an imposing, hunting, bull-wrestling Republican who is rarely seen in public without his cowboy hat – who was not easily ignored in Bush-era Washington. But he didn't have anything to say that the administration wanted to hear. The department of homeland security was still planning the fence in 2006 and had latched on to Eagle Pass as a major problem. It wasn't hard to see why. The town's municipal golf course runs right up to the Rio Grande. Mexico is so close that players have little difficulty in whacking balls across the border. As it was, the real problem for golfers was to avoid hitting illegal immigrants who swam the river and scurried across the course every few minutes. "There were 200-a-day coming across," says Foster. "The Mexicans liked to cross there because they could disappear in to the town within minutes."

Washington told Eagle Pass and other towns strung along the Texas border that it intended to build a barrier on the frontier. Mayors of the towns, grouped under the Texas Border Coalition (TBC), collectively renounced any physical barrier. Officials from Washington arrived to talk to Eagle Pass's council, where opposition hardened as they were told that the fence would run through the golf course. "The number that David Aguilar, the head of border patrol, came up with is that the fence would slow down an illegal entry by three to four minutes," says Foster. "To save three to four minutes and negatively impact our community and relations with our neighbours, you erect a wall between neighbours for no real purpose?"

So far as Foster and much of Eagle Pass was concerned the illegal immigrants weren't a problem. Most got through the town as fast as they could and kept going. Others provided the agricultural and construction labour that the local residents were not prepared to do. But the US's drug tsar, Gil Kerlikowske, has said the Texas border is his greatest concern because of the level of narcotraficante, or drug trafficker, violence on the other side. The Council on Foreign Relations says that Mexico is now more violent than Afghanistan or Iraq with its 20,000-plus deaths in the government's five-year war with the drug traffickers.

Nowhere has been hit worse than Ciudad Juárez, sitting just across the river from the US town of El Paso. Juárez is arguably the most dangerous city in the western hemisphere, with about 3,000 killings there by narcotraficantes last year alone.

"It is more dangerous to walk the streets of Juárez, a few blocks from El Paso, than it is to walk the streets of Baghdad," the attorney general of Texas, Greg Abbott, told Fox News. "There is a very serious problem that is beginning to bulge at our borders and put American lives at risk."

What Abbot did not say was that whatever may be happening in Juárez, El Paso is statistically among the very safest cities in the US. There were just five murders there last year. In 2009 there were 12, still far below other US cities of a similar size.

But then there was the ace up Washington's sleeve: terrorism. When the Border Patrol moved to persuade frontier communities of the need for the fence, the first slide in the presentation was an image of the Twin Towers burning on 11 September 2001. It's an emotional and persuasive argument for many Americans. If Mexicans can breeze across the border, why can't al-Qaida? Never mind that the perpetrators of all the terrorist attacks on the US over the past 20 years have arrived by plane on student or tourist visas. Or in the case of homegrown terrorists, such as Timothy McVeigh, who murdered 168 people by blowing up a federal building in Oklahoma City in 1995, via the local maternity ward.

The government pressed ahead with its plan for the fence, regardless. Bush's homeland security secretary, Michael Chertoff, overrode dozens of federal statutes, and all state, city and tribal legislation governing everything from the environment, to property rights and historic preservation.

As the fence went up through Eagle Pass, across the river in Piedras Negras the local state government was building a "green wall" of trees on the edge of the Rio Grande in repudiation. Residents such as Guillermo Berchelmann, who used to nip across the border to buy cigarettes because they were cheaper in the US, have seen the frontier solidify with closer immigration checks, delays and an erosion of the idea of two towns united by a common border.

"I'm no homeland security strategist but in our view the fence is offensive," says Berchelmann, who runs restaurants in both towns. "People don't understand it. Eagle Pass and Piedras are one community. We intermarry. We have family there. Students from Eagle Pass come to school in Piedras Negras. We cross every day. We used to cross several times a day."

Berchelmann says he is also upset at the rhetoric echoing across the US. "I find it very disturbing. The language is offensive when they talk about anchor babies. It's their country, but are Mexicans taking away jobs from Americans? No. Those jobs were always done by Mexican immigrant workers. If you take those immigrants out, you stop agriculture cold. Corn production, wheat production, apples, oranges, you name it. That's the reality."

But residents of Eagle Pass are increasingly fearful of crossing the border as drug gang violence in its twin has risen sharply. The town's police chief was murdered last April. He had been appointed just three weeks earlier to purge the force of corrupt links to the traffickers. In the following months, the deputy police chief and three other officers were abducted.

Foster and other border mayors came to the view that the fence wasn't being built because of a serious threat to national security but to provide middle America with an illusion. "This fence is a placebo. It gives somebody in mid-America a fluffy warm feeling. It really provides no real deterrent. Look at the wonderfully engineered tunnels under the physical barriers that have been constructed in California, Arizona and New Mexico. They're backing up vehicles and climbing over. It's a very expensive joke," says Foster.

Efrain Valdez, the mayor of Del Rio and chairman of the TBC until last July, has also lashed out at supporters of the fence. "Beginning in the early 20th century, the US government has financed the construction of border fences in El Paso. In 1925, a 'hog-tight, horse-high and bootlegger-proof' barbed-wire fence was built in El Paso. Observation towers were added in 1937," he wrote last year. "As is true of the modern-day fence, the impact was minimal. The observation towers were removed under President Eisenhower, in part because of their resemblance to towers in East Berlin. In 1978, the fence between El Paso and Juárez was replaced with new "impregnable" 12ft-high metal barriers topped with barbed concertina wire. The new fence's manufacturer claimed that the wire strands of which they were made would be so sharp that anyone who tried to scale them might lose his fingers and toes. Within a week after the fences were finished, they were full of holes, some large enough to drive a truck through.

"In the past few years, the US department of homeland security has spent hundreds of millions of dollars constructing new and 'improved' fencing between El Paso and Juárez, obviously with the same ineffective results that have been evident for 85 years."

The ways around the fence have not changed much. In California, pickup trucks pull up, lean a ladder from the back to the top of the wire and the migrants are over in a minute. In the Arizona desert, smugglers have dug short tunnels under the fence. If the border patrol, farmers or Minutemen stumble on them, a new tunnel appears a few miles away. Other smugglers cut through the fence with hacksaws and blowtorches. Some have even built ramps to drive vehicles to the top of the wire and then lower immigrants down the other side.

But the fence has not been without impact. The barrier may deter some immigrants but for others it has added to the physical dangers of seeking a better life. Some now cross deeper into the desert, to remote areas away from assistance, and pay with their lives. They take to backtrails, attempt to head through mountains. Often they are ill-prepared with insufficient water or protection from the relentless sun. Their corpses are increasingly found in Arizona, where the morgue in Tucson is now overflowing with unidentified bodies. Some areas are so isolated that by the time they are found the bodies are little more than skin on bone. Some immigrants, lost and knowing what awaits them, have hanged themselves from trees.

Last year, the bodies of more than 400 suspected illegal immigrants were found in the desert, mostly in Arizona. Governor Brewer misused the rising number of such deaths to justify her popular but constitutionally questionable law requiring the police to detain anyone suspected of being an illegal immigrant. She claimed that some of the corpses were headless, murder victims of drug cartels. But when the police said no headless corpses had been found, Brewer declined to discuss her evidence and ran away from reporters pressing the question.

Foster says Americans look to the fence because they lack the will to do what really has to be done to curb the flow of drugs – reduce the market and end the sale of weapons to the Mexican cartels. "We're funding and arming the narcotraficantes, and then we blame the Mexicans," he says. "Billions of dollars Americans have spent on drugs go back to Mexico. Ninety per cent of the guns the narcos use come from the US. And our government does almost nothing about it."

The Obama administration appears to be recognising the futility of the barrier. Already it is backing away from the "virtual fence" after the US government has spent close to $1bn on the 53-mile network. An investigation by the GAO found that the electronic sensors could not tell the difference between people and small animals or large vehicles. The radar wasn't much better. It would have cost at least another $8bn to fill in the rest of the gaps.

But now the US is left with a fence and wall full of holes. Charlie Bruce, the former sheriff, smiles at the absurdity of it all. And what he sees as the hypocrisy of his fellow Americans who vilify illegal immigrants and then hire them to clean their houses, tend their gardens and build their swimming pools. Americans such as Meg Whitman, the billionaire Republican contender to be governor of California who supports the border fence and opposes amnesty for illegal immigrants, and who was exposed as employing a housekeeper working illegally in the US for nearly a decade.

"A lot of these guys from Mexico, you've heard about them being lazy. They're definitely not lazy people. They're hard-working folks. I wonder in San Antonio and Dallas and Fort Worth and Austin how they'd ever build a goddamn house without Mexicans. They built this house here. I don't know where I would be without them Mexicans helping me," says Bruce.