No myth in American history has been more powerful, more invoked by more presidents, than that of pioneers advancing across the frontier – a word that in the United States came to mean less a place than a state of mind, an imagined gateway into the future. No writer is more associated with the idea of the frontier than Frederick Jackson Turner, who, in the late 1800s, argued that the expansion of settlement across a frontier of “free land” created a uniquely American form of political equality, a vibrant, forward-looking individualism. Onward, and then onward again. There were lulls, doubts, dissents and counter-movements. But the expansionist imperative has remained constant, in one version or another, for centuries. As Woodrow Wilson, who before he was president was a colleague of Turner, said: “A frontier people always in our van, is, so far, the central and determining fact of our national history. There was no thought,” Wilson said, “of drawing back.”

So far. The poetry stopped on 16 June 2015, when Donald Trump announced his presidential campaign by standing Turner on his head. “I will build a great wall,” Trump said.

The border before Trump was no idyll. Conflict grew especially acute in California in the early 1970s. As San Diego’s sprawl began to push against agricultural fields, racist attacks on migrants increased. Vigilantes drove around the back roads of the greater San Diego area, shooting at Mexicans from the flatbeds of their pickups; dozens of bodies were found in shallow graves. Anti-migrant violence was fuelled by angry veterans returning from Vietnam, who carried out what they called “beaner raids” to break up migrant camps. Snipers took aim at Mexicans coming over the border. Led by a 27-year-old David Duke, the KKK set up a “border watch” in 1977 at California’s San Ysidro point of entry, finding much support among border patrol agents. Other KKK groups set up similar patrols in south Texas, placing leaflets with a printed skull and crossbones on the doorsteps of Latino residents, warning “aliens” and the federal government to fear the klan. Around this time, agents reported finding pitfall traps, modelled on the punji traps Vietnamese troops would set for US soldiers, in the swampy Tijuana estuary, an area of the border vigilantes began calling Little ’Nam.

Between 1988 and 1990, 100 migrants had been murdered in San Diego County. Hilario Castañeda, who was 22, and Matilde Macedo, 19, were walking along a back road when the teenage Kenneth Kovzelove, dressed in black, popped up from the bed of a passing pickup. “Die, die, die,” Kovzelove, who had just enlisted in the military because he believed that the US was about to go to war with Mexico, yelled, firing his semiautomatic rifle and killing Castañeda and Macedo. Both victims were legal residents, farmworkers with visas. “So you guys were out specifically looking for Mexicans to kill?” Kovzelove was asked in interrogation. “Yes, sir,” he replied.

The border patrol, for its part, continued being what it had been since its founding: a frontline instrument of white supremacist power. Patrollers regularly engaged in beatings, murder, torture and rape, including the rape of girls as young as 12. Some patrollers ran their own in-house outlaw vigilante groups. Others had ties with groups like the KKK. Patrol agents also used the children of migrants, either as bait or as a pressure tactic to force confessions. When coming upon a family, border patrollers usually tried to apprehend the youngest in the group first, before the others dispersed, with the idea that the rest of the party would give themselves up so as not to be separated. “It may sound cruel,” one patroller told a journalist, “but it often worked.”

Separating migrant families was not official government policy in those decades. But border patrol agents left to their own devices regularly took children from parents, threatening that they would be separated “for ever” unless one of them confessed that they had entered the country illegally. Mothers especially, an agent said, “would always break”. Once a confession was extracted, children might be placed in foster care or left to languish in federal jails. Ten-year-old Sylvia Alvarado, separated from her grandmother as they crossed into Texas, was kept in a small cinderblock cell for more than three months. In California, 13-year-old Julia Pérez, threatened with being arrested and denied food, broke down and told her interrogator that she was Mexican, even though she was a US citizen. The border patrol released Pérez into Mexico with no money or way to contact her US family.

An investigation conducted by John Crewdson of the New York Times revealed that abuses weren’t one-offs but part of a pattern, encouraged and committed by officers up the chain of command. The violence was both gratuitous and systemic, including “stress” techniques later associated with the war in Iraq. Migrants were stripped naked and placed for extended periods in extremely cold rooms. Patrollers pushed illegals off cliffs – done, a patrol agent told a journalist, “so it would look like an accident”.

The remoteness of much of the border region and the harshness of its terrain, the work that straddled the line between foreign and domestic power, and the fact that many of the patrollers were themselves veterans of foreign wars (or hailed from regions with fraught racial relations, including the borderlands themselves) all contributed to a “fortress mentality”, as one officer put it. Patrollers easily imagined their isolated substations to be frontier forts in hostile territory, holding off barbarians. They wielded awesome power over desperate people with little effective recourse.

Just as soldiers use racial epithets for the people they are fighting overseas, border patrollers had a word for their adversaries: “tonks”. Pressed by lawyers in an abuse case to say what the word meant, patroller after patroller claimed they didn’t know. Finally, one witness admitted that tonk is “the sound a flashlight makes when you hit someone over the head”.

In neighbourhoods filled with undocumented residents, the patrol operated with the latitude of an occupying army. Between 1985 and 1990, federal agents shot 40 migrants around San Diego alone, killing 22 of them. On 18 April 1986, patroller Edward Cole was beating 14-year-old Eduardo Carrillo Estrada on the US side of the border’s chain-link fence when he stopped and shot Eduardo’s younger brother, Humberto, in the back. Humberto was standing on the other side of the fence, on Mexican soil. A court ruled that Cole, who had previous incidents of shooting through the fence at Mexicans, had reason to fear for his life from Humberto and used justifiable force.

It wasn’t just the federal border patrol that engaged in such sadism, but local law enforcement as well. In 1980, a Texas lawyer affiliated with the United Farm Workers obtained videos of 72 interrogations of migrants that took place over the course of the previous seven years, recorded by the police department in McAllen, Texas. The images were disturbing: police took turns beating one handcuffed Mexican man, bashing his head on the concrete floor, punching, kicking and cursing as he pleaded for mercy. The tapes were made for enjoyment: as the officers gathered “night after night”, they drank beer and watched “playbacks” of their interrogation sessions. It was, said one of the men involved, a bonding ritual used to initiate new recruits.

This was the border in the 1980s and 1990s, a zone of lawless violence and impunity over a century in the making. For the most part, the borderlands, with all their seething racism and militarised and paramilitarised cruelty, remained apart – a world away from the American heartland. News from the border, no matter how bloody, stayed beyond the nation’s consciousness, even as Ronald Reagan once again launched the US beyond the frontier and Bill Clinton made his pitch that no line separated US interests from the world’s interests.

But awareness of the violence began to break through around 2000. Increasingly, reports of the vigilantism that had long existed but had long been ignored started drawing national attention. Witnesses began to report seeing men wearing camouflage and driving civilian vehicles, shooting and killing migrants. The body of one unidentified male was found with rope burns around his neck, as if lynched. Posses were capturing Mexicans and marching them by the score in coffles to be turned over to the border patrol. “US ranchers take up arms,” ran a headline in the Christian Science Monitor. By early 2001, the border started to attract even more white supremacists, Nazis, nativists and members of the militia groups that had, after the first Gulf war, spread throughout the midwest and west.

Then, suddenly, 9/11 interrupted this gathering of the tribes. The nation mobilised for war, first in Afghanistan and then Iraq. As it did, vigilantism declined. The attacks on the Pentagon and the World Trade Center were galvanising, giving the country a renewed sense of purpose after what many had identified as a decade of post-cold war self-indulgence. Earlier, many liberals and conservatives had pitched the North American Free Trade Agreement (Nafta) – which was ratified in 1993 and opened the border to goods and capital – as providing this purpose, as a continuation of the country’s frontier universalism, a way to resist the temptations of isolation. But free trade, or at least its implementation, is ultimately small-bore, utilitarian stuff. The terms of economic agreements – establishing the minutiae of duty schedules, for example, or defining the distinction between cellulosic and starch ethanol – aren’t matters that give meaning to national life.

And so, in the months after the terrorist attacks, the “global war on terror”, as the US’s post-9/11 campaign came to be called, offered a chance for the nation to turn away from the border and look out at the world anew. The mission was re-sanctified. “We will extend the frontiers of freedom,” George W Bush pledged in the summer of 2004.

Had the occupations of Afghanistan and Iraq not gone so wrong, perhaps Bush might have been able to contain the growing racism within his party’s rank and file by channelling it into his Middle East crusade – the way Reagan broke up the most militant nativist vigilantes in the 1980s by focusing their attention on Central America. For more than a century, from Andrew Jackson forward, the country’s political leaders enjoyed the benefit of being able to throw its restless and angry citizens – of the kind who had begun mustering on the border in the year before 9/11 – outward, into campaigns against Mexicans, Native Americans, Filipinos and Nicaraguans, among other enemies.

A construction crew works on new sections of the US-Mexico border barrier. Photograph: Mario Tama/Getty

But the occupations did go wrong. Bush and his neoconservative advisers had launched what has now become the most costly war in the nation’s history, on the heels of pushing through one of the largest tax cuts in the nation’s history. Yet news coming in from Baghdad, Fallujah, Basra, Anbar province, Bagram and elsewhere began to suggest that Bush had created an epic disaster. Photographs from Abu Ghraib prison showing US personnel cheerfully taunting and torturing Iraqis circulated widely, followed by reports of other forms of cruelty inflicted on prisoners by US troops. Many people were coming to realise that the war was not just illegal in conception but deceptive in its justification, immoral in execution, and corrupt in its administration.

Every president from Reagan onward had raised the ethical stakes, insisting that what they called “internationalism” – be it murderous wars in impoverished developing countries or corporate trade treaties – was a moral necessity. But the disillusionment generated by Bush’s war on terrorism, the velocity with which events revealed the whole operation to be a sham, was extraordinary. The war, especially the war to bring democracy to Iraq, was said to mark a new era of national purpose. And yet a coordinated campaign of deceit, carried out with the complicity of reporters working for the country’s most respected news sources, had to be waged to ensure public support.

As thousands died and billions went missing, the vanities behind not just the war but the entire post-cold war expansionist project – of more, more, more – came to a definitive end. With the frontier closed, some turned back to the border.

Sporadic violence gave way to organised paramilitary extremism. War revanchism usually takes place after wars end – the KKK after the first world war, for example, or the radicalisation of white supremacism after Vietnam. Now, though, it took shape as the war was still going on. And border paramilitarism began to pull in not only soldiers who had returned from this war but veterans of older conflicts, whose fears about the influx of migrants concerned not just the current war but all wars.

In early 2005, Vietnam vet Jim Gilchrist helped found the Minuteman Project, which began patrolling the desert looking for undocumented migrants. He later recalled that he had the idea around the time the Abu Ghraib scandal broke. “Things came out that were in my head swimming around for years,” Gilchrist said. “It was a culmination of fears building up.” He asked: “What did all these people die for in the second world war, Korea and Vietnam?” It wasn’t for open borders, to let in so many migrants that the US would “turn into a country of mayhem”.

“Hunting down Mexicans” is old sport in the US, going back at least to the years after the end of the Mexican-American war in 1848, when Mexican officials used that exact expression in their complaint to Washington about “committees of armed men” robbing and killing Mexicans in Texas. Now the hunt was nationalised. Minuteman franchises started to harass day labourers gathered on city street corners far from the border, in places such as Long Island’s East End. One midwestern detachment targeted Latinos in city parks. “The border is no longer in the desert,” the founder of Kansas City’s Heart of America Minuteman Civil Defense Corps chapter said. “It is all over America.”

By the end of 2006, according to one count, 140 Minuteman branches had been established in 34 states. At its height, the Minuteman Project alone claimed 12,000 members, many of them veterans, retired border patrol agents and other law-enforcement officers. Throughout the country, violence directed at Latinos shot up.

As Bush lost control of his occupations, he lost control of his party. Many at the time thought that modern conservatism was on the wane, done in by its own ideological excess, a contradictory commitment to a militarised national security state and libertarian economics, to its fetish of individual freedom and its stoking of the culture war, including racial grievances. Bush won re-election in 2004. But the lesson many party leaders took from the victory – as they looked at the changing demographics of states such as Arizona, Texas and Florida – was once again that Republicans, to stay viable on a national level, would have to win over Latino voters. To that end, the White House put forth legislation that would further militarise the border but also allow, for those undocumented residents who qualified, a one-time path to citizenship.

The proposed reform electrified vigilantes, who mobilised to oppose the legislation. This, in turn, revived the flagging conservative movement. A blast of nativist fanaticism helped to stay an unravelling caused by the movement’s already existing fanaticism, providing new coherence, vitality and a way forward that didn’t include citizenship for millions of undocumented residents. “The struggle turns inward,” as the historian Richard Slotkin wrote, imagining a moment when the US would no longer possess the ability of regenerating itself through frontier violence; “Wars are followed by witch-hunts.”

Vigilantes formed the core of a larger anti-migrant coalition, which included growing numbers of allies in state legislatures and the US Congress: Alabama senator Jeff Sessions, relatives of victims of crimes committed by undocumented migrants, families of soldiers killed in action, returning veterans and members of law enforcement, including border patrol agents. Leaders of the Minuteman Project appeared regularly on Fox News and talk radio to demand that Bush, instead of pushing “amnesty”, deploy the national guard to the border and build a wall along its entire length.

Donald Trump inspects border wall prototypes in San Diego, California. Photograph: Mandel Ngan/AFP/Getty

Bush tried to placate this rank-and-file rebellion by yet further hardening the border. The 2006 Secure Fence Act appropriated billions of dollars to pay for drones, a “virtual wall”, aerostat blimps, radar, helicopters, watchtowers, surveillance balloons, razor ribbon, landfill to block canyons, border berms, adjustable barriers to compensate for shifting dunes and a lab (located at Texas A&M and run in partnership with Boeing) to test fence prototypes. The number of border agents doubled, and the length of border fencing quadrupled. Agents from Immigration and Customs Enforcement (or Ice, as the post-9/11 reorganised border patrol was called) seized children off school buses and tracked undocumented residents deep in liberal states, including in the Hamptons, New York, and in New Bedford, Massachusetts. All told, throughout his eight years in office, Bush deported 2 million people. To no avail. In 2007, the party’s nativist wing killed his immigration bill.

In the last months of the Bush presidency, with the grassroots rage that had assembled on the border spreading through the nation, and the country bogged down in Afghanistan and Iraq, the housing and credit markets began to collapse. Banks failed. Mortgage foreclosures and evictions spiked. Inequality and personal debt deepened as social services stretched thin. And still, no matter how many patrollers the government put at the border, no matter how many deportations Bush carried out, Mexicans and Central Americans kept arriving. As the historian Gordon Wood said of the Jacksonian period, everything seemed to be coming apart, unravelling. The main difference, though, was that the Jacksonians looked beyond the settlement line and saw nothing but possibility; the promise of free land, and all that went with it, allowed the nation to stitch itself back up. Now the US looked out and saw nothing but peril.

And then the country elected a black man to the presidency.

A number of historians have noted that the same people who hated Barack Obama loved Andrew Jackson, described by more than one scholar as the first “Tea Party president”. That makes sense, for the intensity of the emotions stems from the same source: the frontier. Both presidents came up on the outer edge of the wave, the hither side of the nation’s outermost jurisdiction. The difference, though, is that Jackson, as a cultural symbol, represented the settlers who drove the frontier forward, who won a larger liberty by dispossessing and enslaving people of colour, a liberty that was then defined in opposition to the people they dispossessed and enslaved. Obama, the country’s first African American president, invoked their victims, and so his opponents seized on the idea that he was an alien, raised, if not born, beyond the boundary.

Obama’s election “packed an emotional wallop”, as the historian Daniel Rodgers put it. But his administration produced “only a policy whimper”, seeking to address the multiple calamities inherited from his predecessor not with radical solutions but on familiar terms. In the meantime, the nativist right continued to coalesce. Under Bush, the diverse border vigilante groups expanded nationally and helped set federal policy. Under Obama, they merged with other rightwing organisations into what became known as the Tea Party. Cross-fertilisation occurred at every level, as anti-migrant Republicans rebranded themselves libertarians and anti-Latino organisations mobilised around fiscal “responsibility”. In places like Cochise county, Arizona, long a preserve of rightwing rancher vigilantism, the Minutemen and the Tea Party merged.

“Build a wall and start shooting,” said one featured speaker at a 2010 Phoenix rally. “Line ’em up. I’ll torture them myself,” he said. Cruelty, by this point, was a way of establishing symbolic dominance over foreigners. But it was also a badge of contempt for the political establishment and all its leaders and institutions.

The wars went on, and the military, with its outsized budget, still served as the country’s most effective instrument of social mobility and provider of healthcare and education. But whereas Bush had framed militarism as an ideological struggle, Obama presented it as a matter of utility and competence. As he did so, the country lost its ability to channel extremism outward, and the kind of chaos the US had released in the Persian Gulf was increasingly mirrored at home, in an escalating spiral of jihadist massacres, mass school shootings and white-supremacist and masculinist rampages.

By 2010, the US had lost something more than the ability to vent extremism. For over a century, foreign relations had served as the arena where normative ideas about how to best organise society got worked out, where national leaders could harmonise potential conflicting interests between, say, the individual and society, or virtue and ambition. They could point outward and say that there, beyond the frontier, we’d rise above our differences.

When Clinton started campaigning for Nafta, the world – with the Soviet Union gone – was wide open, which made his insistence that free trade would lead to civic renewal sound credible and the claim that the treaty was the “moral equivalent of the frontier in the 19th century” hard to argue with.

But Obama was fenced in. The collapse of America’s moral and military authority, along with the bankruptcy of the free-trade growth model, meant that there was no aspect of foreign policy that he could use to articulate a larger vision of the common good, no realm of international relations that might help him overcome the polarisation tearing the nation apart: not war, not humanitarian intervention, not trade, and certainly not the Trans-Pacific Partnership, a mega-trade deal described by one critic as “Nafta on steroids”.

With the country unable to imagine a future moving outward, fights over the people trying to move inward grew even more intense. Here, too, Obama tried to meet his opponents halfway. He signed an executive order, the Deferred Action for Childhood Arrivals (Daca), which provided protection to some undocumented residents who had entered the country as minors. But he also increased the funding and staff of the nation’s various border, customs and immigration agencies. The White House was making the same bet Bush did, caught in the same “enforcement first” trap laid out decades ago, which insisted that the border had to be “sealed” – an impossible proposition – before reform could be passed.

Obama hoped that stepping up border security would open a space for compromise. But the situation got away from him. A “surge” of Central American children – tens of thousands every year between 2009 and 2014 – began arriving at the border, mostly from El Salvador, Honduras and Guatemala. Already hit hard by Reagan’s Central American wars, these countries were battered anew by Washington-backed trade, anti-drug and security policies. One reason the children came alone was because border militarisation had closed relatively safe crossing routes, making it too perilous for families to travel together as a group.

In response, the White House diverted more resources to try to secure the border and stepped up deportations. By 2016, the US was spending more on border and immigration enforcement than on all other federal law-enforcement agencies combined.

A group of Central American migrants, mostly Hondurans, climb a metal barrier on the Mexico-US border near El Chaparral in Tijuana. Photograph: Pedro Pardo/AFP/Getty Images

As the power of Ice and the border patrol grew, its impunity continued unabated. Since 2003, patrollers have killed at least 97 people, including six children. Few agents were prosecuted. According to a report by the ACLU, young girls have been physically abused and threatened with rape, while unaccompanied children apprehended by the border patrol experienced “physical and psychological abuse, unsanitary and inhumane living conditions, isolation from family members, extended period of detention, and denial of access to legal medical service”. It’s difficult to process this litany of abuse. The horrors blend into one another, as if the closing of the frontier has brought about a collapse of a sense of time. The violence that had been associated with moving outward in the world, which gave the illusion of leaving problems behind, now just accumulates. “We slash their bottles and drain their water into the dry earth,” writes a border patroller, describing what he and his coworkers did when they came upon a stash of supplies tucked away by hiding migrants. “We dump their backpacks and pile their food and clothes to be crushed and pissed on and stepped over, strewn across the desert and set ablaze.”

Meanwhile, as Obama reached the end of his second term, rightwing grievances continued to spin in a circle, from migrants to healthcare, from taxes, war and guns to Confederate flags, Islamic State, Mexican cartels and environmental regulations, from sharia law, energy policy and gender pronouns to Central American gangs and Black Lives Matter. And finally back to migrants, to Daca recipients and Central American children.

The nativism that rallied at the border under Bush, and that for eight years was expressed in an almost psychotropic hatred of Obama, crystallised into what some have described as “race realism”: a rejection of the legitimating premises of the liberal multilateral order – especially the idea that all could sit at the table and enjoy the world’s abundance, that the global economy should be organised around lines as open as possible, and that diversity rather than, say, Anglo-Saxonism, could serve as the foundation of political communities. The frontier was closed, resources were finite and political systems should be based on an acceptance of those facts.

Such a worldview is often expressed as instinct rather than a worked-out philosophy, and has taken many forms in the US, including a reflexive sympathy for law enforcement agencies and racial resentment. But over the past few decades, the border has provided increasing coherence to the sentiment. In July 2014, for instance, residents of Murrieta, California, just north of San Diego, took to the streets for days, waving US flags and hurling racist slurs, trying to stop buses carrying Central American children to a nearby federal facility. “We can’t take care of others if we can’t take care of our own,” one protester said, offering a concise precis of what would soon be called Trumpism. The buses were turned back – the children shunted into some other federal detention centre – and two years later, Murrieta residents, by a large margin, voted for Trump.

America’s exceptionalism was born on a frontier thought to be endless. Now the only thing endless is history’s endless return, as veterans travel to the borderlands to rehearse how lost wars could have been won. Gilchrist, one of the founders of the Minuteman Project, came home from the war in 1968. “There’s not been one day” since, he says, that he hasn’t “thought about Vietnam”. “We go out in two-man teams and we hit them like we did 40 years ago in Vietnam,” said another vigilante. Other veterans patrolling the border fought in Iraq, in either the first or second Gulf war, or in Afghanistan – or in any of the other 74 countries where the US has been conducting military operations since 2015.

Turner thought the 99th meridian, the place where the prairie meets the desiccated plains, as good a place as any to symbolically mark the frontier. Beyond this line, tenacious, inventive men started to figure out ways to irrigate dry land and began to think of history as progress, as moving forward toward an ever more bountiful future. It was here where the US became liberal and internationalist, where it learned how to “feed the world”.

The novelist Cormac McCarthy called this line the “blood meridian” and thought it signalled a different kind of boundary, across which the conceit of progress gave way to an infernal timelessness, to a land “filled with violent children orphaned by war”, where soldiers and settlers got caught in a dervish swirl, moving in circles going nowhere. That place used to be out there, beyond the frontier. But the US crossed the line so many times that it erased the line. Now the blood meridian is everywhere, nowhere more so than the border itself, a place where all of history’s wars become one war. Vigilantes often describe themselves as the rear guard of the Mexican-American War of 1846-48, standing against an enemy they believe is intent on retaking land they lost at the end of that conflict. “Mexican migrants are attempting a reconquest,” said one of the Minuteman founders, not by force but through migration.

It’s a resonant word, “reconquest”, or reconquista, and often invoked by vigilantes. The Spanish originally used the term to refer to their crusade, starting in 722 and ending in 1492 (a long war if ever there was one), to retake the Iberian peninsula from Arab and Berber Muslims. Today, the border Minutemen imagine the descendants of those Muslims coming north. They say they often spot “Middle Eastern guys with beards” and find Arabic-English dictionaries in the sand. “This is our Gaza,” one Minuteman told a researcher.

In the last years of the Obama presidency, as fallout from Iraq worsened and Central American children arrived, vigilantism surged anew in a more aggressive form. Its ranks were filled with younger, angrier men than its earlier version, outfitted with military hardware and desert camouflage, intent on stopping “fucking beaners”, obsessed equally with Isis, Central American gangs, Mexican cartels and Black Lives Matter. Most have done multiple stints in Afghanistan and Iraq. “For me, it is therapeutic to come down here and join my fellow veterans,” said one veteran, who after four tours in Iraq was left with brain injury and stress disorder.

The desert calms his nightmares. Guarding the border, he told a journalist, helps make “new memories”.

•This is an edited extract from The End of the Myth: From the Frontier to the Border Wall in the Mind of America by Greg Grandin, which will be published by Metropolitan books on 5 March. To order a copy for £20.23, go to guardianbookshop.com or call 0330 333 6846

• Follow the Long Read on Twitter at @gdnlongread, or sign up to the long read weekly email here.