When future historians look back on the presidency of Barack Obama, they will conclude that counterterrorism was the policy area with the biggest gap between the hopes of his supporters and reality of his actions in office.

When running for the presidency in 2008, Obama declared that he would root out al-Qaida but also close Guantánamo Bay, end torture and find a way to pursue counterterrorism policies while improving America's "moral stature in the world". He insisted that pursuing al-Qaida did not mean that the US had to go to the "dark side", as Dick Cheney famously remarked. The US could lead a campaign against al-Qaida in a manner did not present "a false choice between our safety and our ideals", he said.

While Obama has had some important accomplishments, he has failed to deliver on a comprehensive counterterrorism policy that does not undermine American ideals. Although he ended the use of torture by US personnel, his administration has refused to seek accountability for those in the Bush administration that instituted this practice. Rendition of terrorist suspects to foreign countries has continued, with the US now only receiving unverifiable "assurances" that torture will not be used. Guantánamo Bay will remain a national disgrace for the foreseeable future. In the last few months, the US has even resorted to the grotesque spectacle of force-feeding its detainees to keep them alive during Ramadan.

As I have previously argued, Obama has never reset the terms of the "war on terror" in a way which broke decisively with the past. Most of the fundamental conceptual elements of George W Bush's campaign have remained in place, though perhaps with a better sheen of respectability due to Obama's personal appeal.

Moreover, in his own way, the president has gone over to the dark side by introducing a number of dangerous policy innovations and bureaucratic precedents that will shape the foundation of America's future counterterrorism policies. He has presided over a vast increase in domestic and foreign surveillance through the NSA, launched a witch-hunt against government whistleblowers and created a secret drones program to kill terrorist suspects worldwide without public acknowledgement or due process. Under his watch, the US has expanded the field of battle in the "war on terrorism" to a dozen countries and constructed institutions and routines of surveillance and assassination – such as the so-called "disposition matrix" and "Terror Tuesdays" that will have serious consequences beyond his presidency.

What is most infuriating about these decisions is that they are partially traceable to Obama's domestic political position. Obama is shrewd enough to know that the Democrats have traditionally polled poorly on national security relative to Republicans, and that a successful attack carries the risk of derailing his domestic priorities entirely. For Obama, and for many centrist Democrats, the answer is obvious: be more hawkish than the hawks and make it hard for anyone to attack you from the right.

Once in office, Obama pivoted from his original counterterrorism positions and adopted an aggressive, but covert, policy of killing terrorist operatives abroad so that his domestic priorities could proceed unimpeded. Knowing that the American public is largely indifferent to the casualties caused in foreign lands, he gambled that a quiet, dirty war against al-Qaida would allow him to refocus on his domestic priorities and end the wars in Iraq and Afghanistan.

This short-term reasoning carries with it some serious long-term consequences.

By orienting American counterterrorism policies to favor a war in the shadows, Obama has been complicit in expanding the power of the executive branch to dangerous levels. Like George W Bush, Obama has claimed sweeping executive power that allows him to brush aside the courts, denying even their right to review basic information on his polices. His most recent Fisa disclosures – blacked out, and issued without the legal reasoning behind them – are a case in point. The result is that he has contributed to marginalizing the Congress and diminishing the powers of the courts, while pushing the US ever closer to an "imperial presidency" that considers itself beyond even the glance of the other branches of government. This accretion of executive power, which now includes the right to decide who to kill in foreign lands, will certainly be exploited by future presidents with fewer scruples than Obama.

Moreover, Obama – a liberal president, a constitutional lawyer by training – has embraced, and even strengthened, what is sometimes called the "national security state": the vast phalanx of defense and intelligence government agencies, supported by thousands of private contractors, which operate below the surface of American politics. It is this national security state which explains the continuity between presidents, in part because bureaucratic procedures and institutions set in place by previous presidents prove remarkably difficult to unravel. The recent revelations of Edward Snowden are only a glimpse into this world, but they show just how far the US government will go in violating civil liberties as long as the public wasn't watching.

Obama's election should have been an opportunity to reverse the build-up of the national security state under Bush, and to ensure strict rules on its accountability and transparency. Yet Obama chose to ally himself with these agencies, harnessing newly available data to track suspects and eliminate enemies abroad in the hopes of protecting his political position at home.

The irony is striking: a president who came into office pledging to take counterterrorism out of the shadows plunged it deeper into those shadows by expanding the remit of the national security state. In doing so, he wound up forgoing one of the central tenets of liberalism – the distrust of state power, especially when wielded in a non-transparent or arbitrary way – in a Faustian bargain with the intelligence community to hedge against hypothetical future dangers. Now, he is in the unenviable position of pointing to dubious evidence of foiled plots to justify his policy reversal.

Finally, Obama's innovations in counterterrorism have never reconciled the gap between the reality of what the US does in hunting terrorists around the globe and its wider foreign policy agenda. Despite using high-minded rhetoric about forging a long-term strategy which addresses the underlying grievances and conflicts that produce terrorism, until recently, Obama had done relatively little to tackle some of the conflicts (such as the Israeli-Palestinian conflict) that generate worldwide resentment against the US. In much of the Arab world, Obama's noted Cairo speech is seen as a litany of broken promises, testifying to the cynicism of American foreign policy in the Middle East.

In Pakistan, his frequent use of drone strikes have confirmed the narrative of al-Qaida that the US is a hyper-interventionist state willing to bomb civilians (pdf) without a second thought. No matter how at variance with reality that image is, this perception of the United States damages its credibility and cripples its efforts to win the "war of wills and ideas" that Obama describes.

The costs of Obama's counterterrorism policies will be felt for many years after he leaves office. His legacy will be that he confirmed, rather than challenged, the errors made by his predecessor and created a terrifying apparatus of surveillance and assassination that will be tempting for future presidents to use against foreign, and perhaps, domestic enemies.

His insistence that he can be trusted with this apparatus overlooks James Madison's warning that "enlightened statesmen will not always be at the helm." That short-sightedness, and the insincerity which has clothed so much of what he does in aspirational language of hope and change, will ultimately make his counterterrorism polices as disastrous as the ones he inherited.