Within hours of the disgraced US financier Jeffrey Epstein being found dead in his cell in August, Donald Trump was retweeting a conspiracy theory about his death. Epstein hanged himself but the original tweet by the right-wing comedian Terrence K Williams implied that the Clintons had somehow been involved in his demise. Unsurprisingly, neither Williams nor Trump produced any actual evidence in support of this suggestion.

A conspiracy theory isn’t a theory like any other. The official account of the 11 September 2001 attacks is a theory about a conspiracy – an al-Qaeda conspiracy – but not a conspiracy theory. What are called “conspiracy theories” subvert received opinion and are based on the idea that things aren’t as they seem. The official account of 9/11 and the theory that the attacks were planned by the Bush administration are both theories about conspiracies but only the latter is a conspiracy theory.

Another key feature of conspiracy theories is that they tend to be highly speculative rather than based on firm evidence. Many conspiracy theories, like those over the Epstein hanging or 9/11, are not just based on flimsy or contested evidence. They are based on no evidence. They are pure conjecture, without any basis in reality.

In that case, why do people invent conspiracy theories? What do they hope to gain by peddling their theories?

Conspiracy theories are first and foremost forms of political propaganda. They are designed to denigrate specific individuals or groups or advance a political agenda. The theory that the Clintons were somehow involved in the Epstein suicide denigrates the Clintons. The notion that the US government staged the 2012 mass shooting at Sandy Hook Elementary School helped the pro-gun lobby to deflect arguments for greater gun control. What better way to pre-empt calls for greater gun control in the wake of a school shooting than to claim that it never happened?

To say that conspiracy theories are forms of political propaganda is to make a point about their actual political function. There is no need to assume that conspiracy theorists don’t believe their own theories. The deluded Sandy Hook conspiracy theorists who genuinely believe that the whole thing was a hoax will be no less effective at getting the anti-gun control message across than an insincere proponent of the same theory. Indeed, the true believer might be more effective precisely because they themselves believe it.

The sincerity of a person who believes their own conspiracy theories doesn’t mean that these theories aren’t propaganda. Whatever the conspiracy theorist’s beliefs and intentions, the actual function of their theories is to promote a political agenda, be it anti-Semitism, opposition to gun control, hostility to the federal government, or anything else. Conspiracy theories promote a political agenda in a special way: by marketing seductive explanations of major events that are unlikely to be true but are likely to influence public opinion in the preferred direction.

The political agendas of conspiracy theorists are often right wing but the extreme left is as keen on conspiracy theories as the extreme right. Hitler was a conspiracy theorist but so was Stalin. Many conspiracy theories of the extreme right and extreme left are anti-Semitic. Indeed, anti-Semitism is built into the DNA of conspiracy theories. One of the most notorious anti-Semitic conspiracy theories is the 1903 Protocols of the Elders of Zion, a fabricated text that was quoted approvingly by Hitler. The historian Norman Cohn has described how the myth of the Jewish world conspiracy helped enable the Holocaust.

Most people are consumers rather than producers of conspiracy theories. They don’t come up with their own conspiracy theories but endorse those that are already in circulation. But which ones? Research suggests that liberals in the US are more likely to be “truthers”: to believe that the Bush administration planned 9/11. In contrast, conservatives tend to be “birthers” who believe that Barack Obama wasn’t born in America.

This suggests that the specific conspiracy theories that people accept are aligned with their political outlook. Conspiracy theories about climate change are another example of this. There is evidence that such theories more likely to be endorsed by people with free-market ideologies. Why would that be? Presumably because, as committed free marketeers, they dislike the regulations that are needed to combat man-made global warming.

The idea that conspiracy theories function as political propaganda is more controversial than it should be. Many psychologists are keen on the idea that belief in conspiracy theories is largely a matter of personality. People who have a “conspiracy mentality” are more likely to believe conspiracy theories than those who don’t. That may be so, but the particular conspiracy theories that people endorse are much more a matter of ideology than psychology.

If conspiracy theories are political propaganda, that should put paid to the popular notion that they are harmless. They are as harmful and dangerous as the causes they promote. A number of these causes have been extremist, racist causes, like anti-Semitism. Conspiracy theories have a life of their own, with their own history and meaning. By endorsing conspiracy theories, one can’t help associating with the causes that these theories have traditionally promoted.

Despite this, conspiracy theories continue to be popular with people who regard themselves as politically progressive. They like these theories because they fail to understand their dark side. In addition, they see conspiracy theories as anti-establishment and a way of expressing their unhappiness with the status quo. For example, conspiracy theories about 9/11 became a way for people to express their opposition to the Iraq war.

Yet when the president of the United States is a conspiracy theorist, it’s hard to make the case that conspiracy theories are anti-establishment. When people protested against the appointment of Brett Kavanaugh to the US Supreme Court, it was the Republican political establishment that blamed the protests on a conspiracy orchestrated by George Soros. It used to be said that conspiracy theories were contrary to the official view. In the era of Trump, many conspiracy theories are the official view.

Another popular fantasy about conspiracy theories is that they hold governments to account. In reality, conspiracy theories make accountability more difficult because they allow governments to dismiss their critics as cranks. Governments do need to be held to account but the hard work of uncovering real conspiracies has been done by journalists and historians rather than by armchair conspiracy theorists.

For example, Bob Woodward and Carl Bernstein of the Washington Post uncovered the Watergate cover-up but they were investigative reporters rather than conspiracy theorists. Furthermore, their account of what Richard Nixon and his henchman were up to was based on solid evidence, including evidence from a source within the administration. It wasn’t baseless speculation, like the theory that the Clintons were responsible for the death of Epstein.

To say that conspiracy theories function as political propaganda is not to deny that there is a deep human need to find meaning in random events. The popularity of conspiracy theories doubtless has something to do with the fact that they satisfy this need. As conspiracy theorists like to say, things happen for a reason, but sometimes the reasons are much less exciting than they suppose. Opportunities to foil the 9/11 plot were missed not because of a conspiracy but because the FBI and CIA were at loggerheads.

Ultimately, the events that conspiracy theories try to explain often have no deeper meaning. The only thing they prove is that shit happens. It is conspiracy theories themselves that have a deeper meaning and that meaning is political. The best antidote is therefore to stay focused on their political agenda and keep scrutinising their motives.

Quassim Cassam is professor of philosophy at the University of Warwick. He is the author of Conspiracy Theories and Vices of the Mind: from the Intellectual to the Political.

This article is part of the Agora series, a collaboration between the New Statesman and Aaron James Wendland, professor of philosophy at the Higher School of Economics. He tweets @ajwendland.