Vladimir Putin will be hoping that the release of the Russian investigative journalist Ivan Golunov in response to an unprecedented campaign of public protest demonstrates his credentials as a a “good tsar” willing and able to shield his subjects from the excesses of the Russian security and police services. Golunov was arrested in a clumsy frame-up after his exposés of corruption among the Moscow elite made him too many enemies.

But most Russians understand all too well that such cases are a symptom of the degree to which Putin lets the intelligence services plunder and persecute at will. Once seen as his greatest allies, the shadowy siloviki – a distinctively Russian term for the “men of force” of the military, intelligence and security services – increasingly seem like Putin’s biggest problem. The annual St Petersburg International Economic Forum, held earlier this month and meant to showcase investment opportunities in Russia, was haunted by the spectre of the American businessman Michael Calvey,whose arrest earlier this year caused shockwaves in the international investment community. Calvey, one of a dwindling band of foreigners who remain committed to the Russian market, denies any wrongdoing and has been released, but is believed to have been charged after a complaint from a business rival to the Federal Security Service (FSB), the successor organisation of the KGB.

Meanwhile, a string of often ill-thought-through adventures, from the intervention in the Donbass in 2014, through interfering in the 2016 US elections, to the failed assassination attempt on Sergei Skripal in Britain in 2018, have left Russia under sanctions, Nato reunited, Europe rearming and the US with a Russia policy tougher now than at any point since the collapse of the Soviet Union.

From persecutions to adventurism, most of these developments do not originate with Putin, but elsewhere: in the security and intelligence community, and their friends, clients and partners in business and politics. The boss may sign off on their international shenanigans in particular, but it is worth wondering how far he lets himself be guided by them.

The most pernicious and pervasive of the myths the west believes about Putin is the Russian president’s persona as the consummate spook and spymaster. What is clear is that we seriously misunderstand Putin’s relationship with his own intelligence services. Rather than the master spy, he is rather a “spook fanboy”, too easily beguiled and bamboozled by his spymasters.

Arguably, he always has been. While still at school, he went to the regional headquarters of the Soviet KGB in Leningrad, as St Petersburg was still known then. This slab-sided building, known as Bolshoi Dom – “the Big House” – had an infamous reputation, having previously been the offices of Stalin’s secret police. Alleged enemies of the state had been executed in its blood-drenched basement rooms, but teenage Putin popped by for some career advice. A nonplussed KGB officer told him to go away and either do his military service or graduate from university first. And so he did: Putin went to Leningrad State University, graduated and joined the KGB. He would remain in the service for 17 years, a period that many feel shaped him. “I looked into Mr Putin’s eyes,” US Senator John McCain once said after meeting him, “and I saw three things: a K, a G and a B.”

By all accounts, though, Putin was a pretty mediocre KGB officer. For all the artfully crafted mythology built around him, Putin was never some Soviet James Bond. In 1985, in part a result of his good command of German, he was sent to Dresden, in communist East Germany. He was then in the KGB’s First Chief Directorate, its foreign espionage division, but he never left the German Democratic Republic and seems largely to have collated records and debriefed Soviet and East German citizens who travelled abroad. He filed reports for others to read, got plump on German beer (he admits to putting on 25 pounds) and generally lived a comfortable life.

He left the KGB just before it was formally disbanded, but still feels a strong emotional bond with the so-called Chekists (named after the Cheka, the Bolsheviks’ first political police), relying on them for support and in return granting them considerable freedom to spy, embezzle and oppress with impunity. Many of his closest allies are veterans of the KGB and its successor agencies, and though he did not coin the phrase “There is no such thing as a former KGB officer”, he certainly lives it.

Yet time and again, it is clear that he idolises a dream of the spooks rather than the reality. By his own admission, he joined the KGB because of the rich tradition of Soviet spy stories, films and television programmes. Although he briefly headed the FSB in 1998-99, he does not seem to have any deep understanding of how the services work, how raw intelligence is distilled into briefings for the leadership and, especially importantly, how they can be used to colour and shape a policymaker’s view of the world.

Vladimir Putin at the headquarters of the GRU military intelligence agency in Moscow, 2006. Photograph: Dmitri Astakhov/AFP/Getty Images

Putin typically starts his workday in the early afternoon (he is late to bed and late to rise) with a trio of leather-bound briefing files: a report from the FSB on domestic affairs, another from the Foreign Intelligence Service (SVR) on developments around the world and a third from the Federal Protective Service (FSO, his Praetorian Guard) on what is going on within the Russian elite. In other words, his first and main introduction to each day comes from his spooks.

Like so many authoritarian leaders, over time Putin has closed his mind and narrowed his circle of advisers and confidants to people who share his prejudices and do nothing to question his assumptions. As one former Russian spy told me, the intelligence agencies have learned that “you do not bring bad news to the tsar’s table”.

So they all compete to tell Putin not what he needs to hear, but what they think he wants to hear, to reassure him that everything is going well. More than that, they compete to catch his eye and win his favour with a vicious cycle of escalating claims and conspiracy theories. When the Kremlin claims that the west is trying to undermine Putin, that Canada is run by Ukrainian neo-Nazis or that a secret deep state conspiracy dominates Washington, is it just posturing, or in fact repeating eye-catching nonsense from intelligence briefings that aim to enthral rather than educate Putin?

Meddling in the US elections was hardly the only seemingly irrational policy coming from or at least sanctioned by the Kremlin. The intervention in south-east Ukraine, for example, has not only failed in its aim of forcing Kyiv to accept Moscow’s hegemony, it has galvanised Ukrainian national consciousness.

Rather than seeing these as the whims of a capricious and self-indulgent autocrat, the truth of the matter is more likely that they prove how a rational (if ruthless) leader can make dangerous and foolish policy when fed misleading information. Sometimes, it is a case of personal self-interest. Sometimes it is bureaucratic politics, as the agencies vie to win the president’s favour.

Whatever the reason, Putin, a man who, contrary to the macho maverick image, is actually rather risk-averse, appears to be taking increasing risks and allowing the spies and spooks ever greater leeway. They by no means control the Kremlin, but if they can define how their boss sees and understands the world, they don’t need to.

For the outside world this, rather than any paranoid fears about Putin wanting to reconstitute the USSR or invade Europe, is the real danger. As one disillusioned Russian ex-diplomat admitted to me, “The scariest thing about our country today is how smart people can do stupid things, if they don’t realise they have a choice.” From ruining the investment climate to prosecuting undeclared wars abroad, the question is whether the spooks are preventing Putin from realising he has choices.

• Mark Galeotti is the author of We Need To Talk About Putin, an honorary professor of UCL’s School of Slavonic and East European Studies and a senior research associate of the Royal United Services Institute