Street protests and urban riots pose special challenges to modern politicians. All rulers stand for order. Their natural reaction, faced with an angry crowd demanding change, is to turn to the police. Napoleon Bonaparte, driving his opponents off the streets with his whiff of grapeshot, is the one they all still want to copy when the mob is on the march, whether in Paris or Ferguson, Missouri.

True, not all politicians would be as indifferent to the grievances of the streets as Margaret Thatcher, who responded to the Toxteth riots of 1981 with a heartfelt cry of “Oh, those poor shopkeepers!” and who told the Commons a few days later that “Nothing, but nothing, justifies what happened.”

Nor are there many who would have the sheer combative audacity that Franklin Roosevelt displayed in 1936. “They are unanimous in their hate for me,” Roosevelt said of the New Deal’s many militant enemies. “And I welcome their hatred.”

Today’s politicians learn to be less partisan. They operate in a less confrontational and more convergent political culture than that of the past. They are programmed to reach across barricades, not to erect them. The former Liberal leader Jo Grimond once called on his party to march towards the sound of the gunfire. Few modern politicians would say a thing like that.

Barack Obama is a politician of today, not a politician of yesteryear. His response to the real gunfire in Ferguson has nevertheless surprised many people. Faced with a raw racial injustice, the killing of Michael Brown, followed by what is now 11 nights of tension and protest in Ferguson, many expected the US president to march towards the sound of the gunfire too, with a great declaration on race.

But Obama did no such thing. Instead, he stayed on vacation on Martha’s Vineyard, in Massachusetts. Finally, with manifest reluctance, he left the island. But he did not go to Ferguson. When he spoke about it, his tone was remarkably downbeat and dispassionate. The man who was once the great communicator could be heard talking of “the need to foster conversations among local stakeholders.” If he displayed a flicker of animation about the subject, you can be forgiven for having missed it. Of the new “race speech” that so many craved from Obama there was no sign.

What explains this? Admittedly, no one can step into the same river twice. And the death of Michael Brown may not compare for historic moment with the assassination 46 years ago of Martin Luther King. But Obama’s instinct to turn away from confronting Brown’s death highlights the contrast between how an eloquent politician might react to a crowd at a charged moment half a century ago, and how an eloquent politician might react today.

On the evening of 4 April 1968, little more than an hour after King’s killing, Robert Kennedy gave a spontaneous and simple speech in a black neighbourhood in Indianapolis. Police warned Kennedy not to go into the ghetto and provided him no protection. Kennedy gave the crowd the news about the assassination. He talked about the need for love, justice and compassion. He even quoted Aeschylus. It is an electrifying paragon of a political speech. This is what Joe Klein wrote about it in 2006:

“Nearly 40 years later, Kennedy’s words stand as a sublime example of the substance and music of politics in its grandest form, for its highest purpose – to heal, to educate, to lead – but also, sadly, they represent the end of an era: the last moments before American political life was overwhelmed by marketing professionals, consultants, and pollsters who, with the flaccid acquiescence of the politicians, have robbed public life of much of its romance and vigour.”

Obama was supposed to restore all that. It was one of the reasons people voted for him, and why they then voted for him again. This week was a moment when he might have done what Kennedy did in 1968 or what Obama himself did when he spoke so powerfully on race during the nomination campaign in 2008. Instead, Obama talked about conversations between local stakeholders.

Three main explanations seem to combine to create this significant presidential failure. The first, much discussed and puzzled over in the US this week, concerns Obama himself. He has always had a tendency to be aloof from other politicians. He has consistently placed himself above partisan disagreement. He wants differences to be sorted out, but he does little to ensure that this happens. The result is that, from healthcare to Iraq, he doesn’t get what he wants because he fails to make the necessary efforts to mobilise support for it.

On Monday, the New York Times reported a telling anecdote about what many are calling this increasing retreat into isolation. In June, Obama and four congressional leaders from both parties met to discuss Iraq. But the Senate Democratic leader, Harry Reid, used the opportunity to try to enlist Obama’s support against Republican Mitch McConnell’s effort to block confirmation for several of the president’s ambassadorial nominations. “You and Mitch work it out,” Obama told Reid curtly. The impression that the president is not willing to demean himself by doing politics is thus only feeding the gridlock he so deplores.

The second explanation lies in the practices of modern politics itself. Politicians of the Roosevelt, Kennedy or Thatcher eras could take risks and stands that modern politicians, fearful of the sleepless 24/7 media and the polls, and with fewer battalions of supporters at their command, are simply less equipped to take. It is easy to blame today’s leaders as somehow smaller figures than those of the past. But this is unfair. The truth is that not even the most technically skilled modern politicians such as Tony Blair or even Obama can plunge into an issue with quite the same confidence that earlier generations would have had that others will follow them. Modern politicians know how to handle a media interview or a focus group finding. But they do not know what to do when confronted with a lot of people with a shared grievance.

The final explanation is US presidential decline. Roosevelt created the imperial presidency; Kennedy spent his too short political life confident that it still existed. But Obama knows it no longer does. The US president remains uniquely powerful. But he cannot bring order to Iraq, peace to the Israel-Palestine conflict, defend Ukraine’s borders, get his nominations through the senate, or get jobs and skills to the young black men in places like Ferguson, whose route into the middle-class seems as firmly barred as ever.

Obama has all the grandeur of office. But the power, little by little, is beginning to slip away. The signs from this week are that he knows it.