Remember Aidan Burley? He was the Conservative MP for Cannock Chase between 2010 and 2015. You might recall that he described the opening ceremony of the 2012 Olympics as "leftie multicultural crap". Or that he was sacked as a parliamentary private secretary to Justine Greening, then transport secretary, after he was photographed at a Nazi-themed stag party in 2011.

I thought of Burley yesterday, when I saw a picture of Desmond Swayne in blackface. Last week, Swayne, the Conservative MP for New Forest West, inexplicably chose to respond to the controversy over Justin Trudeau's habit of wearing racist make-up by volunteering that he had once done so himself. Today's Telegraph carries a picture of him dressed as an “authentic” (his adjective) James Brown. Blackface, he says, is an “entirely acceptable bit of fun”. He has refused to apologise, and only rules out a repeat performance on the grounds that it is too difficult to clean oneself afterwards.

None of Swayne's colleagues are particularly surprised by his apparently pre-decimal attitude to blackface, peculiar though it is that he has chosen to advertise it publicly. “He's the type of old colonial Tory that still pronounces it Keen-ya,” says one Hampshire neighbour. But a line on his CV means this bizarre and unedifying episode merits a little more reflection that it otherwise might. At the time of Burley's sacking from government, Swayne was David Cameron's PPS.

The entire point of Cameron's leadership was to detoxify the Conservative Party in the eyes of the public. Doing so meant – in theory, if not always in practice – having no truck with unreconstructed bigotry, or, indeed, anything that might make the Tories look like cartoon baddies. That very much included unapologetic, jolly hockey sticks, upper-sixth revue racism. It was a difficult process for a party whose old guard were, by and large, small-c conservatives. Swayne himself said so in a sober and reflective turn in The Cameron Years, the BBC documentary on his old line manager's time in Downing Street. He defended Cameron's decision to legislate for same-sex marriage as brave and necessary, despite the objections of the Tory backbenches and grassroots. Watch it: afterwards, you won't find it particularly hard to imagine him explaining why it had been necessary to sack a junior minister for wearing blackface.

But six years on from that vote, and three years on from Cameron's exit, Desmond Swayne is instead defending his own right to wear blackface – without sanction from the Conservative leadership. It is difficult to imagine someone getting away with this under Cameron. Or, for that matter, Theresa May. But then we know that this administration doesn't really care about its standing among the voters Cameron made it his mission to attract. If it did, its Brexit policy and its cabinet would look rather different. Jacob Rees-Mogg wouldn't be within a 100-mile radius of either. And, even with this much on its plate, it would not be so accommodating of an MP like Swayne. Or at least you would hope not.

Instead, only one of David Cameron's three parliamentary private secretaries has left the Conservative Party: Sam Gyimah, now a Liberal Democrat MP. The first of the other two is not only defending blackface but calling for the Supreme Court and its "fifth columnist" judges to be abolished. The second is now Boris Johnson's education secretary. If in 2013 you had asked David Cameron which of those three he believed had no place in a modern Conservative Party, it would not have been difficult to guess the likeliest answer. Swayne would have probably said the same thing. Though not the ending anyone expected, it is somehow the most fitting.