In Lord Ouseley's damning assessment of the football authorities' record on racism, his most dispiriting insight is that the game's moral compass suffers institutional wobbles when its stars, not fans, are found guilty of racist abuse.

When John Terry was first charged last December with a racially aggravated public order offence for the abusive words he uttered to Anton Ferdinand in Chelsea's match against Queens Park Rangers on 23 October last year, Roman Abramovich's club issued a statement saying: "Chelsea FC has always been fully supportive of John in this matter and will continue to be so. The club finds all forms of discrimination abhorrent and we are proud of the work we undertake campaigning on this important issue."

Ouseley, reflecting on the year in which Terry and Liverpool's Luis Suárez were found guilty of racial abuse by independent Football Association commissions, said he and Kick It Out honoured the legal and FA processes by saying nothing while the cases were ongoing. That contrasted with the astonishing statement by Chelsea's then manager, André Villas-Boas, who said of Terry after he was charged: "We know exactly his human values and personality, so we will support him whatever happens."

Ouseley argues that was "disrespectful" of a process which concluded with Terry acquitted by the chief magistrate, Howard Riddle, after he had been selected to represent England in the European Championship, then found guilty in September by the FA commission. Suárez was found guilty of racially abusing Manchester United's Patrice Evra with Liverpool, under the ownership of Fenway Sports Group, and the club's then manager, Kenny Dalglish, nevertheless displaying unwavering support for their player, complete with those notorious T-shirts.

Ouseley, a former chairman of the Commission for Racial Equality, has a distinguished record of fighting racism, and initiated the original campaign, Let's Kick Racism Out of Football, 19 years ago. A Millwall supporter, he loves the game and always loathed the virulent hatred with which it was riddled in the 1970s and 1980s. He also believes football is an important high profile arena in which an unequivocal message against discrimination of all kinds can and must be sent.

As many people have fairly noted during a year in which racism's unpleasantness has surprisingly thrust itself back into the spotlight, huge progress has been made. The very fact that alleged racist abuse by single supporters is being investigated by the police, following reported incidents at Manchester City and Swansea City this weekend, illustrates how far top-flight football has come from the days when mass racist chants were rife at grounds and nothing disapproving was said at all. Ouseley, unpaid, and his organisation, run on a comparative shoestring, deserve plenty of credit for the clean-up.

That was why, when the Terry and Suárez incidents and the way they were handled sparked the October protests which put Kick It Out in the firing line, Ouseley found it bemusing and hurtful. He accepts that Kick It Out should be subject to critical analysis, and the Reading striker Jason Roberts's view, that the organisation has lost its cutting anti-racist edge, should be honestly considered.

However Ouseley clearly feels that the protest, in which large numbers of players declined to wear campaign T-shirts, made Kick It Out itself the target, rather than Roberts' and other black players' legitimate grievances. Finally speaking out, Ouseley accuses the clubs of lacking morality when faced with racism by their own valuable "assets", the players. He criticises the FA for failing to give clear leadership beyond the formalities of its disciplinary processes, which he says worked well.

He was dismayed the FA did not admonish Villas-Boas or Dalglish, or charge Chelsea or Liverpool themselves for their conduct, or for the racial abuse by one of their players. For that, Ouseley says, the FA was "very slack and weak" and this last 12 months has not sent out a strong condemnatory message.

The FA did strip Terry of the national team's captaincy, leading to the resignation of the coach, Fabio Capello, but Terry remained available to represent the nation at the European Championship. Roy Hodgson, Capello's successor, praised Terry's "warrior" qualities and selected him for the squad, leaving Rio Ferdinand at home. Hodgson said this was for "footballing reasons," that Ferdinand was no longer good enough, but Ouseley is with many others, believing that in fact it was because the two players could not work together and Hodgson and the FA were happy to take the player charged with racial abuse, not the brother of the alleged victim.

Then after Terry had claimed the European Champions League for Chelsea as the club's captain – although he did not play in the final – and been a warrior for England, the FA's independent commission cut through his defence about why he said the offensive words to Anton Ferdinand, and stated they did not believe him. So Terry was finally held to have racially abused a player, effectively lied about its circumstances – and his team-mate Ashley Cole to have "evolved" evidence which was not reliable to support Terry.

On that, the FA said nothing, and, Ouseley said, neither did the Premier League. Terry was given a four-match ban and, although David Bernstein, the FA chairman whom Ouseley respects, said the FA will review its penalties for racism, Ouseley complains it has all now gone very quiet.

Chelsea, having expressed so much support for Terry throughout, imposed an undisclosed fine following the FA commission's verdict, but retained Terry as their captain, a club's leader and ambassador. Chelsea were persistently challenged about the inconsistency with the banning for life in May of a fan who racially abused Didier Drogba, but the club brushed that aside.

No club or authority condemned Terry or Cole for what the commission said about the unreliability of their evidence, but Cole was disciplined by the FA and had to apologise for swearing about them in a tweet. That, Ouseley says, summed up where football's governing body has landed itself in the morality of dealing with racism, and why football, for all the great progress, still has a long way to go.

Throughout all of that Ouseley kept his counsel, and also in October when the dissatisfaction of black players found its vehicle in the refusal to wear the T-shirts of the organisation which has campaigned against racism for 19 years. Ouseley found himself having to defend Kick It Out, when in fact he was frustrated that the black players had not articulated their various grievances — particularly the authorities' failure to deal firmly enough with racist incidents and the lack of coaching opportunities — more coherently.

Now he is considering stepping down from Kick It Out because, important as it is, there are more deserving demands on his time than the struggles of multi-million pound football to be morally convincing. In a recession, he says, scapegoats are hunted and racism increases. With the Premier League now sealing the deals for more bounteous billions from TV channels eager to show "assets" such as Terry and Suárez, the challenge of the modern game to navigate its moral challenges can only intensify.