Kick It Out, football’s leading anti-racism organisation, has told the Guardian it deserves better than to be turned into Rio Ferdinand’s “punchbag” after discovering he has attacked it as “useless” in his latest autobiography for allegedly not supporting his family enough during the race trial involving his younger brother, Anton, and John Terry.

Herman Ouseley, the chairman, decided to speak out after the serialisation of Ferdinand’s book, #2sides, stated that the former England international’s decision not to wear a Kick It Out T-shirt in defiance of Sir Alex Ferguson at Manchester United was taken out of principle because the group had “refused to come to the courtroom with us, so I wasn’t willing to go through the charade … My parents probably wouldn’t have spoken to me if I had.”

A representative of Kick It Out had actually attended the court every day with Ferdinand’s parents, Julian and Janice, sitting alongside them as a show of unity in the public gallery, and Ouseley said he felt compelled to “set the record straight” over what his organisation considers damaging inaccuracies.

Ouseley believes the anger should be directed higher up the chain to “the real power brokers” and he also revealed he resigned from the FA council because it had refused to issue a public apology to the families of Ferdinand and Patrice Evra for the way it handled not just the Terry case but also Luis Suárez’s disciplinary hearing for racial abuse.

“They handled the two cases ineptly and I said to them: ‘You don’t realise the grief and angst you have caused the Ferdinand and Evra families.’ I felt they needed to put out a public apology to the whole of football. They said: ‘We’re not prepared to do that,’ and I said: ‘I’m not prepared to stand any further, goodbye.’”

Ferdinand’s allegation that “it was pure lip service” from Kick It Out has therefore caused dismay among the group’s mix of staff and volunteers, given that they had arranged several meetings with the family in advance of the trial, then publicly supported them in court, as well as lobbying the FA to accelerate its own disciplinary processes and hold its misconduct case before, rather than after, the drawn-out proceedings.

“It rankles in the sense that it sends out the wrong message,” Ouseley said. “Nobody can ever say a player has come to us and we have not supported them. Not one. We have always done what we could whereas there are an awful lot of other people who haven’t.

“Unfortunately we are the punchbag for some people and it’s easy to have a punchbag like us. We’re a soft target and the easiest target. Rio should not be attacking the weakest point – the people who are actually supporting him – but the people who were not supporting him. I would have thought the whole of football should have turned up at that court case wanting justice.”

While the Chelsea chairman, Bruce Buck, was at the court every day, the irony is that Rio Ferdinand stayed away from the trial. Andy Impey, a former team-mate of Rio at West Ham, joined the family in court but that was essentially it apart from Kick It Out. The organisation had expected far more support and were surprised that QPR, Anton’s club at the time, did not send anyone to join them and that the Professional Footballers’ Association were only seen making a fleeting appearance one afternoon.

Ferdinand says the family had wanted Kick It Out to turn up en masse in campaign T-shirts and when it became apparent that was not going to happen Janice had told them: “In that case, get out of my house, and don’t fucking come near us again.” There was never a meeting at the family home, according to Kick It Out, and there is no recollection whatsoever of that outburst.

Danny Lynch, Kick It Out’s representative in court, wore a suit and Ouseley said: “It [the criticism] is absolutely pathetic. He was representing Kick It Out and we expected him to dress in a correct way and respect the court. He was there to observe in a professional capacity.

“The support he offered the family was unbelievable and to make a stupid comment about the fact he didn’t wear a T-shirt is pathetic.”

The complaints, Ouseley continued, should be with “the FA, the Premier League, the clubs, all the people with the real power, not us, and who missed an opportunity and didn’t mobilise. We are a soft target but I don’t hear criticism of David Bernstein or Richard Scudamore. We did as much as we could but what it shows is a complete lack of understanding about our actual powers.”

There is further confusion from the fact Ferdinand was not in court but his book, released on Thursday, appears to suggest otherwise and berates others who were not there. “Outside the court, I saw Clarke Carlisle of the PFA [Professional Footballers’ Association] doing interviews. I said: ‘Come in the courtroom.’”

Terry was acquitted of racial abuse but later found guilty of misconduct at an FA hearing and Ouseley’s belief is that there are wider issues emanating from the trial.

His organisation, he says, has been unfairly scapegoated when it would be better to have a united front. “Ferdinand is selling a book of trivia,” he said. “Meanwhile, the bigger issues remain untouched.”