The struggles for civil rights in the US and South Africa had many parallels. But we didn't just talk about politics – he loved boxing more than anything

I can't remember the first time I ever heard the name of Nelson Mandela. Maybe it was in 1963. Then we were all in jail. I was in jail in Greensboro, North Carolina, he was in jail on Robben Island. Dr [Martin Luther] King was in jail that year too, in Birmingham, Alabama.

We who were involved in the civil rights movement back then were acutely aware of the parallels of the ANC struggle with our own struggles.

They were interrelated, those forces holding us back, those same racist forces opposing civil rights in the south of this country were the same as were operating in the south of Africa, big corporates, big money and interests.

As a young civil rights activist I knew how raw and ugly and violent the apartheid regime was. They were being jailed, we were being jailed. We were being killed, and they were being massacred. The courts were behaving in a similar way in both continents. It was a long time before I met him; by the time I did I had a deep kinship with South Africa.

There were tremendous parallels in our labour struggles. My first arrest was when I was 18, on 16 July 1960, at a civil rights protest at Greenville's segregated public library. We marched there and we were thinking of what had just happened a month or so before at Sharpeville. What was happening in Africa was being used as a basis to justify occupation and murder against black people in the US. Our country was on the wrong side of all this revolution, it was trying to stamp on the freedom movement. [Henry] Kissinger with his talk of terrorists' associations and threat to national security.

So you see we knew what was going on in South Africa, those bridges and links were always there, those parallels just as I saw in Nelson Mandela with our own Dr King. They had an awful lot in common: intellect, courage and high moral authority. Embracing the struggle for others as a way of life. Accepting what happened to themselves with fortitude, with non-violent intent.

Even though the ANC was pushing towards a military campaign, Mandela was a man of peace. He expressed to me that, when he and the others went on trial in 1964, they were going to do some blowing up, something was planned, and he was glad that that had failed, even though it meant he went to jail and he suffered.

And so Mandela was in prison all the time I was making links with South Africa, although I felt him around – you can see Robben Island from the city and Robben Island can see you! I was in Cape Town by chance on the day of his release. I heard the maids in my hotel beating their pots and pans and people screaming and singing, oh I could feel the change coming. It's very difficult to describe the release of glee and joy when the word got out that he was officially freed.

I was that day meeting the wife of Oliver Tambo. Tambo was in exile and we had just been marching together in Trafalgar Square in London before I had gone on to South Africa and I had met with Mrs Tambo. We heard the news and went down to City Hall in Cape Town, just to see what was going on.

Then he was there and I became the first African American to meet Mandela after his freedom from captivity. He immediately recognised me and we embraced and one of the first things he said was that he had seen the 1984 presidential campaign speech I had made where I had called for sanctions and stood up against apartheid and he thanked me.

That is the kind of man Mandela was, he would come to you, to thank you, at such a time for him. He knew everything about the struggle back home, he had followed it so closely. My children were with me that day and it was one of the proudest things that I can say, I showed you this day and you met this man.

We became friends and I hosted him in my home when he visited the United States.

It was not all politics and causes, he had a great sense of humour. He was a very funny man and he loved The Cosby Show. He understood immediately how important The Cosby Show was for African Americans, the first time a black family had been portrayed in a civil and positive light on television.

We were all used to seeing these negative stereotypes, of course, in the movies, of black people, racist stereotypes.

So the Bill Cosby show made some impact, was something we both loved to see and talk about, as well as movie distortions about black people. Boxing was what he loved best, that's what he'd rather talk about than anything. He loved Joe Louis. For a very serious man he had a great sense of humour. But he was also a very serious man who was very focused, his mind was rapier sharp, he was never tripped up in any situation.

He took tough decisions. I remember the flak he took for going to Cuba to see [Fidel] Castro, but he said calmly to the American press: "Your enemy is not my enemy." He did not get tripped up by trying to appease, he was not going to forsake those who had helped him. He didn't adjust his moral compass for anyone else.

In so far as racial reconciliation goes in South Africa, no one could do what Mandela did. There is no doubt Mandela averted a bloodbath, through his reconciliation and rehabilitation at a time when men were thirsting for revenge.

But even with his death there is unfinished business.

Africans are free but not equal, Americans are free but not equal. Ending apartheid and ending slavery was a big deal, Mandela becoming president of South Africa, [Barack] Obama becoming the first African American president was a big deal, but these are stages. We have to go deeper. We were enslaved longer than we have been free and we have a long way to go. We have unravelled our injustices in stages but they remain, in land ownership, in health and life expectancy, in certain aspects of the media and in major business.

It is time to commemorate our great men. Mandela was a great, great man. A champ and a hero with such immense stature. But he has left us unfinished business. In his name we must carry on our struggle.