Last weekend against Crawley Town there was a bloke in the crowd who definitely did not like me. I could hear him because with an attendance of just under 4,000, isolated voices tend to stand out. After 20-odd years in the game you'd think I would be used to it by now but it still catches me out every so often. Off he went, blaming me for England's World Cup defeat in South Africa: I was just waiting for the PlayStation comments to start up.

People may be surprised to hear just how much comments from the crowd affect players. From the reprehensible, such as the alleged racist abuse Tom Adeyemi suffered during the recent FA Cup tie at Anfield, to the bog standard, "you're rubbish", it has been a topic of conversation among players at every club I have played for.

Goalkeepers, defenders and wingers definitely have it the worst simply by virtue of their position on the pitch. Most can tell you which clubs they hate playing against because of the abuse, right down to individual stands and specific areas in the crowd. They will say: "I hate playing at so and so ground…" or "I hate playing in front of that stand, the fans slaughter you." Sometimes they are referring to their own fans. I remember at Liverpool, some feared Anfield's Centenary Stand – there was a group of supporters there who made the players very nervous. It could just be one or two voices close enough to the action to make themselves heard but they would be there every week, shouting abuse. Most of the time you can block it out but it takes only a bad night's sleep, a poor run of form or a bit of an injury niggle and suddenly you feel vulnerable.

Right at the beginning of my career I learned the hard way why you should never listen to the fans. I was at Watford and there was speculation that I might be moving to Liverpool the day we played Southend. At Roots Hall the stands are right up against the pitch and I could hear the fans giving me grief about the rumours. The ball went out for a corner and I turned to them, pointing to one of the fingers on my glove that had my name on it. I felt better for a split second, then we conceded the goal and lost 1-0.

Over the years I have had all sorts shouted at me. From "Liverpool reject" to "England's No5" – which I am taking as a positive because it means I am still in with a chance. (It is when they say "England's No8" that I will really be worried.)

Racist slurs aside, the worst abuse I have experienced was playing against West Ham United when I was with Aston Villa. Around that time I had been in a car accident where the other driver had died. The West Ham fans called me a "murderer". It was the most sickening thing to listen to. Standing in goal that day I literally felt physically disgusted. To make banter out of such a thing was deeply inappropriate for everyone involved, not least the family of the deceased. Ordinarily I would ignore any kind of taunts but in that instance I had to let my disapproval be known.

Nowadays more would be made of such a thing – media reports, perhaps action from the club itself – but back then it was still seen as an inevitable part of the game.

It takes a Herculean effort to ignore a stand full of fans shouting abuse, no matter how experienced you are. I have been caught out before, days where you go out to do your warm-up and somebody says something and it gets you. That is why I decided to train myself to deal with the abuse.

Around the time I moved to Portsmouth I expanded the visualisation technique I was using before games. You may have seen me, standing in goal before the start of a match, rehearsing my catches and positioning. In my head I added a soundtrack, a background wall of noise and insults, to prepare me for anything I might hear during the match. I would anticipate going to certain grounds and imagine a chorus of "you're rubbish" or "England's No5". It helps.

Even the most confident players can be affected by screaming fans. When Paul Ince was photographed in a Manchester United shirt before leaving West Ham, the supporters never forgave him for it. I remember playing at Upton Park with Liverpool and Ince was noticeably reluctant to move out of the centre circle. It was like playing with a completely different person the way he reacted to their taunts.

If players cannot handle abuse then how bad must it be for referees who are modestly paid and certainly don't have any fans?

When I was at Portsmouth I remember speaking to a referee in the tunnel at half-time. He had just given a penalty and was genuinely distressed that he might have made the wrong decision. He came running off the pitch, anxious to catch the replay on the TV to check his decision. Commentators talk about referees being under pressure to "even things up" and in this case I could easily have seen how this referee might have felt obliged to do something of that nature.

To be clear, to abuse players using racist or homophobic language is not acceptable in any way, shape or form. But as footballers we have to be prepared to withstand the other stuff. I was never advised, warned or educated about how to do that. Perhaps, should I become a manager, that is something I could help my players to deal with.

David James has donated his fee for this column to charity