You OK?" said Casey.

"Yeah fine. I'm all right," I said, but I wasn't.

"It was an upsetting morning," he said. "I feel upset myself."

"Tell you what, Case, if I never meet another psychopath again as long as I live, it'll be far too soon." And I knew that I had lost the stomach for the whole damned business. If I carried on in prison, I would have to do it differently; I would have to admit that it was prison.

In 14 years, I had never excluded anyone, no matter how difficult they were. It had all been a bit of a club where everyone was good and bright and sensitive, especially when they weren't. But now, my bottle had gone and I wanted to get under the duvet and stay there. Casey gave me a hug: "Retire, man. You don't need all this shit."

I suppose it had all been creeping up on me for a while. What I had to face up to was that I was becoming more and more indifferent. It was a difficult thing to do. There were moments, more and more of them, when I didn't care about the things that happened, who they happened to. This was the selfishness that prison rubs into your skin. I listened to what Arthur had to say, nodded the right things back at him and then shrugged him off. Arthur led a life of growing desperation. He had an indeterminate sentence and lived from one parole board to the next, with no real idea if release would be sooner or later. He talked to me about his wife, his children, his father and, I said to myself, if I could do anything to help him out I would. Anyone could see that he was a decent sort of a bloke. If he managed to get into the education block a bit early, he gave our room a bit of a tidy, got the office vacuum cleaner and did the floor. If he caught me doing it he told me off: "You shouldn't be doing that." I liked him; we got on.His robust spirit was slowly winding down and sometimes he snapped at people, took refuge in contempt, laughed a lot less than he used to. But I shrugged him off. I caught myself doing it, caught myself thinking of something clever to say, making up something clever to tell myself so that I could slip away.

I had always tried to avoid this failure that begins with indifference. I had never tried to manage the men I met, deflect them, stand behind a platitude, promise anything and then lose the paperwork. That's the track I had started to head down, towards that quiet life that I'd do anything to have. I could see it happening.

And so, after almost 14 years, it was time to go. I couldn't just walk away, just leave the guys flat. I offered them a deal. I would stay as long as they did. There was a certain amount of dark laughter. "You might have a long wait," said one. Sue, the education manager, agreed to let the class run down, no more new faces.

Teaching in a prison means that from time to time, someone who is really difficult walks through the door. I always felt obliged to persist, not to simply chuck them out. "You can't treat people like that," I would say when the guys advised me what they would do to the current nutter. Now, I had started to agree with them and it felt like failure. I was becoming growingly aware that when I heard about someone stabbing a man – "They reckon I stabbed him 47 times," or about pouring boiling, sugary water over someone or about saving up stale piss to throw, there I was nodding and making notes and thinking, "Oh that's good, I might be able to use that in a novel". I knew that something was wrong but, just as it had been when I was a child, the wrongness had no purchase on me.

We got down to four and then, out of the blue, there was a run of golden, other-worldly mornings when we read Chaucer. Chaucer. It started when Ten-Foot thought I was kidding when I said that there was a marxist account of language. They couldn't get enough of the Canterbury Tales, that lovely run of characters. It cheered me up to know again that anything good, no matter how old, obscure or difficult always commanded a hearing. We took our time over the spelling and where words might have come from or gone to. I told them about David Crystal and the notion of polite rather than correct English. We really got into it. The college didn't want to pay me for a class of four; I was willing to pay them. The men's enthusiasm reassured me that I hadn't wasted my time. What, after all, does education offer to people if not a greater sense of being human?

When we don't know about history and art and society we are adrift. Most of you reading this will never have had that experience, but many of the men I taught were ignorant of just about everything, and as grown men felt this keenly. Education was a relief, a route to self-respect.

For the most part, classes in the arts, social sciences and languages have been closed. There won't be much of my liberal nonsense in the future. The government has decided training for work is the way to go and for the most part education, beyond basic numeracy and literacy, has been abandoned. I can't see it myself.

Of course there are men for whom training is just the thing and it will succeed. But in prison, nothing succeeds for everyone; prison is an enormously diverse place. In my classes there were men who were mad, addicted, beyond retirement age. There were men whose intention was to go straight back into crime. There were men who were alienated or simply bewildered. There were of course sensitive, intelligent men who had decided on higher education. So, anyway, good luck with the training.

Then, Ten-Foot was gone. "Released," Casey said. "He never said he was due." "Some guys don't," said Casey. "Yeah, I know, I know." Then, Arthur went. I'd appeared at his parole hearing and been at a loss how to make them understand what a good bloke he was. Everybody there said so.

So it was just me and Casey and I had the full, unrelenting blast of his enthusiasm for everything. I also got all of his thrilling adventures, which had landed him so badly in the soup. I told him some stories of my own, about that first class. "I am not a criminal," Dean had said, "I am a murderer," and it had taken me years to understand. I told him about Jason who had left prison to go to the LSE, Malkie the Glaswegian yogi pool shark, Budgie Blue Legs and on and on. I had been really fortunate with the men who lingered on, intelligent civilised men who clung on to respectability. In the end it was me and Casey. We decided to do quantum physics. He had the maths for it, and for a few months we followed a course of lectures on CD. I know that I tried his patience, making him go over things three or four times, but he was a good teacher.

One afternoon, Sue rang me up. "Casey was shipped out yesterday."

And I was finished.

Alan Smith's columns on teaching in prison can be found at theguardian.com/education/series/philosophy-for-prisoners