It is certainly not fashionable to say this now, but I have always had an affection for most politicians. As sketchwriters, we are verbal caricaturists. Nicholas Garland, the brilliant cartoonist for the Daily Telegraph, once described to me how his profession worked. A new person swims into the public gaze. They may have an apparently quite ordinary face, like John Major or Tony Blair. The cartoonist then finds something, anything, about their appearance that is even slightly off-normal.

Various cartoonists fastened on to Major's upper lip, on which a sort of phantom moustache seemed to hover. My colleague Steve Bell spotted that Blair had, like Margaret Thatcher, one mad staring eye. (I claim credit for noticing that this was sometimes the right, sometimes the left, as if the daemon within took its pick each morning.) By the end of Blair's period in office, Bell was able to draw the eye inside an electricity pylon, as the viewing hole in a prison door, or as an untethered balloon floating away into the sky, and achieve instant recognition.

In the same way, we sketchwriters develop our own shorthand for the people we write about, exaggerating language, absorbing verbal tics, fastening on to some minor eccentricity and pumping it as hard as we can. One way or another, I have met every British prime minister since Macmillan. They are often rather odd, and it seems to be, in some way, a condition of getting the top job. The party leaders who were most like real human beings – Neil Kinnock and William Hague, for instance – never quite made it. I think the electorate wants their leader to be different, slightly set apart, a little bit weird.

Margaret Thatcher

Being Margaret Thatcher was a very demanding, 24-hour-a-day job. Like many people held in the grip of their own convictions, Thatcher had trouble understanding the thoughts and words of others. She was, and no doubt is, a kindly woman. Staff were always pleased when she remembered their birthdays, even more when she remembered their children's birthdays. She could be tolerant of other people's way of living, even when she did not understand or sympathise with it. However, if she had a sense of humour, it was well hidden.

There are many stories about this. When the Liberals adopted a sort of stylised yellow bird as their logo, one of her speechwriters produced a passage based on Monty Python's parrot sketch – "this parrot is no more, it has shuffled off this mortal coil and gone to join the choir invisible . . ." She had to be persuaded that the lines were funny, and would be recognised by the audience. She had one other concern: "This Monty Python, is he one of us?" (Interestingly, Cameron referred to the same sketch in his party conference speech this week, confident that everyone would get the joke.)

Earlier, Peter Jay, the son-in-law of the then prime minister Jim Callaghan, had said that Callaghan saw himself as Moses, leading his people after long travails into the promised land. It was a silly thing to say, and Thatcher's people seized on it. They wrote a line for her party conference speech: "My message to Moses is: 'Keep taking the tablets.'" But she didn't get the joke and tried to change it to "Keep taking the pills." Only after it was carefully explained to her did she agree to the proper version.

She was nonetheless the source of much unwitting humour. Just as, in an early silent movie, when you see a man up a ladder with a pot of paint, you know with near certainty that the star is going to walk under the ladder at exactly the wrong moment, so with Thatcher. When a double entendre appeared in the offing, you felt she was certain to utter it.

I recall Thatcher being asked in the Commons about pacifists handing out leaflets outside an army barracks. "I'm sure soldiers will know exactly what they can do with those leaflets!" she said, to outright laughter from the Labour side and surreptitious giggles from the Tories.

At a training centre in Putney, she was introduced to an extremely large youth who was working with a giant wrench. "Goodness," she said, "I've never seen a tool as big as that!"

But Thatcher saved the best of all for her victory tour of the Falkland Islands. She was taken to inspect a large field gun, basically a ride-on lawnmower with a barrel several feet long. It was on a bluff, overlooking a plain on which another Argentine invasion might one day materialise. She admired the weapon, and the soldier manning it asked if she would like to fire a round.

"But mightn't it jerk me off?" she replied. Chris Moncrieff of the Press Association, who was covering the visit, recorded the manful struggle of the soldier to keep his face, indeed his whole body, straight.

If Gladstone addressed Queen Victoria as if she were a public meeting, Thatcher tended to speak to people as if they were members of her cabinet. I ran into a BBC reporter in the early days of her regime. He had just filmed an interview with her. She had learned from him that his next assignment was to cover an Opec meeting. "If you want to help the British economy," she told him sternly, "persuade them not to put up the price of oil."

The reporter looked fazed by this instruction, though not his cockney cameraman, who said cheerily, "Orright, darlin', I'll do my best!"

Every year at Christmas, the House of Commons press gallery holds a party for reporters' children, and they generally also invite a few dozen deprived children from the Westminster area. Quite often a senior politician turns up to cast their lustre upon the occasion. This time Margaret Thatcher had come. I know the story to be true, because I had it from Father Christmas, or at least the reporter playing the part.

Mrs Thatcher was in a relaxed mood, smiled and chatted to the children, and even those who knew who she was and felt rather awed were put at their ease. Everybody seemed happy except one small boy, who was crying into his bowl of dessert. As the prime minister passed he looked up and said: "Miss, Miss! They've given me blancmange and I don't like blancmange."

"That," she said, smiling sweetly, "is what parties are all about: eating food you don't like!" I felt at the time that this told us a lot about her style of government.

Thatcher's failure to sense other people's mood and concerns extended to the royal family. Every summer, she had to go to the traditional weekend at Balmoral. The visit is relaxed – just the royals, a few close friends, monuments to their ancestors, and around 10 million midges. Every year they have the traditional outdoor barbecue, and afterwards the Queen goes to a little hut to wash up. It is the one time in the year that she washes up, and so has much the same importance as opening the Christmas presents does for a small child. But Thatcher could never believe that anyone could do any job better than she did herself, whether governing the country or doing the chores. Why, it was the images of herself in marigolds, washing up at home, that secured her image as a housewife and mother and helped to win her the Tory leadership in 1975.

My informant said that Thatcher insisted on doing the washing up. The Queen overruled her. She insisted again. The Queen, inevitably, won, but the interesting fact is that Thatcher thought she was offering to do her a favour, and did not realise she was proposing to deprive her monarch of one rare and valued contact with the real world, as lived by her subjects.

Thatcher could be charming when she had the time. At a formal dinner at Chequers, a Wren who was one of the waitresses tripped and spilled lamb casserole into Geoffrey Howe's lap. Mrs Thatcher was on her feet instantly, rushing round to comfort not the chancellor of the exchequer but the serving girl, hugging her and saying, "Don't worry, my dear, it could happen to anyone."

Gordon Brown was often hurt by critical press. Photograph: Tim Graham/Getty Images

But the prime minister's staff became very careful about Denis. On his wife's many tours abroad, he would sometimes go down to the back of the plane where he could hope for a chat and something to drink. Unfortunately, he tended to say exactly what he thought, often complaining about the country they had just visited. "Canada," he once remarked, "is full of fuck all." Another time he was asked what he had made of China. He thought for a moment, then averred: "China is full of fuck all."

One morning, he and his wife were flying to Scotland. The flight attendant asked what he would like to drink, and he said firmly: "A gin and tonic."

"Isn't it a bit early for a gin and tonic?" Mrs Thatcher inquired.

"It is never too early for a gin and tonic," he said.

Discretion sometimes won. Jim Naughtie of the Today programme recalls chatting to Denis on the landing at Downing Street during a reception, standing next to a large potted plant. Denis was drinking gin and tonic. One of the prime minister's detectives approached him and whispered: "The Boss is here, sir."

Denis deftly poured his drink into the plant pot, then reached out to embrace his wife.

Tony Blair

I had lunch with Tony Blair in 1992, when he was still home affairs spokesman for the Labour party. I worked for the Observer at that point. He kept asking me what I thought about various matters, Labour policies, John Major's policies, everything. I'd ask him his opinion, and he would match by asking me mine.

After a while I wanted to yell at him: "Look, the Observer is paying for this lunch because they want to know what you think. What I think, they already know because they pay me to tell them." But I didn't.

Was it a form of flattery? Did he think I would go away and boast, "I believe, and I can tell you, that Tony Blair agrees with me . . ."? Perhaps it was simply a way of avoiding the question, a means of not committing himself to anything that might prove embarrassing later. Possibly a bit of both. But later I saw an interview with Sir Christopher Meyer, who used to be our ambassador in Washington. He was asked why Blair, who had got on so well with Bill Clinton, got on equally well with George W Bush, and he replied that Blair was rather like a radio searching for a signal, then suddenly locking on to it. Once he identified the signal Bush was sending out, he knew exactly what to say.

In 2003, I had gone to MC a charity dinner in Salisbury, held in a marquee in the Cathedral Close. I thought it was odd that we were searched by armed police officers when we arrived – this is not common at cathedrals – and then we saw the table plan, which indicated that we were to sit with the prime minister and his wife, who, it turned out, were acquaintances of the organisers and had spent time with them at neighbouring houses in southern France. Somewhat to my relief, Blair, no doubt deciding he would be more relaxed if he didn't need to worry about having a journalist at his table, sat at the next one over.

David Cameron rode by tube in 2008, but had shown his competitive streak earlier on. Photograph: Kevin Coombs/Reuters

His wife, however, seemed to relish the challenge. Every now and then I had to bob up and introduce the next speaker, or an auctioneer, or the choir, then return to the table and resume eating. In conversation, I said that I had been looking at the silent auction. The first lot was from a hospital in Salisbury, which had offered "mammogram or vasectomy – choice of one only, please".

I tried a small joke: "I hope I don't win that one; I wouldn't know which to pick."

Mrs Blair gave me a stern look. "Men can get breast cancer, too," she said. "There are 300 cases a year in this country."

"I know," I fibbed. "I feel my breast in the shower every morning. But that's only to make sure I still have a heart."

"Of course, you don't," she said tartly. "You're a journalist." Then, to the rest of the table, "Guardian journalists are the worst. And Guardian sketchwriters are the worst of all."

I thought it was going to be a long evening. Conversation turned to teenagers. Euan Blair had recently been found, drunk, lying on his face in Leicester Square. I said that our children were coming up to the same age as their two eldest, and we were beginning to cope with some of the same problems. Her face expressed distaste and disbelief. "You have children?" she asked, as if appalled that such a ghastly person could propagate the species.

At this point, something in me snapped and I said: "Oh, stop it!"

She looked startled for a moment, then a broad smile appeared. Cherie had a very broad smile. Then she took up her cracker and handed me one end to pull with her. After that, she could scarcely have been more charming.

After the dinner, her husband, with guitar, joined a pick-up group with friends who had played together a few times. He looked happy and relaxed. It was the end of the year of the Iraq invasion, and to have nothing more to worry about than the next chord of In The Midnight Hour must have been an hour of blissful relief. Someone standing next to me said: "I saw them in France. But they won't play Summertime Blues this time. It's got that line, 'Gonna take my problem to the United Nations'."

Gordon Brown

Like many people who can treat others brutally, Gordon Brown has a very thin skin himself. I found myself chatting to him at a reception in Downing Street. I had recently had an emergency eye operation, of the kind that makes people's knees buckle if you provide too detailed a description. I knew he had had a similar operation after his rugby accident at school, so I asked him about it. He was clearly quite proud of the fact that he had had the first laser operation on a detached retina ever performed in Britain, and talked about it at some length. As a fellow sufferer I was fascinated, though I think anyone else overhearing would have disappeared, sharpish. I asked him if he had had any further trouble with the eye.

"I've had no complaints," he said cheerily.

"That's very encouraging," I replied.

"In that case, will you write something nice about me for a change?" he said.

I thought it was very uncool for a prime minister to admit to reading critical articles, and even less cool to complain about them. The correct pose is to be standing aloof and above any such trivia. Thatcher and Blair never admitted to reading a newspaper, and in fact probably didn't. They had people to do it for them.

David Cameron

David Cameron has the faultless courtesy of many Old Etonians. This is combined with a nervous awareness that having been to the school is no longer the guarantee of social and financial privilege it once was. Many Old Etonians have stories about being treated almost as social pariahs, finding for example that some Oxbridge colleges are actively opposed to admitting them.

Cameron is also remarkably competitive. This is a trivial example, but I recently asked the questions at an annual charity quiz in Notting Hill, the area of London near his home. I first did this in 2005, shortly before the election. Everyone knew that Labour was likely to win once again, and that Cameron had a good chance of leading the party after Michael Howard had been defeated.

Quizzes are usually hard fought, but Cameron, as the captain of his table, was incredibly determined, nagging his team-mates until the very last second to get crucial extra answers. In the end, I had to threaten that anyone who didn't get their reply sheets in within 10 seconds would lose all points for that round, adding, "Ten, nine, eight . . ." While I counted, Cameron flew from his table, barged to the front, and banged the paper down on the desk just as I shouted, "One!"

I reflected then that it was the last time I would ever be able to issue him with orders.