Recently, I had lunch with my colleague David. Afterwards we went to pick up his dry cleaning. Walking into the dry cleaners was like walking into Arnold's with The Fonz. People cheered as we went through the door. Hands were shaken, backs were slapped. One of the staff stopped pressing shirts round the back of the shop to come out and give David a hug. David greeted everyone by name, asked after their kids, teased someone over a howler their team had made during that weekend's football, and we left. I think David's parting shot was a double thumbs-up. In a couple of minutes, David had made everyone's day a little bit better, a little bit brighter. He'd cheered everyone up. Everyone except me. I felt terrible.

Like David, I'd been going to that particular dry cleaners on-and-off for the past five years (it's round the corner from our office). Unlike David, I had no idea what anyone who worked there was called, let alone what team they supported. I'm pretty sure they didn't even recognise me. Up until that moment I didn't care less. Perhaps your trips to the dry cleaners — or the corner shop, or the pub, it doesn't matter — are like mine. I tend to ask for what I want, hand over my money and go about my business. It's not like I'm not polite. But that's about the extent of my social skills with strangers. Do I want to be friends with my dry cleaners? No, it sounds weird. I want them to take my dirty shirts and give them back to me clean. And I want to pay them for doing that. Still: wouldn't it be nice to have a bit of David's magic?

Everywhere you go with David is like this. Let me explain: David is gregarious, cheeky, Scottish, kind, thoughtful, funny, a flirt, puerile, silly, likes a drink, can't really take a drink, is a chatterbox and is really, really good company. (Perhaps too good company, I used to sit next to him at work and after a couple of weeks I took myself off to John Lewis to buy some noise-cancelling headphones — I wasn't getting anything done.) Whenever a new intern started he'd spend as much time finding out about them as he would chatting to the company boss whenever they did their floor-walk. In summer he'd buy ice lollies for the office.

Then there's my neighbour Paul. Paul is not like David. If you tell Paul about the best act you've just seen at a festival, Paul will tell you that he heard them on the radio and they were rubbish. If Paul sees you taking the bins out he'll tell you you're walking the wrong way — it's actually much quicker the other way. If you tell Paul it looks like it's going to be a nice day, he'll say no, you're wrong — there's rain due later. Paul is like one of the Mr Men: Mr Contrary. Talking to him is the opposite of talking to David. He leaves you feeling annoyed. For conversation to work best it has to flow one way. Unless someone is expressing particularly objectionable views — and even then this isn't a given, how many of us have endured the cliché of the racist cabbie through gritted teeth? No tip, well done, that showed him… but equally no standing up for what's right either — it's generally a good thing to agree with the person you're talking to, even when you don't agree with them, if you see what I mean.

"Nice day."

"No it isn't."

It's not great, is it?

It got me thinking about the difference between David and Paul. And it seemed to me that difference boils down to charm. Do I want to be like David? Not really. It looks exhausting. He's left Esquire now but he does occasionally visit and you can barely get him past reception without someone stopping him for a natter. It's usually girls. (David and girls is another story. I'd like to be like David with girls.) But I definitely don't want to be like Paul. Paul's a dick. This idea of charming versus charmless started me wondering. The idea that random people actually like you, really like you and emotionally respond to you, and that in turn you really like them. Where does that come from? Is it innate, like good teeth or curly hair? Or can it be learned? Is charm actually a bit of an act?

I grew up an only child and for ages I was pretty shy. I can remember not being able to pay for stuff in shops because I was too scared to engage the shopkeeper in a simple transaction. (No, I wasn't getting stuff dry cleaned back then.) My mum mostly brought me up by herself because my dad was away at sea for months on end. My parents sent me to a boarding school precisely because of this — the idea that a fuller extracurricular life would somehow stop me turning into a mummy's boy. It was partially successful: since then I've always had a small but really tight group of mates, two or three very good, very close friends. But I've never been the most popular person in the room. I was always happiest on my own — drawing comics, or making little magazines, or poring over football titles like Shoot! And I'm still like that. I've got better at being sociable, everyone does, it's called growing up.

Now, through work I'm required to go to lunch or evening functions where more often than not I'm sat between two strangers. At first that terrified me. Now I kind of look forward to it — 90 minutes' chat with a couple of people I've never met? I can do that. How bad can it be? Sometimes it turns out to be a really good laugh. At the very least, I can get through it. But I don't for a minute leave thinking that I've charmed the table. But at least I haven't bored them. (I hope I haven't bored them. You'd have to ask them.) And I imagine that's probably a similar experience for most people. (David would have become Instagram pals with a couple of them and got off with the best-looking one in the loos. Paul would have argued with the waiter and cleared off before pudding.)

Charm seems to be a particularly male trait. You seldom hear about a charming woman. Princes are charming. Princesses have other attributes. Famous charmers include people like George Clooney, Barack Obama and Cary Grant. You'll have noticed they all also happen to be good-looking. That helps, but it's not a prerequisite to charm. People talked about John Major in a similar way. Same with Winston Churchill, and he was no oil painting. Bill Clinton famously "lights up" whatever room he enters. People talk about his magnetic personality. How he makes you feel like you're the only person in the room. Nelson Mandela was the same. So was Steve Jobs. Then again, is this charm or charisma? There is a difference. Charm is defined as "the power or quality of delighting, attracting, or fascinating others". Charisma is defined as "compelling attractiveness or charm that can inspire devotion in others". Charm/charisma is a power that isn't always used for good. Adolf Hitler, Charles Manson and Oswald Mosley were charismatic and charming. Donald Trump has obviously got it or half of America wouldn't have voted for him.

There have been plenty of charming serial killers — John Wayne Gacy from Illinois, a contractor and Junior Chamber of Commerce "Man of the Year" who entertained children as Pogo the Clown, had his picture taken with First Lady Rosalynn Carter and murdered 33 teenage boys in the Seventies, burying most of them underneath his house. Kenneth Z Taylor, a popular New Jersey dentist who abandoned his first wife, tried to kill his second wife, horrifically beat his third wife on their honeymoon in 1983, battered her to death the following year, hid her body in the boot of his car while he visited his parents and his second wife, and later claimed self-defence saying she attacked him following his "discovery" she was sexually abusing their infant child. Dr Hannibal Lecter, as imagined by Anthony Hopkins, Mads Mikkelsen, Brian Cox and others, is always charm personified. There's now a hit musical about Patrick Bateman.

American Psycho

In his 1941 book The Mask of Sanity, Hervey M Cleckley uses phrases like "shrewdness and agility of mind","talks enthusiastically" and "exceptional charm" to illustrate his case studies of psychopaths. A more recent study Without Conscience: The Disturbing World of the Psychopaths Among Us by Robert D Hare, PhD, asks us to consider that "there are at least two million psychopaths in North America; the citizens of New York City have as many as 100,000 psychopaths among them. And these are conservative estimates. Far from being an esoteric, isolated problem that affects only a few people, psychopathy touches virtually every one of us." Doctor Hare isn't talking about someone chopping up waitresses with an angle grinder while listening to Huey Lewis and the News. He's talking about white-collar psychopaths: people who use charm in business, who step on others as they literally charm their way to the top. American Psycho was, after all, a capitalist satire.

All this was making my head hurt a bit so I thought I'd better go and speak to an expert. Dr Raina Brands is assistant professor of Organisational Behaviour at the London Business School, and lectures in charm and charisma. I met her one afternoon in her bright, second-floor office, which she shared with one of those computers with a dual monitor, and a running machine. She was pretty charming herself. She laughed when I told her about David and Paul ("Your neighbour sounds very disagreeable") and pointed out that some of what I'd described was down to some people just being more socially skilled than others. Then she said that charm and charisma tend to be in the eye of the beholder. A neural network in your brain figures out that such-and-such a person is emotionally exciting to you so then you project the qualities you think they must have onto them. You follow their directions because you've made an attachment to them.

"Charm is a very indefinable quality," she said. In terms of helping the species survive, evolution has split us into groups: the variation in our temperaments serves a social purpose. One of the ways that helps human beings get along requires followers and leaders, so you have very popular people at the centre of a network, and other people who want to help those people. There is also something called self-monitoring, where people adapt their behaviour and present different versions of themselves depending on the demands of the context. This seemed pretty obvious: the person you are outside the pub on a Friday is unlikely to be the same person you are talking to your mum on the phone on a Sunday. But Dr Brands said that high self-monitors were especially good at adapting, and were often marked out as particularly charming because of their chameleon-like ability to win over very different people in very different types of situations.

The key to charm is empathy. David got along with the dry cleaners and the office interns because he actually liked them and because of that he remembered things about them and asked them about those things the next time he met them. That's really flattering. And it's really hard to fake. (An aside: the 19th-century socialite Jennie Jerome, mother of Winston Churchill, once, on consecutive nights, dined alongside Britain's Leader of the Opposition William Gladstone and then his political rival, Prime Minister Benjamin Disraeli. Recalling her impressions of the two men, Jerome later wrote: "After dining with Mr Gladstone, I thought he was the cleverest person in England. But after dining with Mr Disraeli, I thought I was the cleverest woman in England." Disraeli beat Gladstone in the 1874 election.)

"Everybody is the centre of their own universe, and you spend most of the time in conversation talking about yourself," Dr Brands said. "So if somebody can turn the spotlight onto you, that just makes them more likeable. All that matters is that person recognises it as genuine, whether it is or not, that's enough."

I wondered why women were hardly ever called charming. I mentioned Bill Clinton, that famous lighter-upper of rooms and Hillary Clinton, who was seemingly unable to catch a break. "That's an interesting observation," Brands said. (I felt good about that.) She had studied gender bias in charisma and her findings were sobering to say the least. "Charisma is a leadership characteristic and leadership is a male stereotype. Women are not recognised as good leaders," she told me. "They are held to much higher performance standards. Charisma and charm has a kind of heroic element to it, a visionary element which is a very masculine stereotype. It's a very gendered concept."

Obama Foundation

So, people don't mind Hillary Clinton in office but whenever she seeks power, people turn on her (famously, in the last election, women just as much as men). Because leadership requires so many masculine traits: dominance, assertiveness, being a visionary, taking power, traits that are seen as inappropriate for a woman. During the leadership debates on American TV, Clinton had to keep her cool because she simply wasn't "allowed" to get angry — though Lord knows she was goaded enough. Opposing her, Donald Trump could be very dominant with a very clear, very patriotic vision — "Make America Great Again" — a strategy that, as a woman, simply wasn't available to Clinton.

Brands explained that Bill Clinton can make a joke or a personal connection and nobody is going to worry whether or not he's competent, whereas women have to face a warmth/competence trade-off. "As a woman, if you're warm people immediately start to think of you as not-competent, and if you're competent people immediately start to think of you as kind of a bitch."

I asked Brands about psychopaths. The Without Conscience book mentions a list of traits that share a kind of superficial glib charm. Psychopaths tend to be egocentric, lack remorse, are deceitful and manipulative, have shallow emotions and, most crucially, lack empathy. "Psychopaths are often witty and articulate," the book explains. "They can be very effective in presenting themselves well and are often very likeable and charming." When one psychopath, in prison for a range of crimes including robbery, rape and fraud is asked if he has any faults, his reply is: "I don't have any weaknesses, except maybe I'm too caring."

Playing devil's advocate, I wondered whether this actually mattered in business. If these traits allowed someone to succeed in the corporate world, as long as they weren't attacking people with a meat cleaver on the side, in some circumstances couldn't this behaviour be seen as beneficial?

Brands looked disappointed with me. "I would never, ever, ever advocate hiring a psychopath," she said. "There are obviously a lot of psychopaths represented in the prison population but psychopaths are also over-represented in the corporate world. And interestingly, sending them to counselling just makes them better psychopaths. Because they get a better understanding of how people think and work."

In the corporate world there is one person whom I consider a friend, who also happens to be the most charming man I've ever met. His job relies on it. He is exceptional at it. His name is James Massey and he is the MD of The Massey Partnership, a high-end PR company. His clients work exclusively within the luxury and luxury travel sector — fancy hotels, beautiful watches, handmade shoes, Savile Row tailors and one brand that makes the world's finest underpants. You get the idea. His job requires him to persuade magazines like this one and other glossies, which in The Devil Wears Prada tradition can occasionally be staffed by high-maintenance and idiosyncratic fashion types (though obviously no one at Esquire, we're all great), to write about his clients, who (I'm guessing here) can be equally demanding and occasionally not just high-maintenance and idiosyncratic fashion types but also Italian. It must be a nightmare.

Yet I've never seen James remotely flustered. For every appointment I have with him he is punctual to the minute. (I know this because I always arrive sweating, as the maître d' is helping him to his seat.) He is never less than sharply dressed in a buttoned blazer and polished loafers. His chosen mode of communication is not the, "Hey, how are you?" email but the handwritten thank-you note, double-sided on duck-egg blue card from Westminster's finest stationers, and composed via an impressively nibbed fountain pen. It goes without saying that his manners are exceptional. Naturally, Massey would never be so charmless as to describe himself as charming. But I thought he might have interesting things to say on the subject.

"I think charm is putting people at their ease, fundamentally. I think it's about showing consideration to them," he told me. "And you have to enjoy other people's company. I think that's key. I think it's about noticing things and I think it definitely involves kindness. You need to be as kind, respectful and engaged when you're talking to a waiter as you are talking to a president. I think one of the worst expressions ever is 'turning on the charm'. It's a dreadful expression. It's false and it's easily noticed. If it's unnatural behaviour it comes across as obsequious."

He agreed that charm was not the same as charisma. "Charismatic people walk into a room and people just want to be around them and immediately they're the centre of attention. But I think a lot of charismatic people can be bastards — perhaps not publicly. Whereas I think charm can quite often be a private thing between two people, or a small group of people." (It's a myth that introverted people can't be charming. And think about it: the loudest person in the room is seldom the most charming.)

The cliché of Massey's profession — and surely fashion journalism's too, I suggested — was the blowhard editor or PR banging his or her desk and busting a gasket because their exacting demands hadn't been met.

"I think everyone has their approach," he said, diplomatically. "But I don't think people do things for people who aren't likeable. Not on a sustained basis, anyway."

It's nice to be nice?

"Well, it is. I think it used to be very unfashionable. The days of the super-aggressive Italian PR screaming is just not cool anymore. I genuinely believe life is too short to be an arsehole. It can't be a particularly fulfilling experience to live like that, you know? There's enough horrible stuff going on in the world without adding to it."

Illustration by Shout

Google-search the word "charm" (your journalist is nothing if not diligent with his research) and the topic it's most often linked to is charm schools. Not so much those balancing-books-on-heads finishing schools the like of which Diana, Princess of Wales, was sent to by her father after failing all her O-Levels, although incredibly those do still exist, but courses that promise they'll reveal the secrets to finding love. They appear to be almost universally a) aimed at foreign students and b) extortionately expensive. It seems that part of our current understanding of charm is that it's linked to the wooing of the opposite sex; that, despite what's been discussed above, charm is something that can be learned.

Years ago, men's magazines like this one used to be full of these kind of tips (regrettably a couple of them still are): how to pull a chair out for a lady, the correct cutlery to use to eat your soup (it's the spoon), that kind of stuff. There are plenty of books like The Charisma Myth: Master the Art of Personal Magnetism ("The International Bestseller!") that promise the same kind of thing. "Charismatic people seem to lead charmed lives: they have more romantic options, they make more money, and they experience less stress," writes the author Olivia Fox Cabane.

"Charisma gets people to like you, trust you, and want to be led by you… it makes people want to do what you want them to do." There's chapter after chapter dedicated to life-hacks and advice, the importance of body language — how the MIT Media Lab once did an experiment that concluded it could predict the outcome of negotiations, telephone sales calls and business plan pitches with 87 per cent accuracy simply by analysing participants' body language, without listening to a word of context — and how to neutralise negative thoughts that cause anxiety and stop you presenting your best self at business lunches. How to be comfortable with discomfort. How to locate your "happy place".

Stephen Bayley, the acclaimed design expert and author has written more insightfully about charm. "Charm is mysterious, romantic and appealing," he says. "It is a subtle but irresistible commodity. With charm you can create good situations and extract yourself from bad ones. The charmless and the tedious are surely disadvantaged in a corporate world where character outguns qualifications, where appearances matter, and favourable reactions count. So why is it so little understood? Understanding charm is like trying to embrace fog."

Jokes are never charming. ("A charming man is someone who laughs at your jokes," he says.) Empathy is always charming. So is listening. Never complain. Ever. There is absolutely nothing that people enjoy more than the suggestion you find them fascinating. Be prepared to say: "Do tell me more about how you did your own conveyancing on the basement flat in Chigwell."

Bayley traces the history of charm, from when "gentleman" became a specific social rank, above yeoman but below esquire. Soon caricatures of the gentleman became stock characters in William Shakespeare plays and by 1630 books like Richard Brathwaite's The English Gentleman had begun to establish a literary and social type. A true gentleman was one whose personality transcended his birth and caste. But Bayley concludes that charm has as many vices as virtues.

In business, and in love, the charmer works alone. Charm is one of the great modes of persuasion, a legal way to gain an unfair advantage, he says, adapting advertising legend Maurice Saatchi's description of creativity. The idea of the confidence man is still applauded, particularly in America, where he was first written about in 1849: charmingly persuading passers-by to part with their pocket watches, never to be seen again. Why is this? Because the conman seems to embody the highly motivated, the self-made man. It's true: we tip our hats to Fagin, that inspired, entrepreneurial — and charming — genius.

Did my colleague David think he was charming? I phoned him up and asked him. Funnily enough, he said, he'd been talking to his parents about this recently. They'd told him they didn't know what had happened to him. Up until the age of about 10 he was really shy. But David knew what had happened. He'd changed school and because some of the bigger lads were jealous of the attention he was getting off some of the girls (girls again, gah!), he deduced he was in imminent danger of getting his head kicked in. So he learned to be funny, to clown it up with silly anecdotes and daft behaviour. (His dad has the gift of the gab, he said, so that helped.)

Today, whenever he's got something big on for work, an important pitch meeting for example, he'll often be so nervous that he won't sleep the night before. His way with dealing with the actual event hasn't changed since school: he'll go in and try and make everyone feel comfortable by over-performing and trying to entertain them as much as he can.

"I turn into the fucking town jester," he said. The same applies to the new intern situation. He can remember what that felt like himself and he still remembers who was nice to him and who wasn't in that situation. He said he couldn't bear the idea they might be sitting there feeling uncomfortable, so he steams in and tries to make them feel good.

One night recently, he and a friend were in a restaurant. It was understaffed and overcrowded and their orders got messed up. His friend started complaining so David had a go at him. "It made me feel so awkward," he told me. "'Cos I wanted the staff to feel OK. I said, 'You cannae complain, you can see the situation they're in, they're getting paid £7 an hour.' But he couldn't see the situation they were in. He just thought, 'I'm paying for this.' And it was the most inappropriate piece of complaining I've ever seen in my life."

That's not to say David's behaviour has always been completely altruistic. At school, his mates picked up on something they labelled "The Dinner Lady Technique". One Christmas, David bought the head cook in their canteen a box of chocolates.

"I don't know if that was charm," he said. "It was charm in my head. All my mates were like, 'You're a fucking creep.'"

But deep down, David knew exactly what he was doing.

"I was saying, 'I'm going to get extra chips for the next year.' And I did get extra chips for the next year."

Researching this article I contacted The School of Life, an organisation that offers classes and therapies aimed at developing emotional intelligence. It so happened they had an evening coming up titled "How to be Sociable". "What is charm and how do we acquire it?" it asked, and promised to answer that very question. I emailed someone there who was kind enough to reply and say that I was welcome to come along. She was called Sarah Byrne.

"Thanks very much, Gemma," I typed back in response.

Then the penny dropped.

"Sorry Sarah," I hurriedly emailed. Then I added: "I need to work on my charm."

"That's fine," she replied. "Ha ha."