On a street in east London lies the Pet Shop Boys’ studio; a pile of rubbish is dumped outside. One room is full of synthesizers; the other has mid-century modern furniture and art by Scott King depicting tower blocks amid Technicolor waves. Here, Neil Tennant is talking about Brexit. “I think everything comes down to social media really, and social media promotes emotional illogicality in all its forms: racism, prejudice and of course nationalism.”

He warms to his theme. “Any multinational empire is going to have an irritating bureaucracy – it’s just a fact. Why is it better for that to be a lot of supposed nation states? When I was at North London Poly in the early 70s, I wrote a defence of the Austro-Hungarian empire. I still think I was right. Stability is very easy to find boring, but afterwards you can appreciate it. I like Joseph Roth, who was a Jewish writer who wrote The Radetzky March, and in his books he sees the Habsburg monarchy as the defenders of all the minorities, including the Jewish minority …”

Listening to this well-read and confidently expressed view, it might seem surprising to think that Tennant has devoted his life to writing not academic papers or newspaper columns but pop songs – but the proof is a slim volume on the table. Titled One Hundred Lyrics and a Poem, the book’s minimal white jacket encases his life’s work: songs about sex and politics, love and despair, a whole panorama of British life. “Really quite often, a publisher says, ‘Let’s get Neil Tennant to write his autobiography’ and it’s quite nice that they do,” its author muses. “I’m not convinced my life’s been interesting enough. This is my autobiography.”

One Hundred Lyrics and a Poem collects the Pet Shop Boys songs Tennant thought looked best written down (so no Heart, Love etc or Shopping), with his introductory essay and commentary. There are the words to huge hits such as It’s a Sin; and obscure b-sides such as The Ghost of Myself, in which Tennant remembers living with a girlfriend in the late 70s, before he came to terms with being gay. He has written songs his whole life, first as a teenage hippy in his native Newcastle, then as a Pet Shop Boy. “I remember as a boy hearing Strawberry Fields Forever and also reading John Lennon’s explanation that he wanted it to be like a conversation, and that had a very powerful impact on me,” he says. “And I remember reading an interview with Frank Sinatra where he said you should phrase lyrics like a conversation. I’ve always tried to do that. Someone who you might not think of as the world’s best lyricist is Madonna, but she always gets the emphasis on the right syllable.”

Facebook Twitter Pinterest Neil Tennant (right) and Chris Lowe in 1986, the year before It’s a Sin kicked off the Pet Shop Boys’ ‘imperial period’. Photograph: Mike Prior/Getty Images

He met Chris Lowe, then an architecture student, in an electrical shop on London’s King’s Road in 1981, a year before he started as news editor on the pop magazine Smash Hits. He and Lowe wanted Pet Shop Boys songs to have the raw excitement of the electro, hi-NRG and hip-hop coming out of New York, a city then as scary as it was inspiring: “Every time you left New York in the early 80s you thought, ‘Wow, survived another trip’.” Their lyrics, however, were distinctly English: sometimes direct, even banal (“I always thought banality was a particular talent”), but more often funny and perceptive, with a far wider perspective than most pop songs.

On one level their first No 1 hit West End Girls was about the seedy glamour of a night out in London, but Tennant also slid in a reference to Edmund Wilson’s To the Finland Station, a history of socialism from the French revolution to Lenin’s arrival in St Petersburg. As if slightly embarrassed by this erudition – the Pet Shop Boys were always militantly pop (modern, glamorous, artificial) as opposed to rock (raw, traditional, authentic) – they never used to print their lyrics on their album sleeves. “We probably had some ideological point about it that we lost interest in,” says Tennant. “I wonder if we thought it was rock or something. You see that whole thing went away … because rock lost.” He hoots with laughter.

Tennant’s sharpest lyrics still resonate today. When the Conservative party conference unveiled its slogan Opportunity last month, wits immediately tweeted: “I’ve got the brains, you’ve got the looks/Let’s make lots of money” – the chorus to their song Opportunities, which satirised the Tory zeal for enterprise. “That was a classic early Thatcherite notion,” Tennant says, adding that it was the puckish Lowe who came up with the line “Let’s make lots of money”.

In 1987, It’s a Sin, a disco blockbuster about Catholic guilt, got to No 1 in 11 countries and kicked off what he famously called Pet Shop Boys’ “imperial period”, the stage where a group can do no wrong. Though Tennant’s commentary in the book makes the subject matter of some songs more explicit, the Pet Shop Boys’ sexuality was somewhat coded at the time – obvious to those in the know (they posed in full leather gear on the cover of Smash Hits) but destined to go over the head of most teen pop fans. So was the sin in question homosexuality? “I think it just meant sex,” says Tennant. “When you’re an adolescent boy at a Catholic school you’re taught that sex, apart from reasons of procreation, is a sin. Going out and getting pissed with your friends is a sin.”

Then there was Rent, recently the subject of a tweeted inquiry by Pet Shop Boys fan Cardi B as to what the words are about. “We were trying to write provocative lyrics,” Tennant says. “You’d hear someone in a gay club say, ‘Oh, he’s rent’. It’s nostalgie de la boue, nostalgia of the gutter. We both like the pathos of streetlife.” Rather than a rent boy however, Tennant imagines the subject of the song to be the mistress of a powerful politician, kept in an apartment on New York’s Upper East Side. “I never know quite what it’s about, really. But I always quite like that in pop songs.”

Facebook Twitter Pinterest Pet Shop Boys on stage at the Montreux rock festival in 1986. Photograph: David Redfern/Redferns

Some of Pet Shop Boys’ most moving songs were Tennant’s response to the Aids crisis. “When I decided I was gay was pretty much when Aids came in, so you were paranoid,” he says. Then, “this friend of mine from Newcastle, my closest friend in some ways, suddenly goes down with HIV. And that was when Suburbia was in the top 10.” Tennant spent a good deal of his imperial period in the Aids ward of St Mary’s Hospital in London, “watching [my] friend waste away”. Christopher Dowell, who died in 1989, is commemorated in three songs including Being Boring, a devastatingly sad memorial to their friendship.

It’s something he thinks about still. “I had a very strong group of friends as an adolescent, many of whom I still know – we’ve got a new song which looks back at that,” Tennant says. “It was a very intense part of my life and it sort of ended with the Aids crisis, with so many friends who died, so I will probably never get over that. I don’t mean in a traumatised way, just that it’s always there in my history. It’s part of who I am.”

Tennant finally came out to Attitude magazine in 1994. Does he wish, like Olly Alexander and Troye Sivan today, he had written unabashedly gay songs filled with male pronouns? “In the 80s and the 90s, for that matter, it was such a big deal, being gay,” he says. (In 1987, a British Social Attitudes poll found that 75% of the general public thought homosexuality was “always” or “mostly” wrong; the following year the Thatcher government brought in section 28, which prevented it being “promoted” in schools.) “You knew your audience had a lot of women or girls in it, so you wanted to include everyone. I still sort of think that when I’m writing, to be honest. Also I don’t write about my life in the direct way that most, if not all, artists do nowadays. Sometimes I think, ‘Where’s the art, where’s the poetry in all of this?’ Lily Allen can write these amazingly and actually quite funny direct slag-offs of people and stuff like that. That’s just not who I am, I’m afraid. It isn’t anything to do with pronouns, it’s to do with poetry really.” He laughs.

It’s also the reason why, for all the Pet Shop Boys songs commenting on society and politics (including three about Tony Blair), they have never written a direct protest song. “I wouldn’t write a song called Second Referendum Now, even though I think there should be one,” Tennant says. “We have written a song called Give Stupidity a Chance which is a satire. It’s close to a protest song, but it’s also funny.” Pop music’s strength, he believes, lies not in making a party political point but in summing up the atmosphere of the time – as the Specials managed with Ghost Town. “When that came out in the middle of the Toxteth riots era, everything about it was a political statement, but it wasn’t putting forward a political programme.”

What about just writing a straightforward love song? Tennant is surprised when I suggest that they have become more infrequent on Pet Shop Boys albums. “Maybe there’s been less to write about, I’m afraid.” He pauses, slightly embarrassed. “Not totally. Actually on das neu album is a major love song.”

Would he ever avoid writing about anything too intimate? “Sex or something? No, one has to think of the person that’s the subject, so you’ve got to bear their feelings in mind.” He says he has the “slightly cold and dispassionate” ability to be having an argument with a lover and realise that an accusation like “You only tell me you love me when you’re drunk” will make a good title for a song.

He has never had writer’s block, never considered stopping writing. The closest Pet Shop Boys have ever come to splitting up was in 1999, when the concert promoter Harvey Goldsmith went bankrupt while they were on tour. “We were playing to half-empty arenas, losing a fortune. It came to a head one night at Sheffield Arena. I said to Chris, ‘Why don’t we just pack it in?’ And Chris didn’t answer. So we started talking about something else.” Their forthcoming album will be their 15th, not counting live and compilation albums, and will include a song inspired by the refugee crisis, and another about Berlin, the city where they go to write and – occasionally on a Sunday afternoon – to dance at Berghain, the legendary techno club.

Tennant became famous at 31; he’s now 64. The single poem in the book contemplates his mortality; three songs were inspired by funerals. “People fall away, you know,” he says. “This year we’ve had quite a few friends, all of them quite a bit younger than me, die.” I ask about the death of George Michael, a pop peer nine years Tennant’s junior. “I felt sad and almost angry because it seemed like such a waste. He was so young and also I think he was on a path it would have been possible to reverse. But he was very stubborn, George.”

He adds that while they didn’t know each other well, their relationship spanned three decades. “I first met him in 1982. I interviewed him and Andrew [Ridgley] for Smash Hits and then the last time we saw him it was exactly 30 year later, at the Olympic closing ceremony. We were in these Portakabin-y dressing rooms and the person next to us is playing music unbelievably loudly. I said to our tour manager: ‘Can you go and ask him to turn that down, please?’ And suddenly the door flings open and George, who we hadn’t seen since he’d been in jail, comes in and says: ‘Did you just tell me to turn my music down?’ I said: ‘Yes, I did.’ And he says: ‘Give me a hug.’ And then he went back to his dressing room, put his stereo on and played West End Girls – loudly.”

The front door opens. It’s Lowe. “You know there’s all this rubbish outside? You’re going to have to phone the council.”

“I’m going to, actually,” says Tennant.

“It’s turning into a public … tip!” says Lowe, aghast.

“Actually, a fire risk!”

“Rats, the whole lot!”

It’s time to go. “He’s good at talking, isn’t he,” says Lowe, “especially about himself! Did you ask him about Brexit?”

• One Hundred Lyrics and a Poem by Neil Tennant (Faber, £14.99) is published on 1 November. To order a copy for £12.89, go to guardianbookshop.com or call 0330 333 6846. Free UK p&p over £10, online orders only. Phone orders min. p&p of £1.99.