It was not in New York but in Germany, that the writer, editor and photographer Bob Colacello bought a “spy camera”.

He was with pop artist Andy Warhol, who was commissioned to take Polaroids of Willy Brandt, the then chancellor of West Germany, in 1976.

Colacello bought a Minox 35 EL, a camera used by cold war-era spies. As the right-hand man to Warhol, he pulled it out when hardly anyone in high society was posing.

Now, 170 photos are going on view as part of Pictures From Another Time: Photographs by Bob Colacello, 1976–1982, an exhibition at Vito Schnabel Projects in New York. These behind-the-scenes shots of Truman Capote, Mick Jagger and Liza Minelli trace Colacello’s access as a VIP connoisseur.

“I never had to social climb, Andy knew I could talk to anyone, so he started taking me to parties with him,” said Colacello, sitting in Schnabel’s office, wearing his trademark tortoiseshell Ray Bans.

There is a photo of Warhol and Brandt in the exhibition, which sums up its crux – it’s the meeting of celebrity with politics.

“Andy was a famous artist that had access to everyone,” said Colacello. “Who was buying the art? The rich people. Who was funding the politicians? The rich people. It all blended.”

There are photos of Jimmy Carter’s inauguration in 1977 and Ronald Reagan’s inauguration in 1981, both of which Colacello attended with Warhol. “Andy voted for Carter, I voted for Reagan, my family was Republican,” he said.

There are shots of Warhol with Jerry Zipkin, a real estate heir who was close with the Reagans (and was known for his argumentative nature – a taxi driver once broke his ribs in a fight, but Zipkin apparently still attended the dinner party).

“People wouldn’t typically think Andy would be friends with Nancy Reagan’s best friend,” said Colacello. “Jerry helped Andy get commissioned portraits of rich ladies.”

From Ringo Starr to Robert Rauschenberg, there’s the odd shot of a young John Travolta with sideburns, Estee Lauder having a drink and Robert Mapplethorpe smoking. “I loved taking photos with hands blocking out a face, it’s not people posing, it’s a party,” he said.

Bob Colacello – Gregg Allman and Cher, Carter Inauguration White House Reception, 1977. Photograph: Bob Colacello

There’s also a sneaky photo of Cher and Greg Allman kissing at the White House during Carter’s inauguration reception. “They didn’t know I was taking a photo,” he said. “I wasn’t too obvious.”

He may be discreet, but Colacello calls himself a lazy photographer. “Andy would take four rolls of film in a night, I would only take one,” he said. “It was too much work to change the roll at a party.”

But one wonders if Warhol was running for president of the art world and Colacello was his campaign manager. Or at least, his personal ambassador.

“People always asked, ‘what do you really do for Andy?’ and I said, ‘I’m his translator, from English to English,’” said Colacello. “Andy would say ‘oh gee, oh wow’. And I’d say, ‘Andy loves your apartment.’”

Before he got into writing film reviews for the Village Voice – which ultimately got him noticed by Warhol – Colacello studied international affairs at Georgetown University School of Foreign Service in 1969 (then went on to study film at Columbia University).

“A big part of the foreign service was how to be a diplomat,” he said. “My own personality, I wasn’t shy, but I was never pushy.”

His diplomatic skills helped Warhol’s career but also his own as editor of Interview from 1971 to 1983, then as a staff writer for Vanity Fair from 1984 to 2017. “My great grandmother used to say, ‘it’s not what you say but how you say it,’” he said. “That has been a real rule for me and my journalistic career, I always tried to keep the tone neutral, not smarmy. You get so much more out of people if you pretend to be friendly.”

Bob Colacello – Self Portrait, 1976. Photograph: Courtesy the artist and Vito Schnabel Projects

In 1990, he wrote a memoir called Holy Terror: Andy Warhol Close Up, detailing his role working for Warhol, then in 2004, wrote a biography of the Reagans called Ronnie and Nancy: Their Path to the White House.

In one shot from 1978, he captures the former Vogue editor Diana Vreeland, who is raising her manicured hand in conversation at the Doubles Club, a private members club on Fifth Avenue.

“I think people are all the same,” said Colacello. “I realized it when I went to Italy and saw these fancy Italians that were just like my grandmother and my aunts in Brooklyn, sitting around the dining room table talking about the neighbors in a way to make each other laugh, entertaining each other.”

“OK, these people are a million times richer, they have big titles and live in palaces,” he adds, “but it could be Aunt Jenny and Grandma”.

The photos capture the height of the disco era, which symbolized a different kind of freedom than the 1960s. Many shots were taken at Studio 54. “It was uptown, downtown, gay, straight, rich, poor, everyone had fun,” said Colacello. “Cocaine was a way to stay up even later and dance to I Will Survive.”

Colacello isn’t widely recognized as a photographer, per se. In the photo book Andy Warhol’s Exposures, roughly 20 photos were taken by Colacello and credited to Warhol (they have “BC” on the backs of them). “It was a job,” he said. Though, he used to show with Steven Kasher Gallery, he has shown at the Andy Warhol Museum in Pittsburgh and published a photo book with Steidl in 2007.

When he wasn’t attending Jackie Kennedy’s Christmas party, he was dining with Bianca Jagger. “We were the only two people in the whole group who sat down to talk about the situation in Nicaragua and actually talk geopolitics,” said Colacello.

Bob Colacello – Liza Minnelli with Marisa Berenson on her wedding day, Beverly Hills, 1976. Photograph: Bob Colacello

When he saw the former secretary of state Henry Kissinger at a party, he didn’t rush over. “I would not go up and put a flash in his face, my photo of Kissinger is blocked halfway,” said Colacello. “I held back and took it at a distance.”

His photos not only capture a more carefree time, but a time when the country was far less divided than it is today. “It was just before Aids, but after Watergate and Vietnam,” he said. “I see artists at the Carter inauguration, ladies in fur coats at the Reagan administration, there was much more humor. You could still make jokes about yourself. Today there’s a vast humorlessness over everything.”

He recalls Kissinger making fun of his own accent and Fran Lebowitz complaining about anything too trendy. “It just was another time,” he said. “It was a period where we were resting on our laurels.”

The debates between Reagan and Carter were less vicious. “They weren’t calling each other liars, it was still rather civilized,” he said. “It was Walter Cronkite who just read you the news, we all watched the same three TV channels.”

It might be easy to write off dinners and cocktail parties as superfluous or unnecessary fodder for the worlds of business and politics, but casual conversations were the starting point for more important meetings. “That’s a part of why Washington got more and more polarized, because people didn’t know each other,” said Colacello. “It’s easy to hate a name, it’s hard to hate a human being.”

Could this be a call for more people to come together and sit face-to-face, rather than hiding behind their screens? “Even if you sit down beside someone at a dinner party who doesn’t share the same beliefs, it takes the sting out of our opposition to one another,” he said.

“In a way, what Donald Trump lacks is a sense of humor about himself,” said Colacello. “If he could admit he’s kind of a funny character, people might like him better.”