"It's almost painful to watch the president and first lady together," historian Douglas Brinkley tells PEOPLE about Donald and Melania Trump's marriage

Donald and Melania Trump's Marriage Is Most 'Awkward' Presidential Union in Modern Times: Historian

Prominent presidential historian Douglas Brinkley doesn’t think Donald and Melania Trump‘s marriage is “hinged right.”

“In modern times, we haven’t had such an awkward marriage — one that seems to be, if not unraveling, it seems to be frozen. It doesn’t seem to be a very organic thing,” Brinkley, a professor of history at Rice University and CNN’s presidential historian, tells PEOPLE.

Get push notifications with news, features and more.

The first couple’s public image has been defined by viral video of Melania, 48, swatting away her husband’s hand when he’s tried to hold hers.

“That scene of the hand swat,” says Brinkley, “is a visual metaphor for a marriage that isn’t hinged right.”

Image zoom Donald and Melania Trump SAUL LOEB/AFP/Getty

It’s quite a departure from the many loving images of Michelle and Barack Obama, and of a doting George and Laura Bush during their tenures in the White House. But those husbands didn’t have a porn star and a Playboy model alleging affairs, or 19 women accusing him of sexual misconduct.

“It’s almost painful to watch the president and first lady together — they seem uncomfortable in the same photo frame,” Brinkley says. “There’s not a lot of warmth that is being exuded.”

Most likely, the viral news stories about Trump’s womanizing “all gets through to the first lady,” Brinkley says. “She knows this is happening.”

Earlier this year, Playboy playmate Karen McDougal said she had a 10-month sexual relationship with Trump that began in 2006.

And Rudy Giuliani shockingly confirmed the president reimbursed his personal attorney Michael Cohen for the $130,000 paid to porn star Stormy Daniels to stay quiet about an affair she alleges she had with Trump shortly after Melania gave birth to their son, Barron, in 2006. Daniels passed a lie detector test when asked if she had unprotected sex with Trump.

Trump has denied both affairs.

RELATED VIDEO: PEOPLE Writer Natasha Stoynoff Breaks Silence, Accuses Donald Trump of Sexual Attack



“There’s always the chance they are working things out, but I have to imagine Stormy Daniels is a major psychic blow to Melania Trump,” says Brinkley. “To have your spouse be having unprotected sex with a porn star is not something that causes endearment.”

Brinkley says that while some marriages can “sometimes get healed by apologies,” saying ‘I’m sorry’ is “not Trump’s strong suit.”

Image zoom Barron, Melania and Donald Trump BRENDAN SMIALOWSKI/AFP/Getty

“I think you have Melania wanting to put Barron first,” he says. “That’s the man in her life, taking care of her child.”

As devoted as Melania may be, the visuals of a happy home life are largely absent to the public.

“Usually, when there is a child in the White House, there is a feeling of joy and laughter,” Brinkley says, referencing the Kennedys and Obamas. “That isn’t emanating from the current first family.”

Image zoom Donald and Melania Trump Carolyn Kaster/AP/REX/Shutterstock

Meanwhile, Melania is getting some love from the public, with her approval rating at an all-time high — and beyond her husband’s.

Brinkley thinks the first lady has “positioned herself well in the public eye,” with the issues she’s decided to champion.

RELATED VIDEO: Did Melania Trump Reject Donald’s Attempt to Hold Her Hand for a Second Time?

“Melania Trump picking cyber bullying almost seems to be a jab at her husband,” he says.

Brinkley also sees far beyond the presidency for the Solvenian native.