“He made it uncomfortable for the cast and Adam Driver by calling them back onstage and not saying why and then went off on them,” the source tells PEOPLE

Kanye West got a lot off of his chest during his appearance on Saturday Night Live — and ended up making some of the show’s stars uneasy, a source tells PEOPLE.

While wearing a Make America Great Again hat, the rapper, 41, wrapped up the show’s season premiere with a third musical performance and a lengthy speech about his support of President Donald Trump that did not make it to air.

Get push notifications with news, features and more.

“He made it uncomfortable for the cast and [host] Adam Driver by calling them back onstage and not saying why and then went off on them,” the insider tells PEOPLE.

“The show had been supportive of all of Kanye’s visions throughout the week, even giving him the opportunity for the third song during the goodnight [portion],’ and then he surprised everyone,” the source adds.

Image zoom Kanye West NBC

The insider went on to explain that although West’s rant wasn’t televised, the show wasn’t trying to silence the outspoken musician.

“The show was already off the air when he started his rant. It goes off at 1,” the source shares, adding that West’s specific claim that he had been “bullied backstage” about wearing the MAGA hat wasn’t accurate.

Image zoom Kanye West, Adam Driver and Keenan Thompson Rosalind O'Connor/NBC

“No one bullied him about wearing the MAGA hat. He wore it in promos and all week before the show even aired and was asking everyone if they thought he should wear the hat, interns, anyone walking by who would listen,” the insider remarks.

RELATED VIDEO: Kanye West’s Behavior ‘Can Be Exhausting’ for Kim Kardashian: Source

Despite some of the negative reactions to West’s speech, his family — including wife Kim Kardashian West, 37 — didn’t seem bothered at all by the surprise rant.

“While Kanye talked, Kim just stood in the back and watched,” the source shares. “The kids were there, too. She didn’t seem like anything [was up], she was just watching.”

“Trump’s been good to them, so it seems they do really love him, even Kim,” the source continues, adding that while the family didn’t attend the afterparty, “they all hung out for a bit afterward.”

Kardashian’s daughter North, 5, was also seen dancing along to his performances during the show, while Kourtney Kardashian, 39, and Scott Disick, 35, were also in attendance for the spectacle.

Of course, there is one person who was very impressed by West’s performance: the president himself.

Responding to West’s support, Trump tweeted on Sunday, “Like many, I don’t watch Saturday Night Live (even though I past hosted it) – no longer funny, no talent or charm. It is just a political ad for the Dems. Word is that Kanye West, who put on a MAGA hat after the show (despite being told ‘no’), was great. He’s leading the charge!”

West also shared that he enjoyed his time on the program.

“Had so much fun at SNL,” he tweeted on Sunday. “[Producer] Lorne [Michaels] agreed that I would host before the year is out. Need to set a date. Gonna be sooooooooo lit. In the words of Lil Pump …Esskeetit !”

Hours later, he went on to share a separate message on Instagram while wearing the MAGA hat, which he previously claimed to have “redesigned” in order to fill the controversial fashion choice with “positive energy.”