Conservative writer Charles Murray said he has “never encountered anything close to” the environment Thursday night at Vermont’s Middlebury College, where hundreds of student protesters disrupted the controversial social scientist’s scheduled lecture.

Protesters turned their backs to Murray, a scholar at the American Enterprise Institute think tank, and chanted, “Racist, sexist, anti-gay, Charles Murray, go away” and “Your message is hatred. We cannot tolerate it.” Some stomped their feet and set off fire alarms, according to the Associated Press, prompting administrators to move Murray to a video studio, where he livestreamed and recorded his remarks.

Murray sparked national controversy with his 1994 book, The Bell Curve: Intelligence and Class Structure in American Life, which was widely criticized for linking intelligence level with race. He has been described as a “white nationalist” by the Southern Poverty Law Center, which accused him of “using racist pseudoscience and misleading statistics” in his work. Last year, after the president of Virginia Tech criticized Murray’s work ahead of a lecture there, he published an open letter defending his scholarship.

The Middlebury event is the latest recent incident to raise the question of free speech on college campuses, pitting protesters against visiting speakers. At the University of California, Berkeley, an event featuring provocative right-wing pundit Milo Yiannopoulos was canceled last month after protests turned violent. And at New York University, multiple people were arrested last month during protests over a speech by conservative comedian Gavin McInnes.

“The inmates ran the asylum last night at Middlebury,” Murray told TIME on Friday. “The administration really did very well, and I have no complaints about the Middlebury administration. In fact, they stood up for free speech.”

In a letter published Wednesday, more than 450 Middlebury graduates called the college’s decision to host Murray “unacceptable and unethical.” Murray, who was invited by the student American Enterprise Institute Club, said he had intended to speak about his 2012 book, Coming Apart: The State of White America.

“His invitation to campus, then, is not an educational opportunity, but a threat,” the alumni letter said. “It is a message to every woman, every person of color, every first-generation student, every poor and working-class person, every disabled person and every queer person that not only their acceptance to and presence at Middlebury, but also their safety, their agency, their humanity and even their very right to exist are all up for ‘debate.'”

Murray said he was “physically assaulted” by protesters when leaving campus. “We were confronted by a mob that man-handled us that — had it not been for the security guards — at the very least, I would have been on the ground,” he said.

“I’ve never encountered anything close to this, both in the open-ended protest — not letting me speak at all — and the ferocity,” he added.

College spokesman Bill Burger said he, Murray and professor Allison Stanger were confronted by a group of about six to eight protesters — some wearing masks — who shoved them as they walked to the parking lot, at one point pulling Stanger’s hair. Burger said the group of protesters later surrounded their car, climbing on the hood and banging on the windows.

Middlebury President Laurie Patton said in a statement Friday that she believes many of those protesters “were outside agitators, but there are indications that Middlebury College students were involved as well.”

Burger defended the decision to host Murray at the school.

“We’re confident we did the right thing,” he said. “We were determined that this event go forward. We felt it upheld a core principle of this institution and of higher education, which is one of free expression and free speech.”

Get The Brief. Sign up to receive the top stories you need to know right now. Please enter a valid email address. Sign Up Now Check the box if you do not wish to receive promotional offers via email from TIME. You can unsubscribe at any time. By signing up you are agreeing to our Terms of Use and Privacy Policy . This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply. Thank you! For your security, we've sent a confirmation email to the address you entered. Click the link to confirm your subscription and begin receiving our newsletters. If you don't get the confirmation within 10 minutes, please check your spam folder.

Write to Katie Reilly at Katie.Reilly@time.com.