By 1966, Sidney Poitier was already a bona fide movie star. He had recently made history as the first black man to win a best-actor Oscar and was on the cusp of releasing three movies that would go on to be his most iconic works: To Sir, with Love, Guess Who’s Coming to Dinner, and In the Heat of the Night. However, he was still a black man working in America—a reality that reared its ugly head in the deep South while he was working on the latter film with director Norman Jewison.

In a column for The Hollywood Reporter, as told by T.H.R. executive editor Stephen Galloway, Jewison recalls that Poitier was excited to make the film. It was a leading role in a drama about a Philadelphia-based police officer named Virgil Tibbs investigating a murder in Sparta, Mississippi. However, the location presented an issue for Poitier. Jewison wanted to work in “believable” locations, but the actor had previously made a vow to never go south of the Mason-Dixon line.

“He said it with such emphasis that I realized it was very important to him,” Jewison recalls. “I said, ‘Why is that?’ And he says, ‘I had an unsettling experience with Harry Belafonte in Georgia, where our car was chased and we were threatened, and I don’t want to go down there.’ So I said, ‘I’ll do my best to stay north of the Mason-Dixon line. What can I say? I want you to do the picture.”

Jewison and his team began an exhaustive search to find new locations, but ultimately couldn’t find anything fitting north of a town called Dyersburg, Tennessee. After two or three weeks filming, the filmmaker personally went to Poitier and presented his case—he only needed Poitier in two scenes, which could be shot in just a couple of days.

“I remember pleading with him, and Sidney said, ‘I understand, I understand,’” the director remembers. “He realized we were up against the gun and the picture was going well.”

So they picked up and went south. Jewison remembers assuring Poitier that the crew was “very loyal” and was full of “big guys” who would protect him from potential protests. However, they couldn’t escape the town’s racist policies. “We were forced to stay in the Holiday Inn because it was the only place that accepted African-American people,” Jewison says. “The main hotel in Dyersburg was a real Southern hotel, and it was whites only. You’ve got to remember, we were shooting in 1966, so things were a little uptight. Martin Luther King Jr. had just done the march on Selma. The country was in the midst of a racial revolution, if you could call it that. There were marches; there was a lot of friction between the races; and most of it was in the South. That was the state of the union, and so we knew this was a very controversial film. And I had no idea how it was going to be received.”

Poitier’s scenes in Dyersburg were ultimately filmed without a hitch, and In the Heat of the Night went on to win five Oscars—including best picture, best actor, and best adapted screenplay. Jewison was also nominated for best director. The film further cemented Poitier’s status as a true movie star and provided one of the most iconic lines of his career: “They call me Mr. Tibbs.”