In its two seasons so far on Hulu, The Handmaid’s Tale has become known for its message of resistance—and for its relentless darkness. Joseph Fiennes’s character, Fred Waterford, has proven to be one of the most horrifying characters of all—a man who helped shape a society that enslaves fertile women, and whose authoritarian tendencies come to light more and more with each passing episode.

But there was one scene in Season 2 that Fiennes thought went too far—passionately enough that he refused to do it, making his case in multiple “long e-mails.” The scene in question? Commander Waterford was supposed to rape his wife, Serena Joy, during their ill-fated trip to Canada.

“I guess in many ways, as abhorrent and nasty and evil as Fred is, I have to defend parts of him,” Fiennes recently told Entertainment Weekly. The actor said that right after his character ran into June’s husband, Luke, at a protest outside his hotel in Canada, he was supposed to rape Serena Joy in their room. “It just didn’t track for me,” Fiennes told E.W. “I had to go out on a limb and refuse to do it, because I felt that even though Fred is who he is, he’s human. And I think that he would be reeling from the interaction with Luke, and that suddenly the reality comes face to face with him, and he would be digesting that and trying to understand it, and he wouldn’t necessarily be switched on by being in Canada in a new hotel and trying to heavily persuade his wife to do something that she wouldn’t want to do.”

Fiennes said he made his case in a series of “long e-mails and defending and pushing.” He argued that actress Yvonne Strahovski, who plays Serena Joy, had already made her character’s disenfranchisement clear—so the drama “didn’t need a heavy scene to kind of suddenly push her over the edge.”

That instinct feels appropriate, given how this season played out. By the time Fred and Serena Joy arrived in Canada, Fred had already beaten his wife for collaborating with June while he was indisposed—not only subverting his authority, but also breaking the law. (June and Serena Joy were writing memos in Fred’s name; in Gilead, women are not permitted to read or write.) As seen in the episode, Serena Joy’s loyalty to Fred and the regime she helped create and install was already wavering—and so, in many ways, watching her get raped would have seemed gratuitous.

Given how controversial the series’s brutality is already, it’s understandable that everyone involved wants to avoid going too far unless absolutely necessary. As Bruce Miller put it to V.F. in a postmortem interview following the Season 2 finale, “I don’t want to see anything that I don’t absolutely have to see to understand the plot.” And while Serena Joy getting raped certainly may have pushed her over the edge, the series had already given her plenty of reasons to question her loyalty to Gilead—which means that Fiennes drawing this line was probably a wise decision.