A FedEx driver in Oregon will not face charges in connection with the September death of a Portland man he fatally punched after the stranger called him racial slurs, the Multnomah County District Attorney’s Office announced Monday.

Senior Deputy District Attorney Adam Gibbs said in a memo that Timothy Warren “was in his right” to challenge Joseph Magnuson, who witnesses said called the FedEx worker the n-word and “other aggressive and abusive phrases,” The Oregonian reported.

The incident occurred Sept. 26 in Northeast Portland, when witnesses said Magnuson started the altercation by yelling at Warren “very aggressively” to slow down, the paper reported. When Warren tried to speak with Magnuson after stopping the vehicle, Magnuson called him racial slurs and threw a drink at the FedEx employee.

Some witnesses said Magnuson also hurled a bag of food at Warren, the paper reported, before he threw a punch that missed. Warren hit Magnuson above his left eye, knocking him to the ground unconscious, the report said. Magnuson later died, according to the paper.

“Oregon law contains no ‘duty to retreat’ and Mr. Warren was within his right to exit his vehicle and verbally challenge the manner in which Mr. Magnuson was addressing him,” the memo said, according to the paper. “Mr. Magnuson was the initial verbal aggressor; Mr. Warren responded in kind. Mr. Magnuson then escalated and became the initial physical aggressor; Mr. Warren again responded in kind.”

While Magnuson might still be alive if Warren had not stopped, Gibbs noted that “the decision by Mr. Warren, who is black, to not let the racist vitriol to which he was being subjected go unanswered is not of legal significance,” according to the paper.

Warren waited at the scene and cooperated with police, the paper reported.

A medical examiner also determined Magnuson was in “extremely poor health,” which contributed to his death, according to the paper.