Washington (CNN) The state lawmaker who received an expletive-laced voicemail from Maine Gov. Paul LePage said Friday he did not call LePage a racist, as the governor claims.

"I've never received a voicemail like that before," Democratic State Rep. Drew Gattine told CNN's Jim Scuitto on "OutFront." "Every time you think he's crossed a line -- you think he can't go any further -- but then he draws a different line, and he crosses it."

LePage left a message goading Gattine to "prove that I'm a racist" after a series of controversial comments once again put the governor in the news, according to the Portland Press Herald, which also was the first to obtain audio of the voicemail. The uncensored audio message, also shared with CNN, contains several explicit phrases.

The bombastic Republican governor is known for his loose style, which sometimes draws criticism. He is a strong supporter of Donald Trump, and Trump's campaign recently hired his daughter to work on the race in Maine.

Earlier this week, LePage had doubled down on comments he has made in the past about the racial background of drug dealers in his state, saying that "90-plus percent ... are black and Hispanic people."

According to the Press Herald, Gattine was identified by local media as having called LePage racist, though Gattine denies doing so.

"I'm not a name-caller. I'm not going to play that game with the governor," Gattine told Sciutto. "Calling somebody a racist is one of the worst things you can ever call somebody. And it's not something that I've ever called anybody."

LePage, however, was adamant in the recording.

"Mr. Gattine, this is Gov. Paul Richard LePage," the audio recording says. "I would like to talk to you about your comments about my being a racist, you (obscene term). I want to talk to you. I want you to prove that I'm a racist. I've spent my life helping black people and you little (obscene term), socialist (obscene term). You -- I need you to -- just friggin'. I want you to record this and make it public because I am after you. Thank you."

LePage apologized for the voicemail in a statement on Friday, but justified his reaction saying the label of racist is "the absolute worst, most vile thing you can call a person."

"I didn't know Drew Gattine from a hole in the wall until yesterday," LePage said. "It made me enormously angry when a TV reporter asked me for my reaction about Gattine calling me a racist. ... So I called Gattine and used the worst word I could think of. I apologize for that to the people of Maine, but I make no apology for trying to end the drug epidemic that is ravaging our state."

Gattine told CNN he had not heard from LePage since.

Challenges Gattine to a duel

On Thursday, the governor took credit for leaving the voicemail in a later interview at his home with local media, and said he would like to challenge Gattine to a duel.

"When a snot-nosed little guy from Westbrook calls me a racist, now I'd like him to come up here because, tell you right now, I wish it were 1825," LePage said, according to the Press Herald. "And we would have a duel, that's how angry I am, and I would not put my gun in the air, I guarantee you, I would not be (Alexander) Hamilton. I would point it right between his eyes, because he is a snot-nosed little runt and he has not done a damn thing since he's been in this Legislature to help move the state forward."

Friday, LePage walked back that rhetoric, saying he was purely making a historical reference.

"Obviously, it is illegal today; it was simply a metaphor and I meant no physical harm to Gattine," LePage said.

Maine Democratic Party Chairman Phil Bartlett called LePage's remarks a "threat."

"Gov. LePage's direct threat against Rep. Gattine is both erratic and disturbing, and he is clearly unfit to lead our state," Bartlett said in a statement. "Not only did the governor blatantly say he would take violent action against a sitting lawmaker, he also twice invoked a homophobic slur to drive home his point. Those reckless remarks may incite others to violence. ... Paul LePage is an increasingly menacing figure who does not reflect the values of our state."