The White House is moving to permanently ban CNN reporter Jim Acosta from the White House after a temporary legal setback, CNN’s Brian Stelter revealed Sunday evening.

The White House sent a letter to Acosta notifying him that his pass granting him temporary access to the White House grounds would be suspended after a temporary restraining order runs its course. The letter is the latest in the fight between the White House and the CNN reporter who caused a fracas at a recent presidential press conference when he refused to give up the microphone.

Judge Timothy J. Kelly issued an injunction Friday morning ordering the White House to reinstate Acosta’s credentials, saying his Fifth Amendment right to due process was violated. The judge was clear, however, that he made no ruling on whether the First Amendment right applies for any reporter to be at the White House. (RELATED: CNN Sues The White House Over Jim Acosta’s Revoked Press Credentials)

White House press secretary Sarah Huckabee Sanders presaged the letter in a Friday statement warning Acosta, “in response to the court, we will temporarily reinstate the reporter’s hard pass. We will also further develop rules and processes to ensure fair and orderly press conferences in the future. There must be decorum at the White House.”

The White House pulled Acosta’s press pass nearly two weeks ago, alleging that he behaved in an unruly manner during the presidential press conference and put his hands on an intern assigned to take the microphone away from him.

CNN responded to the letter in a Sunday statement saying, “the White House is continuing to violate the First and 5th Amendments of the Constitution. These actions threaten all journalists and news organizations. Jim Acosta and CNN will continue to report the news about the White House and the President.”

CNN’s Brian Stelter speculated in his Sunday evening newsletter that the White House is likely trying to establish a paper trail to indicate that due process was given to Acosta, as the remainder of the legal battle remains to be fought out in court.

President Donald Trump told The Daily Caller in an exclusive interview Tuesday he was unsure his administration would prevail in the lawsuit, but noted “we’ll see how the court rules. Is it freedom of the press when somebody comes in and starts screaming questions and won’t sit down?”

Trump suggested later in a sit-down Sunday with Fox News that in response to the lawsuit “I think one of the things we’ll do is maybe turn the camera off that faces them because then they don’t have any air time, although I’ll probably be sued for that and maybe, you know, win or lose it, who knows.”