Washington (CNN) In April, after the Supreme Court heard arguments in one of the most significant cases of the term -- whether the Trump administration can add a citizenship question to the 2020 census -- the justices met behind closed doors and likely took a preliminary vote to decide the case.

By now, the majority opinion and dissent are probably well underway, with drafts flying between chambers in order to meet an end of June deadline.

But challengers of the citizenship question have launched an explosive new allegation : they say they have "new evidence" that the decision to add the question was politically motivated in order to intimidate minority households from responding in order to reduce Democratic Party representation.

The controversy is of enormous consequence because the census, which only occurs once every 10 years, yields data that goes to the core of political representation. It determines how to divvy up congressional seats as well as the allocation of billions of dollars of federal funds.

At oral arguments in April, the justices appeared to be split along their familiar ideological 5-4 conservative-liberal divide, although it's a risky endeavor to ascertain how a case will come out based solely on that brief public appearance.

If they followed their usual protocol, the justices met later in their stately conference room to discuss the case. Speaking in order of seniority, they each likely stated their initial vote. No one else is allowed in the room -- no clerks, no aides. If there is a knock on the door, Justice Brett Kavanaugh, the junior-most justice, is tasked with responding. Soon after the initial vote, Chief Justice John Roberts, if he is in the majority, assigns the opinion or takes it for himself. If he is in the minority, the most senior member of the majority assigns the opinion.

Although some are more technologically apt than others, they still communicate between chambers via paper. Sometimes that includes one justice encouraging a colleague to add or delete certain language so that the justice feels more comfortable joining the opinion, as with Justices Antonin Scalia and Anthony Kennedy in a landmark 2008 Second Amendment case, retired Justice John Paul Stevens told The Washington Post

Most famously, Roberts switched his vote in 2012 after oral arguments in order to save the Affordable Care Act. After first voting to strike down the law, he decided to uphold it -- calling it a tax -- and worked with liberals Elena Kagan and Stephen Breyer on a controversial final opinion, as CNN's Joan Biskupic reported in "The Chief."

The process means that nothing is ever set in stone until the court releases its opinion in public, and the challengers to the citizenship question likely hope that at least some of the justices might be interested in what is going on elsewhere in the legal system.

In a dramatic turn, the American Civil Liberties Union has taken its new allegations back to the district court judge who initially heard the case in New York. The ACLU and a team of Arnold & Porter lawyers are seeking sanctions against the Trump administration and alleging that officials gave misleading testimony about the genesis of the question. The group sent a six-page letter to the judge, as well as the Supreme Court, describing the material as "damning new evidence" that they say reveals "hyper-partisan" motives were at the root of the citizenship question.

"The Justice Department's hands are not clean," the letter states

"It's almost certainly too late for these new allegations to impact the outcome of the decision the court is likely to hand down by the end of the month," said Steve Vladeck, CNN Supreme Court analyst and professor at the University of Texas School of Law.

"But it's hard to imagine that the justices who are inclined to side with the challengers won't see this as further proof of the strength of the challengers' claims -- and the wrongfulness of the government's conduct here," Vladeck said.

In a talk Monday before the Supreme Court Historical Society, Justice Clarence Thomas was asked if the justices ever trade votes from one case to the next. He responded sternly: "I don't horse trade."

But he said there are instances when the justices try to convince each other of their position after arguments.

"Justice Kennedy was great at it," Thomas said. "He would call and say 'I know we have a different point of view about this, but I'd like to tell you what my thinking is.'"

"This building is a bit different from the rest of what we're seeing in society. People will actually consider your argument," Thomas said.

New York hearing

On Wednesday, the ACLU went to Judge Jesse Furman of the US District Court for the Southern District of New York, who said the allegations that there is new evidence are "serious" but indicated he didn't want to get in the way of the Supreme Court.

Furman said he's "acutely" aware that the justices are currently reviewing his own January opinion blocking the government from including the question, and he didn't want to interfere in that case.

So he ordered lawyers to submit additional briefs next month regarding the new evidence, which would conceivably come after the Supreme Court has ruled.

At one point, he said "there is no reason" to rush the process.

The justices are reviewing the Furman's scathing opinion from last winter when he held that Commerce Secretary Wilbur Ross's decision to add the question was unlawful for a "multitude of independent reasons."

Furman said the Commerce Department violated the Administrative Procedure Act, a federal law that governs the way agencies can establish regulations. Most critically, Furman said Ross' stated rationale for the question -- that the Justice Department said it was necessary for better enforcement of the Voting Rights Act -- was "pretextual -- in other words that he announced his decision in a manner that concealed its true basis."

The Administrative Procedure Act gives agencies broad discretion, but they must offer the actual reasons for their decisions.

When the Supreme Court heard arguments in the case back in April , the conservatives on the court seemed ready to side with the Trump administration.

Justice Brett Kavanaugh noted at one point that "the statute that Congress has passed gives huge discretion to the secretary how to fill out the form, what to put on the form."

But liberals such as Kagan pushed back on Ross' justification for adding the question, citing the fact that Ross deviated from the advice of some census officials.

"The secretary needs reasons to do that, and I searched the record and I don't see any reason," Kagan said.

"You can't read this record without sensing that this -- this need is a contrived one," she added.

New information

Late last month, the ACLU obtained a hard drive with some 75,000 documents, including a 2015 study from a Republican redistricting expert, Dr. Thomas Hofeller, who they say played a significant role in the decision to add the question. Hofeller, who has since died, wrote that using "citizen voting age" population as the redistricting population base would be "advantageous to Republicans and Non-Hispanic Whites."

The challengers allege Hofeller also ghostwrote part of a draft Justice Department letter requesting that the Commerce Department, which oversees the census, add the question. They say that the evidence reveals that Ross adviser A. Mark Neuman and senior Justice official John Gore "falsely testified" about the genesis of the request to add the question.

In a fiery letter to Furman on Monday, the Justice Department said that Hofeller's work did not play "any role whatsoever" in the department's request to add the question.

"There is no smoking gun here; only smoke and mirrors," the Justice Department said, adding that Gore's testimony was "entirely truthful."

"Substantively, the 'new' evidence is irrelevant because the critical" issue in the case "is whether (Ross) provided an objectively rational basis for his decision to reinstate the citizenship question," the department added. "Nothing in the private files of a deceased political operative can affect the resolution of that issue."