(CNN) The Supreme Court wades into a bitter controversy on Tuesday over whether the Trump administration can ask all recipients a citizenship question on the 2020 census for the first time since 1950.

The justices will hear the administration's claims that the question is necessary to better comply with federal voting rights law, versus arguments by critics who say it represents a veiled attempt to intimidate noncitizens and Hispanic households and will lead to a decrease in response rates.

Every lower court to consider the issue has so far blocked the administration from adding a question about citizenship status to the census questionnaire, holding that Commerce Secretary Wilbur Ross, who has jurisdiction, exceeded his authority under federal law and the Constitution by doing so.

How the justices rule in the case -- with the 2020 count fast approaching -- could impact the critical data derived from the census, which is used for issues such as the allocation of congressional seats and the distribution of billions of federal dollars to states and localities over the next decade.

"The Constitution requires that everyone in America be counted, it's a basic principle of our democracy that representation be allocated equally to states based on their populations," said American Civil Liberties Union lawyer Dale Ho, who is set to argue for the challengers on Tuesday. "The addition of this question is an attack on that basic constitutional principle that will impact the entire country, and hurt the representation of minorities."

Lower courts have ruled against the Trump administration

Lower courts have ruled against the government, pointing to the administration's shifting rationale for reinstating the question, and held that the way the government proceeded was illegal.

Ross' decision was unlawful for a "multitude of independent reasons and must be set aside," Judge Jesse M. Furman of the US District Court for the Southern District of New York ruled in January in a 277-page opinion after holding an eight-day trial.

Furman said Ross' decision to add the question violated the Administrative Procedure Act, a federal law that governs the way agencies can propose and establish regulations. He said Ross failed to consider several important aspects of the issue and "alternately ignored, cherry-picked or badly misconstrued the evidence in the record before him."

Most critically, Furman said Ross' stated rationale for the question, to promote the enforcement of the Voting Right Act, was "pretextual — in other words, that he announced his decision in a manner that concealed its true basis rather than explaining it" as the Administrative Procedure Act required him to do, the judge held.

It was back in March 2018 that Ross announced in a letter to the Commerce Department's under secretary for economic affairs that the Department of Justice -- led by then-Attorney General Jeff Sessions -- had requested that the Census Bureau reinstate the question in order to obtain "more effective enforcement" of the Voting Rights Act.

Ross acknowledged that adding the question "could reduce" response rates, but said that more accurate citizenship data would outweigh such fears.

But after the trial, Furman said Ross' justification in the letter was "materially inaccurate."

Furman -- drawing from documents and testimony -- wrote that Ross, almost as soon as he arrived in office in February 2017, began asking questions about adding a citizenship question. Ross spoke about the issue with Sessions, as well as immigration hardliners such as then political adviser Steve Bannon and Kansas Secretary of State Kris Kobach. Those consultations, as well as the new timeline, undermined the stated justification for the question, Furman held.

The judge also noted that the Census Bureau believed there were other ways to obtain more accurate measures of citizenship.

In testimony, Dr. John Abowd, the Census Bureau's chief scientist, was asked whether he was under the impression that his work on the question mattered.

"Yes," he said.

Abowd was then asked if he had ever been told that Commerce had in fact initiated the process to insert the question. "No one told me that," Abowd testified. Furman noted that the doctor "choked up" and visibly held back tears.

Constitutional questions at play

Although Furman didn't rule on the constitutional questions, a judge sitting on the US District Court for the District of Maryland held that Ross' action violated the Enumeration Clause of the Constitution, which provides for an "actual enumeration" every decade.

"Because the Secretary ignored evidence regarding the impact of the question and provided no legitimate rationale to support it, the addition of the citizenship question would unreasonably compromise the distributive accuracy of the Census and the addition violates the Enumeration Clause," Judge George J. Hazel of the US District Court for the District of Maryland held in April.

Solicitor General Noel Francisco stressed in court briefs that the citizenship question was legal and that "nothing in the record supports the district court's extraordinary charge that the Secretary of Commerce" lied about his rationale for the decision.

Francisco accused the court of straining "to read every statement and action of the Secretary in the worst possible light."

As a threshold argument, Francisco argued that the challengers in the case --18 states, cities and nonprofit groups, among others -- do not have the legal right or "standing" to bring the case.

In order to come to court, they would have to prove an injury, and Francisco argued they can't base that injury on a third party's "speculative" refusal to answer the question.

"None of respondents' alleged injuries will materialize if individuals completely and truthfully answer the census questionnaire, as required by federal law," he said.

Francisco argued that if the justices allow the plaintiffs to bring the case it would "permit any demographic question on the census to be challenged so long as a group of individuals disproportionately residing in certain States announce their intent to illegally boycott" the question.

He also said Ross' action is not reviewable under the Administrative Procedure Act because "courts do not have the authority to second guess the Secretary's decision" in part because the Constitution vests Congress with "virtually unlimited discretion" in conducting the census. Congress, he said, delegated the authority to the secretary of commerce.

But even if the action were reviewable, Francisco argued, "it simply cannot be 'arbitrary'" or "irrational" to reinstate to the decennial census a question whose pedigree dates back nearly 200 years.

Francisco's position drew the support of 16 states, all Republican-led, which wrote Ross asking him to adopt the question.

Democratic-led states are leading the opposition

But the 18 challenging states, cities and nonprofit groups disagree.

They say the Constitution and the Census Act require the federal government to count every person in the country, every 10 years, and that Furman held that adding the citizenship question would reduce responses among households with noncitizens, totaling roughly 6.5 million.

"There is just one chance, each decade to get the enumeration right," New York state Attorney General Letitia James told the justices in court papers.

She outlined the injury to states, including the possibility of losing seats in the House of Representatives and funds from federal programs.

James said that Ross not only violated the Administrative Procedure Act but also the Constitution's Enumeration Clause, which requires an "actual" enumeration of the population every 10 years. She said adding the question would "affirmatively undermine the accuracy of the enumeration."

Another set of challengers represented by the ACLU argued that since 1950 the government had realized that a "differential undercount" of racial and ethnic minorities would threaten census accuracy.

"The government stopped asking this question, along with dozens of others on the census, when it realized that these questions were harming the accuracy of the population count and were specifically causing an undercount of communities of color," Ho said.

Lawyers for the Democratic-led House of Representatives will also have the opportunity to argue against the addition of the question before the Supreme Court justices on Tuesday.

In addition, five former directors of the Census Bureau, who have served under both Democratic and Republican administrations, filed a friend of the court brief supporting the challengers. They argued they have "unique expertise" on the processes and procedures required to conduct an "accurate, high-quality" census.

"The longstanding view of the Census Bureau -- reaffirmed by several recent Census Bureau analyses -- is that addition of the question will reduce the accuracy of the population count," their lawyer told the justices.