WASHINGTON (CNNMoney) -- Should the Postal Service get its way on laying off thousands of workers, the cuts would weigh heavily on two segments of the population hard hit in recent years: minorities and veterans.

The U.S. Postal Service, teetering toward default, wants the power to cut 120,000 jobs by voiding labor protections for currently protected workers.









Layoffs can't happen unless Congress steps in -- and they could, in fact, violate the Constitution. But the timing of the layoff talk couldn't be worse. The odds of a double-dip recession are increasing and the nation added no new jobs last month.

The Postal Service is one of the largest employers of all minorities, led by African-Americans, who make up 21% of workers. Last year, Black Enterprise magazine praised the service as one of the top 40 companies for diversity.

The post office has historically been one of the largest employers of African Americans, said Philip Rubio, a former mail carrier and an assistant professor at North Carolina A&T State University.

"It was a job that brought status, security, decent pay and benefits, and elevated many African Americans into the middle class, enabling them to buy homes and send their children to college," Rubio said. "The proposed US Postal Service layoffs and post office closures -- harmful to the nation as a whole -- would therefore be especially devastating to the African American community."

The unemployment rate for African-American men hit a 25-year-record-high of 16.7% last week.

In addition, the Postal Service is the single largest employer of veterans, which make up 130,000, or 22% of its ranks, according to a 2010 report on postal operations. Of those veterans, 49,000 -- nearly a third -- are disabled.

In a memo discussing potential layoffs, the Postal Service proposes giving veterans preference in layoffs. But unions insist veterans' jobs would be at stake.

"Whenever the Postal Service closes a whole plant, as they have said they want to do, all the employees in that plant, including veterans, would be subject to being laid off," said Cliff Guffey, president of the American Postal Workers Union, in a prepared statement.

Layoffs aren't merely controversial, they may just be unconstitutional, according to an independent research arm of Congress.

Existing union contracts with one of seven different unions for post office workers prohibit layoffs of employees with more than six years of service.

But the Congressional Research Service said in a July note to lawmakers that if Congress passes a law possibly allowing layoffs of Postal Service workers with labor contracts, those workers could challenge the move in court -- and stand a good chance of winning.

The possibility of future lawsuits hasn't stopped Congress before, but it could bolster opponents of tweaking union contracts to allow layoffs.

Other moves under consideration by the Postal Service to save money are the closure of some 3,700 post offices and an end to Saturday mail delivery.