WASHINGTON (CNN) -- The United States is boycotting a U.N. conference on racism next week over a document that "singles out" Israel in its criticism and conflicts with the nation's "commitment to unfettered free speech," the U.S. State Department said Saturday.

The Congressional Black Caucus criticized the boycott, saying President Obama's decision "set the cause back."

The Obama administration made the decision not to attend the Durban Review Conference in Geneva "with regret," a State Department statement said.

Two months ago, the administration had warned that it would boycott the conference if changes were not made to the document to be adopted by the conference. In recent weeks, discussions over the document have fueled several revisions, but the changes to the language didn't meet U.S. expectations, the statement said.

The current draft is "significantly improved," but "it now seems certain these remaining concerns will not be addressed in the document to be adopted by the conference next week."

State Department officials say the document contains language that reaffirms the Durban Declaration and Programme of Actions from the 2001 conference in Durban, South Africa, which the United States has said it won't support. The 2001 document "prejudges key issues that can only be resolved in negotiations between the Israelis and Palestinians," the statement said.

Australia also said it would boycott the conference. Foreign Minister Stephen Smith said the 2001 document unfairly "singled out" Israel.

""Regrettably, we cannot be confident that the Review Conference will not again be used as a platform to air offensive views, including anti-Semitic views," he said.

Disagreements over the Middle East and slavery that year had threatened to derail the conference goal of creating a global blueprint for fighting discrimination. At the time, Israel had said it was disappointed so much of the conference had focused on its relations with Palestinians.

The Obama administration also said recent additions to the document regarding "incitement" contradict the United States' stance on free speech.

Still, the United States "will continue to work assiduously" with all nations "to combat bigotry and end discrimination," the statement said.

Meanwhile, the Congressional Black Caucus said it was "deeply dismayed" by the decision made by the nation's first African-African president, saying it was inconsistent with administration policies.

"Had the United States sent a high-level delegation reflecting the richness and diversity of our country, it would have sent a powerful message to the world that we're ready to lead by example," the statement said. "Instead, the administration opted to boycott the conference, a decision that does not advance the cause of combating racism and intolerance, but rather sets the cause back."

CNN's Scott Spoerry contributed to this report.

All About Racism and Bigotry • Israel • U.S. Department of State • Middle East Conflict • U.S. Congressional Black Caucus