Washington (CNN) The Democrats who support President Barack Obama's free-trade push are growing increasingly concerned about organized labor's hardball tactics.

The AFL-CIO and other leading labor unions have made defeating trade promotion authority -- which greases the wheels for Obama's massive Trans-Pacific Partnership by guaranteeing it an up-or-down vote with no amendments -- their top legislative priority.

"Labor is going a little overboard and I think there is some potential backlash for how far they are going," said Louisiana Rep. Cedric Richmond, a Democrat, on Wednesday.

The measure has already cleared the Senate , and it's expected to get a vote in the House in the coming weeks.

But unions are skeptical about Obama administration promises that the deal would protect labor rights and the environment. They also fear that the agreement would result in countries having a price advantage over the United States, which they say would drive down wages for U.S. workers and siphon away more manufacturing jobs.

"Even though this trade deal will affect 40% of the world's GDP, and even though no prior trade deal (a U.S. Trade Representative) has negotiated has ever lived up to its promises, and even though the administration has promised that this would be the most progressive, transparent trade deal in history, we still can't see the text," said Liz Shuler, secretary-treasurer at the AFL-CIO, in a statement.

So union leaders are telling pro-trade Democrats, and those considering voting for the deal, they will fund a primary challenge against them, or they won't back them in a future tough re-election fight in 2016.

Playing hardball

They've made an example of California Democratic Rep. Ami Bera, putting the early supporter of Obama's trade efforts in the crosshairs with paid television advertisements in his district and promises to unseat him in 2016, even if that means a Republican takes his seat.

Illinois Democratic Rep. Mike Quigley, an early supporter of the bill, told CNN the pressure from labor is the reason why there are fewer than 20 House Democrats committed to voting for trade promotion authority.

"If you just look at this from a rational view, you'd have a lot more yeses," Quigley said.

Lawmakers' frustration spilled out in a closed-door meeting with pro-trade Democrats and House Minority Leader Nancy Pelosi on Wednesday.

Photos: Barack Obama's presidency Photos: Barack Obama's presidency U.S. President Barack Obama attends the Nuclear Security Summit in the Hague, Netherlands, in March 2014. As Obama's second term nears its close, here's a look at some key moments of his administration. Hide Caption 1 of 55 Photos: Barack Obama's presidency First lady Michelle Obama brushes specks from the coat of then-Sen. Obama in Springfield, Illinois, just before he announced his candidacy for President in February 2007. Their daughters Malia, left, and Sasha wait in the foreground. Hide Caption 2 of 55 Photos: Barack Obama's presidency Obama appears on "Meet the Press" with Tim Russert, right, in Des Moines, Iowa, in November 2007. Hide Caption 3 of 55 Photos: Barack Obama's presidency Obama shakes hands with supporters after addressing a rally in Concord, New Hampshire, in January 2008. Hide Caption 4 of 55 Photos: Barack Obama's presidency Obama gives a speech at the National Constitution Center in Philadelphia in March 2008. Hide Caption 5 of 55 Photos: Barack Obama's presidency Obama speaks at the 2008 Democratic National Convention in Denver. Hide Caption 6 of 55 Photos: Barack Obama's presidency Obama stands on stage in Chicago with his family after winning the presidential election on November 4, 2008. Hide Caption 7 of 55 Photos: Barack Obama's presidency Obama poses in the Oval Office with several former U.S. Presidents in January 2009. From left are George H. W. Bush, Obama, George W. Bush, Bill Clinton and Jimmy Carter. Hide Caption 8 of 55 Photos: Barack Obama's presidency Obama is sworn in by Chief Justice John Roberts as the 44th President of the United States on January 20, 2009. Hide Caption 9 of 55 Photos: Barack Obama's presidency As retired military officers stand behind him, Obama signs an executive order to close down the detention center at Guantanamo Bay, Cuba, in January 2009. Hide Caption 10 of 55 Photos: Barack Obama's presidency Obama and Vice President Joe Biden look at solar panels as they tour the solar array at the Denver Museum of Nature and Science on February 17, 2009. That same day, Obama signed the American Recovery and Reinvestment Act. Hide Caption 11 of 55 Photos: Barack Obama's presidency A soldier hugs Obama during his surprise visit to Camp Victory just outside Baghdad, Iraq, in April 2009. Hide Caption 12 of 55 Photos: Barack Obama's presidency Obama bends over so the son of a White House staff member can pat his head during a visit to the Oval Office in May 2009. The boy wanted to know if Obama's hair felt like his. Hide Caption 13 of 55 Photos: Barack Obama's presidency Obama kisses Sonia Sotomayor's cheek after announcing her as his nominee for Supreme Court justice in May 2009. Hide Caption 14 of 55 Photos: Barack Obama's presidency Palestinian security forces in Jenin, West Bank, listen to Obama speak from Cairo University in Egypt in June 2009. The Palestinian Authority hailed as a "good beginning" Obama's speech to the Muslim world in which he reiterated his support for a Palestinian state. Hide Caption 15 of 55 Photos: Barack Obama's presidency Obama and the first lady meet with Pope Benedict XVI in Vatican City in July 2009. Hide Caption 16 of 55 Photos: Barack Obama's presidency Obama hosts the Apollo 11 astronauts -- from left, Edwin "Buzz" Aldrin, Michael Collins and Neil Armstrong -- in the Oval Office on July 20, 2009. It was the 40th anniversary of the moon landing. Hide Caption 17 of 55 Photos: Barack Obama's presidency Police Sgt. James Crowley, second right, of Cambridge, Massachusetts, speaks with Harvard Professor Henry Louis Gates Jr., second left, alongside Obama and Biden as they share beers on the South Lawn of the White House in July 2009. The so-called Beer Summit was held after Crowley arrested Gates at his own home, which sparked tensions and racial furor. Hide Caption 18 of 55 Photos: Barack Obama's presidency Obama salutes during the transfer of Sgt. Dale R. Griffin at Dover Air Force Base in Dover, Delaware, in October 2009. Obama traveled to the base to meet the plane carrying the bodies of 18 U.S. personnel killed in Afghanistan. Hide Caption 19 of 55 Photos: Barack Obama's presidency Obama walks along the Great Wall of China in November 2009. Hide Caption 20 of 55 Photos: Barack Obama's presidency Obama delivers a speech after receiving the Nobel Peace Prize in Oslo, Norway, in December 2009. Hide Caption 21 of 55 Photos: Barack Obama's presidency Obama and former Presidents Clinton and George W. Bush walk to the White House Rose Garden to speak about relief efforts for earthquake-stricken Haiti in January 2010. Hide Caption 22 of 55 Photos: Barack Obama's presidency First daughters Sasha and Malia Obama play in the snow with their father after a snowstorm hit Washington in February 2010. Hide Caption 23 of 55 Photos: Barack Obama's presidency Obama's signature on the Affordable Care Act is seen at the White House in March 2010. Hide Caption 24 of 55 Photos: Barack Obama's presidency Obama throws out the opening pitch before a baseball game between the Philadelphia Phillies and the Washington Nationals in April 2010. Hide Caption 25 of 55 Photos: Barack Obama's presidency Obama and his daughter Sasha swim in Panama City Beach, Florida, in August 2010, to encourage people to come back to the Gulf Coast after a devastating oil spill. Hide Caption 26 of 55 Photos: Barack Obama's presidency Obama laughs as he makes a statement on his birth certificate in April 2011. Obama said he was amused over conspiracy theories about his birthplace, and he said the media's obsession with the "sideshow" issue was a distraction in a "serious time." Hide Caption 27 of 55 Photos: Barack Obama's presidency Obama, Biden, Secretary of State Hillary Clinton and members of the national security team receive live updates on the mission to capture or kill Osama bin Laden on May 1, 2011. Hide Caption 28 of 55 Photos: Barack Obama's presidency U.S. Marines watch from Afghanistan as Obama announces the death of bin Laden on May 2, 2011. Hide Caption 29 of 55 Photos: Barack Obama's presidency Obama enjoys a pint of Guinness in his ancestral home of Moneygall, Ireland, in May 2011. Hide Caption 30 of 55 Photos: Barack Obama's presidency Obama and the first lady meet with Britain's Prince William and Catherine, Duchess of Cambridge, at Buckingham Palace in May 2011. Hide Caption 31 of 55 Photos: Barack Obama's presidency Obama and Missouri Gov. Jay Nixon walk together in May 2011 during a tour of the tornado devastation in Joplin, Missouri. Hide Caption 32 of 55 Photos: Barack Obama's presidency Obama delivers remarks to troops and military families at Fort Bragg, North Carolina, on December 14, 2011, marking the exit of U.S. soldiers from Iraq. Hide Caption 33 of 55 Photos: Barack Obama's presidency Obama pays for a dog toy as he shops with his dog Bo at a PetSmart in Alexandria, Virginia, in December 2011. Hide Caption 34 of 55 Photos: Barack Obama's presidency Obama sits on the famed Rosa Parks bus at the Henry Ford Museum in Dearborn, Michigan, in April 2012. Hide Caption 35 of 55 Photos: Barack Obama's presidency British Prime Minister David Cameron, Obama, German Chancellor Angela Merkel and others watch the overtime shootout of the Champions League final between Chelsea and Bayern Munich in a conference room at Camp David, Maryland, during a G-8 Summit in May 2012. Hide Caption 36 of 55 Photos: Barack Obama's presidency Obama sits in his chair during a Cabinet meeting in July 2012. This image was tweeted by his official Twitter account in August 2012 in response to Clint Eastwood's "empty chair" speech at the Republican National Convention. The tweet simply said, "This seat's taken." Hide Caption 37 of 55 Photos: Barack Obama's presidency Obama casts a shadow in this picture as he accepts the 2012 Democratic nomination for President during the final day of the Democratic National Convention in September 2012. Hide Caption 38 of 55 Photos: Barack Obama's presidency Obama and Republican presidential candidate Mitt Romney participate in the first presidential debate of the 2012 election. Hide Caption 39 of 55 Photos: Barack Obama's presidency Obama celebrates on stage in Chicago after defeating Romney on Election Day in 2012. Hide Caption 40 of 55 Photos: Barack Obama's presidency Obama pauses during his speech at a memorial service for the victims of the Sandy Hook Elementary School massacre in December 2012. Hide Caption 41 of 55 Photos: Barack Obama's presidency Hundreds of thousands gather at the U.S. Capitol building as Obama is inaugurated for his second term on January 21, 2013. Hide Caption 42 of 55 Photos: Barack Obama's presidency Israeli President Shimon Peres, left, and Prime Minister Benjamin Netanyahu, right, stand with Obama after Obama arrived in Tel Aviv, Israel, in March 2013. Hide Caption 43 of 55 Photos: Barack Obama's presidency Obama adjusts an umbrella held by a Marine during a White House news conference with Turkish Prime Minister Recep Tayyip Erdogan in May 2013. Hide Caption 44 of 55 Photos: Barack Obama's presidency Jay Leno interviews Obama on "The Tonight Show" in August 2013. Hide Caption 45 of 55 Photos: Barack Obama's presidency White House press secretary Jay Carney fields questions from reporters during a daily press briefing at the White House in September 2013. Obama had just pushed for congressional approval for limited military strikes against the Syrian government. Hide Caption 46 of 55 Photos: Barack Obama's presidency Actor and comedian Zach Galifianakis interviews Obama during his appearance on "Between Two Ferns," a digital video series with a laser focus on reaching people aged 18 to 34. The President urged young people to sign up for his new health care plan in the video posted on the website Funny or Die. Hide Caption 47 of 55 Photos: Barack Obama's presidency Obama walks to the Oval Office on August 7, 2014, the same day he announced the beginning of airstrikes on ISIS. Hide Caption 48 of 55 Photos: Barack Obama's presidency Obama speaks to the nation about normalizing diplomatic relations with Cuba in December 2014. Hide Caption 49 of 55 Photos: Barack Obama's presidency From left, House Majority Leader Kevin McCarthy, House Minority Leader Nancy Pelosi and House Speaker John Boehner listen as Obama speaks during a meeting in the Cabinet Room of the White House on January 13, 2015. Hide Caption 50 of 55 Photos: Barack Obama's presidency Obama responds to a heckler who interrupted his speech during a White House reception for LGBT Pride Month in June 2015. Hide Caption 51 of 55 Photos: Barack Obama's presidency Obama cries in January 2016 as he delivers a statement on his executive action to reduce gun violence. Hide Caption 52 of 55 Photos: Barack Obama's presidency Cuban President Raul Castro tries to lift up Obama's arm at the end of a joint news conference in Havana, Cuba, in March 2016. Castro hailed Obama's opposition to a long-standing economic "blockade," but said it would need to end before ties between the two countries are fully normalized. Hide Caption 53 of 55 Photos: Barack Obama's presidency Obama visits Prince William, Duchess Catherine and their son, Prince George, during a trip to London in April 2016. Hide Caption 54 of 55 Photos: Barack Obama's presidency Obama hugs Hillary Clinton after speaking at the Democratic National Convention in July 2016. Obama told the crowd at Philadelphia's Wells Fargo Center that Clinton is ready to be commander in chief. "For four years, I had a front-row seat to her intelligence, her judgment and her discipline," he said, referring to Clinton's stint as secretary of state. Hide Caption 55 of 55

"We've always known the history of trade debates and local politics can be pretty tough. This time is no different," said Rep. Ron Kind, the Wisconsin Democrat who organized the meeting.

"I just hope that they are not losing the forest to the tree here," Kind told CNN, noting "a lot of issues affect the causes that labor stands for, and Democrats, by and large, are extremely supportive of that. To isolate this one issue and make that the end-all, be-all, I think, is a bad strategy."

While labor's threats have thrown the fate of a key trade bill into doubt, some Democrats say they could backfire.

Richmond, who did not attend Wednesday's meeting said he was "leaning no" on the bill -- but the more he hears about labor groups' tactics, the more inclined he is to support Obama.

"I'm watching them do it, and it bothers me," Richmond said.

Obama and his administration have spent recent months lobbying Democratic lawmakers to support the bill in phone calls, one-on-one meetings and group briefings. Both Democratic and Republican aides say with around 50 tea partiers likely to break away from GOP members who typically support free trade, Obama's party will need to supply about 30 votes -- and the President is somewhere around a dozen shy.

White House optimistic

The White House is optimistic, though, since Pelosi has remained on the sidelines -- not publicly supporting the bill, but not twisting her members' arms to oppose it, either.

Democrats on both sides of the issue have praised Pelosi, who hasn't publicly taken a position on trade, or giving her members time and space to decide how they will vote. Pelosi has made sure they get regular briefings, and has bombarded offices with a steady stream of background on the details of the agreement.

"It's like drinking through a fire hydrant," Richmond said.

Photos: The hand-off: How Speakers Ryan to O'Neill came to power Photos: The hand-off: How Speakers Ryan to O'Neill came to power The current Speaker of the U.S. House of Representatives is Paul Ryan, a Republican from Wisconsin. He gained his power from his predecessor, former Speaker John Boehner, after the Ohio Republican shocked the political world by deciding to vacate his position. Click through for other recent speakers: Hide Caption 1 of 8 Photos: The hand-off: How Speakers Ryan to O'Neill came to power Former Speaker John Boehner, a Republican from Ohio, gained his power from his predecessor, former Speaker Nancy Pelosi, when the GOP gained the majority of seats in the House in the 2010 midterm elections. Boehner announced his intention to leave the position in September 2015, and Paul Ryan succeeded him in October. Hide Caption 2 of 8 Photos: The hand-off: How Speakers Ryan to O'Neill came to power U.S. Rep. Nancy Pelosi was the first and only female speaker of the House. Her speakership lasted from January 4, 2007, to January 3, 2011. Pelosi, a Democrat, lost her seat to the Republican majority in the 2010 midterms. John Boehner took the gavel. Hide Caption 3 of 8 Photos: The hand-off: How Speakers Ryan to O'Neill came to power Dennis Hastert remains the longest serving Republican speaker in history, from January 6, 1999, to January 3, 2007. However, the GOP lost its majority in the House of Representatives, leaving Democrat Nancy Pelosi to become speaker. On Thursday, May 28, Hastert was accused in an indictment of lying to the FBI and evading currency reporting requirements as he sought to pay off a subject to "cover up past misconduct." On Thursday, October 28, Hastert pleaded guilty in the case. Hide Caption 4 of 8 Photos: The hand-off: How Speakers Ryan to O'Neill came to power Newt Gingrich broke the four-decade line of Democratic speakers by becoming speaker from 1995 to 1999 and was named Man of the Year by Time magazine for the accomplishment. He then fell from grace after a disappointing 1998 midterm election for the GOP, prompting him to step down from both the speakership and Congress. Gingrich's resignations came as a complete surprise to many, as the speaker had been fighting to keep his top job until the announcement. Hide Caption 5 of 8 Photos: The hand-off: How Speakers Ryan to O'Neill came to power Tom Foley, a Democrat who represented Washington state in the House for 30 years, took over the office of the speaker after the resignation of Jim Wright. Foley served as speaker from 1989 to January 1995 but was defeated in the 1994 election by George Nethercutt. Hide Caption 6 of 8 Photos: The hand-off: How Speakers Ryan to O'Neill came to power Jim Wright of Texas served two years as speaker, after Massachusetts Democrat Tip O'Neill retired. But he stepped down in 1989 after facing a House Ethics Committee investigation on improprieties with the sale of his book and fees from speaking engagements. He was the first speaker to resign in the face of a scandal. He died on May 6, 2015, at 92. Hide Caption 7 of 8 Photos: The hand-off: How Speakers Ryan to O'Neill came to power A Massachusetts Democrat who served as speaker from 1977 until retirement in 1987, Tip O'Neill was well-known for his deal-making as well as his collegiality with former President Ronald Reagan. Former Speaker Nancy Pelosi thought to commemorate the 34-year congressional career of O'Neill with an office building on the edge of Capitol Hill that bears his name. O'Neill was the longest continuously serving House speaker of any party in U.S. history. Hide Caption 8 of 8

Some pro-trade groups are hopeful that members of the Congressional Black Caucus, feeling some loyalty to President Obama, will provide the final bloc of votes to pass trade promotional authority.

But Richmond, a member of the Congressional Black Caucus, told reporters that it's "unrealistic and unfair to group the black caucus like that."

"I've heard them (Republicans) say, 'because he's y'all's president' a lot of the time, and I think that's offensive because I think you are insinuating he's not everybody's president."

Eating their own?

House Democrats are not only griping about labor's tactics, but about their fellow Democrats actively working against them.

One outspoken opponent of the trade deal, Rep. Alan Grayson of Florida, circulated an email to anti-trade groups and other members who have been critical listing the names of all the House Democrats who committed to voting yes, urging grassroots supporters to bombard their offices and pressure them to vote no.

Texas Rep. Henry Cuellar attended the session with Pelosi and told CNN he specifically called out Grayson in the meeting. He quoted a fellow Texas Democrat, former President Lyndon B. Johnson, who once joked about the difference between cannibals and liberals.

"Cannibals don't eat their own," Cuellar told his fellow Democrats. "Here, we are seeing people going after their own, just because we have a disagreement."