A celebrity roster of protesters, including climate scientist James Hansen, actor Daryl Hannah, civil rights leader Julian Bond and environmental advocate Robert Kennedy Jr, were arrested on Wednesday after handcuffing themselves to the gates of the White House to demand that Barack Obama shut down the Keystone XL pipeline project.

In all, about 50 people were detained. The protests, while entirely orderly and co-ordinated in advance with local police, delivered a sharp reminder to Obama the morning after his state of the union address of the enormous political challenges ahead as he tries to make good on his promise to act on climate change.

Other campaign groups vied for Obama's attention on Wednesday. The Natural Resources Defense Council and the League of Conservation Voters released separate polls showing a majority of the American public – roughly two-thirds in both polls – wanted Obama to act on climate. Democratic senators are preparing to formally launch a new attempt at climate legislation on Thursday. None of those issues, however, are as immediately problematic for Obama as the Keystone XL project.

Obama did not mention Keystone XL in the state of the union speech. But the pipeline has become one of the most contentious issues on his second-term agenda, a touchstone for environmental campaigners. To protesters, the project has become the defining symbol of Obama's promises to cut the greenhouse gas emissions that cause climate change.

"The Keystone pipeline project is the purest test of whether the president is serious about doing something about climate change or not," the author Bill McKibben, whose 350.org group has led opposition to the project, told a few hundred protesters outside the White House.

Pipeline supporters – the Canadian government, the oil industry and a large swath of the country – argue that it will create jobs and shore up supplies of North American energy. Pipeline opponents say the project will unlock vast stores of carbon, overwhelming Obama's other efforts to cut emissions that cause climate change.

"You can't really parcel it out," said Hannah. "You need to have a full-scale approach if you really want to be serious about addressing climate change."

Over the past 18 months, the protests, initially dominated by students, have grown in size, gathering supporters from across the environmental movement. On Wednesday the Sierra Club, the biggest mass environmental group, broke a 120-year tradition and authorised its president, Michael Brune, to risk arrest.

What was once seen as a fringe environmental cause has gone mainstream. Protesters last year forced Obama to call a halt to the project, because of concerns about the pipeline's proposed route through ecologically sensitive areas of Nebraska. Now opponents are thinking they have a chance of shutting down the project completely – if they can make a show of force. Obama is due to make his decision later this year.

"I think this helps him because he is getting tremendous pressure from the energy industry which puts more money into the political process than any other entity," said Robert Kennedy Jr. "They have all the money on their side, but we have the intensity on our side. I think the longer we can delay it the less likely it is to happen."

McKibben agreed. "Everybody told us it was a done deal, and that it was going to be built," he said. "But it's been 15 months so far, and that pipeline still hasn't been built."