Japan's powerful Fisheries Agency has publicly floated the idea of ending the country's controversial whaling program.

A report by an official agency panel has raised the option stopping the whaling program, saying constant harassment by the militant Sea Shepherd conservation group has made whaling too dangerous for Japanese crews.

Harassment by Sea Shepherd has limited the whaling fleet's effectiveness, and last season it was forced to return to Japan after harpooning only a fifth of its total quota of whales.

The agency panel was convened to examine the impact of Sea Shepherd's actions and make recommendations for the upcoming Southern Ocean whaling season.

Panel member Hisa Anan says she does not believe Sea Shepherd will ever stop its anti-whaling campaign.

"The crew of the whaling fleet should not be exposed to danger, and the government should not force the fleet to go out," Ms Anan said.

"So, I think Japan should end its research whaling."

Only a minority of the fisheries panel agreed with Ms Anan's call to end the whaling program.

Other options include keeping the whaling expeditions as they are, or scaling back operations in the Antarctic and taking a smaller catch.

For advocates of the whaling program, any talk of putting the harpoons away is defeatism and a capitulation to what is denounced as Sea Shepherd's "eco-terrorism".

Masayuki Komatsu, a former Japanese representative to the International Whaling Commission, argues the whalers are doing nothing wrong.

"We believe this is perfectly legal and scientifically justified," he said. "We should not give up."

Mr Komatsu's assertion that the whaling program is scientifically vital is dismissed by Ms Anan.

"Research whaling has been conducted for more than 20 years now," Ms Anan said.

"I think they've gathered enough scientific data. And even if they want more, they can conduct non-lethal research."

But increasingly, more and more Japanese are calling an end to whaling because of the high cost of the program.

According to Japan's Nikkei newspaper, it costs around $40 million a year to equip and supply the fleet and send it thousands of kilometres south.

Few Japanese want to eat the meat the fleet harvests, and it mostly ends up in growing stockpiles in freezers around the country.

With the whaling fleet due to embark on its annual hunt in November, Japan's Fisheries Agency has just three months to make a decision whether to continue with the program.