Tokyo, Japan (CNN) -- Japan's prime minister visited Okinawa and delivered an unpopular message Tuesday: it will be challenging to move a U.S. base off the island.

"I have determined that transferring all of the functions at Futenma [Air Base] outside the prefecture or abroad is difficult," Prime Minister Yukio Hatoyama said.

Okinawans say the U.S. military has been responsible for a number of blights in the city, from serious crimes like rape and drunken driving, to environmental and noise pollution.

Nearly 100,000 residents protested last week demanding that the base be moved off the island.

And a recent poll said two out of three Japanese say they disapprove of the prime minister's performance and 59 percent said Hatoyama should resign if he can't resolve the fight over the future of the Futenma Air Base in Okinawa.

One of the issues that sparked a backlash against the U.S. base recently was the 2008 charges against a Marine for the rape of a 14-year-old Japanese girl in Okinawa.

The case revived simmering anger in the local community, some of whom remembered the 2006 case of the sailor who was convicted of fatally beating a 56-year-old Japanese woman and the 1995 case of three American servicemen who were found guilty of kidnapping and raping a 12-year-old Okinawan schoolgirl.

CNN's Kyung Lah contributed to this report.