(CNN) -- President Obama and his family are spending time away from Washington but their weeklong vacation on Martha's Vineyard is unlikely to be free of controversy.

It's the president's second time on the island off the coast of Massachusetts. In 2009, Obama spent time golfing and hanging out with family and friends.

Vacations are often considered political low-hanging fruit: Zing a president or first lady anytime they go away for some rest and relaxation.

The topic of where the Obama would go has been fodder for cable television and radio talk show critics who ask: Why did the Obamas go to Maine instead of the Gulf Coast as Obama had encouraged Americans to help bolster tourism there? Once in the Gulf, why only for 27 hours? Who footed the bill for Michelle Obama's glitzy trip to Spain?

As for Martha's Vineyard, isn't that an elite playground?

Presidential historian Douglas Brinkley said some voters pay close attention to the message a destination sends -- especially in tough economic times.

"People would rather see their President in Put-in-Bay, Ohio, or the Wisconsin Dells than in the Kennedy realm of Cape Cod," he said.

Journalist Ken Walsh, who wrote a book on the history of presidential retreats, said the level of criticism goes up no matter who is in office.

"The opposition party is almost like a parlor game in Washington," he said. "There's a ritual that the out party criticizes the president for being profligate with spending taxpayers' money for flying away from Washington, for not being sensitive to people's concerns and that's what is happening with President Obama now."

Walsh recalled how President Clinton was concerned about how his vacation destination played in middle America.

"President Clinton, for instance, took polls to figure out where he should go on vacation when he ran for re-election," he said. "He took polls to figure out where Americans want the president to go. Well [it was] national parks, so President Clinton went to national parks."

But Clinton also spent plenty of time on Martha's Vineyard.

The view of Martha's Vineyard, a vacation spot of the rich and famous, has the potential to hurt a president's image with average Americans when times are tough.

"When they run into trouble, public relations-wise, [is] if they look like they are being insensitive to the problems of the country and whatever people are going through," he said. "And that could be the problem President Obama has in Martha's Vineyard."

Presidential vacations have been scrutinized by the public for more than a century.

George W. Bush was often criticized by Democrats for taking long vacations to his ranch in Crawford, Texas, during the Iraq war.

For Theodore Roosevelt, leaving Washington for months to ride the rails or a horse created an uproar. The same was true for Franklin Roosevelt, who spent time floating around on his private yacht.

"Criticism's always been there ... whether it's Theodore Roosevelt disappearing and hunting, there's criticism," Brinkley said. "Whether it's FDR living on a yacht and not being at the White House, there's criticism."

Even John Adams came under fire when he spent around eight months away from the White House caring for his sick wife, Abigail.

"When he was away, the critics said he kind of abdicated the presidency," Walsh said. "And there were a lot of attempts to organize a sort of a war against France when he was away. So, he had to come back to deal with all that."

CNN's Ed Hornick contributed to this report.