Editor's note: David Frum writes a weekly column for CNN.com. A special assistant to President George W. Bush from 2001 to 2002, he is the author of six books, including "Comeback: Conservatism That Can Win Again," and is the editor of FrumForum.

Washington (CNN) -- How do you score partisan points against a president who looks to have won a military victory in Libya at very low cost? Simple: Attack him for being away from Washington at a summer house when the victory was won.

That line of attack was tested by some Republican talkers and bloggers this weekend. I wonder if we'll hear more of it in the coming days.

Almost nothing in American politics drives more people to say more ridiculous things than the subject of presidential summer vacations.

Back during the George W. Bush years, Democrats sneered and scoffed at the 43rd president's extended visits to his Crawford, Texas, ranch. Where was the guy's work ethic?

Now the shoe is on the other foot. President Barack Obama has, for the second summer in a row, rented an expensive compound on the Atlantic island of Martha's Vineyard. What a snob!

Let's dial back and introduce some reality to this partisan point-scoring.

The president of the United States never gets a vacation, not really. The nuclear football follows wherever the president goes. He receives the daily intelligence briefing every morning, including Christmas. The decisions never stop, the cares of state never lighten, the burden of responsibility is never lifted.

When a president goes "on vacation," here's what happens:

1) He or she is spared the ceremonial parts of the job: the state dinners, the meetings with the girl who sold the most Girl Scout cookies that year, that kind of thing;

2) The other members of the first family are liberated from living inside the White House, aptly described by Harry Truman as "the crown jewel of the federal prison system."

But this game of tallying "vacation days" to make a point about presidential work ethic tells us nothing. Franklin Roosevelt devised the concept of Lend Lease, which provided aid to Britain and other nations in the early years of World War II, while on a two-week cruise through the Caribbean in December 1940. That seems a very good week's work -- even if he also managed to find time for a little sunbathing.

We don't measure presidential productivity by hours spent behind a desk in Washington. We measure by results.

Even sillier than complaints about days away from Washington is the tit for tat whereby presidential vacations are used by one set of partisans to depict another set of partisans as "out of touch."

This summer Republicans urge Americans to be shocked that Obama rented a vacation house that reportedly goes for $50,000 a week. Five years ago, Democrats wanted Americans to be shocked that Bush took his vacations on a 1,200-acre hobby ranch. Before that ... it was Bill Clinton again on Martha's Vineyard, and before that it was the elder Bush at his Kennebunkport, Maine, compound, and so on and on.

The ironies jump out here.

The holidays that raise the most serious concerns are those taken by the poorest presidents, not the richest.

Truman was probably the poorest president of the 20th century. He took 11 vacations on U.S. Navy property in Key West, Florida, where a large and pleasant lodge house was made available to the president and his poker-playing friends.

Truman paid nothing for the use of the house, which still stands as a museum.

As president, Clinton had little more wealth to his name than Truman. He accepted the use of the vacation homes of supporters, a deviation from the usual rules that strongly frown on presidents accepting valuable gifts for their own use.

By contrast, both Obama and George W. Bush paid their own way. Isn't that better?

Apparently not. As the country has become more ideologically polarized, even vacation choices have become opportunities for culture war, for battles over who is a real authentic American and who is an out-of-touch East Coast elitist or alternatively phony cowboy.

Is this not nuts? There's only one president at a time, occupying the toughest job in the world. If they like to clear brush, let them clear brush. If they like to bike on the beach, then go ahead. Leave them be.

A tense, edgy, unhappy and overtired president is good for nobody.

The opinions expressed in this commentary are solely those of David Frum.