WASHINGTON (CNN) -- Former Secretary of Homeland Security Tom Ridge says he successfully countered an effort by senior Bush administration officials to raise the nation's terror alert level in the days before the 2004 presidential vote.

Ex-Homeland Security Chief Tom Ridge says he was pushed to raise the terror alert before the 2004 election.

"An election-eve drama was being played out at the highest levels of our government" after Osama bin Laden released a pre-election message critical of President George W. Bush, writes Ridge in his new book, "The Test of Our Times."

Attorney General John Ashcroft and Secretary of Defense Donald Rumsfeld strongly advocated raising the security threat level to "orange," even though Ridge believed a threatening message "should not be the sole reason to elevate the threat level."

The former Pennsylvania governor also writes that he saw no reason for the move, which he now calls a bad idea, because additional security precautions had already been taken in advance of the election.

"We certainly didn't believe the tape alone warranted action, and we weren't seeing any additional intelligence that justified it. In fact, we were incredulous," Ridge said of the push. "... I wondered, 'Is this about security or politics?' " Watch panelists discuss Ridge's allegations »

The idea that an attack might take place had been discussed, he says. "But at this point there was nothing to indicate a specific threat and no reason to cause undue public alarm. ... It also seemed possible to me and to others around the table that something could be afoot other than simple concern about the country's safety."

In the end, the threat level was not raised.

Frances Townsend, a former Homeland Security adviser to President Bush and now a CNN contributor, denied politics played any role in the request to raise the threat level.

"There was a debate," Townsend said Thursday on CNN's "The Situation Room." "Tom Ridge wasn't the only person in that meeting who suggested that the terror alert shouldn't be raised. At no time was there any discussion of politics at that meeting. And the president was made a recommendation, a consensus recommendation from the council that he accepted, not to raise the terror alert."

According to 2004 exit polls, one in five voters said terrorism was the most important issue to them. Of those, 86 percent voted for Bush, 14 percent for Democratic candidate John Kerry.

Ridge said that he had mixed emotions about the incident at the time, and that after it ended, he knew he had to follow through on his plans to leave the government.

He also writes that the White House added language praising the president to an August 2004 statement on a newly discovered terror threat, and that he believes he should have delivered the warning as written, and pretended that he had accidentally left out the new wording.

"I should have known better," wrote Ridge, calling the incident one of his few regrets related to his time in public service.

All About Tom Ridge • U.S. Department of Homeland Security