Story highlights Colin Powell endorsed President Barack Obama twice and came out in favor of the Iran nuclear deal

But, he says, "I'm having difficulty with the party now"

Washington (CNN) Former Secretary of State Colin Powell, who endorsed President Barack Obama twice and came out in favor of the Iran nuclear deal days before the Congressional vote, says that he is still a Republican.

When asked at the Washington Ideas Forum Wednesday if he still considers himself a Republican, Powell said, "Yes, I'm still a Republican," and highlighted his votes for "five presidents in a row who were Republican," as well as his work for President Ronald Reagan and former Sen. Howard Baker.

"I want to continue to be a Republican because it annoys them," Powell said, prompting laughter from the audience.

"In Virginia, you don't have to declare a party ... but I'm still a Republican because I believe in a strong defense, because I believe in the entrepreneurial spirit that is so typical of the Republican Party in the past. But I'm having difficulty with the party now," he said.

Read More