A federal appeals court ruled last week in favor of a Hawaii police department’s decision to ban a man convicted of misdemeanor harassment 20 years ago from owning firearms.

The Ninth U.S. Circuit Court of Appeals said Friday the Honolulu Police Department didn’t violate Kirk Fisher’s constitutional rights when it denied him a gun permit in 2009 based on a prior misdemeanor conviction for harassing his wife and daughter 12 years earlier.

Fisher pleaded guilty in 1997 to two counts of harassment and served six months probation. Honolulu police then returned the firearms Fisher surrendered per court order after his initial arrest the year before. When he applied for a permit to own a gun more than a decade later, police refused, citing his prior conviction, and forced him to transfer ownership of his existing weapons to his wife.

Fisher filed a civil suit against the city and police department in 2011, according to the Honolulu Star-Advertiser. A federal judge ruled against him in 2014 and he appealed.

All three appeals court judges struck down Fisher’s case Friday, though Circuit Judge Alex Kozinski criticized state law’s “notably slender” path to Second Amendment rights’ restoration: a gubernatorial pardon. Comparatively, federal law provides four different ways to get ownership rights back.

“The time has come to treat the Second Amendment as a real constitutional right,” he wrote in court documents Friday. “It’s here to stay.”