Story highlights Scuffle happens at Palm Beach County, Florida, polling place, authorities say

One man says the other pulled a gun, CNN affiliate WPBF reports

(CNN) A Florida man was charged with simple battery stemming from a scuffle outside a polling site, the Palm Beach County Sheriff's Office said.

The sheriff's office said a 39-year-old man was videotaping a 68-year-old man with his cell phone outside a Lake Worth poll, but the older man objected and slapped the phone out of the younger man's hand, causing it to fall to the ground.

The younger man struck the older man and caused him to fall to the ground, the sheriff's office said.

"Fearing for his safety," the older man said he had a weapon, authorities said. He didn't actually make a threat and had a concealed weapons permit for the weapon he was carrying, the sheriff's office said.

The older man was charged with simple battery and criminal mischief, the sheriff's office said. He has not been identified. The younger man was not charged.

Read More