CHESTERFIELD COUNTY, Va. -- An Amelia man was charged with brandishing a firearm after he pointed a gun at a Chesterfield family that was waving Confederate Flags on Hull Street Road outside their Moseley home, police said. James Baker, 46, was released following his arrest on the class one misdemeanor. He is due in court on September 3.

No one was hurt in the Monday night incident and no charges have been filed yet as the investigation remained on-going, a Chesterfield Police spokeswoman said.

Mark Wilson, whose family stood outside and waved the flag Monday night and again on Tuesday, called the incident scary.

"He slammed on the brakes and when he got right beside me, he pulled out a gun, chambered a round and told me my cause wasn't worth anything now," Wilson said. "He got out of the car and took three steps towards me and the gun was maybe six inches off my head."

Wilson said he was worried the man was going to shoot his children. Wilson's son was concerned for his father's safety.

"I was just hoping the man didn't make a bad decision to pull that trigger," Wilson's son 15-year-old Mark Jr. said. "The guy starts to yell at my father and got out of his car, pulled the weapon and chambered a round in it. [He] put it to [my father's] head.

The man eventually got back into his car and drove off. The Wilsons called the police.

"They found him in Amelia and he said we were threatening his life," Mark Wilson said. "This is a busy highway and he is in a vehicle going 70 miles per hour, how are we threatening him by walking across?"

The Wilson family said to them, the Confederate Flag is about heritage, not hate.

"We just don't believe our flag means that we're racist," Mark Wilson said. He said his family planned to continue waving the flag on Hull Street until the public understood their side of the flag debate.

As for the man accused of pulling out the gun?

"He brandished a weapon, he chambered a round and put it to my head," Wilson said. "I think he needs to go to jail."