TOCCOA, Ga. — At the Waffle House here, no one can believe that the gray-haired men who came in almost daily for egg sandwiches and coffee could have been terrorists plotting to blow up government buildings and kill masses of people using poison from a bean plant that people in this rural part of the state grow to ward off moles.

The four men — bespectacled and so hard of hearing that they strained on Wednesday to hear a federal judge at their initial court appearances in Gainesville, Ga. — have been accused of conspiracy and may be linked to a loose association of fringe militia groups targeting a government they believe is out of control, according to court papers.

“There’s two schools of thought on this: go for the feds or go for the locals,” Frederick Thomas, 73, a Navy veteran who was described as the leader of the group was recorded saying to an F.B.I. informant. “I’m inclined to consider both. We’d have to blow the whole building like Timothy McVeigh.”

Those statements and dozens of others were cited in court documents that also charged Dan Roberts, 67; Ray Adams, 65, and Samuel Crump, 68. The men were arrested at their homes on Tuesday by a wave of federal agents so big it stunned people in this mountain town of about 27,000.