Atlanta, Georgia (CNN) -- A federal judge in Georgia has been arrested on drug and weapons charges after federal agents say he bought drugs for a stripper with whom he was having an affair.

Senior U.S. District Judge Jack Camp Jr. has been charged with possession of cocaine, marijuana and the painkiller roxicodone, according to court documents. He is also charged with possessing a firearm as an unlawful user of controlled substances and with aiding and abetting the possession of drugs by the stripper, who had a prior drug conviction.

Camp was arrested Friday after federal agents said he bought drugs from an undercover agent. He made an initial appearance Monday in a federal court in Atlanta and was released on $50,000 bond.

The charges allege that Camp had been paying the stripper for sex since the past spring and was fronting money to buy drugs for both of them. Before the sting that led to his arrest, he told her, "Let me pay him, because you've already got a record," the affidavit states.

Camp, 67, is the former chief judge for the Northern District of Georgia. He was appointed to the bench by President Ronald Reagan in 1987 and took senior status in 2008. His lead attorney, Bill Morrison, would not discuss the facts of the case but played down his client's public office.

"This is really a case between Judge Camp and his wife," Morrison told reporters outside the Atlanta federal courthouse. "It's not a case of Judge Camp being a judge, it's a case of Judge Camp being a husband."

Tuesday, Morrison stressed his client's case has not been presented to a grand jury.

"If there is an indictment, then the next step after that will be a hearing," he said.

A Justice Department spokeswoman confirmed the next court appearance in the case has not yet been scheduled.

The dancer, who is not identified in court papers, had a prior drug-related federal felony conviction on her record and "recently began cooperating with the FBI," according to the charges -- but she began recording the judge before she became an informant, the affidavit states.

The woman has been promised immunity from prosecution, but also had fewer tapes than she originally told agents from other law enforcement agencies, according to a footnote in the court papers.

During the deal that led to his arrest, Camp carried at least two pistols, his arrest affidavit states. According to the paperwork, it was one of at least three transactions in which he was armed.

"I'll watch your back anytime 'cause I'm afraid, and I not only have my little pistol, I've got my big pistol," the affidavit quotes Camp as telling the woman beforehand. "So, uh, we'll take care of any problems that come up."

They took separate cars to the meeting, which took place in the parking lot of a suburban Atlanta nightclub. According to the affidavit, Camp pocketed the drugs and told the undercover agent, "We'll call you again."