Washington (CNN) Sen. Lindsey Graham, R-South Carolina, says his GOP colleague and possible competitor for the White House was wrong to suggest vaccines could be the cause of mental disorders, and challenged him to back his claim with scientific evidence.

"The statement of [Sen. Rand] Paul would create anxiety because he's a doctor, he's a respected senator and there is no evidence that I'm aware of that a vaccine causes mental retardation," Graham told CNN during an interview in his Capitol Hill office. "If he has that evidence he should come forward and let the scientific community do something about it. You cannot scare people by anecdote."

On Monday, Paul said on CNBC he has heard of "many tragic cases of walking, talking, normal children who wound up with profound mental disorders after vaccines."

Graham suggested that was irresponsible.

"These are the most unsettled times I can remember since I have been in government. The last thing we need are politicians creating anxiety for no good reason," said Graham.

Last week Graham formed a political action committee to explore a possible White House run. Paul, too, has made no secret about his presidential ambition.

"Rand Paul is a nice man. He is a very committed libertarian. I think he is looking at this issue through a libertarian's eyes, not a physician's eyes," said Graham.

Even so, Graham rejected any libertarian notion that not vaccinating children is an individual right.

JUST WATCHED Gupta: Vaccines do not cause autism Replay More Videos ... MUST WATCH Gupta: Vaccines do not cause autism 01:30

"As to freedom, it is cherished, it is hard to come by, it is hard to hang on to. But freedom without responsibility is chaos, so to those who push the idea that freedom would allow an individual to do anything, anywhere, at any time, I reject," Graham said. "Your freedom ends where my ability to raise my family safely begins. So I would urge every American to vaccinate their children and I would reject any effort to stop vaccinations until someone can show me a scientific reason to do so."

Graham said he is not ready to call for a federal mandate for vaccines.

"For a federal mandate to come about for measles or any other particular illness or disease you'd have to convince me it was necessary," he said. "But the question is, is there scientific evidence to justify a politician suggesting that vaccinations programs we have today are unsound? My answer is no."

When asked by CNN's Athena Jones Tuesday, Paul declined a chance to clarify his comments.

On Monday, a Paul spokesman told CNN he does believe vaccines save lives and that his children were all vaccinated.

"He also believes many vaccines should be voluntary and like most medical decisions, between the doctor and the patient, not the government," said Sergio Gor, a Paul campaign spokesman.

On Tuesday, Paul further clarified his stance, saying he didn't say vaccines caused disorders.

"I did not say vaccines caused disorders, just that they were temporally related -- I did not allege causation. I support vaccines, I receive them myself and I had all of my children vaccinated," Paul said in a statement. "In fact today, I received the booster shot for the vaccines I got when I went to Guatemala last year."