Washington (CNN) The Justice Department inspector general's office referred former FBI Director James Comey for potential prosecution over his handling of memos that the FBI later determined contained classified information, a person familiar with the matter confirmed Thursday.

But Justice Department prosecutors declined to prosecute Comey, in part because they didn't believe there was evidence to show Comey knew and intended to violate laws on handling classified information.

At issue were memos that Comey shared with a friend and attorney, Daniel Richman, who then shared the information with a New York Times reporter.

Two memos Comey shared contained information the FBI deemed as "confidential," the lowest classification level.

CNN reached out to attorneys for Comey. Richman declined comment.

Read More