BARRON, Wis. — The father of Jake Patterson, the 21-year-old man accused of kidnapping and holding 13-year-old Jayme Closs in captivity for nearly three months, says he has a letter for the Closs family.

Patrick Patterson told a CNN reporter at the Barron County Justice Center on Tuesday that he was hoping to pass along a letter he wrote to the Closs family.

"All I care about right now is Jayme's family. I want to get them a note," he told CNN. Patrick Patterson declined to provide more details on the letter and apologized for not being able to elaborate on the situation.

"I'm sorry, I can't talk," he said, before he was escorted away from the public area by a deputy.

Patrick Patterson, according to CNN, was there for his son's arraignment Monday and could be seen crying and burying his head into the shoulder of a nearby relative as he watched his son appear in Barron County Circuit Court.

Jake Patterson is also accused of killing Jayme's parents, James and Denise Closs, in their home outside Barron on Oct. 15 and taking Jayme against her will.

More:Jake Patterson's car was seen the night of Jayme Closs' kidnapping

Jake Patterson's grandfather, Jim Moyer, told ABC News that the family had no clue that any of this was happening.

"Something went terribly wrong, nobody had any clues ... We are absolutely heartbroken. It's wrenching to deal with," he said. "He was shy and quiet, he backed off from crowds, but a nice boy, polite. Computer games were more of a priority than social interaction."

Jake Patterson was charged in Barron County Monday with two counts of first-degree intentional homicide, one count of kidnapping and one count of armed burglary. He is being held on a $5 million cash bail.

On Tuesday, Patterson was moved to the Polk County Jail from his cell in Barron. Barron County Sheriff, Chris Fitzgerald, said he made the decision but didn't offer a specific reason for doing so. However, he did say there were no threats against Patterson and that a relative of Jayme's works in the Barron jail.

More:The details of Jayme Closs' kidnapping are terrible. Are more to come?

More:911 call log reveals details of Jayme Closs' escape from Jake Patterson

More:Jake Patterson targeted Jayme Closs after seeing her board school bus, complaint says