(CNN) Paul Manafort's lawyers have new concerns about where the formmer Trump campaign chairman should be held in jail.

For weeks, Manafort has implored judges to alleviate the distance between him and his lawyers while he's at Northern Neck Regional Jail, about two hours outside Washington. On Tuesday, federal Judge T.S. Ellis stepped in to move Manafort to an Alexandria, Virginia, detention center so he "has access to his counsel and can adequately prepare his defense."

But Manafort doesn't want that, his attorneys said Tuesday. Five hours after Ellis said he could move closer to his lawyers, Manafort told the court he'd prefer to stay at Northern Neck. His safety among other inmates is one concern, his lawyers say.

"In light of Mr. Manafort's continuing detention and after further reflection, issues of distance and inconvenience must yield to concerns about his safety and, more importantly, the challenges he will face in adjusting to a new place of confinement and the changing circumstances of detention two weeks before trial," Manafort's attorneys wrote to the judge Tuesday afternoon.

Manafort has been held in a "VIP" unit for one at the rural Warsaw, Virginia, facility since a federal judge in DC revoked his bail on June 15. The judge found Manafort had violated the terms of his house arrest by contacting potential witnesses in his case -- an action that also led to two obstruction of justice criminal charges piled on top of the bank fraud, lobbying and other financial crime allegations he faces. He is set to face a jury in Virginia for a three-week trial on alleged financial crimes beginning July 25.

Read More