By Bill Mears, CNN Supreme Court Producer

The Obama administration has begun limiting the legal rights of terror suspects held at the Guantanamo Bay military prison in Cuba, telling a federal judge Tuesday the government alone should decide when the prisoners deserve regular access to their counsel.

In a 52-page filing, Justice Department lawyers said they have started restricting when Guantanamo prisoners can challenge their detention in a Washington-based federal court. If approved, any relaxing of the rules would be made on a case-by-case basis at the exclusive discretion of military officials, not by the courts.

At issue is whether a Supreme Court decision on detainee rights from 2008 gives federal courts the ultimate power to control so-called "habeas" petitions from foreign combatants in U.S. military custody. Volunteer private lawyers say they deserve regular access to their imprisoned clients, even if there is no active habeas challenge pending in court, or any pending charges.

Under the proposed changes, the Navy base commander at Guantanamo would have sole veto power over attorney access, as well as access to classified material, including information provided directly by the detainees from interrogations.

"The dispute thus before the Court, though important, is quite narrow," said the government in its legal filing. "The only question presented is whether detainees who have neither current nor impending habeas petitions are entitled to" challenge continued access to counsel. "The answer to that question is 'no.'"

The case is before Chief Judge Royce Lamberth. His court has been handling the many appeals filed by the prisoners. There are currently 168 detainees - all male - in the Guantanamo facility, most of whom do not have pending charges. Five Muslim men labeled "high-value detainees" are being prosecuted before a military commission for their alleged leadership roles in the 9/11 terrorist attacks.

In the so-called Boumediene ruling in 2008, the high court said "enemy combatants" held overseas in U.S. military custody have a right to a "meaningful review" of their detention in the civilian legal justice system. It would force the government to present evidence and justify keeping the prisoners indefinitely, without charges. But a federal appeals court in Washington has since refused to order the release of any detainee filing a habeas corpus writ, in some cases rejecting such orders from lower-court judges.

The administration has argued it does not seek to restrict lawyers who have an active legal appeal, but that the rights of detainees shrink once they have filed their first habeas challenge. The military says lawyers must now agree to the new conditions in order to have continued access to their clients and to any classified information the military would deem to release.

And lawyers would be prohibited from using any information they gather that might help the prisoners appearing before a Periodic Review Board. PRBs are newly designed panels of military officials to decide whether a Guantanamo inmate should continue to be held, and whether that person is a national security threat. Those boards were put in place by President Barack Obama by executive order, but have not been fully implemented.

"Executive Order 13,567 does not provide detainees who undergo PRB review with a judicially enforceable right to counsel, or any justification for asking the Court to impose a counsel-access regime on the PRB process other than the one developed, per the Order's direction, by the Secretary of Defense," said the government. "As a general matter, executive orders are viewed as management tools for implementing the President's policies, not as legally binding documents that may be enforced against the Executive Branch."

The government said the court's power to intervene was limited, and urged Judge Lamberth to deny the request guaranteeing attorney access. A court hearing is set for August 17 on the legal question.