Justice Ginsburg seemed troubled by the board’s criteria. “It’s a nebulous standard,” she said.

Justice Stephen G. Breyer, joined by Justice Kennedy, pressed Mr. Keller for the standard the state uses to allow some plates but not others.

“I just think you have to have some kind of legitimate reason,” Justice Breyer said. “It doesn’t have to be much. It could be just a little.”

But neither justice seemed satisfied with Mr. Keller’s responses, which were vague and abstract.

The veterans’ heritage group won in the United States Court of Appeals for the Fifth Circuit, in New Orleans. The court said the proposed message on the license plate was private speech and that Texas had discriminated against the group’s view that “the Confederate flag is a symbol of sacrifice, independence and Southern heritage.”

In the early part of Monday’s argument the group seemed likely to prevail. But some justices seemed to grow uncomfortable with the implications of a ruling in its favor, which could make states choose between having any specialty license plates and allowing all sorts of vile and vulgar speech.

Justice Scalia asked about “dirty words” on personalized vanity plates. Mr. Keller said a ruling against Texas would require it to allow such plates along with specialty plates celebrating Al Qaeda and the Nazi party.

Chief Justice Roberts proposed a solution. “There is an easy answer to that,” he said, “which is they don’t have to get in the business of selling space on their license plates to begin with.”

By the time R. James George Jr., a lawyer for the heritage group, began his argument, he seemed to have amassed a majority. But he took a hard line, and it may not have served him well.