For Joe Temas, the gypsy cab business has been a lifeline. A former cab driver, Temas was barred from driving a licensed taxi after a stint in state prison. For a few years, he worked at an auto detailing shop. But after having a double hip replacement, he was unable to return to that work.

Forced to live on a $700 monthly disability check, Temas turned to an improvised gypsy cab stand in Dudley Square to supplement his income.

“You can make $50 in a day if you’re lucky,” he said.

Gypsy drivers are unlicensed. While they commonly refer to the cars as “cabs,” they operate without sanction of the law and, as such, are not regulated or legal. As is the case in Roxbury, they commonly operate in communities not served by licensed cab companies. The drivers commonly haggle on the price of a fare or leave it to the passenger to dictate the cost of a ride.

Like many gypsy drivers, Temas supplemented his fares by selling loose cigarettes for $1 each, earning little more than $10 profit for selling the contents of each pack. That lifeline was nearly severed two weeks ago when a Boston police officer ordered Temas out of his car at 8:45 a.m. and arrested him for selling cigarettes without a license.

Police held Temas in a cell at the station until 3 p.m., when he was brought before a judge, who promptly dismissed the case after learning Temas had been held for the better part of the day in jail without being charged. The police, however, kept 20 cartons of cigarettes they found in his car.

Temas was swept up in what many say is an ongoing crackdown against gypsy cabs in Dudley Square that has pitted merchants and property owners against the drivers and hustlers who populate the informal economic sector in the busy commercial district.

“It’s part of the black market economy,” said Luther Pinckney, a Dudley Square Main Streets board member. “When cabs wouldn’t come here, gypsy drivers stepped in to fill the need.”

On one side are drivers and their customers, who say they’re providing a needed service for the thousands of shoppers and commuters who pass through the square on a daily basis. On the other are merchants and business district boosters who complain of drivers who aggressively jockey for fares, loose cigarette sellers and other hustlers who crowd the sidewalk on Dudley Street, sometimes leading to arguments, fights, cat-calling and other public nuisances.

“There needs to be some kind of management there,” Pinckney said.

The gypsy cabs and hangers-on who congregate on Dudley Street are part of a broader pattern in the square, some say, that includes heroin addicts whose use of public bathrooms as injection sites recently prompted the management at the Bruce Bolling Municipal Building to close the bathrooms there to the public.

Temas admitted that some drivers contribute to the disorderly atmosphere in the square.

“A lot of them are yelling ‘Taxi, taxi, taxi!’” he said. “And some of them don’t have their cars registered and insured.”

Others, like Temas, mostly wait in their cars for regular customers who prefer to ride with a known driver. But loud or quiet, all are vulnerable to the recent sweeps. On a recent afternoon, the curb in front of Joe’s sub shop was bare, save for a BPD cruiser overlooking the usually bustling block.

David Price is executive director of Nuestra Comunidad CDC, the nonprofit that owns the 66-unit Dartmouth Hotel building that the gypsy drivers park in front of. While acknowledging that the drivers fill a need in the area, he said he’s heard complaints about the people who congregate in front of the building.

“It’s a narrow sidewalk,” he said. “It’s not good for the community. Retail businesses suffer because people can’t park there. In a place like Dudley Square, you want people to come in, park and shop. That’s not happening on that stretch.”

Price suggests the gypsy drivers move their operation to the opposite side of the street, by the Dudley branch of the Boston Public Library, where an unused cab stand is often occupied by parked cars.

“It would really help improve the retail environment, which has been struggling,” he said.

Carlos Castillo, who owns Castillo Wine and Spirits on Washington Street, agrees.

“Give them a location where people can get a ride that doesn’t disturb the merchants,” he said. “If it’s a nuisance to the merchants, relocate them.”

Such a move could be difficult. While gypsy drivers who congregate in areas such as the Washington Park Mall have for years self-managed or been managed by owners or security at supermarkets, the gathering at Dudley has never been under the control of a central authority, leaving it to the police to operate as a blunt instrument of control.

Temas said that from his perspective, efforts to sweep nuisances from the square shouldn’t come at the expense of the hard-working drivers who cause no problems. Their livelihoods are on the line.

“There’s not much else I can do,” he said. “I’m not a young kid anymore.”