Jason Scherr had a lot on his mind the day after he opened his fifth Think Coffee shop in Manhattan last week. The fan was blowing too hard, the classical music was playing a little too loudly—and he was trying to figure out how to get more customers to pay with cash.

A new law that was supposed to reduce costs for merchants that accept debit cards has instead sent Mr. Scherr's monthly processing bills much higher and forced him to reassess the way he does business.

"My choice is to raise prices, discount for cash or get an ATM," says Mr. Scherr, a lawyer who has been in the coffee-shop business for more than a decade.

Just two months after one of the most controversial parts of the Dodd-Frank financial-overhaul law was enacted, some merchants and consumers are starting to pay the price.

Many business owners who sell low-priced goods like coffee and candy bars now are paying higher rates—not lower—when their customers use debit cards for transactions that are less than roughly $10.