Every time shoppers return purchases to Best Buy Co., they are tracked by a company that has the power to override the store’s touted policy and refuse to refund their money.

That is because the electronics giant is one of several chains that have hired a service called Retail Equation to score customers’ shopping behavior and impose limits on the amount of merchandise they can return.

Jake...