More consumers are finding out that stores are using a third-party service to monitor their shopping behavior and limit the amount of merchandise they can return.

Best Buy Co. has been the target of a large share of consumer complaints on Facebook, Twitter, Yelp and other online forums about its efforts to police returns with the help of a firm called Retail Equation. But the same service is used by a variety of other retailers, including J.C. Penney Co., Sephora, CVS Health Corp., Advance Auto Parts Inc., Dick’s Sporting...