Stock-exchange operator BATS Global Markets Inc. said late Wednesday that a "system issue" allowed hundreds of thousands of transactions in a four-year period to be executed at prices that may have violated securities rules.

At issue are rules the Securities and Exchange Commission has in place requiring orders to go to the best bid or offer in the market. BATS said problems also occurred with transactions in which investors "short" shares, or bet a stock will fall. Short sellers borrow shares to sell them, then buy the stock at a later date to repay their loan. They profit if the stock falls.

According to a notice sent to BATS customers Wednesday, the systems problem caused some orders to be executed at prices that were equal to or less than the so-called national best bid and offer price—the price exchanges are required by regulators to honor.

The net amount of the "inferior prices" customers received as a result of the erroneous executions was less than $500,000, a BATS spokesman said Wednesday night.

BATS recently discovered the problems internally and hadn't received any complaints from customers, a person familiar with the matter said.