For years, CBOE Holdings Inc. was the reluctant acquisition target: a profitable options exchange with an old-school trading floor ripe for change.

But the company turned the playbook upside down Monday, announcing its own high-profile acquisition of Bats Global Markets Inc. for $3.2 billion in a largely unexpected deal that would create a new global exchange behemoth to rival the likes of New York Stock Exchange owner Intercontinental Exchange Inc., Nasdaq Inc. and CME Group Inc.

The...