Earlier this month, Brett Wentworth took Level 3 Communications Inc. into territory that most rivals have been reluctant to enter. The director of global security at the largest carrier of Internet traffic cut off data from reaching a group of servers in China that his company believed was involved in an active hacking attack.

The decision was reached after a broad internal review. The Broomfield, Colo., company is taking an aggressive—and some say risky approach—to battling criminal activity. Risky because hackers often...