Concerned a brain drain could hurt its long-term ability to compete, Google Inc. is tackling the problem with its typical tool: an algorithm.

The Internet search giant recently began crunching data from employee reviews and promotion and pay histories in a mathematical formula Google says can identify which of its 20,000 employees are most likely to quit.

Google...