At a 2007 meeting to discuss a potential buyout of Avaya Inc., some employees of private-equity firm TPG expressed concerns that the company was at risk of becoming technologically outmoded, a “buggy-whip business,” as one put it.

To them, Avaya’s business of installing and managing corporate phone systems appeared vulnerable to the same forces that were making landlines scarce in households across the U.S., according to people familiar with the meeting.

But...