Yahoo Inc.'s rejection of Microsoft Corp.'s buyout bid will test whether the software giant is willing to pay a lot more for the Internet company -- or risk a truly hostile takeover attempt.

Microsoft may sweeten its offer, say people familiar with the matter. But any increase is likely to fall short of what Yahoo's directors believe would fairly value the company, the people say, setting the stage for a protracted battle.

People...