When Nasdaq Inc. named Adena Friedman as its next chief executive in November, she won attention for being young, tech-savvy and the first woman to lead a major U.S. stock-exchange operator. Now, the hard part has begun.

Ms. Friedman, a 47-year-old black belt in taekwondo, has inherited a company whose traditional core business of U.S. equities trading has gotten much tougher since the heady 1990s when Nasdaq touted itself as “the stock market for the next 100 years.”

Regulations in the late 1990s and 2000s brought competition to exchanges, opening the door to high-tech upstarts like Bats Global Markets Inc., which surpassed Nasdaq as the second-biggest U.S. exchange group by volume in 2014, and dozens of dark pools. Two major market crashes in the 2000s soured many investors on stocks altogether.

Analysts link a recent slide in Nasdaq’s share of U.S. equities volume to a broad shift in trading toward securities listed on the New York Stock Exchange and its sister exchanges, including NYSE Arca, which specializes in exchange-traded funds.

Adding to Ms. Friedman’s challenges, Nasdaq faces fierce competition from its historic rival NYSE for listings—even in the realm of hot tech stocks where Nasdaq was once king. Earlier this week the Big Board won the coveted initial public offering of Snap Inc., owner of the popular messaging app Snapchat.