The Dow industrials and the S&P 500 ended lower Wednesday as a drop in oil prices fueled a selloff in energy shares while a drop in International Business Machines Corp. was responsible for half the session’s losses in the blue-chip average.

The Dow Jones Industrial Average DJIA, -0.46% fell 118.79 points, or 0.6%, to 20,404.49, with IBM by far the biggest drag, the lowest close for the blue-chip benchmark since Feb. 10.

IBM IBM, +0.56% shed 4.9% after the tech giant posted weaker-than-expected quarterly sales late Tuesday. It was the biggest one-day drop for the company since June, and the decline took the stock to its weakest level since December.

IBM has an outsize impact because the Dow is price-weighted, meaning the most expensive stocks—rather than the largest companies—have the biggest pull. IBM is the fifth-most influential company in the 30-stock average by this metric, and contributed to a 57-point drag on the average.

“IBM is constantly disappointing, but it’s a nonplayer for me,” said Wayne Kaufman, chief market analyst at Phoenix Financial Services. “There are a few companies that people look at to read the tea leaves of the market, but IBM isn’t one of them anymore.”

Also, shares of Chevron Corp. CVX, +0.29% fell 1.4%, adding further drag to the Dow.

The S&P 500 SPX, -0.84% , which had earlier traded back and forth between slight gains and losses, finished down 4.02 points, or 0.2%, at 2,338.17, as energy shares closed down 1.4% with oil futures US:CLK7 settling down 3.8% at $50.44 a barrel. Health-care and industrial shares posted slight gains.

The S&P 500 had been clinging to modest gains earlier in the session, but soon swung into negative territory following the release of the Fed’s Beige Book, which noted that climbing wages aren’t really boosting inflation.

On the other hand, the Nasdaq Composite COMP, -1.26% rose 13.56 points, or 0.2%, to finish at 5,863.03, but well off its session highs, with big gains from shares of Intuitive Surgical Inc. and Lam Research Corp. LRCX, -0.23%

Seven of the S&P 500’s 11 primary sectors finished lower on the day. Financials, which had been the lead gainer earlier in the session, finished with a 0.3% loss. Morgan Stanley MS, -1.76% closed up 2% following stronger-than-expected earnings, which helped offset the weak read out of Goldman Sachs MS, -1.76% on Tuesday. Goldman shares shed 0.7%.

“We’re off to a good start with earnings, with three-quarters of companies beating expectations, though it’s a small data set so far,” said Kaufman. “Financials have been good, with Goldman the only land mine. But we need to continue seeing good earnings in order to spark demand for investors to keep coming in.”

Everything you need to know to invest in marijuana

Intuitive Surgical ISRG, -1.52% shares rallied 6.4% after the surgical robot maker topped Wall Street estimates for the quarter.

Facebook FB, -3.30% shares rose 0.9% a day after the social media company announced a new cameras-based augmented reality platform.

BlackRock Inc. BLK, -2.37% , the world’s largest money manager, posted increases in revenue, profit and assets under management. Shares fell 1.7%.

Yahoo Inc. US:YHOO was down 1.2% after the internet pioneer posted better-than-anticipated earnings. It also said the planned acquisition of the company by Verizon Communications Inc. VZ, +0.36% is expected to close in June.

Other markets: European stocks SXXP, -0.18% ended mostly higher, while most Asian markets closed lower. Gold futures US:GCM7 settled down 1% at $1,281.40 an ounce, as a key dollar index DXY, -0.07% gained.

Read:Why the snap U.K. election is a ‘game-changer’ for the pound

And see:What’s a ‘snap election’ and why does Theresa May want one?

Economic news: Boston Federal Reserve President Eric Rosengren said at Bard College’s Levy Economics Institute that he would like the Fed to start shrinking the balance sheet but at such a gradual rate that it doesn’t disrupt the central bank’s raising of interest rates.

See: Evidence the Fed can shrink its balance sheet without sparking a market tantrum

Check out:MarketWatch’s Economic Calendar

—Victor Reklaitis in London contributed to this report.