WASHINGTON—President Donald Trump’s reservations about sending more troops to Afghanistan have triggered a new exploration of an option long considered unlikely: withdrawal.

Unable to agree on a plan to send up to 3,900 more American forces to help turn back Taliban advances in Afghanistan, the White House is taking a new look at what would happen if the U.S. decided to scale back its military presence instead, according to current and former Trump administration officials.

“It’s...