Washington (CNN) The U.S. assesses that ISIS is ramping up the numbers of militants in Libya and that it has become harder for the group's fighters to enter Syria.

The U.S. estimate of ISIS militants in Libya has doubled as it has become harder for them to enter Syria, according to U.S. intelligence assessments.

There may now be up to 6,500 ISIS fighters in Libya, twice the number previously thought, according to several U.S. intelligence officials.

They attributed the increase to the U.S. analysis that ISIS is diverting more fighters to Libya from Syria -- and from Turkey when they cannot get into Syria.

"ISIS is investing heavily in Libya," one U.S. official said.

While the estimate of up to 6,500 is the most recent from military intelligence sources, others in the intelligence community don't agree and believe the number could be half that.

The trend began to emerge over the last six months of 2015, as indicated by the growth in numbers.

However, the official also strongly emphasized that the estimate is made up of "best guesses with low confidence," underscoring that the U.S. is not certain at this point how many fighters are there.

The build-up is one of the key reasons the Pentagon wants to increase aerial surveillance and reconnaissance over Libya.

At the same time, the U.S. believes the number of ISIS fighters in Syria and Iraq has declined slightly to somewhere between 19,000 and 25,000 -- compared to the previous estimate of 20,000 to 31,500 -- according to two U.S. officials familiar with the latest information.

Both emphasized that these are all estimates at best.

The U.S. believes the decline is due in part to the rate at which coalition airstrikes are killing ISIS fighters, combined with a decline in the numbers coming into both countries.

The Pentagon has recently noted the rise in reports of ISIS defections and the increase in localized conscription as the group tries to maintain its strength, as well.

But U.S. officials emphasize that ISIS is still capable, for now, of maintaining its strongholds in the Iraqi coty of Mosul and the Syrian city of Raqqa, despite the decline.

White House spokesman Josh Earnest cited the declining numbers of ISIS fighters in Iraq and Syria as evidence that the Obama administration's strategy is working in those countries. He noted that the intelligence assessments show 19,000 to 25,000 fighters in Iraq and Syria, down from 20,000 to 31,500 in a previous assessment.

"ISIL is having a more difficult time than they've had before in replenishing their ranks," he told reporters, using a different term for ISIS. "The reason the numbers inside Iraq and Syria are encouraging is that the numbers are moving in the right direction."

Earnest also denied that there's consideration of adding Libya as a "new front" in the war on ISIS, saying the administration has long been aware of the terror group's attempts to gain a foothold outside of Iraq and Syria.

"We've been mindful of these other places like Libya and Afghanistan, where ISIL may turn some of their attention," he said. "But we know they are focused on expanding ISIL's footprint in Iraq and Syria."

On a broader scale, the U.S. has recently estimated that more than 36,500 foreign fighters, at least 6,600 from the West, have traveled to Syria from at least 120 countries since ISIS emerged as a force of strength. That number is up from the previous 34,500 estimate. Not all of them have joined ISIS, however. Some, the U.S. believes, have joined al Qaeda-related groups such as al Nusra.

The U.S. also estimates 250 Americans have traveled or attempted to travel to Syria and Iraq to potentially fight or otherwise support the conflict.