Washington (CNN) Department of Homeland Security Secretary John Kelly said Tuesday that "thousands" of fighters from Syria could possibly make it to the US, though he added it wouldn't be easy, as he defended the administration's controversial travel ban executive order in front of Congress.

Under questioning from Democratic Rhode Island Rep. Jim Langevin as to whether any new information prompted the ban, Kelly said the measure was based on concern about countries that couldn't provide adequate vetting information on potential immigrants and noted that "thousands" of Syrian fighters could possibly make it into the US. He later qualified that they would not be able to "easily" enter the US

The individuals "have the kinds of papers we have confidence could get them passed into Europe and by extension into the United States," Kelly told the House Homeland Security Committee. "We know there are thousands of fighters coming out of the caliphate that could easily -- not easily -- could bring them to Western Europe and then the United States."

Kelly did not give more details, suggesting that he had presented similar information to the committee previously in classified session and could not provide more at the public committee hearing.

Speaking with reporters after the hearing, Kelly clarified that he was referring in general to the thousands of individuals fighting for ISIS in Syria and parts of Iraq, saying that as terrorists lose territory in that fight, they could try to make their way to Europe or the US.

"So there are thousands, we know, of fighters that when they make the decision, will move out of places like Iraq, the part of Iraq that's controlled by the caliphate and parts of Syria, and they'll be available to create, you know, their brand of mayhem in Europe and potentially the United States," Kelly said. "Papers are not hard to get in that part of the world, particularly since in many cases ... ISIS and the affiliates overran government offices that make papers, so it's not a small threat."

Kelly was testifying before Congress for the first time since being sworn in, speaking at a hearing focusing on border security.

He faced numerous questions about President Donald Trump's controversial order -- which temporarily suspended nationals from seven high-risk countries from entering the US and put a pause on the refugee program. It has since been put on hold pending court challenges and has seen numerous walk-backs in implementation guidance, including carifying that green card holders are exempted.

Kelly repeatedly referred to it as a "travel pause," taking painstaking care to emphasize that it was not a "Muslim ban" and was nothing more than an effort to get up to speed on what kind of vetting can be done on people coming into the US.

"The ban was based on countries we don't have any real confidence in right now that they can help us vet people that come into the United States. Countries that are clearly in disarray," Kelly told the committee.

He said despite repeated questions about why, with countries like Saudi Arabia, Lebanon and Egypt having sent far more documented terrorists to the US than the seven countries affected by the order, more countries were not being considered for adding to the list.

"We are right now contemplating no other countries, because even though some of these other countries are questionable, in terms of their internal organization, police, that kind of thing, we're satisfied that most other countries have enough that they can provide the information we're looking for," Kelly said.

Kelly also took responsibility on Tuesday for the quick rollout of the executive order, saying he wished he had delayed enacting it.

Chairman Mike McCaul, a Texas Republican, told Kelly he was unhappy with some aspects of the implementation of the order, including the lack of consultation with members of Congress and reports that it was not thoroughly put through the agencies involved in putting it in place.

Kelly said he knew the order would be coming out the Friday it was signed and released late in the evening, and that he and his team had a chance to make changes the week before.

"In retrospect, I should have -- this is all on me by the way -- I should have delayed it just a bit so I could have talked to members of Congress," Kelly said.

The secretary has defended the ban despite reports that his awareness of the content was limited and that career staff at the department and others did not have opportunities to fully review the order.

"The desire was to get it out, the thinking was to get it out to quick so potentially the people were coming to harm us could not take advantage," Kelly said, echoing the Trump administration's claims that if the order was announced in advance, "bad" people could have rushed into the country.

A federal district court in Washington has blocked the order from going into effect nationwide while it works its way through the courts, and several other district courts blocked aspects of it in their jurisdiction.

The wall

The hearing also repeatedly returned to Trump's pledge to build a wall along the border with Mexico. Trump released an executive order in his first week in office ordering its construction and increasing the number of border patrol and immigration agents.

Kelly left vague what the wall would actually look like, though, and made clear the order would take years to executive.

"We're not going to be able to build a wall everywhere all at once," he said.

He didn't give a clear timeline or cost estimates.

"I'd like to see we'd be well underway in two years," Kelly said of the wall. "This is going to take some time, but there's places I think we can right away get at this problem."

Kelly said he was talking to agents in each sector of the border so he could get sense of what each area needs. For example, he said, some agents emphasized the importance of a barrier they can see through.

"The people locally, though, and that's really more important to me, they can tell you exactly where they want 10, 12, 15 miles tomorrow, and then 50 miles tomorrow and 100 miles the next day," Kelly said.