The US is to open direct talks with Taliban leaders within days, it was revealed on Tuesday, after Washington agreed to drop a series of preconditions that have previously held back negotiations over the future of Afghanistan.

In a major milestone in the 12-year-old war, political representatives of the Taliban will shortly meet Afghan and US officials in Doha, the capital of Qatar, to discuss an agenda for what US officials called "peace and reconciliation" before further talks take place with Afghan government representatives soon after.

The move came on the day that Nato forces handed official control of nationwide security to Afghan troops. Less than 12 hours later the US confirmed that four US personnel died at Bagram air base near Kabul, in what was thought to be a mortar attack. The Taliban claimed responsibility.

Earlier the Taliban, in a statement announcing their plans for peace talks and an office in Qatar, said they would not allow anyone to threaten or harm other countries from Afghan soil – a move senior US administration officials described as an important first step to the Taliban severing ties with al-Qaida.

The US has agreed that a formal rejection of al-Qaida by the Taliban leadership would now be a "negotiating aim" rather than a precondition for talks. It will also seek a commitment from the Taliban to end its insurgency in Afghanistan and recognise women's rights in the country.

"This is an important first step but it will be a long road," said one senior US official. "We have long said this conflict won't be won on the battlefield, which is why we support the opening of this [Doha] office."

White House officials say they believe the Taliban delegation at the talks represents the movement's leadership, and includes more radical groups such as the Haqqani network. Officials said the US would have a direct role in the talks starting starting this week in Doha, but the substantive negotiations over the future of Afghanistan would then be led by the Afghan government.

Speaking later, Ben Rhodes, deputy national security adviser, said: "The United States will be supporting a process that is fundamentally Afghan-led … We can play a role in talking to the Taliban as well in supporting that peace process – and because we have issues of our own to bring up with them."

A Taliban spokesman said the group was opening the Doha office to "reach understanding and initiate talks with countries of the world for the purpose of improving relations with them", and to support a peaceful, political solution to end the "occupation of Afghanistan".

The proposal for a Doha office has been on the table since 2011, and several senior Taliban figures have been living in Qatar for many months now, but the group had not publicly embraced plans for peace talks.

In Kabul, Afghan president Hamid Karzai said he hoped the opening of the Taliban office would bring the start of talks between the High Peace Council he set up to lead government negotiation efforts, and the insurgents.

However the Afghan leader, who has long been lukewarm about efforts to set up a Taliban base in Qatar, also called for any negotiations to move back to Afghanistan as soon as possible. "We hope that our brothers the Taliban also understand that the process will move to our country soon," he told a news conference in Kabul, although US officials stressed that moving talks to Afghanistan would take time.

Karzai also announced that Nato forces had handed official control of nationwide security to Afghan troops on Tuesday. Foreign soldiers will still be fighting on the ground and supporting Afghans with air power, medical evacuation and other key capacities until the end of next year.

Barack Obama is understood to have informed G8 leaders of the breakthrough at a dinner at the Northern Ireland summit on Monday night.

The deal on talks with the Taliban was partly brokered by Pakistan and the emir of Qatar after "months of diplomatic spadework" also involving Germany, Norway and the UK. In 2011, Hillary Clinton suggested that Taliban leaders would have to renounce violence for a peace process to work.

"Over the past two years, we have laid out our unambiguous red lines for reconciliation with the insurgents: they must renounce violence; they must abandon their alliance with al-Qaida; and they must abide by the constitution of Afghanistan," she said. "Those are necessary outcomes of any negotiation. This is the price for reaching a political resolution and bringing an end to the military actions that are targeting their leadership and decimating their ranks."

But on Tuesday, that position appeared to have soften somewhat. "We don't expect them to break ties with al-Qaida [immediately]," said one of the US officials speaking on an off-the-record conference call. "That is an outcome of the process." He said the expected Taliban statement opposing the use of Afghan soil for foreign attacks was "a first step in distancing them from international terrorism".

The Taliban also appeared to have softened on their long-term demand that foreign troops leave before talks can start. Karzai, despite his misgivings about overseas talks and initial opposition to the Qatar office visited the Gulf state twice this year, apparently paving the way for Tuesday's breakthrough.

Afghan president Hamid Karzai, who has always said he would prefer talks to take place in Afghanistan, was initially lukewarm about the Qatar plans, but has visited the state twice this year, apparently paving the way for today's breakthrough.

Additional reporting by Mokhtar Amiri in Kabul and Spencer Ackerman in Washington

