Labour has responded to a backlash among voters of Indian heritage in the UK by shifting its stance on the Kashmir dispute and insisting it is a bilateral matter between India and Pakistan in which Labour will not interfere.

The clarification has been made in a letter sent by the Labour party chairman, Ian Lavery, in which he admits an emergency motion on Kashmir passed by Labour at its annual conference had caused offence to some British Indians and India itself.

The motion said there was a humanitarian crisis in the disputed territory and that the people of Kashmir should be given the right of self-determination. It also called for international monitors to be allowed into the region.

The motion has led some Indian groups in the UK to call on their community to vote Conservative.

More than 100 Indian groups wrote to Jeremy Corbyn in protest, and more recently the criticism of Labour has spread to social media. There have also been complaints that only one candidate of Indian heritage has been selected in a safe Labour seat, and none in a target seat.

In his letter Lavery promises “the Labour party will not take a pro-Indian or pro-Pakistan stance on Kashmir”.

The latest controversy follows India’s decision in August to revoke the special status of Indian-administered Jammu and Kashmir, bringing it under direct rule from Delhi.

India and Pakistan both claim Kashmir, but control only parts of it.

Since August political activity in Indian-administered Kashmir has been repressed, journalists have been refused free access, and NGOs have reported human rights abuses, claims denied by India.

The former UK national security adviser Mark Lyall Grant has predicted rising extremism in the UK unless the issue is settled.

Lyall Grant told a meeting at the security thinktank Chatham House, organised by strategic advisory firm CTD Advisors, that India’s decision to revoke Kashmir’s special status was likely to lead to greater extremism in the region. He said the temptation for the Pakistan intelligence services would be once again to support cross-border militancy, even if the Pakistan civilian authorities opposed such a move.

He said greater extremism in Kashmir would have a direct impact on the UK, adding that 60-70% of British Pakistanis had origins in the Mirpur district in Kashmir. “Therefore there is a risk of radicalisation in this country of British Kashmiris. We all know that diasporas tend to be more radical than communities left behind and I do not see why this should be any different.”

In his letter Lavery wrote: “We are adamant that the deeply felt and genuinely held differences on the issue of Kashmir must not be allowed to divide communities against each other here in the UK.

“Kashmir is a bilateral matter for India and Pakistan to resolve together by means of a peaceful solution, which protects the human rights of the Kashmiri people and respects their right to have a say in their own future.”

Attention will now turn to how Labour words its election manifesto on this issue.

In a sign of the controversy the issue creates inside Labour, the former foreign secretary Jack Straw called on Pakistan to withdraw its claim to Kashmir. Speaking at the same Chatham House event as Lyall Grant, Straw said he was “extremely sympathetic” to the plight of Muslim people in Jammu and Kashmir, and said the actions of India’s prime minister, Narendra Modi, in revoking the state’s special status were “outrageous, preposterous, a complete breach of human rights and without any strategy attached”.

But he continued: “The thing I am clearest about is the way in which the whole of Pakistan’s politics and economics has become distorted in this vain search or attempt to redraw the boundaries of Kashmir and to take Jammu Kashmir into the Pakistan Republic.

“That is completely unobtainable, impossible, but that goal has led to Pakistan sponsoring terrorism across the line of control – without any question, we all know that – and it has also led to a bloated defence spending and disproportionate power to the defence forces of Pakistan.”

He said Pakistan needed to reflect on why its economy had not grown at the same rate as India’s, adding that Pakistan’s state sponsoring of terrorism reduced the country’s diplomatic traction.

He added: “If the ISI [Pakistan’s intelligence services] think the answer to the situation is a bit more terrorism they will be gravely mistaken.”

Under the new arrangement, Jammu and Kashmir is designated one territory, and Ladakh, which borders China, is separate. Almost 98% of the state’s population will be in the territory of Jammu and Kashmir, comprising two regions – the Muslim-majority Kashmir valley, which has about 8 million people, and the Hindu-majority Jammu, which has about 6 million.