China, India and the US would be able to set the terms of the UK's future trade agreements if Britain broke away from the EU, Lord Peter Mandelson has warned. The former Labour business secretary, speaking at the Institute of Directors (IoD) annual convention in London on 6 October, argued that the UK should stay within the 28-nation bloc in a bid to use the market's negotiating powers.

"It is not true, as they say, that we can leave the EU one moment and re-enter the single market the next. If we want access to the EU's market, so as to maintain our exports, we have to pay for it and we have to obey their rules without influencing them – there's a straight trade off in other words," Mandelson said.

"As for the rest of the world, we can in theory negotiate new trade agreements if Britain is no longer to be covered by the EU. But we are a country, let's remember, of only 64 million people. Without Europe's 500-million-strong market behind us, if we went into those negotiations, we would be settling on China's terms, we would be settling on India's terms, we would be settling on America's terms – that's the reality of the choice we face."

Mandelson, who is also a former European trade commissioner, concluded that a Brexit would make the UK more "notionally independent" but "less influential". The alternative from staying inside the EU is "very far from golden", the Labour grandee added.

Mandelson debated with Eurosceptic former Conservative chancellor Lord Nigel Lawson, who accused his fellow peer of having an "astonishing" lack of self-confidence in the UK. The exchange comes ahead of the UK's referendum on the issue, which David Cameron has promised to hold before the end of 2017.

The latest opinion polls are neck-and-neck over the historic vote. The latest YouGov study, of more than 2,700 people between 17 and 22 September, put the Leave camp three points ahead of Remain (41% versus 38%, respectively). Elsewhere, the IoD's general director has warned the prime minister of an "accidental Brexit" and urged Cameron to hold the referendum "sooner rather than later".

"If, after deliberation, the public votes to leave Europe, our members will have to accept it, and the period of uncertainty for business that will follow. They will be less philosophical if carelessness and domestic discontent led to an 'accidental Brexit'," Simon Walker said.

"As with all votes, the IoD will not be telling anybody what they should think as we approach the referendum. But I have rather less compunction in suggesting to everybody that they should be focusing squarely on the real issue – would we be better or worse off leaving the EU."