Labour must wholeheartedly embrace patriotism to win back support, the new leader Keir Starmer has said as he faced ex-Labour voters as part of a virtual tour of the country.

He said the Labour movement and patriotism were “two sides of the same coin” and the party should not shy away from displaying national pride.

During a video call with people in Bury, there was criticism from one ex-voter who said he had been made to feel his support for the monarchy, Brexit and even waving the union jack was tantamount to racist behaviour.

Starmer said: “I’m really proud of my country and I wouldn’t be leader of the Labour party if I wasn’t patriotic,” adding that he has felt this even more strongly during the Covid-19 response.

“What I desperately want for our country is for our country to get better. In the Labour party we should be proud of being patriotic. We’re all working, knocking on doors in the rain or shine, to try to put in place a team that can go into government to improve the country we live in because we love the country we live in.

“I don’t think we should shy away from that. That is a really good thing to be proud of, and [to] want your country to be the best it possibly can be,” he said.

The former leader Jeremy Corbyn was regularly criticised for a perceived lack of patriotism owing to some of the international causes he supported and past comments that he would prefer an elected head of state.

Both Bury North, which had been Labour, and Bury South, which had been Labour and then independent, switched to Conservative at the last general election as part of the sweeping Labour losses in the north of England.

Around 120 people from Bury tuned in to the video call. A call with people in Tees Valley – another home of former “red wall” seats – was planned for Thursday night. Next week Starmer will take calls from the public in Scotland.

The “Call Keir” initiative has been put into action after Starmer’s plan to spend his first months in charge visiting seats Labour has lost had to be called off because of coronavirus.

In a question on how he would tackle “anti-patriotism”, Starmer was also asked to condemn the former shadow home secretary Diane Abbott for her comments on Brexit supporters, and Clive Lewis MP for saying the Brexit campaign had “racism at its heart”.

Abbott was reported as saying in 2016, at a fringe meeting at the Labour party conference, that people who voted for Brexit wanted “less foreign-looking people on their streets” and she raised repeatedly the rise in hate crime after the referendum.

Starmer said he thought the hate crime statistics were of justifiable concern. “The vast majority of people who voted leave – nothing to do with racism at all and I don’t think Brexit was really about that,” he said.

“I was concerned about the spike in hate crime that happened at the same time, and that we do need to take very seriously. Although it may be a very, very small element, and I don’t think the EU is really to do with that at all. But there are racist comments and incidents that are said and done and there has been hate crime out there and we need to be very firm about tackling that.”