Dozens of police and security guards have rescued Julia Gillard and Tony Abbott from a group of angry protesters who had surrounded a Canberra restaurant.

The Prime Minister stumbled as she was rushed to an awaiting vehicle and was helped up by security officials who were confronted by the protesters.

It appears the mob was incensed by remarks made by Mr Abbott earlier in the day in which he said he thought it was probably time to reconsider the relevance of the Aboriginal Tent Embassy.

Around 200 protesters gathered outside the restaurant near Old Parliament House where Ms Gillard was handing out medals to emergency services representatives.

Mr Abbott was also inside the building.

At least 50 police, including the riot squad, were called to the scene shortly after 2.30pm (AEDT).

The protesters, involved in a nearby event to mark the 40th anniversary of the embassy, banged on the three glass sides of the restaurant chanting "shame" and "racist".

The two leaders, protected by police and security officers, escaped out a side door after about 20 minutes.

Protesters chased their car down the road, banging on its roof and bonnet.

There had been false reports that the Prime Minister had been tackled. Ms Gillard's office confirmed she lost her footing, and a shoe, as she was dragged by security.

ACT Police Sergeant Chris Meagher says no-one was injured in the fracas and he has praised the police response.

"Immediate response - we had about 35 to 45 police respond also with the assistance of uniform police from Parliament House as well," he said.

Some of the protesters scuffled with police after the Prime Minister left but there have been no arrests.

At least 50 police were called to help escort the leaders from the restaurant. ( AAP: Lukas Coch )

'Invasion day'

Hundreds of people marched in the capital on the anniversary of the tent embassy, calling for land rights and Aboriginal sovereignty on what many call "invasion day".

Embassy founder Michael Anderson addressed a rally at the site.

"To hell with the Government and the courts in this country. You haven't got a high hope to take us on," he said.

"We will force these issues. Too many of our families have suffered for some bastard to get in the road."

Earlier, Mr Abbott told reporters in Sydney that he thinks it is probably time to reconsider the relevance of the tent embassy.

He says he can understand why the embassy was established but he thinks times have changed for the better since then.

"I think the Indigenous people of Australia can be very proud of the respect in which they are held with every Australian," he said.

"Yes, I think a lot has changed and I think it's probably time to move on from that."

'Madness'

Mr Anderson said the comments were disrespectful.

"He said the Aboriginal embassy had to go; we heard it on a radio broadcast," he said.

"We thought no way, so we circled around the building."

He said the protesters wanted the leaders to clarify their position and whether Mr Abbott was serious about removing the embassy.

"You've got 1,000 people here peacefully protesting, and to make a statement about tearing down the embassy - it's just madness on the part of Tony Abbott.

"What he said amounts to inciting racial riots."

Tent embassy protesters face off against police. ( AAP: Lukas Coch )

ABC/AAP