Some wealthy residents of Trump Tower are planning to move out of their apartments in the luxury skyscraper because they’re annoyed with the protesters and heightened security, a real estate broker has claimed.

In the wake of Donald Trump’s shock victory in the presidential election, crowds of protesters – as well as his supporters and security guards – have swarmed the building on Manhattan’s Fifth Avenue.

The chaos has brought traffic in the area to a standstill and left stores in the vicinity deserted. It means that the tower’s residents have had difficulty getting in and out of the building and have had to show ID before being allowed entry into their own homes, the New York Post reports.

As a result, some of the building’s affluent residents have decided they no longer want to live there.

Scroll down for video

New York City Police adjust barricades across the street from Trump Tower on Fifth Avenue on Friday

Some wealthy residents of Trump Tower are planning to move out of their apartments in the luxury skyscraper because they’re annoyed with the protesters outside. Above, protesters outside the Manhattan skyscraper on Thursday

‘These are wealthy people, they don’t need this, and they can’t take it any longer,’ a top real estate broker, who says he’s received calls from residents intending to move out, told the Post.

‘Some of them are already planning on moving out, and they’ll decide later whether or not they want to sell.’

He added: ‘They can’t get into their own homes without being stopped and frisked and having to show ID.’

The extreme security measures began going up around the landmark Fifth Avenue skyscraper on Election Day, when authorities brought in a fleet of heavy Sanitation Department trucks filled with sand to wall off the front of the glittering, 664-foot glass tower and protect it from a potential car bomb attack.

Those trucks were gone by Friday, replaced by concrete barriers stamped with the NYPD logo. But the stepped-up security — a team effort by the Secret Service, the New York Police Department and Trump's private security personnel — isn't going away.

The Secret Service and NYPD wouldn't detail what lies ahead. That will depend largely on how Trump decides to divide his time between Washington and New York and on an assessment of the vulnerabilities of Trump Tower, where the president-elect lives in a penthouse condo and his Trump Organization is headquartered.

'It's going to take a lot of planning, and it's going to take a lot of creativity,' said NYPD Deputy Commissioner Steve Davis.

The cost is uncertain, too, though the city might seek reimbursement from the Homeland Security Department for some of its expenses.

The heightened security has already become a concern for some high-end retailers in one of the city's busiest shopping districts, especially with the holiday season ramping up.

A worker washes the windows at Trump Tower in New York City on Friday

People stop to look at the front of Trump Tower on Fifth Avenue in New York following Trump's election victory

Police officers manning metal barricades asked visitors and shoppers where they were going before they could get onto the block Friday.

People who said they were headed for Trump Tower or the flagship Tiffany & Co. store next door were being let through, but the jewelry shop canceled the unveiling next week of its annual Christmas light display.

Tiffany spokesman Nathan Strauss said the store has been in frequent contact with the Secret Service and NYPD on the issue. 'It's obviously an important time of year for us,' he said.

Metal barricades also restricted access to a Gucci store on the ground floor of Trump Tower. Gucci declined to comment.

In the weeks leading up to the election, visitors at Trump Tower were subjected to bag checks and other screening but otherwise had free access to a five-story atrium that has shops and restaurants, including Trump Grill and a Starbucks.

Under a zoning deal Trump made with the city in the late 1970s, he is required to keep the atrium open to the public between 8am and 10pm.

By late last week, though, access was restricted and occasionally cut off completely.

Inside the atrium, law enforcement was everywhere, visitors were few and business was slow for the Starbucks and Trump gift shops and restaurants.

Police also closed off the block south of the high-rise and set up guard towers on either end. And they have gotten stricter about diverting delivery trucks away from tower.

This week, the Federal Aviation Administration barred aircraft from flying below 2,999 feet over Trump Tower, saying the airspace restrictions are needed until late January because of 'VIP movement'.

Trump blasted the protests against his election victory as ‘very unfair’ and blamed the media for deploying ‘professional’ protesters

Hours later, the president-elect backtracked to commend the protesters for being passionate

More permanent safeguards will take into account the effect on businesses and residents, Secret Service spokesman Martin Mulholland said.

'People think we shut stuff down because we can, but we pay close attention to what happens when we close a street or a sidewalk,' he said. 'We look for the best solution.'

The president-elect took to Twitter to complain about the protests that have erupted across the country after he won the White House.

Trump blasted the protests as ‘very unfair’ and blamed the media for inciting ‘professional’ protesters.

‘Just had a very open and successful presidential election,’ Trump wrote on Twitter on Thursday evening.

‘Now professional protesters, incited by the media, are protesting. Very unfair!’ he moaned.

Hours later, he backtracked, commending protesters for being passionate.

Current residents of Trump Tower include art dealer Helly Nahmad (left) and Trump’s former campaign manager Paul Manafort (right)

'Love the fact that the small groups of protesters last night have passion for our great country. We will all come together and be proud!' he wrote.

Current residents of Trump Tower include Trump’s former campaign manager Paul Manafort and art dealer Helly Nahmad, who was jailed for his part in a $100 million international gambling ring that attracted celebrities including Leonardo DiCaprio.

Trump himself lives in the skyscraper’s gold-encrusted penthouse triplex – which is estimated to be worth $90million.