(CNN) Residents of a UK tower block have distanced themselves from a xenophobic poster in their apartment block ordering people in the building to speak English following Brexit .

Roger Blackwell, 68, told CNN "you don't want to give these people attention" after the "Happy Brexit Day " sign was seen on several fire doors in his building, Winchester Tower in Norwich, Friday. He said he thought someone "energized" by Brexit and "probably a fascist" had put it up.

Someone I follow just shared this photo of a sign he saw attached to a front door in a block in Norwich. Brexit has encouraged and emboldened these people. It will get worse. Do whatever you can to support immigrants who face this shit. We all need to stand strong against it. pic.twitter.com/9IbB3ztOkO

Police are investigating the poster, which told residents "we do not tolerate people speaking other languages than English in the flats."

The sign, which was spotted just hours before the UK left the European Union at 11 p.m. Friday, has been attacked as "racist" on social media.

The poster said "we finally have our great country back" and people wanting to speak other languages should return to their home countries and give their apartment back to the local authority "so they can let British people live here and we can return to what was normality before you infected this once great island."

Brexit supporters waved placards at Parliament Square in London as the UK officially left the European Union on Friday.

It added: "We do not tolerate people speaking other languages than English in the flats. We are now our own country again and the the Queens English [sic] is the spoken tongue here."

Residents received a letter Saturday from Lee Robson, head of neighborhood housing at Norwich City Council, which read: "Some of you will be aware that there was an incident yesterday where someone put an unaceptable [sic] and offensive poster in Winchester Tower."

Robson said the caretaker had swiftly removed the poster and remained on site to "monitor the situation and offer reassurance."

Brexiteers celebrate after the UK finally leaves the EU.

Blackwell, a retired photographer, told CNN he had not seen the poster before he received the council's letter. "I was surprised," he said. "The building is quiet, it's a quiet street."

He said the block was "pretty self-contained" and he did not often see people. "With Brexit, it's energized these people to come out of their homes.

"You don't want to give these people attention. In some ways, it'd be better buried."

He said people were "taking advantage" of the political situation. "I think it'll die down."

Blackwell said that after the UK voted to leave the EU in the 2016 referendum "they were energized then, and it did die down."

Music journalist Simon Price , who shared the photo of the poster taken by an anonymous Twitter account, wrote: "Brexit has encouraged and emboldened these people. It will get worse. Do whatever you can to support immigrants who face this s**t. We all need to stand strong against it."

Pro-EU, anti-Brexit protesters held a banner and smoke flares during a demonstration on Westminster Bridge in London Friday.

MP David Lammy said on Twitter that the poster's words were "chilling," adding: "Bigots and xenophobes have been emboldened.

"Our fundamental values are under attack."

Robson said the council had reported the matter to Norfolk Police, who are investigating.

"Norwich has a proud history of being a welcoming city, and, rest assured, we will not tolerate behaviour of this kind," said Robson, in a message also posted on Twitter by the local authority.

"We take this very seriously and urge residents to contact Norwich City Council or the police if they have any concerns."

A digital Brexit countdown clock projected on to the front of 10 Downing Street showed 00:00 as the UK left the EU at 11 p.m. Friday local time.

A Norfolk Police spokeswoman confirmed that the force was investigating the incident, which was "being dealt with as a racially aggravated public order incident."

She said police visited Winchester Tower on Saturday night after the matter was reported at 5.40 p.m. and all posters had been removed.

"Those posters kept by residents have since been seized for forensic enquiries and we will be working with the council to examine any available CCTV," she said.

"There is no place in society for hatred and intolerance. Nobody should have to face intimidation because of who they are and it is more important than ever that we stand together in the face of hostility. We remain committed to helping people feel safe and secure as they go about their lives."