A 30-year-old hardhat briefly unhooked his safety harness to climb to another floor at a construction site on the Upper East Side on Friday — and plunged to his death, police sources and coworkers said.

The man fell about three stories down a planned elevator shaft at what will be a 19-story residential building on East 87th Street near Lexington Avenue around 9:20 a.m., according to the sources.

He was rushed to Lenox Hill Hospital, where he died of severe head trauma, sources said.

“His coworkers told me he was on the third floor, and he was wearing a harness but wasn’t hooked to a cable, and when he fell, he just went down,” said Fabian Para, 38, who works at a nearby parking garage. “He was wearing a helmet. His co-workers seemed really sad and worried about his situation. They only knew he was unconscious, but didn’t know if he was dead or alive.”

Another witness said, “People were yelling and there [was] a lot of commotion, ‘Oh my God, something happened, something terrible happened.’ When they put him the ambulance it was super quiet, nobody spoke, and everyone looked somber.”

A city official familiar with the investigation said the worker was removing concrete form work from the second floor and unhooked his harness to go to the third floor on a ladder before he plummeted.

“He was wearing a harness and was clipped in,” the official said. “He goes up to the third floor and unclips, and that’s where he falls.”

The dead man worked for Noble Construction Group, which logged three violations in March after a federal Occupational Safety and Health Administration inspection at one of its sites in Flushing, Queens, for not meeting the standards for fall protection, online records show. They were fined $3,500, according to the records.

The company didn’t return a telephone message seeking comment.

“It’s horrible. I was lucky enough to be bought into the union two years ago where I received real training, unlike these poor guys,” said Brian Houser, 37, a union elevator construction worker.

“These developers aren’t being held accountable and are saving a few bucks on some manpower and putting these people’s lives at risk,” he added.

Noble Construction Group, which is overseeing work at the site, and Highbury Concrete, Inc., a subcontractor that employed the victim, issued a joint statement saying, “All those affiliated with this project are deeply saddened by today’s event and extend our heartfelt condolences to the family, friends and co-workers.

“We are working closely with all parties to determine the cause of the accident and will provide any assistance requested by investigators.”