Accompanied by some of Puerto Rico’s most famous performers, thousands of people marched to the governor’s residence in San Juan on Wednesday chanting demands for the embattled governor, Ricardo Rosselló, to resign after the leak of online chats that show him making misogynistic slurs and mocking his constituents.

The crowd ranged from teenagers to retirees, with some waving the island’s flag printed in black and gray rather than red, white and blue to symbolize their discontent with a government they call corrupt and unresponsive to its people. Musicians Ricky Martin, Residente and Bad Bunny marched and addressed the crowd.

Police erected concrete barricades and shop owners covered store windows with metal sheeting or plywood as if a hurricane were coming. The multicolored umbrellas that form a photogenic awning over the street in front of the governor’s mansion were taken down.

The turnout filled several city blocks in colonial Old San Juan but appeared to fall short of the many tens of thousands that some Rosselló opponents had predicted. Many older protesters went home before nightfall as chanting young people filled Old San Juan’s Totem Plaza and the first few blocks leading up to the 16th century fortress where the governor resides.

Karla Villalon has three elementary-age children and an 81-year-old grandmother. Her kids have been uprooted twice in two years when first one school, then another, was closed by budget cuts under Rosselló. Her grandmother, a retired teacher, is anguished over the possibility of losing her pension in future rounds of cutbacks.

Villalon was outraged when Rosselló’s former education secretary was arrested and accused of steering millions in improper contracts to politically connected contractors. Then hundreds of pages of online chats between Rosselló and members of his administration leaked, revealing the men mocking women, the disabled and victims of Hurricane Maria. Villalon has had enough.

“It’s the final straw,” the homemaker said before the march. “My kids’ classrooms have mold in them ... There’s just so much outrage that’s been building over time.”

Demonstrators chant and wave Puerto Rican flags in San Juan. Photograph: Gabriella N Baez/Reuters

The Rosselló administration has remained under siege since the weekend after leaked text messages between the governor and a number of his inner circle revealed a slew of misogynist and homophobic comments shared between the group.

A number of senior members of the administration have already resigned in the wake of the scandal, but on Monday Rosselló refused to tender his resignation, claiming that while the messages were inappropriate they were not illegal.

“I’m not proud of what I did,” Rosselló told reporters on Tuesday. “Those were merely comments – but they were hurtful comments. So, I apologize for what I’ve done but again, I need to move forward and continue on the work we’re doing for Puerto Rico.”

The affair only adds to sustained criticism of Rosselló’s leadership as sweeping austerity and privatization measures imposed after Hurricane Maria decimated the island almost two years ago drew public backlash.

Puerto Rico, an unincorporated US territory, is in the midst of a multibillion-dollar debt crisis now managed by an unelected oversight board appointed in Washington that oversees much of the island’s economic affairs.

Shortly before the text message scandal, referred to as “RickyLeaks”, a number of administration officials and contractors, including the former education secretary Julia Keleher, were arrested by the FBI over allegations of corruption and misappropriation of $15.5m in federal funds apportioned to Puerto Rico after Hurricane Maria.

Workers cover shop windows with wood in preparation for protests against Governor Ricardo Rosselló near La Fortaleza in San Juan, Puerto Rico, on Wednesday. Photograph: Carlos Giusti/AP

Key figures in the movement to oust Rosselló remained hopeful that the protests on Wednesday would remain peaceful.

A number of high-profile Puerto Ricans, including the actor and playwright Lin-Manuel Miranda, singer Ricky Martin and trap artist Benito Martínez Ocasio, known by his stage name Bad Bunny, have also lent vocal support to the protests. Martin and Ocasio are expected to appear at the protests.

San Juan’s firebrand mayor, Carmen Yulín Cruz, an outspoken critic of Rosselló, told the Guardian by text message the demonstrations would mark “a historic day in Puerto Rico”.

Cruz, who announced she would challenge Rosselló in elections next year, became the face of resistance to the Trump administration’s faltering efforts to assist during the immediate aftermath of Hurricane Maria.

Both Mayor Cruz and Ricky Martin also appeared as targets of abuse in the leaked text messages.

According to the messages, Rosselló referred to Cruz as “off her meds” while other administration officials mocked Martin’s sexuality.