After 42 years of selling jewelry on Union Street, Terry Brumbaugh is calling it quits.

His shop in San Francisco’s Cow Hollow neighborhood is plastered with signs advertising discounts on every last pearl necklace and sapphire ring in the display cases. Brumbaugh is retiring, a decision he made in part because commercial rents are high and competition from online retailers is ramping up.

“It’s a good time to go out of business,” he said, with a smug grin.

Brumbaugh has plenty of company. Of the 197 storefronts that line Union Street between Van Ness Avenue and Steiner Street, 20 are vacant, according to a Chronicle count last week. The empty spaces are even more pronounced on Fillmore Street. Thirty-two stores dot the three-block stretch between Union and Greenwich streets, but seven of them are empty shells.

“I can never recall a time when there were more vacancies,” said Jeremy Scherer, who for 20 years has worked at Plumpjack Wine and Spirits in the bustling Fillmore-Greenwich Street intersection known as the Bermuda Triangle. Scherer is now president of Plumpjack.

Vacant storefronts are prevalent throughout the city, leading some to wonder whether the demise of small businesses is a warping side effect of economic prosperity. It’s particularly unsettling in Cow Hollow, a patchwork of old bars, hair salons and beachy cafes in one of the most affluent areas of San Francisco.

“Union Street was always the Madison Avenue of San Francisco — it’s artsy, it’s boutique-y, it had a reputation for taking care of locals and enticing a great deal of tourist traffic,” said David Blatteis, board chairman at Blatteis Realty, a brokerage firm that represents many of the street’s property owners.

Santiago Mejia/The Chronicle

Yet in recent decades, the composition of Union and Fillmore streets has changed dramatically. Mom-and-pop shops are shuttering. Chain stores and temporary pop-ups are moving in. Fitness centers and Pilates studios are replacing the antique shops and galleries.

“Landlords on this street just have to get real about what it takes to make a small business work,” said Brumbaugh, noting that commercial rents have ballooned as high as $30,000 a month for some spaces — an insurmountable figure for most small businesses. At the same time, brick-and-mortar stores are struggling to retain customers who now have the ease of shopping on eBay or Amazon.

“People like to go online and hit ‘pay,’” said Renee Cocke, manager of Krimsa, a high-end rug store that her sister opened on Union Street 15 years ago.

The question is how far the city should go to preserve its retail enclaves.

“It’s really tough to legislate a free market,” said longtime commercial broker Cameron Baird, noting that real estate prices and business trends fluctuate a lot faster than the law. “By the time you put legislation in place, it may already be outdated,” he said.

Blatteis acknowledged that most small businesses can’t afford the average rent on Union Street, which he said is about $8 a square foot, or $12,000 a month for a medium-size store. That’s a slight drop from the average of $10 per square foot two years ago, he said, but it comes with the added cost of bringing an old building up to code. Add to that the rising cost of labor and city fees, and some shop owners can’t make the cut.

He said the city’s efforts to preserve those small businesses are counterproductive. Over the years San Francisco has tightened its controls on formula retail — any business with 11 or more stores — and as a result, Blatteis said, buildings that could be filled by chain stores remain bare, hurting the neighborhoods.

“It often takes a year for a formula retail store to get through the Planning Department and the Planning Commission,” Blatteis said. “So you either have to have an owner that’s willing to give up rent, or a tenant who is willing to pay speculatively.”

Others say the city isn’t doing enough to prevent landlords from holding storefronts captive and setting the rent unreasonably high.

“The average profile of a landlord has changed,” Scherer said. “These are not people that care if a space is vacant for 10 years. They don’t need the money — they own 20 other properties.”

San Francisco already requires owners of empty storefronts to register with the city and pay an annual fee of $711 until their spaces are filled. A recent report from the city’s Budget and Legislative Analyst showed that many businesses don’t pay the fee.

Some Cow Hollow merchants are calling for aggressive measures. Ben Bleiman, owner of Tonic bar, said he would support a blight tax on landlords who turn down prospective tenants and leave storefronts empty by setting unreasonably high rents.

“There’s a few bad actors who own a lot of property, and they will hold out for years to get an exorbitant price per square foot,” Bleiman said. “It kills the neighborhood.”

Paul Kerr, owner of the small home decor shop Atys, suggested the city impose some form of commercial rent control in San Francisco’s retail hamlets.

“I want our neighborhood to thrive and I want retail to be wonderful and elegant again,” said Kerr, who opened Atys in 1997.

Mayor Mark Farrell, who represented Cow Hollow as a district supervisor, said he focused on public safety and aesthetic improvements, including new benches and trees, to make the commercial areas attractive.

“Look, our small businesses are the heart and soul of San Francisco, and, in my opinion, the city government cannot do enough to protect them,” Farrell said.

His successor, Supervisor Catherine Stefani, promised to prioritize vacant storefronts in her legislative agenda.

But she and other supervisors haven’t yet found the right solution. Some economists say the onus is on businesses themselves — not the city — to adapt to market conditions. That could mean taking to social media or launching an app-based delivery service. Or it could mean that only certain types of merchants survive.

“Bars, cafes, services, unique new products — those will remain competitive,” Baird, the broker, said. “The reality is San Francisco is expensive, and rents are just going to follow a northern curve.”

Rachel Swan is a San Francisco Chronicle staff writer. Email: rswan@sfchronicle.com Twitter: @rachelswan