Lunchtime customers at Whole Foods in Manhattan’s Union Square had little trouble expressing the shortcomings that have led the once high-flying, organic-focused retailer to become linked with a takeover.

“I love the sushi, but I wouldn’t shop here except maybe for a special ingredient,” said Argentinian software designer Benjamin Vinas. “People say Whole Foods is for pretentious people, and I can see why. It’s too expensive. I don’t have the budget.”

Vinas was not the only customer to express a similar point of view. Others said that for their groceries they went several blocks north and west to lower-cost rival Trader Joe’s, where products may not be so exquisitely selected but are, in general, more uniformly discounted.

Maria Johnson, a postgraduate student, said Whole Foods’ pricing, with some items marked competitively and other expensive, was inconvenient.

“I only buy body lotion and lunch here. And maybe spices,” Johnson said. “There are so many different price points you feel like you are missing out on the more fun, expensive things – and when you are shopping for the cheaper, more affordable things, you’re reminded of the things you can’t afford.”

But the views of Manhattan’s grocery shoppers point to only part of the problem for Whole Foods, sometimes called Whole Paycheck, which has been facing a backlash from consumers.

Founded in Austin, Texas, in 1980, Whole Foods Market, to give it its official name, has about 462 locations and a market value of almost $12bn. The chain helped make health food and organic food mainstream, and in its boom years shook up the food retail industry. Whole Foods had grand plans for a UK expansion too, opening its first outpost in Kensington in 2004 with plans for 40 more. But Whole Foods has stalled: like much of the retail sector, it faces economic headwinds including razor-thin margins, competition from other retailers offering organic food, and increasingly price-conscious consumers.

In February, the company announced it would close nine stores: in Chicago, New Mexico, Utah, Arizona, and Georgia, and two each in Colorado and California. The closures, which followed six straight quarters of sales declines, represented the first downsizing since 2008. Founder and chief executive John Mackey explained that the business had changed because “the more conventional, mainstream supermarkets have upped their game. The world is very different today than it was five years ago.”

One rival chain, Sprouts Farmers Market, was found to be on average 19% cheaper than Whole Foods. Other rivals, including Kroger, picked up Whole Foods customers. Last month, Barclays advised that Whole Foods had experienced a “staggering” decline in foot traffic that it estimated at 3%, or roughly 14 million customers.

Whole Foods has long been the butt of jokes for its prices – although it disputes it is more expensive than it rivals – and its bougie products. Comedian John Oliver is particularly fond of its asparagus water.

The difficult transition from expensive behemoth to a more nimble grocery store is reflected in its share price, which has dropped by almost half since October 2013. Investors are increasingly petitioning the company to move faster with reforms.

An acquisition would underscore the consolidation of the US food retail industry as the sector prepares to compete against online retailers including Amazon, stores like Walmart and discount chains such as Aldi.

Earlier this week, the Financial Times reported that Cerberus Capital Management, the New York private equity firm that owns Albertsons and Safeway, had initiated talks with bankers about making a bid.

Other potential suitors include Jana Partners, an activist investor group that has amassed a 9% stake. In a regulatory filing, Jana said it wanted Whole Foods to “improve its technology and operations to better compete with larger rivals, shake up its board, and explore how much potential bidders might be willing to pay”.