For the best part of two decades, it was almost impossible to walk down any street without seeing a Gap logo hoodie or baseball cap. The logo was a powerful symbol of the strength of the US fashion store, which dominated the high street with its simple, clean-cut, college style.

But the brand has fallen out of favour, and with sales dropping at an alarming rate, Gap has admitted that the decline had reached crisis point and announced the immediate dismissal of its chief designer, Patrick Robinson.

In a statement released on Thursday, Pam Wallack, the head of Gap's creative division, said: "I have made the decision to make a change within our Gap adult design team."

Wallack added that Robinson was "leaving the company, effective immediately" and that the Gap leadership were "taking the necessary steps to compete and win around the world". The brusque wording, unusual when dealing with a figure of Robinson's seniority, may be an attempt by Gap to signal to the industry that the turnaround will be swift. First quarter earnings revealed on Thursday were below market estimates.

Gap seemed in an unassailable position 15 years ago. It was the Starbucks of the fashion world, a label that symbolised an aspirational but achievable modern lifestyle. When the dress-down Friday phenomenon swept offices from Manhattan to Manchester, Gap chinos were the mainstay of the new uniform.

Gap was a label that cut across geography, class, gender and age. Bill Clinton wore Gap jeans. Cindy Crawford wore Gap denim skirts. Students wore Gap sweatshirts and Silicon Valley executives wore Gap button-down Oxfords. The brightly-coloured polo shirts were sported by the children of fashion-conscious parents, and by retired people on the golf course.

But early in the 21st century, Gap began to lose touch with its core American customers and sales began to slide. Robinson was appointed, to much fanfare, in 2007, to reverse the trend. He was very much a name from fashion's top table: he had designed for Giorgio Armani, Paco Rabanne, Perry Ellis and Anne Klein. During his four-year tenure at Armani, he was credited with turning the ailing Armani's Collezioni line into profit.

A one-time nominee for a Council of Fashion Designers of America award, the highest honour in US fashion, Robinson is married to Virginia Smith, one of the most senior editors on American Vogue.

His arrival was much celebrated within the industry. He was described as a "megabrand messiah", and compared by one powerful fashion editor to Barack Obama. He staged his first collection for Gap on a catwalk at New York fashion week, and Anna Wintour attended.

Robinson declared it his mission to take Gap back to its roots in easy, classic pieces and give the customer clothes they could wear every day of the week. "I grew up with Gap," he told Interview magazine shortly after his arrival. "I am the focus group. You go to Gap to find khakis. It's not like it has to be hard."

As it turned out, it was hard. In its heartland of affordable preppy style, Gap was increasingly shaded by J Crew, a label whose more colourful and glamorous take on the American aesthetic was boosted by the support of Michelle Obama. When the Obama daughters wore jewel-coloured J Crew coats to the January 2009 inauguration, Gap's neutrals looked ever more tired by comparison. Detractors queried Robinson's decision to pull back from the brightly coloured clothes that they felt had been key in making an emotional connection with shoppers, particularly at Christmas.

Gap's invisibility, in comparison to the Clinton 90s, was stark. Meanwhile, the punchy marketing of Abercrombie & Fitch was wooing the teen market, while the standard of fast fashion available on the high street was being pushed ever higher by the fierce global rivalry between H&M, Topshop and Zara.

Robinson has continued to enjoy critical success, particularly in international markets. His range of Perfect Black Trousers, launched in 2010, quickly became a front row uniform for fashion editors attending catwalk shows. He masterminded sellout collections in collaboration with Stella McCartney, and with the cult shoe designer Pierre Hardy. But sales have failed to pick up, and this season he has attracted criticism for sticking to a muted, faded-pastel palette which is losing out in comparison to the juicy brights and stark monochromes at Zara, which is having a hit spring run.

A source close to Robinson said that his departure was not a shock, but will be a blow as he is a popular figure within the company. "Patrick has been in an almost impossible position, in a company trying to please so many people. He has fantastic ideas, but some of them work better in Europe than in the USA.

"Great though it is to have a figurehead, it is hard for one person to encapsulate such a big brand. It may be that a team, rather than another 'name' designer, is the best next step for Gap."

In an interview in March Seth Farbman, recently appointed the company's chief marketing officer, made reference to the problems facing Gap. "It's hard to be in the middle," he said, "so we have our work cut out for us – to design what a middle brand really means."

For Gap, the search for the designer who can identify this starts now.