During the depths of the great depression, the British economist John Maynard Keynes laid out a vision for his own profession: “If economists could manage to get themselves thought of as humble, competent people, on a level with dentists, that would be splendid.” Today, many think that Keynes would be disappointed if he could see what his fellow economists have become.

According to their critics, economists are less like dentists and more like mad scientists. Economics, its detractors claims, is a failing field in desperate need of fundamental reform. Unsurprisingly, economists disagree. They think such criticisms are part of the lurch “away from evidence-based policy” and come “dangerously close to surrender to special interest groups, gut feelings and superstitions”. So what does the evidence tell us? Are they dentists, mad scientists or something else entirely?

Economics started out as a relatively humble occupation, but by the 20th century it had become increasing influential in shaping national policy. In the second half of the 20th century, economics gradually became a transnational occupation with a fairly coherent set of ideas about how the world worked. Influential members of the profession would often be trained in the doctoral programmes of elite US economics departments and then be dispatched to work on policy problems around the world. When working in their home countries, they would often look back to the elite economics departments for support and guidance. This myopic focus on elite US economics departments has created a steep professional hierarchy, where the sign of being a good economist is having your work published in five scientific journals that are mainly controlled by US-based or US-trained economists.

Economists are a fairly ideological, diverse group of people who are striving to address the grand challenges we face

As well as being very hierarchical, economics is insular. Economists tend to mainly refer to the work of other economists. One study found that economists cite the work of other disciplines 19% of the time. In comparison, sociologists cite other disciplines 52% of the time and political scientists cite external disciplines 59% of time. When asked whether interdisciplinary knowledge is better than knowledge derived from a single discipline, only about 42% of economists agreed, whereas, 72% of sociologists, 78% of psychologists and 68% of historians thought interdisciplinary knowledge was better.

Within the profession, economists also have fairly narrow horizons. A recent experiment found economists were more likely to accept a message if it was labelled as being from someone working in mainstream economics rather than a non-mainstream economist. Even in online discussions, economists tend to prefer to stick to their own. A recent analysis comparing how economists and other scientists used Twitter found that economists were much more likely to focus their responses towards other members of their profession. This insularity means economists often hold views that are very different from those of the wider populace.

Economics is also a discipline dominated by men. Women are significantly under-represented at all levels of the profession – from undergraduate students all the way up to professorship level. The same is true for minority-ethnic groups. While women have made up increasing proportions of other disciplines during the last 20 years, gender imbalances have not budged in economics. Some corners of disciplines such as finance and macro-economics are particularly dominated by men. Even informal online interaction tends to marginalise female economists. A recent study found that online rumours about female economists tended to focus on their personal lives, while online rumours about male economists were focused on their professional lives.

Economists may not be the most intellectually diverse, outwardly focused or demographically diverse group. There is, however, more ideological diversity than we might suspect. A recent analysis of the political ideology of US economists found that about 60% have a broadly liberal or left orientation, while 40% swing more to the right. What’s more, their personal ideological preferences are reflected in their work. Left-leaning economists tended to work on topics that were typically more progressive (such as studying labour markets), while right-leaning economists worked in areas that were more conservative (such as finance). Their policy recommendations also tend to follow their ideology. The slight majority of left-leaning economists were more likely to support higher government spending and higher taxation, while the slight minority of right-leaning economists tended to urge government fiscal restraint and lower taxes.

As well as being more ideologically diverse than we might think, economists are also more focused on collecting real-world data than their critics often assume. In 1983, theoretical papers that relied on abstract mathematical models made up about 57% of those published in the top five economics journals. By 2011, that number had dropped to 19%. An analysis of macro-economics found that abstract mathematical models dominated the discipline between the 1960s and the early 1990s. Since then, economists have favoured a more experimental approach. With the increasing availability of big data sets, economists are more interested in empirical questions than ever before.

Economists also tackle a wider range of topics than many assume. For instance, the Nobel prizewinner Robert Shiller has recently been exploring the role that the narratives we tell each other play in shaping economic crises. George Akerlof (another Nobel prizewinner) and US economist Rachel Kranton have been looking at the role identity plays in shaping economic decisions such as where we work and what we buy. The economist Deirdre McCloskey has highlighted the role that moral virtues play in economic growth.

Economists have also made important contributions to tackling some of the biggest social issues we face, including how we raise our children, how to tackle the climate crisis, how to make the international financial system less crisis-prone, how to deal with the rise of “deaths of despair” in the United States, and how to deal with the impact of poor mental health on people’s lives.

Economists may not become the humble and practical dentists than Keynes had hoped they might. But they have not become the kind of mad scientists many of their critics think they are. Economists are a fairly ideological, diverse group of people who are focused on real-world problems and who strive to address the grand challenges we face. But if they want to play a valuable role in shaping the future, they need to become less elitist, less insular and less masculine. That means being less obsessed with a handful of top journals, being more open to ideas from outside their discipline, and being more welcoming to women and other groups who are under-represented in the profession.

• André Spicer is professor of organisational behaviour at the Cass Business School at City, University of London