Let’s check the science: How can I decide which experts to trust on scientific issues?

Most of us can’t hope to fully understand every scientific issue that impinges on our lives. Even scientists at the pinnacle of their own fields are novices outside those fields. Nearly always, we must turn to the opinions of experts in the field which concerns us. But how do we decide who to trust, who is the real expert? The world is full of pretenders who are eager to persuade us that they have discovered the indisputable truth on issues from cosmic consciousness to the secrets of perfect health. How can you judge who is really an expert?

Quite often, when an idea that could be helpful presents itself, we do not appreciate it, for it is so inconspicuous. The expert has, perhaps, no more ideas than the inexperienced, but appreciates more what he has and uses it better.

George Pólya, Hungarian mathematician (1957)

What is an expert?

I’ll regard a person as an expert in a field if she:

has superior factual knowledge in the field and exceptional understanding of how to apply the facts to produce concrete results.

has a better ability than her peers to differentiate between similar, but not identical cases.

shows consistency in judgements that is better than her peers.

is identified by her peers in the field as an expert.

(These criteria are my own and I don’t claim any expertise in the field of experts. They are a kind of condensation of the characteristics of experts used by Ericsson et al and Shanteau et al in references 1 and 2 below.)

Becoming an expert is not easy. Ericsson et al1 explain:

The journey to truly superior performance is neither for the faint of heart nor for the impatient. The development of genuine expertise requires struggle, sacrifice, and honest, often painful self-assessment. There are no shortcuts. It will take you at least a decade to achieve expertise, and you will need to invest that time wisely, by engaging in “deliberate” practice— practice that focuses on tasks beyond your current level of competence and comfort.

Deciding which experts to trust

Alvin I. Goldman (see reference 3 below) has proposed five criteria for deciding which experts to trust. His criteria are meant to be used to select which of two competing experts we should prefer. I’ve used his criteria as a basis, but modified and broadened them in an attempt to produce a sort of checklist for deciding whether a particular expert is on the level. (Goldman would probably not approve.)

For each criterion, I’ve given some descriptors that you might be able to use as indicators. Don’t expect any particular expert to tick all the boxes, but obviously, the more the better. I realize that, for many of the criteria, you won’t be able to say one way or the other. Hopefully, though, you’ll find enough to either reject or accept your “expert” candidate without too much trouble.

A real expert’s arguments make sense and they are consistent The putative expert … has arguments to support his position and they make sense

shows obvious familiarity with opposing positions and addresses them confidently

has arguments that do not contradict accepted facts, concepts and models

has arguments that do not contradict each other

has arguments that show no glaring logical fallacies, cognitive biases, conspiracy theories or deceptive tactics

shows an obvious grasp of factual detail within his field and an ability to discriminate between situations that seem, to a novice, to be similar

recognises that non-experts depend on his interpretations and takes account of this in his explanations

does not claim expertise on topics extending beyond his field A real expert’s arguments are regarded as valid by most other experts in the field The putative expert … has a position that coincides with the consensus position in the field, if there is one

has a position that is supported by significant numbers of other experts in the field , if there is no consensus position

, if there is no consensus position has a position that is supported by multiple lines of evidence from the field and related fields A real expert’s expertise is vouched for by third parties that have authority in the field The putative expert … is a member of professional bodies within the field which testify to her expertise.

which testify to her expertise. has relevant qualifications, work experience, research publications, awards and certifications within the field

has professional registration within the field

has a long period of intense, problem-focused experience in the field

has a current or previous professional appointment within the field A real expert has no conflict of interest or prejudices which may influence her judgements The putative expert … is not associated with groups that would benefit from bias in her arguments

does not receive funding from any person or group with a vested interest in one side of the issue at hand

is not allowing an ideological, political or religious position to bias her argument

is not part of a group that has been set up to promote a predetermined outcome from the issue in question

is not artificially exaggerating the significance of her arguments (e.g. by appealing to the media rather than peer-reviewed publications) A real expert has a successful track record of judgements in his field. The putative expert … has a good record of giving correct answers to questions within his field

has a good record of assessing and solving problems within his field

has a good record of predicting events within his field

has a good record of exposing sources of misinformation and deception in the field

Red flags when seeking experts

There are some signals that tell you immediately you are not dealing with an expert. You should be very alert to these signals. Among them are:

Fudging the boundaries of expertise – the putative expert tries to imply that because he’s an expert in one field, his opinion in a different field should be given credence.

Appeals to testimonials – the putative expert rolls out glowing recommendations from clients or grateful followers.

The Galileo Gambit – the putative expert insists that he is an unacknowledged genius, a maverick who is shunned by mainstream science because of his unconventional ideas.

Conspiracy theories – the putative expert claims that some form of “scientific establishment” is suppressing his ideas.

The Gish Gallop – the putative expert slickly rattles off a long list of assertions without providing evidence or allowing questions.

Overconfidence – it’s been found that most of us judge the quality of experts by the amount of confidence they project. Unfortunately, the well-known Dunning-Kruger effect means that those who are least competent in a field are precisely the ones who overate their own performance. Those who are most competent are more likely to underrate their performance. True experts will admit that they are never completely certain of their stance and can specify the uncertainties in their explanations.

For an extended list of similar signals, see my Science Red Flags page.

Further reading

Why we need to listen to the real experts in science at The Conversation.

The science of why you really should listen to science and experts by Chris Mooney at The Washington Post.

Whose word should you respect in any debate on science? by Peter Ellerton at The Conversation.

George Pólya’s quote is from How to Solve it: A New Aspect of Mathematical Method (1957), 223.

The Impostor cartoon is from xkcd.

This page is based on ideas developed in these articles:

1. K. Anders Ericsson et al (2007), The Making of an Expert, Harvard Business Review, July – August 2007

2. James Shanteau et al (2003), How Can You Tell if Someone is an Expert? Empirical Assessment of Expertise, in Emerging Perspectives on Judgment and Decision Research, Cambridge, U.K.: Cambridge University Press.

3. Alvin I. Goldman (2001), Experts: Which Ones Should You Trust? Philosophy and Phenomenological Research, Vol. 63, No. 1., pp. 85-110.

