Opinion & Features

How experts get it wrong

Weather forecasters Ian McCaskill and Michael Fish battling with the elements (PA)

It’s 7:30 on Sunday morning. My wife wakes me. Her body has been in intense agony for some hours. In 40 minutes we are at the nearest hospital. She undergoes a series of tests for her kidneys, without result. On the third day in the kidney ward she shows a young doctor, passing by, a sore place on her stomach. He says immediately “Shingles.” She is out of the hospital within the hour. Experts!

Yes, we all need experts; it is no more than common sense to get advice on our important needs from those who are appropriately qualified and have long experience in their field. Some of them are acknowledged as such and are paid rich fees by the government, businesses and litigants. They will be well worth the money if they solve a big problem or make accurate predictions about the future.

But hold hard. I have a recent study in front of me which claims that experts are highly susceptible to subjective influences – from individual values and mood, to whether they stand to gain or lose from a decision. It is published by the most reputable of journals, but then I remember the occasion when several articles by known experts previously published in scientific journals were resubmitted to the original journals under an invented name. Almost all were rejected as not meriting publication.

The expert we most frequently consult is a doctor. How do they perform? We know that they tend to diagnose according to their own specialty, and are unduly influenced by recent cases, but that may not be the end of it. The expert paediatrician at the trial of Sally Clark simplistically calculated that the random chance of cot death of both her children was one in 73 million. This was wildly out, and Clark’s conviction was reversed. She died soon after. Extreme, but not unusual: when 60 staff and students at Harvard Medical School were tested on a straightforward probability calculation, only one in five got it right. Doctors do not have a good record on conditional probabilities, which are at the heart of diagnosis.

How do judges fare? Surely we can rely on their objective wisdom? Well, perhaps not. Israeli judges who had to decide whether to grant prisoners parole were strongly influenced by whether or not they had had a recent refreshment break. Perhaps we should pit them against monkeys who choose our investments for us by throwing darts. They have a surprisingly good record compared with expensive investment experts.

Many of us need the help of financial advisers, particularly if we are making plans for retirement. Historical abuses have led to a web of regulations to ensure their objectivity. But even the many honest advisers are prone to subjective influences. We do not know how widely our adviser has examined other options. We do not know the extent of his knowledge. We do not know how his own temperament and values have influenced his recommendations.

It is a threatening scenario, and we must look to our defences. We do so knowing that directly or indirectly we are paying for advice, and we are entitled to interrogate it. Experts tend to use general phrases such as “It’s highly likely that” or “there’s little risk that”. So ask them to put a number to it. One expert thinks “highly likely” means 99 per cent probability, another means 80 per cent. Asking for numbers makes the expert think more precisely, and leads us to getting better information.

How much does the expert really know? Ask more questions. An expert tells you what he already knows, but searching questions, whether you need the answer or not, will quickly tell you how much he doesn’t know. Socrates taught us that. You may want to prepare a little list before you meet your expert. How often do you say afterwards, “Oh, I wish I’d asked him that”? Every year the NHS puts aside more than £20 billion, around a fifth of its budget, to meet negligence claims. Yet every operative
is employed as an expert at their level.

We need experts to give us reliable forecasts – whether the issue is the likelihood of a cancer recurring or the reduction of the UK deficit by 2020. There are as many opinions as there are experts.

Ironically, it is not the experts in the subject but forecasters using professional methods who tend to do best. They use a wide range of data, a good deal of mathematics, and continually revise their outcomes with every passing relevant factor. Perhaps the ultimate examples are the weather forecasters, who follow just this pattern. A three-day forecast is now as accurate as a one-day forecast in 1980. But, notwithstanding gallons of data and a supercomputer, even they can get it wrong.