I have a friend who has held several senior level executive positions at very large companies over the past 20 years. She is currently looking for a job and, times being what they are, she finds that unless her experience completely matches up against the new opportunity, she’s not considered as desirable a candidate as someone else whose experience is a more perfect fit. As a strategy and innovation consultant, I’m frequently asked to provide referrals for work that is the same as the work I’m proposing to do. In effect, there is an assumption that my previous success can be replicated if somehow the situation is close enough and if I do exactly the same thing again.
We treat experience as extremely valuable – as if it embodies deep knowledge and insight and is highly predictive of success in the future. We believe that because someone has managed a particular situation in the past successfully, he or she is a better bet to achieve success in the future when placed in a similar situation. But what if experience at best is irrelevant and at worst is misleading? What if how we understand correlation is deeply flawed? What if the truth wears off?
If we define experience as the proven ability to achieve a particular result under certain conditions, it is analogous to the scientific principle of replicability which is a foundation of the scientific method. In a scientific experiment, recreating the conditions under which a particular outcome has been observed is supposed to yield the predicted outcome. But, scientists are finding is that this is not the case. In many scientific experiments, the probability that predicted outcomes will occur actually decreases relative to the number of times the experiment is recreated increases. This is called the decline effect.
Many of the obvious culprits – regression to the mean, publication bias, selective reporting – do not provide a satisfactory explanation for the problem (which is a BIG one for science). After all, if a fact is not a fact – that is, something true that remains true – then what are these things that are being observed? What is the truth?
Regression to the mean is a statistical phenomenon that common wisdom captures in the phrase “the law of averages.” So, any experiment that shows a big effect at first might be expected to show a small effect another time and then bounce around until the effects cluster about an average. However, for many experiments with statistically solid data sets, the data doesn’t regress to a mean, the data thoroughly degrades, for which there is no satisfactory explanation.
Publication bias is the tendency of scientific journals (or almost any publication) to prefer to publish research that supports previously published research. After some period of time, the insight becomes orthodoxy, and then these same journals are more open to disconfirming research. While this explains some of the reasons why certain scientific facts become enshrined and then discredited, it doesn’t explain early positive results.
Selective reporting is the perceptual bias that humans bring to observation. An example of selective reporting can be found in an experiment where scientists look for physical symmetry, which is supposed to be a marker for a preferred mate, in animal subjects. If the scientist knows that the observed subject has successfully mated and is looking for symmetry that is expressed in something as minute as tail feathers or whiskers, there is a predisposition to see it. Experiments that test the efficacy of acupuncture provide another window into the problem of selective reporting. During a 30 year timeframe, experiments were conducted in Asia and the US/Europe. All of the experiments in Asia supported the effectiveness of acupuncture and only about 60% of the US/European experiments reached the same conclusion.
The most disturbing new information that I learned about the scientific method was not that scientists and scientific publications are biased or that judgment based on beliefs turns out to be more important than decisions based on a so-called solid set of facts, but that the notion of a threshold for statistical validity, that a phenomenon’s occurrence is not likely to be caused by chance – the 95% probability level – was originally set because it made “pencil and slide-rule calculations easier.” And much research chases after achieving results that meets this criterion so that they can be treated as facts.
If it turns out that facts are slippery and no more solid than beliefs, how do we understand the value of experience? Our contention that experience matters derives from the scientific model of replicability. But that model is not as firm a foundation as we may have hoped. If we can’t rely on the scientific construct of replicability – find someone whose experience matches up precisely against the current situation to better predict a successful outcome – what can we rely on? Turning to science for answers means a long wait, because science is only now beginning to grudgingly acknowledge that it has a problem. Even though it is much more challenging, I believe that we need to acknowledge that we are not sure what the value of experience actually is and that we might be as wrong about it as much scientific research is turning out to be.
Source: “The Truth Wears Off,” Jonah Lehrer, The New Yorker, December 13, 2010




