The Seven Most Common Thinking Errors of Highly Amusing Quacks and Pseudoscientists (Part 2):
Thinking Error 3: Post Hoc Correction of Hypotheses:
Before we get into this “thinking error”, I need to make abundantly clear what it is we’re talking about. One of the most fundamental characteristics of real scientists is that they are always revising, modifying and – when necessary – discarding their theories and hypotheses in light of new data. To many people outside of the scientific disciplines, this looks like indecision or just plain waffling – a “bad” thing if you’re a politician trying to stake out a position for legislation or electioneering.
However, a failure to “change with the wind” – as one politician put it – is a certain sign that a researcher has abandoned science and turned their hypothesis into a religion. Real science often means having to abandon a cherished hypothesis, one that you have nurtured and raised from a mere pup of an idea, like it was yesterday’s newspaper – if the data warrant it. To fail in this most solemn duty is to turn down the path to the “dark side” – to pseudoscience and quackery.
At issue here is the basic purpose of a hypothesis or theory. Although many people outside of the sciences (and, regrettably, some inside as well) equate “theory” (and “hypothesis”, if they are acquainted with that term) with “idea”, the fact is that it is much more than that. A hypothesis or theory (more about the difference later) is a model of how the universe works. Now, it may be a model of a very small part of the universe (such as the replication of a virus) or it may be a model of the entire universe (e.g. theBig Bang).
No matter what the scale, the purpose of a hypothesis or theory is to give us a deeper understanding of our world by showing us the workings of the parts we can’t see. Or, sometimes, by explaining why the parts we can see do the things they do. Either way, the model – the proposed explanation – has to conform to the behavior of the real world if it is to survive. And that – in a nutshell – is the difference between a hypothesis and a theory. A hypothesis is a model that has not yet been extensively tested to see if it predicts what the real world does – a theory has already survived a number of tests successfully.
Having survived testing does not necessarily mean that the hypothesis (or theory, if it has gotten to that point) has survived unchanged. In the process of testing even the most inspired hypothesis, discrepancies are found between what the hypothesis predicts will happen and what actually does happen. Sometimes these discrepancies can be explained by flaws in the measurements or data collection, but any consistent difference between what the hypothesis predicts and what the data show must be seen as evidence that the hypothesis – the proposed model of how the world works – needs to be modified or abandoned.
The problem is knowing when a hypothesis or theory should be revised and when it should be abandoned – something that is often difficult to see until enough contradictory data has amassed. But, like that favorite old pair of jeans that you keep patching and patching, eventually a hypothesis becomes more patches than whole cloth and needs to be revamped or rejected. On the other hand, many of today’s solid, tried and tested theories went through a period where they needed some “tweaking” (or even major overhauls) in order to function.
The thinking error of post hoc correction occurs when someone tries too hard to keep a failing hypothesis “in the game”, crossing from legitimate modification of the hypothesis to frantic attempts to keep it alive at all costs. This can be – and probably usually is – done without any intent to deceive. And it can be done by people who have an impeccable record of excellence in science – as the mutual fund people always say, “past performance is no guarantee of future yields”.
The hallmark of post hoc correction is the modification of a hypothesis in response to contradictory data in a way that is:
[a] Not supported by any existing data
[b] Not tested or not testable
Let me make this clearer by two examples – one of a legitimate modification of a hypothesis and one of an illegitimate post hoc correction:
 Lost a star but gained a planet.
In the early 1800’s, the French astronomer Alexis Bouvard undertook to publish a corrected table of the orbit of Uranus due to observed discrepancies from the orbital tables published by Jean Baptiste Delambre in 1792. He was unable to get all of the observations to fit into the predicted orbit (predicted by the theory of gravity) and so published his new tables in 1821 with the comment that he was unable to determine if the discrepancy was due to errors in the earlier observations or a “foreign and unperceived cause”.
By 1841, however, it was clear that even Bouvard’s calculations were failing to account for the actual orbit of Uranus. At this point, there were two theories in play, one of which was in need of modification – the theory of gravity or the theory that the Solar System had only seven planets. Although the majority of astronomers at the time were “betting” on the existence of an eighth planet (which we know as Neptune), there were others (i.e. George Airy, the Astronomer Royal) who felt that the theory of gravity was in need of an overhaul.
The British astronomer John Adams and the French astronomer Urbain Le Verrier began a search for a new planet, using the mathematical basis of the theory of gravity to predict where this new planet might be, based on the irregularities in the orbit of Uranus. In 1845, they both (more or less simultaneously) found the planet – despite resistance, reluctance and a good deal of old-fashioned mule-headedness on the part of their more senior colleagues.
To diagram the process:
a. Hypothesis (Theory, actually): Gravitational attraction is proportional to the product of the masses involved and varies with the inverse square of the distance between them.
b. Problem: The orbit of Uranus is not following the course predicted by the Theory of Gravitation.
c. Possible Explanations: The Theory of Gravitation does not apply at large distances from the Sun OR there is another planet beyond Uranus.
d. Resolution: After calculating where a planet would have to be to cause the observed perturbations of Uranus’ orbit, astronomers found a planet – Neptune – in the expected location. The Theory of Gravitation had survived another test!
The “take-home points” of this example are looking for supporting data (the planet Neptune) before deciding which theory to revise and the fact that they did not automatically assume that one theory was “privileged” and therefore not subject to scrutiny.
 When low means high
A few years ago, an unlikely group of researchers – a PhD academic chemist, an MD oncologist and an MBA – embarked on a project to prove that mercury caused autism. Since tests on hair, blood and urine had previously failed to show any significant difference in mercury content between autistic children and “normal” controls, they tested hair specimens that had been collected at the child’s first haircut – the so-called “first baby haircut” – and retained as a keepsake. This, they felt, would be the definitive proof that autistic children had been exposed to a significantly higher mercury levels as infants (as stated by one of the researchers, Dr. Holmes, during the 2000 DAN! Conference).
Unfortunately, the mercury levels in the “first baby haircut” samples from autistic children were significantly lower than those from the “normal” controls. This might have proved to be a difficulty, had not the researchers applied a post hoc correction to their hypothesis. They concluded that, based on their data, autistic children are unable to excrete mercury as effectively as their “normal” peers. They made this conclusion despite numerous studies, many dating back a few decades, that showed mercury was passively taken up by hair rather than excreted.
In addition, a later national study showed that the hair mercury levels that they measured in the autistic children were very close to the national average for children of the age when these hair samples were taken (remember, the hair samples were taken when the children were one to two years old – the analysis was performed many years later). In addition, this same national study – which was not studying autism – showed that the hair mercury levels of the “normal” controls was greater than the national average by over fifteen times!
To diagram the process:
a. Hypothesis: Mercury causes autism (subhypothesis: previous studies have failed to demonstrate high mercury levels in autistic children because the mercury “washes out” by the time of diagnosis some years later).
b. Problem: Hair mercury levels in hair taken at the “first baby haircut” of autistic children are lower than those of “normal” controls.
c. Possible Explanations: Mercury is not related to autism (apparently not considered by the authors) OR mercury protects children from autism (supported by the data, but nonsensical) OR children with autism cannot excrete mercury as well as “normal” controls (consistent with their data but not supported by it – also, not consistent with over forty years of data on how mercury and hair interact) OR the laboratory assays were in error.
d. Resolution: Rather than opt for an explanation that is consistent with known physiology, the authors chose an “explanation” that supported their hypothesis that mercury causes autism at the expense of being almost certainly wrong. In short, either dozens of researchers’ work over the past forty years (and more) is wrong or the authors of this “study” are wrong in their conclusion.
The “take home points” of this example are that a hypothesis (e.g. “autistic children cannot excrete mercury as well as non-autistic children, leading to low hair mercury levels”) which contradicts previous well-established hypotheses or theories (e.g. “mercury is not excreted in the hair – the hair mercury concentration merely reflects the blood concentration at the time the hair was formed”) needs to have data supporting it, not merely the assertions of its authors. Additionally, most of the time, many conclusions can be drawn from the data of a single study – the authors of this study were blinded to those alternative explanations by their single-minded desire to “prove” their hypothesis.
In short, post hoc corrections of a hypothesis are those which “save” the hypothesis at the expense of making it unsupported by data. You can properly “save” a hypothesis that fails to correctly predict reality by either changing the hypothesis so that it predicts reality better (as was done when Neptune was added as the 8th planet). Or you can try to change reality itself by asserting that your hypothesis only predicts reality in your laboratory or in the absence of “negative thought energy”. Or you can take the route of adding another unsupported hypothesis to the mix in order to make the whole thing “work”, as the authors of the “study” in example 2 did. The latter two processes are post hoc corrections and only add more unsupported assertions to a hypothesis that is – by definition – already in trouble.
Coming Up: Conspiracy! (or, Et tu, Brute!)