A Perfect Example of how NOT to do a Study - Part Two
Welcome back to the second part of our dissection of "Reduced levels of mercury in first baby haircuts of autistic children" (Holmes AS, Blaxill MF, Haley BE. Int J Toxicol. 2003 Jul-Aug;22(4):277-85)! This is the part where things may get a little messy, so be sure to have your gloves and goggles on before we start. I'll give you a minute to get ready.
The dissection of the Results section will require us to dissect a few parts of the Methods section, as the two are interconnected. I will try to make it clear when I switch areas.
Results:
Despite a "...first-order hypothesis of heavy metal toxicity in autism.", the hair mercury concentrations were as follows (parts per million [ppm] +/- standard deviation):
We will get to conclusions later, but I know what I thought when I read those numbers.
The numbers were different enough that, despite having standard deviations almost as large as the values, there was a very strong statistical significance (p<0.000004).From the Methods section, we know that they were using a two-tailed test and these standard deviations make a one-tailed test more appropriate (for a good introduction, see here), but let's just pass over that for now.
The authors further break down the autistic group into three sub-groups - "mild", "moderate" and "severe" - and show that the hair mercury levels are lowest in the "severe" group (0.21 ppm +/- 0.18), higher in the "moderate" group (0.46 ppm +/- 0.19) and highest in the "mild" group (0.71 ppm +/- 0.30).
A number of potential mercury "exposures" are listed, with data given only for injections of Rho D immunoglubulin, some of which contains thimerosal, a mercury-containing preservative. Of the rest of the "exposure data" collected, the authors only give the correlation between the number of maternal "amalgams" [amalgam dental restorations, also known as "fillings"] and hair mercury level. This is presented in a graph which is not particularly helpful. The other "exposure data" (vaccinations and fish intake) are not displayed.
Discussion:
The most important part of this section is the formula, derived by "multiple regression", that correlates three of the four exposure "variables" to hair mercury level - but only in the control group. You'll note that exposure to Rho D immunoglobulin, even though its association with the autistic subjects was highly significant and much discussed, is not part of the formula. The resulting equation is:
Hair mercury level = 5.60 + 0.04( amalgam volume ) + 1.15( fish consumption ) + 0.03( vaccinations )
Since the authors make extensive use of the results of this equation, it is worthwhile to take some time to examine it closely.
From the Methods section, "amalgam volume" was calculated from the number of amalgam dental restorations (no data about actual size was collected) and was simply the square of the number of amalgam restorations. In this way, two small restorations on the sides of the teeth were calculated to release four times the amount of mercury that one large restoration on the chewing surface - a conclusion not supported by either common sense or experimental data.
The variable "fish consumption" is presumably an integer derived from the "none, little, moderate, heavy" scale and the variable "vaccinations" is presumably the number of vaccinations received prior to the haircut.
I must admit to a certain degree of confusion about the formula, since it would appear that the lowest hair mercury level would be 5.60 ppm, assuming that all exposures were zero. I could find no subsequent correction, so this is the only formula we have.
I note that the constant in the equation (5.60) is in parentheses in the article. Mr. Blaxill has (in various Internet postings) subsequently claimed to have done the actual writing of the article and since his education is in business, perhaps he was using the accounting practice of placing negative numbers in parentheses. This would allow for hair mercury levels below 5.60 ppm, but would mean that zero exposure would lead to a negative hair mercury level - a clearly nonsensical result. Either way, the formula is suspect - at best!
Conclusions:
This is the smallest section, but it has the biggest bang! After leading the reader down the path of mercury toxicity and the consequences of poor mercury excretion (and without showing any studies or supporting data that autistic children excrete mercury poorly), the authors veer sharply away from what their data can support:
Earlier, I asked you to remember that the mercury exposures were calculated and not measured and I believe I have shown that the calculations are also fairly questionable. Now, in the conclusion, the authors seem to have forgotten that they never measured mercury exposure. How amazing! Even more amazingly, the reviewers and editors failed to catch this BIG mistake!
Even more amazing is that the authors, the reviewers and the editors failed to notice two other points:
[1] There are no data or citations provided to support the implied assertion that mercury is actively excreted in the hair.
[2] There are no data citations provided to supports the assertion that mercury elimination in the hair is a significant means of clearing mercury in humans - especially infant humans, who have little hair relative to their body weight.
The reason for [1] is that there are abundant data and studies showing that mercury enters the hair - in all studied species - passively from the blood, attracted by the sulfur-containing amino acids in hair. The only way to "impair excretion" is to cut off the blood supply to the hair - which causes it to fall out.
The reason for [2] is that humans, and especially human infants, have a very small mass of hair relative to their body weight - a much lower relative amount than many of the other species studied (e.g. rodents and primates). For this reason alone, excretion in the hair cannot be a major route of mercury excretion in humans.
Wrap-up:
The authors imply, in the introduction, that one of the reasons they began this study was that some autistic children had presented to Dr. Holmes' practice with low hair mercury levels. Finding that all 94 autistic subjects in this study had low hair mercury levels, they then proceded to a conclusion that was not supported by their data.
Given that the only thing they measured in this study was hair mercury level, the only thing that their data support is that autistic children have low hair mercury levels. Since mercury moves passively from the blood to the hair, the explanations for this finding are:
[1] These autistic children had low mercury exposure, despite what historical data collected a median of 5 1/2 years after the fact may suggest.
[2] The low hair mercury level is the result of a laboratory or specimen collection error that was sytematically applied to only the autistic subjects. There is not enough information in the paper to address how this might have happened, but it is my personal favorite.
[3] Something about autism changes the amino acid composition of hair in such a way that it has less affinity for mercury.
You will note that none of these hypotheses requires the reversal of decades of previous study results or the discovery of currently unknown biochemical pathways. They also don't support the hypothesis that mercury causes autism (and they don't refute it), which is probably why they were rejected by the authors of this silly study.
Class dismissed!
Prometheus.
The dissection of the Results section will require us to dissect a few parts of the Methods section, as the two are interconnected. I will try to make it clear when I switch areas.
Results:
Despite a "...first-order hypothesis of heavy metal toxicity in autism.", the hair mercury concentrations were as follows (parts per million [ppm] +/- standard deviation):
Autistic children: 0.47 ppm (+/- 0.28)
Controls: 3.63 ppm (+/- 3.56)
We will get to conclusions later, but I know what I thought when I read those numbers.
The numbers were different enough that, despite having standard deviations almost as large as the values, there was a very strong statistical significance (p<0.000004).From the Methods section, we know that they were using a two-tailed test and these standard deviations make a one-tailed test more appropriate (for a good introduction, see here), but let's just pass over that for now.
The authors further break down the autistic group into three sub-groups - "mild", "moderate" and "severe" - and show that the hair mercury levels are lowest in the "severe" group (0.21 ppm +/- 0.18), higher in the "moderate" group (0.46 ppm +/- 0.19) and highest in the "mild" group (0.71 ppm +/- 0.30).
A number of potential mercury "exposures" are listed, with data given only for injections of Rho D immunoglubulin, some of which contains thimerosal, a mercury-containing preservative. Of the rest of the "exposure data" collected, the authors only give the correlation between the number of maternal "amalgams" [amalgam dental restorations, also known as "fillings"] and hair mercury level. This is presented in a graph which is not particularly helpful. The other "exposure data" (vaccinations and fish intake) are not displayed.
Discussion:
The most important part of this section is the formula, derived by "multiple regression", that correlates three of the four exposure "variables" to hair mercury level - but only in the control group. You'll note that exposure to Rho D immunoglobulin, even though its association with the autistic subjects was highly significant and much discussed, is not part of the formula. The resulting equation is:
Hair mercury level = 5.60 + 0.04( amalgam volume ) + 1.15( fish consumption ) + 0.03( vaccinations )
Since the authors make extensive use of the results of this equation, it is worthwhile to take some time to examine it closely.
From the Methods section, "amalgam volume" was calculated from the number of amalgam dental restorations (no data about actual size was collected) and was simply the square of the number of amalgam restorations. In this way, two small restorations on the sides of the teeth were calculated to release four times the amount of mercury that one large restoration on the chewing surface - a conclusion not supported by either common sense or experimental data.
The variable "fish consumption" is presumably an integer derived from the "none, little, moderate, heavy" scale and the variable "vaccinations" is presumably the number of vaccinations received prior to the haircut.
I must admit to a certain degree of confusion about the formula, since it would appear that the lowest hair mercury level would be 5.60 ppm, assuming that all exposures were zero. I could find no subsequent correction, so this is the only formula we have.
I note that the constant in the equation (5.60) is in parentheses in the article. Mr. Blaxill has (in various Internet postings) subsequently claimed to have done the actual writing of the article and since his education is in business, perhaps he was using the accounting practice of placing negative numbers in parentheses. This would allow for hair mercury levels below 5.60 ppm, but would mean that zero exposure would lead to a negative hair mercury level - a clearly nonsensical result. Either way, the formula is suspect - at best!
Conclusions:
This is the smallest section, but it has the biggest bang! After leading the reader down the path of mercury toxicity and the consequences of poor mercury excretion (and without showing any studies or supporting data that autistic children excrete mercury poorly), the authors veer sharply away from what their data can support:
"Despite hair levels suggesting low exposure, these infants had measured exposures at least equal to a control population, suggesting that control infants were able to eliminate mercury more effectively." [my emphasis]
Earlier, I asked you to remember that the mercury exposures were calculated and not measured and I believe I have shown that the calculations are also fairly questionable. Now, in the conclusion, the authors seem to have forgotten that they never measured mercury exposure. How amazing! Even more amazingly, the reviewers and editors failed to catch this BIG mistake!
Even more amazing is that the authors, the reviewers and the editors failed to notice two other points:
[1] There are no data or citations provided to support the implied assertion that mercury is actively excreted in the hair.
[2] There are no data citations provided to supports the assertion that mercury elimination in the hair is a significant means of clearing mercury in humans - especially infant humans, who have little hair relative to their body weight.
The reason for [1] is that there are abundant data and studies showing that mercury enters the hair - in all studied species - passively from the blood, attracted by the sulfur-containing amino acids in hair. The only way to "impair excretion" is to cut off the blood supply to the hair - which causes it to fall out.
The reason for [2] is that humans, and especially human infants, have a very small mass of hair relative to their body weight - a much lower relative amount than many of the other species studied (e.g. rodents and primates). For this reason alone, excretion in the hair cannot be a major route of mercury excretion in humans.
Wrap-up:
The authors imply, in the introduction, that one of the reasons they began this study was that some autistic children had presented to Dr. Holmes' practice with low hair mercury levels. Finding that all 94 autistic subjects in this study had low hair mercury levels, they then proceded to a conclusion that was not supported by their data.
Given that the only thing they measured in this study was hair mercury level, the only thing that their data support is that autistic children have low hair mercury levels. Since mercury moves passively from the blood to the hair, the explanations for this finding are:
[1] These autistic children had low mercury exposure, despite what historical data collected a median of 5 1/2 years after the fact may suggest.
[2] The low hair mercury level is the result of a laboratory or specimen collection error that was sytematically applied to only the autistic subjects. There is not enough information in the paper to address how this might have happened, but it is my personal favorite.
[3] Something about autism changes the amino acid composition of hair in such a way that it has less affinity for mercury.
You will note that none of these hypotheses requires the reversal of decades of previous study results or the discovery of currently unknown biochemical pathways. They also don't support the hypothesis that mercury causes autism (and they don't refute it), which is probably why they were rejected by the authors of this silly study.
Class dismissed!
Prometheus.
posted by Prometheus at 5:00 PM
7 Comments:
Nice work, Prometheus.
I know *I* want my medical problems to be explained to me by a guy with an MBA!
--HG
I have wracked my brain trying to find a connection that would explain how an otherwise respectable journal like the International Journal of Toxicology would let this article slide by without proper review and editing. This is not the first time that I have read an article and thought "Who let THIS piece of trash get published?", but it a particularly egregious example.
If anyone out there has any ideas how Holmes, Blaxill and Haley managed to circumvent the normal review and editing process, I'd love to know.
Prometheus
I find it interesting that the 4,500+ petitioners in the US Court of Federal Claims Autism Omnibus Proceeding are not willing to rely on the Geier study or the Holmes, Blaxill and Haley study, either. These claimants are required to prove that their injuries were caused by either thimerosal in vaccines or the MMR vaccine, and they must submit expert reports to do so. Last month the Petitioners' Steering Committee requested an open extension of time, to mid-2006 or beyond, to submit their expert reports. They said:
"[I]t is critical in this proceeding - involving very serious injuries to thousands of children - that legal decisions are based on the best possible science. It is the PSC's position that the scientific and medical evidence needed to resolve issues of general causation in the Omnibus Proceeding has not yet matured to the point that it can support a sound adjudication of the causation issues presented by these claimants."
In re: Claims for Vaccine Injuries Resulting in Autism Spectrum Disorder, or a Similar Neurodevelopmental Disorder, Various Petitioners v. Secretary of Health and Human Services,
Petitioners' Filing Re: Submission of Expert Reports in Support of General Causation, filed June 14, 2005, at page 2.
http://www.uscfc.uscourts.gov/OSM/Autism/6%2014%2005%20pet%20filing%20re%20timing%20of%20expert%20reports.pdf
Their protestations to the contrary, these parents appear to be aware that the studies you have commented upon would not hold up in a court of law.
Anne
If this is a journal that accepts comments on previously published papers, you may want to consider refining your analysis and submitting a comment on this paper.
TonyL
Thanks for that! I'd come across a summary of this study and it seemed fishy, but I didn't have the information or the knowledge to dissect ti. Keep up the good work.
I'm not surprised it got published. It's new data on a topical problem, and many cranks don't even bother to take data.
I'm curious why you say that a one-sided test would be more appropriate given the large CV. My gut reaction would have been that, given the large SD difference between the groups, I'm not certain that the difference in means between the groups is meaningful, regardless of whether I can establish its statistical significance. I.e., I've got to fix my methodological problems with the data before I can figure out what it's telling me.
Anonymous,
You may be right about the reason for publication. I still can't understand why the editor(s) allowed so many errors in form (no information about subjects selection, sketchy statistical methods description, etc.) through. That is actually more concerning to me.
On reflection, I think that even a one-sided test would be inappropriate. This sort of skewed distribution needs non-parametric methods.
Even a casual look at the data from this study shows that there is a serious problem. The two populations are so different, you have to wonder if they are even related - which suggests a systematic problem with the study itself.
In addition, a large US study in 2000 measured the hair mercury in over 800 children aged 1 - 5 years. It found a mean hair mercury of around 0.2 ppm - the Holmes et al study's control group had a mean hair mercury (from stored specimens tested by a lab of unknown quality) of 3.6 ppm. Clearly, the Holmes et al control subjects' results are not consistent with this much larger study.
Prometheus.
<< Home