Survey says..... Nothing!!!
Generation Rescue just published their long-awaited telephone survey results that showed - surprise! - that vaccinations cause autism.
Well, actually, it doesn't really show that vaccines cause autism, what it shows is that vaccines cause "neurological disorders", loosely defined as autism (or autistic spectrum disorder) and/or ADD/ADHD.
Except that it really doesn't show that, either.
So what does the survey show? Well, let's just take a look.
Not surprisingly, they did not publish the questions they asked, which would have given us the opportunity to analyze their survey for systematic bias. What they did do was publish the raw data, which allows us to analyze their results.
Assessing the survey:
When looking at a survey, the first step is to examine the survey structure - particularly the questions, but also the area covered and population contacted - for any flaws or biases. This survey was conducted in nine counties of California and Oregon - generally the more populous counties - which gives it a West Coast bias. Since autism, ADD/ADHD, asthma and diabetes do not show a coastal preference, this bias should not create a serious problem for the survey.
Next, looking at the data, we need to see if there are age or sex biases. Since the data do not include a detailed breakout of the ages year-by-year, it is impossible to analyze for age biases, but the sex ratio (52% male, 48% female) - although it is close to the US Census Bureau data for that age range - shows a statistically significant difference from the US population.
Looking at the sex ratios of the unvaccinated and vaccinated (both fully and partially) groups, the differences are not statistically significant, so the imbalance in the sex ratio is equal in both groups.
It would be helpful to know how many people they had to call to get the number of responses they did - this gives a good measure of how skewed the responses might be. The higher the percentage of people refusing to take the survey, the more likely that the people who do answer are not a representative sample of the population.
Unfortunately, we are not provided this information, so we have no way of knowing how many of the people who were called hung up or otherwise refused to participate, which makes it impossible to know the degree of participation bias. We'll just have to soldier on.
Assessing the data:
Looking at the data, one thing that pops up as an immediate red flag is that the prevalence of autism is greater than the prevalence of PDD-NOS. This is an inversion of the usual finding, which is that autism (with stricter diagnostic criteria) is less common than PDD-NOS (which is essentially some of the criteria for autism but not enough to diagnose autism). This suggests that the people involved in the survey may not be asking the right questions.
Looking a bit closer, the prevalence of the autistic spectrum disorders is way out of line with the most recent data. In February of this year, the CDC published a report on the prevalence of autism in six states. This prevalence was widely reported and was frequently cited as more evidence of the "autism epidemic" by the chelationistas, so it should be familiar.
This CDC report found that the prevalence of autistic spectrum disorders ranged from 4.5 to 6.5 per 1000 children, which would be 0.45 - 0.65%.
The GR survey found that the prevalence of autistic spectrum disorders was 4.7% in their overall sample and ranged from 1.3% (female, fully vaccinated) to 8.4% (male, partially vaccinated). This is nearly ten times the CDC prevalence.
To put the number into perspective, the CDC data say that somewhere between 1 in 222 to 1 in 154 children have an autistic spectrum disorder. The GR survey would suggest that 1 in 21 children have an autistic spectrum disorder. Ridiculous!
Clearly, there is a serious problem with this survey.
This is reminiscent of the infamous Holmes et al study, which found that their control subjects had over ten times the amount of mercury in their hair than the NHANES study found in children of similar ages less than a year later.
Now, the people who ordered this survey may argue that it's the difference between the two groups - not the absolute number - that is important. This sounds reasonable, but it isn't. There is no way to know if the degree of error is the same in all the groups.
If the "over-reporting factor" is only 5% less in the "unvaccinated" group than in the other groups, the difference between the groups disappears. This completely invalidates the "results".
So, with this glaring evidence of error, is there any point in looking any further?
No, not really. But we should examine the major claim being made about the survey results, just to be thorough.
"Vaccinated boys have a 155% greater chance of having a neurological disorder like ADHD or autism..."
Let me start off by saying that - even if the survey was accurate, which it clearly is not - an increase in autism and ADD/ADHD would not constitute an increase in "neurological disorders", which their conclusion implies. There may, in fact, be neurological disorders (e.g. measles inclusion-body encephalitis) that are reduced by vaccination. So, even without looking at the data, this assertion is pure nonsense.
But, back to the survey data.
To get a statistically significant difference between the unvaccinated and vaccinated boys, they had to lump the partially vaccinated and the fully vaccinated boys together and they also had to lump autism, PDD-NOS, Asperger's syndrome, ADD and ADHD together.roups, with the partially and fully vaccinated boys having - as a group - 55% more reported "neurological disorders".
However, if you look at the "autistic spectrum disorders" by themselves, there is no statistically significant difference between unvaccinated and vaccinated (partially plus fully) boys.
Curiously, there is a statistically significant difference between the partially vaccinated boys and both the unvaccinated and fully vaccinated boys. Partially vaccinated boys have a higher prevalence of autistic spectrum disorders than either the unvaccinated or fully vaccinated boys.
Now, if you want to show that an exposure causes a disorder, it is generally a good idea to show a dose-response relationship. In the case of this survey, a little vaccination is worse than none or a lot. This doesn't sound like a dose-response relationship to me.
Well, so the autism-vaccine connection doesn't work - how about ADD/ADHD?
Out of 17,674 children in the survey, 1,875 had either ADD or ADHD. That works out to a prevalence of 106 per thousand or 10.6%.
That seems a bit high.
If you look at the USDE data from 2005, the percentage of children ages 4 - 17 years with "other health impairment" (the catch basin for ADD/ADHD and many other disorders) was 9.5 per thousand or 0.95%. This is over ten times lower than reported in the GR survey. And remember - the USDE "other health impairment" category includes a variety of diagnoses other than ADD/ADHD.
So, it seems that there are some serious problems with the GR survey data. I don't know what the cause of the problem is (although I have my suspicions), but one thing is painfully clear:
The survey data is garbage.
So, what does the GR survey show?
Absolutely nothing.
Prometheus
Well, actually, it doesn't really show that vaccines cause autism, what it shows is that vaccines cause "neurological disorders", loosely defined as autism (or autistic spectrum disorder) and/or ADD/ADHD.
Except that it really doesn't show that, either.
So what does the survey show? Well, let's just take a look.
Not surprisingly, they did not publish the questions they asked, which would have given us the opportunity to analyze their survey for systematic bias. What they did do was publish the raw data, which allows us to analyze their results.
Assessing the survey:
When looking at a survey, the first step is to examine the survey structure - particularly the questions, but also the area covered and population contacted - for any flaws or biases. This survey was conducted in nine counties of California and Oregon - generally the more populous counties - which gives it a West Coast bias. Since autism, ADD/ADHD, asthma and diabetes do not show a coastal preference, this bias should not create a serious problem for the survey.
Next, looking at the data, we need to see if there are age or sex biases. Since the data do not include a detailed breakout of the ages year-by-year, it is impossible to analyze for age biases, but the sex ratio (52% male, 48% female) - although it is close to the US Census Bureau data for that age range - shows a statistically significant difference from the US population.
Looking at the sex ratios of the unvaccinated and vaccinated (both fully and partially) groups, the differences are not statistically significant, so the imbalance in the sex ratio is equal in both groups.
It would be helpful to know how many people they had to call to get the number of responses they did - this gives a good measure of how skewed the responses might be. The higher the percentage of people refusing to take the survey, the more likely that the people who do answer are not a representative sample of the population.
Unfortunately, we are not provided this information, so we have no way of knowing how many of the people who were called hung up or otherwise refused to participate, which makes it impossible to know the degree of participation bias. We'll just have to soldier on.
Assessing the data:
Looking at the data, one thing that pops up as an immediate red flag is that the prevalence of autism is greater than the prevalence of PDD-NOS. This is an inversion of the usual finding, which is that autism (with stricter diagnostic criteria) is less common than PDD-NOS (which is essentially some of the criteria for autism but not enough to diagnose autism). This suggests that the people involved in the survey may not be asking the right questions.
Looking a bit closer, the prevalence of the autistic spectrum disorders is way out of line with the most recent data. In February of this year, the CDC published a report on the prevalence of autism in six states. This prevalence was widely reported and was frequently cited as more evidence of the "autism epidemic" by the chelationistas, so it should be familiar.
This CDC report found that the prevalence of autistic spectrum disorders ranged from 4.5 to 6.5 per 1000 children, which would be 0.45 - 0.65%.
The GR survey found that the prevalence of autistic spectrum disorders was 4.7% in their overall sample and ranged from 1.3% (female, fully vaccinated) to 8.4% (male, partially vaccinated). This is nearly ten times the CDC prevalence.
To put the number into perspective, the CDC data say that somewhere between 1 in 222 to 1 in 154 children have an autistic spectrum disorder. The GR survey would suggest that 1 in 21 children have an autistic spectrum disorder. Ridiculous!
Clearly, there is a serious problem with this survey.
This is reminiscent of the infamous Holmes et al study, which found that their control subjects had over ten times the amount of mercury in their hair than the NHANES study found in children of similar ages less than a year later.
Now, the people who ordered this survey may argue that it's the difference between the two groups - not the absolute number - that is important. This sounds reasonable, but it isn't. There is no way to know if the degree of error is the same in all the groups.
If the "over-reporting factor" is only 5% less in the "unvaccinated" group than in the other groups, the difference between the groups disappears. This completely invalidates the "results".
So, with this glaring evidence of error, is there any point in looking any further?
No, not really. But we should examine the major claim being made about the survey results, just to be thorough.
"Vaccinated boys have a 155% greater chance of having a neurological disorder like ADHD or autism..."
Let me start off by saying that - even if the survey was accurate, which it clearly is not - an increase in autism and ADD/ADHD would not constitute an increase in "neurological disorders", which their conclusion implies. There may, in fact, be neurological disorders (e.g. measles inclusion-body encephalitis) that are reduced by vaccination. So, even without looking at the data, this assertion is pure nonsense.
But, back to the survey data.
To get a statistically significant difference between the unvaccinated and vaccinated boys, they had to lump the partially vaccinated and the fully vaccinated boys together and they also had to lump autism, PDD-NOS, Asperger's syndrome, ADD and ADHD together.roups, with the partially and fully vaccinated boys having - as a group - 55% more reported "neurological disorders".
However, if you look at the "autistic spectrum disorders" by themselves, there is no statistically significant difference between unvaccinated and vaccinated (partially plus fully) boys.
Curiously, there is a statistically significant difference between the partially vaccinated boys and both the unvaccinated and fully vaccinated boys. Partially vaccinated boys have a higher prevalence of autistic spectrum disorders than either the unvaccinated or fully vaccinated boys.
Now, if you want to show that an exposure causes a disorder, it is generally a good idea to show a dose-response relationship. In the case of this survey, a little vaccination is worse than none or a lot. This doesn't sound like a dose-response relationship to me.
Well, so the autism-vaccine connection doesn't work - how about ADD/ADHD?
Out of 17,674 children in the survey, 1,875 had either ADD or ADHD. That works out to a prevalence of 106 per thousand or 10.6%.
That seems a bit high.
If you look at the USDE data from 2005, the percentage of children ages 4 - 17 years with "other health impairment" (the catch basin for ADD/ADHD and many other disorders) was 9.5 per thousand or 0.95%. This is over ten times lower than reported in the GR survey. And remember - the USDE "other health impairment" category includes a variety of diagnoses other than ADD/ADHD.
So, it seems that there are some serious problems with the GR survey data. I don't know what the cause of the problem is (although I have my suspicions), but one thing is painfully clear:
The survey data is garbage.
So, what does the GR survey show?
Absolutely nothing.
Prometheus
posted by Prometheus at 7:22 PM
22 comments
