Let's do a study!
A reader wrote in with concerns that I was overlooking a key point of the abysmal telephonic autism "survey" - namely, showing how many unvaccinated children were available to study the interaction of vaccines and autism... or "neurological disorders" (AKA autism spectrum and ADD/ADHD).
Actually, I think that the folks at the CDC and university medical centers all over the country (and in other countries) are acutely (even painfully) aware of the number of people who aren't vaccinated.
What GR and other advocacy groups have been agitating for is a study comparing groups that don't vaccinate their children (e.g. the Amish, although they actually do vaccinate their children) with vaccinated children. The folks at GR are pushing for this despite the fact that their baloney "survey" showed no connection between autism and vaccination.
Here are a few reasons why that study wouldn't work, followed by my own "modest proposal" for a study that would work.
[1] The unvaccinated population is not necessarily the same as the vaccinated population: People who don't vaccinate their children may not be the same as those who do. In fact, studies have shown that those who don't vaccinate their children tend to be at either end of the socioeconomic and educational spectrum. It would not be valid to draw any conclusions about the impact of vaccines on autism (or ADD/ADHD) with such marked differences between the groups (apples and oranges).
[2] Selection biases: Recruiting people for a study is one of the biggest sources of error, especially when looking for something that is a public controversy. You will tend to attract more people who believe in a connection than those who do not.
[3] Inadequate power: If the GR survey is correct, about 3% of the overall population in the 4 - 17 year age range is completely unvaccinated. On the other hand, the prevalence of autism is only 0.65% in the same age range. It will be easier to detect a statistically significant difference in unvaccinated children between autistic and non-autistic groups than it will be to detect a difference in autism prevalence between vaccinated and unvaccinated children.
[4] Loss of useful information: It would be a shame to spend the time and money to do a study and only be able to answer a single question. After all, if there is no connection between autism and vaccination - as the GR "survey" suggests - wouldn't it be nice to be able to "mine" the data for other possible connections? By looking for subjects based on their vaccination status, the only question that can be answered is whether there is a correlation between the two.
My Modest Proposal:
This is a study that could be done rather quickly and with a minimum of expense. It would eliminate many of the sources of bias and would fairly easily generate balanced study populations that would be a good match to most of the general population.
[a] Contact a large HMO with actual facilities (e.g. Humana or Kaiser) and arrange to get access to their patient medical records. This is routinely done, although the HMO will want assurances that patient confidentiality will be maintained.
[b] Obtain a list of patients with autism diagnosis in the proper age range (I would suggest 6 - 12 years).
[c] Select one thousand of these patients at random. This would allow you to detect a difference if the prevalence of unvaccinated children is less than 1/3 that in the general population (alpha error level 5%, beta error level 5%). If the difference is less than that, you'll need to select more subjects.
[d] Confirm the diagnosis by having a child psychiatrist or psychologist review the records.
[e] For each of the remaining children, select a non-autistic control child from the HMO database that is of the same age, sex, geographical region, etc.
[f] Determine how many of the children in each group have received all, none or some of their vaccinations (keep track of which vaccines, when, etc.). If the GR "survey" was right (a very big "if") about the number of children unvaccinated, each group should have around 30 unvaccinated children, unless there is a correlation between vaccination and autism.
[g] If the autism and non-autism groups have statistically significant differences in their vaccination rates, then a correlation can be claimed. If the study shows no correlation, then the relative risk is less than 3. You'd have to have twice as many subjects to bring the minimal relative risk to below 2.
Using HMO patients eliminates any issues of affordability (which is minimal) or access to health care. Although the population of people who have HMO coverage is not necessarily the same as the overal US population, using the case control design ensures that the two groups are as similar as possible.
This study could have been done in the time - and for the reported cost - of the GR "survey". If they had done a real study - like the one I've outlined above - the answer would already be here.
If anybody would like to fund such a study, please contact me. If I can't find anyone else to do it, I'd be willing to coordinate it myself.
I doubt that any of the federal funding agencies will be interested in funding this study - not because of any conspiracy, but because the only people who want the study will probably not believe the likely outcome (i.e. no association - just as in the GR "survey").
So, this is my challenge to those who want this study: if you want it, do it. The advocacy groups have the money - all they need to do is find the will to take a chance. And make no mistake - this is a risk. There is a better than even chance that the study - if it isn't "fudged" - will show no correlation.
If they really want to know, they can find out for less than the cost of a full-page ad.
Prometheus
Actually, I think that the folks at the CDC and university medical centers all over the country (and in other countries) are acutely (even painfully) aware of the number of people who aren't vaccinated.
What GR and other advocacy groups have been agitating for is a study comparing groups that don't vaccinate their children (e.g. the Amish, although they actually do vaccinate their children) with vaccinated children. The folks at GR are pushing for this despite the fact that their baloney "survey" showed no connection between autism and vaccination.
Here are a few reasons why that study wouldn't work, followed by my own "modest proposal" for a study that would work.
[1] The unvaccinated population is not necessarily the same as the vaccinated population: People who don't vaccinate their children may not be the same as those who do. In fact, studies have shown that those who don't vaccinate their children tend to be at either end of the socioeconomic and educational spectrum. It would not be valid to draw any conclusions about the impact of vaccines on autism (or ADD/ADHD) with such marked differences between the groups (apples and oranges).
[2] Selection biases: Recruiting people for a study is one of the biggest sources of error, especially when looking for something that is a public controversy. You will tend to attract more people who believe in a connection than those who do not.
[3] Inadequate power: If the GR survey is correct, about 3% of the overall population in the 4 - 17 year age range is completely unvaccinated. On the other hand, the prevalence of autism is only 0.65% in the same age range. It will be easier to detect a statistically significant difference in unvaccinated children between autistic and non-autistic groups than it will be to detect a difference in autism prevalence between vaccinated and unvaccinated children.
[4] Loss of useful information: It would be a shame to spend the time and money to do a study and only be able to answer a single question. After all, if there is no connection between autism and vaccination - as the GR "survey" suggests - wouldn't it be nice to be able to "mine" the data for other possible connections? By looking for subjects based on their vaccination status, the only question that can be answered is whether there is a correlation between the two.
My Modest Proposal:
This is a study that could be done rather quickly and with a minimum of expense. It would eliminate many of the sources of bias and would fairly easily generate balanced study populations that would be a good match to most of the general population.
[a] Contact a large HMO with actual facilities (e.g. Humana or Kaiser) and arrange to get access to their patient medical records. This is routinely done, although the HMO will want assurances that patient confidentiality will be maintained.
[b] Obtain a list of patients with autism diagnosis in the proper age range (I would suggest 6 - 12 years).
[c] Select one thousand of these patients at random. This would allow you to detect a difference if the prevalence of unvaccinated children is less than 1/3 that in the general population (alpha error level 5%, beta error level 5%). If the difference is less than that, you'll need to select more subjects.
[d] Confirm the diagnosis by having a child psychiatrist or psychologist review the records.
[e] For each of the remaining children, select a non-autistic control child from the HMO database that is of the same age, sex, geographical region, etc.
[f] Determine how many of the children in each group have received all, none or some of their vaccinations (keep track of which vaccines, when, etc.). If the GR "survey" was right (a very big "if") about the number of children unvaccinated, each group should have around 30 unvaccinated children, unless there is a correlation between vaccination and autism.
[g] If the autism and non-autism groups have statistically significant differences in their vaccination rates, then a correlation can be claimed. If the study shows no correlation, then the relative risk is less than 3. You'd have to have twice as many subjects to bring the minimal relative risk to below 2.
Using HMO patients eliminates any issues of affordability (which is minimal) or access to health care. Although the population of people who have HMO coverage is not necessarily the same as the overal US population, using the case control design ensures that the two groups are as similar as possible.
This study could have been done in the time - and for the reported cost - of the GR "survey". If they had done a real study - like the one I've outlined above - the answer would already be here.
If anybody would like to fund such a study, please contact me. If I can't find anyone else to do it, I'd be willing to coordinate it myself.
I doubt that any of the federal funding agencies will be interested in funding this study - not because of any conspiracy, but because the only people who want the study will probably not believe the likely outcome (i.e. no association - just as in the GR "survey").
So, this is my challenge to those who want this study: if you want it, do it. The advocacy groups have the money - all they need to do is find the will to take a chance. And make no mistake - this is a risk. There is a better than even chance that the study - if it isn't "fudged" - will show no correlation.
If they really want to know, they can find out for less than the cost of a full-page ad.
Prometheus
posted by Prometheus at 11:05 PM
26 Comments:
Another population of children who are unvaccinated are those who canNOT be vaccinated. These would be children with existing health issues that prevent them from receiving the vaccine, yet also make them very vulnerable to the actual disease.
My oldest son's history of seizures precluded him from getting the pertussis vaccine. Yet... he was one of those kids who became hospitalized everytime he got a virus infection that led to croup. Pertussis would have definitely killed him in his first year of life, and possibly up until he was three years old (the last time he was hospitalized with croup).
Another example would be these boys who became permanently disabled from measles:
http://www.timesonline.co.uk/tol/news/uk/article1055533.ece
Also, a study using HMOs has already been done:
http://pediatrics.aappublications.org/cgi/content/full/112/5/1039
It could be modified to find kids without vaccines, since I am pretty sure due to Washington State's overly lax rules on vaccines that there would be plenty of unvaxed kids at Group Health.
According to this article:
http://seattletimes.nwsource.com/html/health/2003130272_resisters16m.html ... There are some communities (some served by Group Health) that are actively resistent to vaccines. Enough that Washington state is in the lower end of vaccine compliance.
Another article is here:
http://query.nytimes.com/gst/fullpage.html?sec=health&res=9F00E3DB1038F933A05752C1A9649C8B63
Scientists have settled the thimerosal/autism question to their satisfaction. And so the burden of proof to counter the overwhelming scientific evidence and consensus rests squarely on the thimerosal/autism advocates.
Yet for all their wealth, the thimerosal/autism lobby hasn't funded the kind of research that might lend empirical traction to their unsubstantiated belief.
Moreover, these advocates are very clear that they will not accept scientific findings that fail to find a causal association. So, if they want something done right, why don’t they do it themselves?
One has to wonder whether they truly have the courage of their convictions
If they find an association, they’ll be hailed as saints. If they don’t, they can work to find the real causes of autism. What’s not to like?
Inherient problems with your proposed suggestions:
[a] The demographics of covered individuals and areas of coverage versus other regions, uninsured, and self insured individuals may skew the results.
[b] and [d] Patients may be un-diagnosed or misdiagnosed in the records also what ASD threshold to use, strict adherence to DSM-IV for autistic, include Aspergers, PDDNOS….
[c] Statistically you would need 100,000 patient records to find 1,000 ASD individuals assuming a very liberal 1% rate and HMO may not allow that broad of a search or cover that many ASD individuals.
[d] If the original records are faulty, the confirmation will be faulty (GIGO).
[f] A majority of individual most likely will not have been in that HMO their entire lifetime. Vaccine records often do not follow the patient from HMO to HMO or even from doctor to doctor.
HCN, Tom and Chuck,
All very good comments!
HCN - I didn't go with the group of children who cannot be vaccinated because they have other health problems that make them different from the general population. It's the same problem with looking at Christian Scientists (for example) and comparing them to the general population.
I hadn't thought about Group Health - I was thinking about my recollections of all the unvaccinated kids in the Berkely, CA area who are seen in the Kaiser system.
Tom - I completely agree with you. As far as I'm concerned, the vaccine/thimerosal issue hasn't panned out. Unless some new data shows up, I see no point in doing ecological studies. However, if the proponents of the vaccines-cause-autism hypothesis want a study, this would be a good way to do it.
Chuck - Good points. The reason I went with a case-control design was to have the two groups matched as closely as possible except in the variables under study. Even if the study groups are different from the general population, they would be similar to each other, thus isolating the variables.
Diagnosis is also an issue, but I suspect that over-diagnosis is more likely.
The reason I suggested using Humana or Kaiser is that they may have enough patients to supply the necessary numbers. An alternate approach is to accept a higher beta error (chance of accepting the null hypothesis in error). By accepting a 25% beta error, the sample size shrinks to about 500. If you were willing to accept a beta error limit of 50% (keeping alpha at 5%), the sample could be as small as 300 and still be able to detect a relative risk as small as 3.
The issue of vaccine records is also a valid concern, but most pediatricians will get previous records when they get a new patient. At least in the Humana system, they make a good effort to get old records on new "members". Fortunately, those children with no vaccinations will be pretty easy to find, as pediatricians will want to cover their butts in the event a child gets a vaccine-preventable illness and so will extensively document the fact that the parents refused vaccination.
Prometheus
It's tough to answer the question, "why would you not support a study?". Countless mercury enthusiasts have asked that one but I think that you've hit the nail on the head: because the study that is proposed is flawed to such an extent that it would render the results meaningless.
Great post.
Actually flying under the "ASD Diagnosis" and "vaccine" radar is pretty easy. The last two doctors my children had did not have a vaccine history of any of my children.
My background:
The last time I got a flu shot was the last time I got the flu. (2000)
My son did get Chickenpox and was vaccinated.
0 for 2 is a POOR track record in my eyes.
Group Health was one of the HMOs included in the study I posted (I'm too lazy to put in the html tags).
Chuck,
I'm not sure that I get your point. Are you implying that children with autism of a clinically significant degree (language delay, stereotyped behaviors, etc.) would not be diagnosed as such?
If your point is that some children "assessed" with autism in the schools do not have a medical diagnosis of autism, then I agree. I also think that is a good feature of my proposed study. The "educational" (and Cal. DDS) diagnosis of autism is too broad to be useful. By limiting it to children with the DSM-IV(TR) definition of autism, Asperger's syndrome and PDD-NOS, we would eliminate a great deal of the "noise" and focus more on the "signal".
As for your experience with vaccination records, I can only extend my sympathy. We've moved around a good deal and our children's doctors have always sent away for records from previous doctors. As a result, they have a continuous medical record. This is the policy of many of the larger HMO's, such as Humana.
Also, your comment of "0 for 2 is a POOR track record in my eyes" was a bit odd. Are you referring to vaccine efficacy? In that case, this posting had nothing to say about vaccine efficacy. But, since you brought it up, did you have your "flu" confirmed? It may very well have been an unrelated adenovirus, rhinovirus or echovirus.
Prometheus
I'm trying to think through how differential health care utilization would affect this kind of study. It seems to me that there are parents who are plugged into the health care resources available to them, get well checks and immunizations on schedule, and would be likely to seek screening and eventual diagnosis if autism seemed like a possibility. Conversely, there are also parents who for whatever reason don't keep up with regular visits, use the ER for primary care, skip immunizations to some degree, and would be less likely to pick up on the possibility of autism.
If there are clumps of parents fitting the descriptions I'm envisioning, the study group might end up comparing children of the on-the-ball parents to those of the loosey-goosey ones and end up concluding that vaccines were associated with autism when they were really just associated with conscientious parenting. Does that make sense?
Being a "High risk" individual that is supposed to get the vaccine every year I did get a flu shot in Oct 99. Being a "high risk" individual I did go to the doctors upon getting sick and influenza was confirmed. The doctor said the strain I had wasn't covered that year.
My child is not professionally diagnosed or labeled as ASD in the educational system although he meets all the requirements in the DSM-III (and IV)
Why waste time and money going to a doctor when they are of no benefit after the diagnosis?
I have convinced many associates of this as a professional diagnosis is unnecessary for educational or treatment services. We have done just fine without it.
Isles,
You have hit at the major problem with any study trying to look at a connection between vaccines and autism. If a correlation is found, the question remains: is it a connection between vaccines and autism or a connection between greater health care utilization and identifying autism?
One of the reasons for looking at HMO patients is that the HMO's emphasize preventive care and make it easier to get well child care. Again, these people are not neccessarily representative of the US population as a whole, but the two comparison groups would be identical, which would give the best chance of detecting a real difference.
Prometheus
Actually, outside of CA, going to an HMO is either useless or counterproductive for a diagnosis or follow-up treatment for any ASD disorders. When I first became aware of my child’s autism, I specifically looked at the underwriting of our medical coverage. Specifically, a diagnosis of any “developmental delay” would not have been covered and would have further limited future treatments or services. I know of no one who has stayed “in-network” for any treatments for ASD disorders. These are all parents that are “on-the-ball”.
There is a potential problem with Kaiser--and this will only increase with time.
Kaiser is known in California for refusing to provide services under the mental-health parity act (AB88). Kaiser identifies kids, then tells the parents that most of the expensive therapies are "educational" rather than medical.
From my experience watching autism groups in California, those parents who have the opportunity to switch health insurance providers drop Kaiser when they have an ASD kid.
Matt
One source of information could be Texas--they track immunization records on the state level:
http://www.dshs.state.tx.us/immunize/immtrac/default.shtm
Matt
I'm not too familiar with Kaiser plan details - the university only offers a Humana plan in their HMO offerings. Also, Kaiser isn't too active where I am.
For a lot of folks, at least where I am, the HMO offering of their employer is the only one that allows them to have well-child care and other routine visits covered. For that reason, a lot of families with children opt for the HMO.
Although there are plenty of problems with the study I've outlined above, it still is much cleaner and more likely to give good data than the sort of study advocated by many of the vaccines-cause-autism groups.
I think the "take-home message" is pretty clear: it's not easy to do a good study looking for potential connections between vaccination and autism. However, it would be very easy to do a bad one.
Prometheus
Actually, there is a database that likely has the records for which Prometheus is looking for.
It's called the VSD.
You know, that same VSD that the mercury loons always say they can't get access to. Of course, that's a lie, since right here the NCHS (which administers the program) has the modest fee structure.
http://www.cdc.gov/nchs/r&d/rdcuserfees.htm
Groups like SafeMinds and Generation Rescue will spend tens of thousands of dollars on useless phone surveys and piss-poor biological research, but they won't use this database. Maybe it's because they know they're not going to like the answer so they do sham research instead.
It's never been about science with these clowns. Ever. It's about pushing their agenda no matter what the cost.
Re: utilization as a confounder, I suppose you could screen out any child who hadn't had a certain number of well checks in his or her record, to get rid of the underutilizers and make the case and control groups more similar.
I still have the nagging feeling that you're trying to reinvent the VSD wheel, though. I wonder why Verstraeten and colleagues did not use a case-control approach. No offense, but I would think there was a reason they went the route they did.
I think the VSD data is still available, isn't it? All you have to do is go to Bethesda or wherever and *not* try to spirit personally identifying data out of the room by hamfistedly disguising it as an SAS program file. :-)
My completely unscientific polling confirmed to me why using any HMO would not gain any real insight into ASD or it’s causes.
83% of the people who responded with health insurance (not just HMOs) said ASD diagnosis was covered by insurance
70% of the people who responded with health insurance (not just HMOs) said they had “Some” or “No” recommended treatments covered by insurance.
60% of the people who responded with health insurance (not just HMOs) and covered treatments said they still used uncovered treatments.
Strangely enough, I had completely forgotten about the VSD. As for why the Verstraeten study didn't use a case-control format, I can't say for sure, but my suspicion is that they felt they had a large enough sample to "even out" the heterogeneities.
Part of the proble with a case-control study is that people are always second-guessing about the selection of the controls. Are they too similar to the cases, thereby reducing the study's ability to see differences or are they too dissimilar, leading to spurious differences?
The user fees from the NCHS seem quite reasonable - $500 a month for remote access, if I read the schedule correctly. Even if I needed to create a new file, it would probably come in for under a few thousand dollars.
I called the number and spoke to a very helpful person about getting access - they assured me that it would be no problem at all. Their data specialists would make sure that no confidential data "leaked" into the files.
Strange that the autism advocacy groups haven't done that. It would be so easy. Heck, I'd do it myself, if I came across a couple thousand dollars to spare.
It really makes you wonder, doesn't it? Do they really want to know?
Prometheus
I have not read all of the posts of this blog, but the autism advocacy groups have done that before.
http://www.patsullivan.com/blog/2006/03/geiers_link_aut.html
Chuck,
Actually, the Geiers apparently had their VSD access privileges revoked because of some "hanky-panky" they tried.
See:
http://www.casewatch.org/fdawarning/rsch/geierk.shtml
and:
http://www.casewatch.org/fdawarning/rsch/geier.shtml
Researchers who do not try to get around the cofidentiality safeguards in place are allowed access. Those who do get barred. It's pretty simple.
The study in question, published in JAP&S, was reputedly showing the "early downward trend" in autism diagnosis from the VAERS database (a notoriously unreliable source, since it is "stuffed" with cases from lawyers) and the California DDS.
While the VAERS data is pointless, the California DDS data later showed a resumption of the upward trend, rendering Dr. Geier's work moot.
Not that his "interpretation" of the California DDS data was valid in the first place, but later data showed that he was clearly wrong.
Prometheus
I thought that we already determined that any statistical analysis of CA DDS is junk science at best?
Chuck,
Um, you brought up the California DDS study by the Geiers, not me. The study cited in the 'blog posting you provided looked at VAERS and California DDS data - not VSD data.
Yes, the California DDS data is not valid for year-to-year comparisons. In addition, the analysis done by Dr. Geier was erroneous - even if the California DDS data could be used for such a comparison.
It is pointless to do an erroneous analysis of valid data - it is doubly pointless to do an erroneous analysis of invalid data.
Perhaps you meant to give a link to a different study?
Prometheus
Neither of your casewatch links work, so I attributed this quote to you.
"While the VAERS data is pointless, the California DDS data later showed a resumption of the upward trend, rendering Dr. Geier's work moot"
The logic of using a flawed database (DDS) as a counter argument to another flawed database (VAERS) is pointless making the conclusion worthless.
Chuck,
The Casewatch links worked for me - it must be something with your system. Regardless, the "quotes" are mine.
Let me be more clear:
[1] The VAERS database is worthless, especially for looking at autism, because it makes little effort to confirm the data. It is easy to add a case to the database, even if you just make something up. Try it!
[2] New point, not related to the above. The people who run the California DDS database explicitly warn people that their data cannot be used to compare one year to the next.
[3] New point. Even if [2] weren't true, the analysis by Geier was flawed and invalid. The subsequent quarters' data from CDDS showed clearly that the "decline" was an analytical artifact.
I'm not defending either database as a valid source of epidemiological data. Quite the opposite! VAERS, California DDS and the US Dept. of Education are all useless sources of data to make year-to-year comparisons.
Chuck, since you were the one who brought up the Geier "study" of the VAERS and CDDS databases, was there a point you were trying to make?
Prometheus
Anonimouse,
Do you know if the NCHS has any restrictions on who can have access to the VSD database? My call seemed to indicate that there were none (except for people who have tried to "break" the confidentiality), but I would like to be sure before I 'blogged about it.
Also, do you know if NCHS has a "sliding scale" for their fees - say, for poor academics who don't have a grant to cover this research?
Prometheus
<< Home