You may have read recently about a new meta analysis that purportedly shows that intercessory prayer works. If you had read my posts on intercessory prayer you might have been surprised by this since other good studies have shown that prayer doesn’t work, and prayer still doesn’t work.
Steven Novella at NeuroLogica Blog explains why this prayer meta analysis is bogus:
But meta-analysis is tricky business. First, it should be pointed out that it does not represent new data – it is just taking a fresh look at old data. It can be useful but only when it is very carefully applied. For example, the studies that are lumped together should have very similar design, they should be looking at the same type of subjects and should use similar outcome measures. The results of a meta-analysis are only meaningful if data from the different studies can be reasonably combined.
Also, a meta-analysis does nothing to address the quality of the studies being looked at. The old adage of “garbage in-garbage out” still applies. If you lump together 10 bad studies, you don’t get one good study, you get a useless meta-analysis. For these reasons meta-analyses have a poor track record of predicting the ultimate outcome of a question, once definitive studies are done, failing over a third of the time.
A meta-analysis is not always the best method for coming to an overall conclusion about an area of research. There are aspects to the pattern of results that are important to consider, and are white-washed in a meta-analysis. For example, what is the trend between study quality and size of the effect? If we see a real tendency for the better studies to have a smaller effect (which we do, in my opinion, in the intercessory prayer literature), that strongly suggests that the effect is not real. By combining these studies, however, these differences are erased. In effect, the good data is diluted in bad data.
More at the link.
Unfortunately, as Novella laments, this study has already produced headlines of the “new study finds prayer works” kind, while the later retractions, as the data is analyzed properly, will get little airtime.
Ironically, Paul Kurtz of the skeptical Center for Inquiry will be undergoing cardiac surgery today. I wish him well, and I won’t be praying for him.