I’ve written before about Shawn Carlson’s “A Double-blind Test of Astrology”, published in the journal Nature, in 1985. To recap, 116 people completed California Personality Index (CPI) surveys and provided their natal data (date, time and place of birth). One natal chart and the results of three CPI surveys (one of which was for the same person as the natal chart) were given to an astrologer who was to interpret the natal chart and determine which of the three CPI results belonged to the subject whose natal chart it was. In only 40 of the 116 cases did the astrologers choose the correct CPI. This is the exact success rate expected by random chance. The conclusion: the test gave us no reason to suppose astrology works.
That was 24 years ago. As far as I am aware, no serious challenge has been made to Carlson’s conclusions, perhaps until recently. I have now been informed that Suitbert Ertel, professor of psychology at Göttingen University, has claimed to have found serious flaws in Carlson’s paper. Ertel’s paper is apparently not available online, although I have read a summary of it by Ken McRitchie: Reappraisal of 1985 Carlson study finds support for astrology.
Normally, articles in Nature, or any scientific journal, are peer reviewed before publication. The peer review process subjects scientific beliefs and claims of fact to critical analysis by qualified experts. Yet, even though the Carlson study makes claims of scientific fact, it had not been peer reviewed. Nature had published the article as editorial content in the Commentary section, a detail that undoubtedly has been overlooked by countless authors who have cited the study and regarded it as definitive.
I found this a strange if interesting claim – some Nature articles are not peer reviewed? The “commentary” section is somehow less “definitive” than if it had been published elsewhere in the journal? I emailed Shawn Carlson to see what he had to say about this. His reply began:
In general I stop reading an article when I come across the first absurdity. Ken McRitchie states that my paper was not peer reviewed. Not so. It survived a rigorous peer review that included a famous psychologist whom I will reveal in a later publication. To the best of my knowledge, Nature never publishes research articles that have not been properly reviewed, and I doubt that any serious scientist thinks otherwise. I talked personally with then Editor-In-Chief John Maddox about whether or not the "Commentary" section was the appropriate venue and he assured me that they often published original results there that they believed were likely to be of general public interest.
So it was peer reviewed, and the commentary section is not reserved for non-definitive reporting.
I have to say, I found McRitchie’s criticisms here to be laughable, especially considering that Ertel’s “reappraisal” is published in the so-called Journal of Scientific Exploration – a journal that includes papers by Dean Radin and Ian Stevenson among others. Ertel’s paper is rather laughingly referred to as “peer-reviewed.” Well yes, but if his “peers” are the likes of Radin and Stevenson, I’m unimpressed. According to Wikipedia, the journal is not indexed in Web of Science, an indexing service for leading scientific journals, that covers over 10,000 of the highest impact journals worldwide. Of course, that doesn’t make Ertel’s paper wrong, but it shows this criticism of Carlson’s paper to be absurd at best.
Instead of presenting the astrologer participants with pair choices, which is the normal format for such tests, and the format followed in an earlier well-known astrological study by Vernon Clark (1961), Carlson presented a three-choice format, consisting of one genuine object and two selected at random. This three-choice format, Ertel notes, is less powerful than a two-choice format.
Maybe, but that doesn’t mean the one in three choice is invalid. If astrology were real, it should have been possible for the astrologers to pick out the correct one with greater odds than you would get with just guessing. They didn’t. If you were to pay money to an astrologer, would you be happy to be given advice based on an interpretation of your horoscope that, while not accurate for you, was accurate for the second closest person? I doubt it, and this certainly isn’t what astrologers claim.
Ertel is also critical of Carlson’s piecemeal analysis of the sampled data, in which only sub-samples are examined instead of the total effects. The correct analysis for a three-choice format, Ertel asserts, is to calculate the proportion of combined first and second choices according to the normally accepted protocol. Carlson initially states his intention to do this but then disregards the protocol for no given reason. Re-analysis shows that the astrologers correctly matched CPI profiles to natal charts better than would be expected by chance with marginal significance (p = .054). This positive result, Ertel found, was replicable with even better results (p = .04) for the astrologers’ ten-point rating of profiles fit to birth charts, a procedure that Carlson requested of the astrologers but ignored in the end, again without giving reasons.
Firstly – Carlson didn’t ignore the “ten-point rating of profiles.” He uses the ten point rating results and concluded (page 424):
Next we took the weights into account, by a method established before studying the data.
The scientific hypothesis predicts that 1/3 of the choices at any weight should be correct choices. Figure 4 shows the percentage correct for each weight with the appropriate error bars, and the best linear fit with slope – 0.01 +/- 0.02. The slope is consistent with the scientific prediction of zero slope.
In other words, the astrologers’ confidence that they were right (on a scale of 1 to 10) did not fit the actual correct choices any better than the incorrect ones.
Regarding the correctness of the second choice options, the Nature paper (page 424) states:
The correct CPI was chosen as the second place choice at the 0.40 +/- 0.044 rate which is also consistent with [random chance].
It looks to me as though the astrologers' first and second choices were consistent with guessing, rather than with any information provided by astrology. In addition, Carlson did use the ten point rating scale (despite what Ertel claims), and I see nowhere in the Nature paper where Carlson initially states his intention to calculate the proportion of combined first and second choices but then changes his mind, as Ertel also claims. (If anyone can see where he says this, I’d be interested. I’ve read the paper several times, and I can’t find it.)
The eminent psychologist Hans Eysenck, late author of the book Astrology: Science or Superstition, argued that the CPI explicitly states that it should be interpreted only by trained and experienced users, and the astrologers lacked the necessary qualification. Other critics questioned whether the CPI and astrology evaluate personality in the same ways, and whether there was enough common ground for astrologers to make valid matches.
This one really had me rolling my eyes. McRitchie really needs to make his mind up. Is the CPI a valid way of testing astrology or not? Because if it isn’t, then Carlson’s whole test is invalid regardless of what any “reappraisal” of the statistics tell us, which means that Ertel can’t claim that Carlson’s test now supports astrology. He can’t have it both ways. Which is it?
Of course, if the CPI is not a valid way of testing astrology, then all this means is that there is still no evidence astrology works. Remember, proponents of astrology are the ones making the claim, and so they are the ones with the thing to prove. Null hypothesis, anyone?
Furthermore, if the CPI is not a valid way of testing astrology, and if asking the subject which natal chart interpretation is correct (Carlson’s test #1) is not valid (as Carlson concluded), then how should astrology be tested? It’s a bit rich for proponents of astrology to criticize a method of testing astrology to see if it works, if they don’t have a valid way of testing it themselves. As I wrote in January 2007 in Testing Astrology – Again, if none of these methods are acceptable, and therefore if astrology can’t be tested, then this means that astrology is almost certainly bogus. For one, astrology’s doubtful provenance (no known method by which it is supposed to work, no known way its rules were derived, its absurd premises), means we need extraordinary evidence that it works. By this I mean better evidence that we demand for many other things. And we certainly don’t have this extraordinary evidence. And for two, if astrology can’t be tested, then clearly no one would ever have been able to work out all the detailed rules astrologers use. How would they have been able to work out the rules if there is no way of ever testing them to see if they were right? On what basis do astrologers claim that astrology works? How do they know?
Proponents of astrology want to have it both ways – they claim astrology can’t be tested, and yet they also claim they know it works. And you can see this mindset in all its absurdity with this from Ertel:
“The results are regarded as insufficient to deem astrology as empirically verified,” Ertel warns, “but they are sufficient to regard Carlson’s negative verdict on astrology as untenable.”
- which is an epic FAIL. If the results of Ertel's study are insufficient to deem astrology as empirically verified, then astrology failed the test. The null hypothesis is that astrology doesn't work. Come on, this is basic experimental design. Ertel, with this comment, shows himself to be a pseudo-scientist who is clueless about how real science works. This is further confirmed by the fact that in Ertel’s “reappraisal” methodology, he decided to change the things that he thinks the test should have been looking for in the first place. This is a strict no no in experimental design. In a well designed test, you state your objectives clearly before you do the test, and you compare the results with what you stated at the beginning you would be looking for. What you should never do, is do the test, decide the results were not what you wanted, and then go looking for something else that might show the result you did want. It’s well known that this technique is likely to return false positives, which is why it is never done by real scientists. Or as Carlson wrote in his email:
I could go on at length about Dr. Ertel's flawed analysis. I will take up that effort at a later time. For now, let me just point out that a scientist never figures out how to analyze his data after the fact with all the data in view. This leads to selection biases that can skew the results to favor the experimenter's hopes. Apparently Dr. Ertel has done just that and from many possible approaches he has selected a particular method of analysis that gives him a result that he says is "marginally significant" in favor of the astrologers. That won't surprise anyone who understands statistics and knows how certain subtle pitfalls often turn a "great discovery" into a fool's errand (and sometimes vice versa).
To get there however, Dr. Ertel ignores the direct and extremely significant null result that I obtained using data analysis methods that were perfectly reasonable and selected before the fact. The astrologers failed directly--thereby refuting the notion that astrology is so effective as a discipline that a selection of astrologers who are held in high esteem by their peers can gain access to vital information that is not available to ordinary psychologists.
This is the best astrology proponents can do after 24 years – no new information, no new tests that show astrology can do anything, just some dishonest and scientifically dodgy sniping at a comprehensive test that all involved (including the astrologers) agreed before the test was done, was a valid test. Proponents of astrology are the ones who need to show evidence that their magic fortune telling system is real, and yet even after 24 years, and after the efforts of a university professor with a history of writing papers supportive of astrology, the best they can do is say, and I quote again:
The results are regarded as insufficient to deem astrology as empirically verified
If Ertel, or anyone else, believes that astrology works in any measurable way, then they need to stop sniping at Carlson’s well designed and genuinely peer reviewed test, and they need to design and perform their own experiment, have it peer reviewed and published in Nature or another real scientific journal. As Carlson notes, such a successful test would undoubtedly win its author the Nobel Prize. If astrology is real, it should be possible to design and conduct such an unequivocal test. Why don’t they?