John Cook (with others) published, recently, a paper called Quantifying the consensus on anthropogenic global warming in the scientific literature. The paper used Web of Science to search for papers on “global warming” or “global climate change”. They restricted their search to the physical sciences and to articles only and it returned 12474 results. They then analysed the abstracts of these papers and scored them according to whether or not they endorsed anthropogenic global warming (AGW) or had no position with regards to AGW. Their results indicate that of those papers that addressed (in the abstract at least) AGW, 97.1% endorsed AGW while only 2.9% rejected or were uncertain about AGW.
According to Watts Up With That (WUWT), Richard Tol has statistically deconstructed the 97% consensus. The basic claim attributed to Richard Tol is that by searching for articles on “global climate change” rather than “climate change” the Cook et al. study ignored 75% of relevant papers and changed the disciplinary distribution. Maybe so, but is this at all relevant? Let’s consider how one might design a study such as that carried out by Cook et al.
Well, typically one would consider the resources available and hence how many samples they can realistically analyse. Next one needs to determine how to extract the sample. One could do a very broad search (such as “climate change” rather than “global climate change”) and then discover that the sample was too large to be analysed given the resources. One could then choose to change the search terms, or to select randomly from within the bigger sample. Both could be valid ways of extracting a suitable sample. The advantage of using a randomly selected subgroup of a big sample is that your large sample may be – in a sense – the “correct” full sample of all possible papers. Your randomly selected sub-group would then be a representative sub-sample of all possible papers.
Alternatively, one could argue that the full sample includes papers that aren’t actually relevant. In this case, the interest is in whether or not man is responsible for “global warming” or “global climate change”. By searching for “global warming” or “climate change” the results may well include many that aren’t really relevant (or that one could argue aren’t relevant). By changing your search terms to be more specific, you can extract a full sample of papers that cover the topics of interest and that is of a manageable size.
Essentially I’m suggesting that although Richard Tol’s strategy may be fine, there’s nothing fundamentally wrong – in my opinion at least – with the strategy adopted by Cook et al. It’s certainly a perfectly reasonable strategy that returned a large number (12474) of relevant articles, one third (4000) of which directly – in their abstracts – address AGW. If Richard Tol really wants to show that there is a problem with this sampling strategy, he could do so quite easily. His two claims are that Cook et als. strategy ignored 75% of relevant papers, and that it changed the disciplinary distribution. What Richard Tol would need to do would be two different things, neither of which should be too onerous. First, randomly select a sample of papers from his larger sample and then apply the same study criteria as applied by Cook et al. One could then assess if the larger sample was likely to produce a different result. The next thing would be to do the same – but individually – for some of the different disciplines. This would indicate if there was a discipline dependence.
If Richard Tol did this and showed that the result were quite different, then maybe his argument would have some merit. But to say that the “sampling strategy is a load of nonsense” simply because he can get a different sample by doing a different search and that his different search produces a different disciplinary distribution doesn’t really have much merit without a study to show how the results are affected by these differences. One of the most extreme variations between the two searches considered seem to be for his own papers. The Cook et al. search produced 10 papers by Richard Tol, while Richard Tol’s search produces almost 90 of his own papers. Given that Richard Tol is actually an economist, that only a few of his papers are relevant to a study of whether or not climate science papers endorse AGW or not, seems quite reasonable to me.