There is another post over at Watts Up With That (WUWT) claiming that John Cook’s consensus paper is falling apart. They seem rather fixated by this paper, which is interesting in itself. Anyway, it’s based on a post on Bishop Hill called landmark consensus study is incomplete.
The Bishop Hill post, by Shub Niggurath, says
A Web of Science search performed following the authors’ description to the letter actually returns 30,940 entries, not 12,464. Excluding the ‘Arts and Humanities Citation Index’ (A&HCI), this becomes 30,876. This is when search phrases are not enclosed in double-quotes (i.e., ‘global warming’ instead of “global warming”).
Well, if you do use double-quotes instead of single-quotes and restrict the search to articles only, you find 12,574 articles – essentially the same as Cook et al. Well, that explains why they got a different result to Cook et al., and rather disproves Shub Niggurath’s claim that he followed the authors’ description to the letter.
However, have they found that Cook et al. missed a lot of relevant papers. I don’t think so. If I do a search using single-quotes it returns results for ‘climate change’ and ‘global’, whereas using double-quotes restricts it to results for “global warming” and “global climate change”. So, basically the author of the Bishop Hill post has shown that if you do a topic search different to that of Cook et al., you get a different number of articles. They haven’t really shown that Cook et al. missed a lot of relevant papers. However, even if they have, it doesn’t really matter. Even if there are 30000 relevant articles, rather than 12474, it doesn’t invalidate the study. What they used was a large fraction of the relevant articles and hence would, typically, be regarded as a representative sample. The uncertainty in the result might increase slightly, but not enough to make the consensus particularly uncertain.
Wotts Up With That Blog 
“So, basically the author of the Bishop Hill post has shown that if you do a topic search different to that of Cook et al., you get a different number of articles.”
That is clearly not correct.
Cook and co-authors restricted their search to just one database of the possible 6 databases, SCIE, SSCI, A&HI, MEDLINE, INSCAPE and SCOPUS. They did not mention this in their paper.
Cook has no doubt missed a lot of relevant papers and there is no way of knowing what these papers are and what impact they might have had on his percentage calculations and assessments. This has taken place because he chose “global warming” and “global climate change” as keywords. This is unavoidable as you need to settle on a few search phrases, which means in any combination, there will be papers that are relevant but missed (for lack of the searched phrases)
What’s important however is not relevance, but application the very described search strategy. Cook defined their search phrases and conditions, but failed to apply the search to an appropriate database. Which means the search outcome, though it produces a large number of results, is not representative or comprehensive. There are thousands of papers out there which fulfill the exact criteria used by Cook et al, but not included in the paper.
Thanks for commenting. I do appreciate you coming here to discuss this.
Let’s address this step-by-step. You say that they restricted their search to one database out of 6 and that they do not mention this in the paper. The paper says
To me, that’s fairly clear. They’ve use ISI Web of Science which – I believe – is a database. Maybe you can clarify what you meant by them not mentioning the database in the paper.
As an aside, I will grant you that in the paper they use single-quotes for the search terms and to match their results one needs to use double-quotes. I think this is being pedantic. Single-quotes do not restrict the search as clearly as double-quotes (in ISI WoK) and so this seems to be simply a typographical difference, rather than anything major.
You go on to say that Cook has clearly missed a lot of relevant papers and there is no way of knowing what these papers are and what impact they might have had on his results. Firstly, I would say that one can know because one can replicate his search to find the papers he used (which I’ve done) and hence could determine which papers he’s missed. Secondly, if the papers he missed were to change his result that would imply that a search using “global warming” and “global climate change” somehow produces papers with a different bias to a different search (for example, ‘global warming’ and ‘global climate change’). Possible I guess, but why? Why would the topic of papers that endorse AGW (for example) be different to the topic of papers that don’t. It’s really hard to see how the 12474 papers that they used in the Cook et al. survey isn’t a reasonably good representative sample of the relevant papers. It’s clearly a big fraction of all possible papers and I can’t see why their search term would have preferentially produced papers that endorse AGW when compared to some other search term.
Again, I think your final point is incorrect. I will grant you that the paper indicates that the search terms had single-quotes and that to replicate the survey sample you need to use double-quotes. However, it is clear that using single-quotes versus double-quotes in ISI Web of Knowledge produces a different set of results. Using double-quotes restricts it to papers that satisfy the topics of “global warming” and “global climate change” while single quotes returns papers with the ‘climate’, ‘climate change’, ‘global’ and ‘warming’ (according to my quick check of the sample). Hence using “double-quotes” restricts it to a sample that is more likely to be relevant than using ‘single-quotes’. In fact, I’ve just redone the search on WoK and using single-quotes produces the same result as using no quotes at all. Hence using single quotes produces papers that satisfy the topics of ‘global’, ‘warming’, ‘climate’ and ‘change’ and hence has many papers that would not be relevant. Consequently, I would suggest that your final statement There are thousands of papers out there which fulfill the exact criteria used by Cook et al, but not included in the paper. is essentially wrong. The Cook et al. study may well have missed some relevant papers, but these are not simply all the papers that your alternative search returned as that search returns many papers that clearly aren’t relevant.
Would be interested in what you now think. It seems obvious that you have simply done a different search to that done by the Cook et al. team, and it seems that your search clearly produces papers that are not about “global warming” or “global climate change”. Do you disagree with this?
I should correct something about my comment. It seems that if you do a topic search using ‘global warming’ or ‘global climate change’ on ISI Web of Knowledge, it returns papers with the words ‘global’, ‘climate change’, ‘climate’, and ‘global warming’ highlighted. However, my point stands that it clearly does not restrict it only to those on the topics of “global warming” and “global climate change”.
Marcott et al. all over again. Something is published that the fake sceptics don’t like, so they create various straw man arguments and an astonishing amount of fake fuss.
After a while, the *impression* is created that there is some sort of “problem” and the fake sceptics have achieved their aim. They have calmed and reassured their dupes.
Well, that would be my general impression. I’m interested to see if Shub responds to my comment. It seems clear that his choice of search terms produces more results simply because it is less restrictive than that used by Cook et al. and, consequently, produces results that – in my opinion – are not relevant.
At the end of Shub’s post it says
I take this to mean that on Scopus, Richard Tol found many more records than found by Cook et al. using ISI Web of Knowledge (I’m happy to be corrected if this isn’t what is meant by this final credit). I’ve just gone to Scopus, used the following search term “global warming” or “global climate change”, restricted my search to the Physical Sciences only and refined it to only include articles. The search finds 13127 articles that satisfy the search criteria and that were published between 1991 and 2011. Doesn’t seem particularly different to that found by Cook et al. using ISI Web of Knowledge.
This is what makes the fake sceptic’s favoured tactic effective. You ignore them and they howl that you are refusing to acknowledge the legitimacy of their claims. Engage and you amplify and protract the fake controversy exactly as they wish you to.
BTW I wouldn’t take Shub too seriously. As a veteran of many long exchanges with him I can confirm that his views on physical climatology place him way out on the lunatic fringe.
wuwtb
Thanks.
First, you state: “To me, that’s fairly clear. They’ve use ISI Web of Science which – I believe – is a database. Maybe you can clarify what you meant by them not mentioning the database in the paper.”
The authors say they used ISI Web of Science in the paper. One of the authors Dana Nutticelli has now stated on Twitter what they actually used was Science Citation Index, which is a subset of the Web of Science. This is not stated in the paper. SCI or SCI-Expanded is different from Web of Science.
The authors’ failure to describe the exact search phrase, vis a vis their use of singe-quotes, is certainly a minor error. However it is not an insignificant error. Using single quotes as they described returns results vastly different from when double-quotes are used. Search for specific phrases should be enclosed in double-quotes or braces ‘{}’. I think we can both agree that enclosing search terms in double-quotes produces the most appropriate results.
If one uses search terms enclosed in double quotes, restricts to ‘articles’ in English, in the years 1991 – 2011, published only in journals, in Web of Knowledge – one gets ~21,488 records.
If one uses search terms enclosed in double quotes, restricts to ‘articles’ in English, in the years 1991 – 2011, published only in journals, in Scopus- one gets 19,417 records.
If one uses search terms enclosed in double quotes, restricts to ‘articles’ in English, in the years 1991 – 2011, published only in journals, in Web of Science – one gets ~13,563 records.
There may be several papers out there that are very much relevant to question of anthropogenic global warming and its consensus, but yet are not trapped by a search with the chosen phrases. This cannot be helped and it is completely understandable. You choose a search term or a set of search terms and implement them consistently. On the other hand, the authors’ chosen phrases were “global warming” and “global climate change”. *But they failed to search for these phrases in the full complement of academic literature databases that are available.* Nowhere do they state that they restricted themselves to ‘Physical Sciences’.
What is the impact of this failure? The authors miss 70-100% more papers that are present in the literature, that contain the authors’ own chosen key-phrases.
You can try this yourself: search Scopus for “global warming” OR “global climate change” for articles only, in the years 1991 to 2001, published in the English language. Use the “Article Title, Abstract, Keywords” option which is the equivalent of a WoS/WoK ‘Topic’ Search (if you have any doubts, you can help files from both sources to confirm). You should get close to 20,000 records.
… and there’s still virtual unanimity amongst Earth System scientists that AGW is a reality and its effects will become ever-more pronounced as the atmospheric fraction of CO2 increases.
So let’s talk up a fake controversy over Cook et al. instead! All day and all night, for weeks and weeks!
Yay!
Okay, so it really does seem that one of the things that you’re criticising is that they haven’t made it clear that they’ve restricted their search to the physical sciences. Okay, maybe but it seems like a reasonable restriction and they did refer to the database as Web of Science.
Furthermore, if I do restrict it to the physical sciences, in Web of Knowledge this returns (if you use double-brackets and restrict it to articles only) 12547 results and in SCOPUS – if I restrict it to keywords – it returns 13157 results. If I remove the restriction to it being physical sciences only, WoK returns 13761 articles and SCOPUS returns 14748 articles.
Okay, so if I change the option in SCOPUS to be “Article Title, Abstract, Keyword” it does indeed increase to 18725. A bit odd, but fair enough. Maybe there are 6000 other papers out there that could be regarded as relevant but it’s hard to see how this would influence the outcome unless you’re suggesting that for some reason those that don’t overlap in the two searches are fundamentally different from those that do. I don’t believe Cook et al. claimed that it was a complete sample. They’ve defined their parameters (okay, maybe they should have clarified the physical sciences thing, but I worked that out in a couple of minutes and it is a reasonably obvious choice) and defined their working. You might disagree with the result (although not quite sure why) but to claim that the methodology is flawed seems rather disingenuous.
Cook et al (a) do not provide a clear description of what exact search was carried out. The fact that you had to tweak and fiddle around with the different options to obtain a result close to theirs proves this. A colleague of mine was able to reproduce Cook’s numbers by searching for ‘Abstracts’ instead of the ‘Article Title, Abstracts, Keywords’ field. The issue is not whether users can do something or the other to recreate the paper’s results, but rather whether the search methodology captures a self-contained, openly described body of literature that fulfills a pre-determined set of uniform criteria relevant to identifying a putative consensus in the climate field. This is not the case.
(b) Secondly, Cook and co-authors lay great stock in their paper being a ‘comprehensive’ search of all literature. If one follows their paper, one gains the distinct impression that they captured every possible paper that fits the search phrases. Lewandowsky, mistaking Web of Science for Web of Knowledge, claims that the authors included (every single possible paper with “global warming” and “global climate change”. Again, in making these claims, the authors and their supporters are entirely mistaken.
Thirdly, do not mistake the above to mean that I intend to distract from the amount of work that is required to implement a project of this type. It does take a lot of work and that has been done. However, the authors (i) kept their definition of ‘consensus’ purposefully vague, and broad, so it became relatively easy to somewhat mindlessly push papers into the ‘endorse’ category. (ii) kept their search restrictive owing to an inconsistent and incomplete application of the search algorithm. These considerably weaken the paper. There are other powerful objections to exercises of such kind, both methodological and otherwise, but I don’t intend to get into them here.
Essentially, I disagree. In a matter of minutes I had replicated their sample. Didn’t take much and maybe you can nit-pick and claim that they should have been more precise, but it wasn’t difficult to work it out.
As far as I can tell, they essentially do find every paper in Web of Knowledge that does satisfy their search criteria. You can find 6000 more if you do an “abstract, title and keyword” search in SCOPUS, but they didn’t use SCOPUS, they used WoK. Therefore, their search is pretty comprehensive. This SCOPUS results is also rather odd. SCOPUS is newer than WoK and, certainly, in my field is regarded as less reliable than WoK. Maybe there really are 6000 papers on “global warming” or “global climate change” that WoK ignores, but maybe SCOPUS is identifying papers in a way that is different to WoK.
I essentially disagree with your last paragraph. They define their criteria quite well. Yes, the assessor will have to make a judgement, but that’s the nature of this process.
Here’s a question for you, though. A typical theme on skeptic sites is that the existence of a consensus in climate science is in itself an indicator of a problem with the science. The basic idea is that science doesn’t work via consensus building, hence that climate science has a consensus means that they are approaching the science incorrectly. I think this argument is nonsense and completely misunderstand what scientists as a consensus in science. However, it does indicate that even skeptics accept that there is a general consensus within the climate science community that global warming is anthropogenic (at least at some level). Now there is a paper that indicates that a study of the literature confirms that such a consensus exists (to be more precise – that a large majority of papers that state a view about AGW, endorse AGW). Now all of a sudden, those who are skeptical of climate science claim the paper is flawed. Why? Surely this paper is entirely consistent with the general skeptical view of climate science. The rhetoric about this paper amongst skeptics just makes them seem contrary and lacking in objectivity. It makes it seem like they will attempt to discredit anything that makes their views seem less credible. Do you agree with and can you explain this? If skeptics genuinely believe that a consensus makes a scientific area seem less credible, surely they should embrace this paper and then use it to show why this means that climate science has some fundamental flaws.
[1] I replicated their results in a matter of minutes too. What I did to replicate the results (a) does not match what the authors say they did (b) does not represent what the authors claim it does. We can disagree but it changes little. From your description, (a) is true in your case as well. Additionally, you may be very satisfied that their search is comprehensive. Having seen that thousands of papers are excluded, a number almost as many that are included, I don’t think it is comprehensive.
[2] The issue I addressed was how the ‘consensus’ was determined in the Cook paper. Whether a ‘consensus’ of any kind is a valuable and/or an useful thing in a scientific discipline is a larger question in the background. You are correct that ‘consensus’ in scientific fields may always not be a good thing (and that ‘skeptics’ believe this to be true). If I were take this view to slam climate science (as you assume skeptics are apt to do), I could employ the Cook paper to do it.
But in order to imply the overwhelming consensus as shown by Cook to mean something bad, I must believe his results are valid first. What I find, rather, is another example of poor science, done badly, described sketchily and defended in a worse manner. Surely, many of the problems with the paper are simply inherent in the nature of the project itself, i.e., trying to discern some form of agreement in tens of thousands of papers. I appreciate the Cook team’s efforts. I appreciate Cook’s efforts in getting so many people to work productively (for what its worth) toward a single goal. But, in their over-enthusiasm, they have drawn strange, headline-friendly inferences (like 97%) which demean the underlying hard work of the volunteers. Nor do they seem to have the necessary chops to perform a more rigourous and water-proof analysis that would withstand scrutiny.
It is my own belief, contrary to my supposed instincts of harming the interests of climate science, that a greater diversity of opinion exists amongst climate scientists. There are explicitly conducted surveys in the climate area by von Storch, Dennis and Bray, Annan and others which paint a different picture. The studies that portray an overwhelming consensus – Doran, and now Cook – have faulty methodologies.
It’s your last paragraph that illustrates the problem. Yes, I’m sure there are a great diversity of opinion amongst climate scientists. Noone is claiming that climate science is absolutely settled and that we understand everything. However, it is my view that the disagreement amongst climate scientists about the basics of anthropogenic global warming (AGW) is minimal. I suspect that most agree that the Earth is warming (and I mean gaining energy rather than simply having rising global surface temperatures) and I suspect that a significant fraction agree that this warming is driven by increasing levels of CO2 in the atmosphere which is present as a consequence of our use of fossil fuels.
The exact consequences of this warming is clearly not absolutely clear and there is clearly much more work to be done. The Cook et al. survey was based (as far as I can tell) on determining the “consensus” with regards to the basics of AGW, not to determine that all of climate science in this area is settled. In some sense, such a study shouldn’t really be necessary. It shouldn’t matter. In climate science it seems that it does because there are numerous people making claims about climate science and about the views of climate scientists based on nothing other than their own opinions. The Cook et al. study is at least an attempt to quantify the level of consensus in that science.
You may disagree with the methodology and how they presented the work. I think you’re nit-picking and if this is how you think we should assess bits of research, you could probably criticise and ignore a significant fraction of what has taken place in the last 50 years. I also think you need to be careful about making claims about consensus given that this study now exists. If this is regarded as important (and given the level of criticism it has attracted it seems that it is seen as important by some) it seems that someone else should redo this in what you think is the correct fashion so as to show where Cook et al. went wrong.
I think that last part of the comment indeed shows the problem: Shub “believes”, and whatever does not fit his “beliefs” must use faulty methodology. It’s odd he wishes to point to Bray & Von Storch’s survey, which contained many multiply-interpretable questions as well as leading questions. Also, their last survey had 373 responses, or less than 20% of those contacted. The self-selection bias in surveys is well-known, so the results may well be skewed (the name “von Storch” does not go well everywhere, even before 2006). The same goes for the Brown, Pielke, Annan survey, where the number of respondents and response rate was even lower (140, less than 8%) – here at least the questions were less multiply-interpretable. However, one also needs to interpret one category in a rather specific way to claim that that survey “paint[s] a different picture”. About 7% of respondents fitted in categories 1 to 3.5, which state anthropogenic forcings are small/it’s all a hoax). You’d have to add the one’s in category 4 to get a different picture (another 10%). But what is category 4?
“There is warming and the human addition of CO2 causes some of it, but the science is too uncertain to be confident about current attributions of the precise role of CO2 with respect to other climate forcings. The IPCC WG1 overestimates the role of CO2 relative to other forcings, including a diverse variety of human climate forcings.”
That category would include quite a few that *do* think the current climate change is mainly anthropogenic, but who are not so sure about the role of CO2. They may, for example, think methane is more important than stated in the IPCC report.
Note that the Brown et al survey also had about 17% claiming the IPCC underestimates what is going on, with 9% even making it a *serious* underestimation.
wuwtb,
There should be no problem with my last paragraph. Me making up my own stuff about scientific abstracts and inferring what they mean is definitely worse than asking the question directly to the scientists concerned. It is also funny you say “not that this study exists.” Why are you so eager to swallow its conclusions? What if I demonstrate that it has major problems? We’ve talked about just one issue in this thread. In fact, the Cook team directly admit that asking scientists directly what they feel is the best way to obtain opinions.
Marco
As far as sampling goes, the Cook survey is actually the worst. In terms of papers, the overall rate of explicit acceptance and quantification is 0.54%. The rate of explicit acceptance (with or without quantification) is 8.3%. In terms of scientists, their attempted coverage was ~30%, and the actual response rate was a meagre 4.1% overall, i.e., 14% of coverage.
I think you misunderstand my issue. I’m not swallowing the conclusions of this study. What I was suggesting by my comment regarding the existence of this study is as follows. Consider a particular study. As someone who is considering the work presented you have a number of options. You can accept that they’ve done the work as described and accept the results in context (by which I mean no single study will ever be definitive). You could look closely at what they’ve done and possibly find some flaws. If these flaws are minor you could accept that these flaws probably don’t influence the ultimate conclusions. If you regard the flaws as major, you could dismiss the study as being inconclusive or disregard it. What you can’t do – without repeating the study – is assume that some alternative is now more likely. You’ve pointed out what you think are some issues. I think you’re nit-picking and nothing you’ve highlighted is sufficient to make me think that there is anything fundamentally wrong with what they’ve presented.
You mention that asking scientists would be best. Maybe so, but that’s in some sense irrelevant. They didn’t. I’m slightly worried that we’re just going in circles and will get nowhere with this. Let me try one thing though and see what your response is. Here’s my summary of the paper.
The used Web of Science to find all articles in the physical sciences published between 1991 and 2011 on “global warming” and “global climate change”. Their search returned 12464 articles. Agreed?
The abstracts of all of these articles was assessed by 2 of 24 volunteers to determine if it satisfied one of 8 possible conditions related to AGW and ranging from “explicit endorsement with quantification” to “explicit rejection with quantification”. The paper describes each category and gives an example for each category.
Their results indicate that (based on the abstracts) 66.4% had no position on AGW, 32.6% endorsed AGW, 0.7% rejected AGW, and 0.3% were uncertain. Overall, therefore, 97.1% of the abstracts that stated a position on AGW endorsed it and 2.9% rejected or was uncertain about AGW (or at least, this is how they were assessed by the volunteers).
Do you agree with that general summary.
Here’s my summary of your issues.
1. They didn’t explicitly state that the search was restricted to the physical sciences.
Okay, that may be true but it was a study of papers published by climate scientists. It might have been good if they had stated this, but it is fairly obvious.
2. You can find 6000 extra papers if you use a “keyword, title, and abstract” search using SCOPUS.
Fair enough, but they didn’t use SCOPUS – they used WoK. Also, their sample appears to at worst be 2/3 of all suitable papers. To suggest that the result might have been different had they used the larger sample seems absurd.
3. It would be better to have asked the scientists directly.
Possibly true, but it isn’t what they did. It is generally disingenuous – in my opinion – to criticise a study because they didn’t do it in a way that you thought would be better (or rather to assume that not doing it the way you thought would be better suddenly invalidates the study). There are probably always going to be better ways of doing something. However, people design studies in certain ways for various reasons. Also, as far as I’m aware most studies that have tried to ask scientists directly get very poor response rates and so it is not clear that asking the scientists is actually likely to produce a particularly large sample.
As far as your response to Marco is concerned, I have no idea what those numbers mean. Could you clarify?
Most papers do not address the question “AGW, yes or no?” and as such should not be taken into account. It’s like asking the question “Will you vote for Obama or Romney” and then calculate the percentages by including those who state they will not vote at all.
Regarding coverage: a 16% response rate (about 2100 papers were rated by the authors) is about the same response rate as Bray & Von Storch, and twice as large as the Brown et al survey. But numberwise we’re talking about 4* higher compared to Bray & von Storch, and almost 10* compared to Brown et al. Thus, sample wise it is much bigger than either of the other two surveys.
Finally, your “4%” appears to be based on the total number of authors on all papers combined. But they never asked all authors, but mostly the 8500 first/corresponding authors (which should make sense to anyone who has ever done research). Of those 14% responded. Thus, the response rate is 14%, not 4%. Perhaps you switched the two numbers around, but a response rate of 14% is clearly not bad (see Bray & Von Storch and Brown et al, with the latter not even managing 8% response rate). With 4% coverage they still did vastly better than Bray & Von Storch and Brown et al.
In my opinion you will always consider something a “major problem”, as long as the outcome of a study does not fit your preconceived ideas.
Shub writes: “What if I demonstrate that it has major problems?”
~ ~ ~
Shub have you subjected your “demonstration of major problems” to the experts in the field?
You do agree it’s important to present any serious argument to the critique of the expert’s skeptical eye…
{rather than the merely the admiration of uncritical under-educated partisan spectators} . . .
don’t you?
No not blog posts – Have you written any papers that present your scientific complaints and your supporting data/arguments to scientists in a coherent constructive manner?
Please share links or citations.
thank you
A bit late to this party, I just noticed this claim from Shub.
“One of the authors Dana Nutticelli has now stated on Twitter what they actually used was Science Citation Index, which is a subset of the Web of Science.”
First of all, the name is Nuccitelli. Second, that is a false claim – I said no such thing. John Cook did the search and I wasn’t watching over his shoulder (his shoulder being in another hemisphere). I have seen him provide instructions how to replicate his search, but I’ve never paid much attention because honestly this is a really stupid argument. Different searches yield different results. Unless you can come up with an argument why our search was somehow unrepresentative, your argument has no merit.
Pingback: Watt about the 97% consensus, again? | Wotts Up With That Blog