So I’ve had a look through the second draft of Richard Tol’s paper Qunatifying the consensus on anthropogenic global warming in the literature:a comment. It doesn’t seem to have changed much, so I stand by much of what I said in my earlier post, Watt about Richard Tol?. It seems unfortunate that Richard has chosen to comment on the paper in the manner that he has. It’s very adversarial and quite insulting to the authors of the Cook et al. survey. Referring to authors of a paper as being “incompetent” in discussions with colleagues is one things. Doing so in the academic literature (at least explicitly) is quite another.
One thing that Richard does pick up on is that the assessments of the abstracts by the volunteers in the Cook et al. survey differed to the author self-assessments. The table below is from Cook et al. (2013) and shows the abstract ratings and author self-ratings for the 2142 articles that were assessed both by the volunteers and by the authors. It seems clear from the table that (unless something very odd happened) an abstract assessed to endorse AGW by the volunteers would be assessed by its authors to endorse AGW. It does seem, however, that there was a high chance that the volunteers might assess an abstract to be uncertain or to have no position, when the authors would regard the paper as endorsing AGW. There were somewhat more articles that were assessed by the authors to reject AGW than determined from the abstracts by the volunteers, but it is still a small number when compared to the number that endorse.
Is this discrepancy an issue? Well, it’s not necessarily that surprising. The Cook et al. survey explicitly used only the abstracts and it seems quite likely that there will be articles that don’t state a position on AGW in the abstract, but do so in the body of the paper. Also, the goal wasn’t to determine if one could use an abstract to determine the position of an individual paper with respect to AGW; it was to see if abstracts could be used to quantify the consensus in the literature with regards to AGW. What would be concerning would be if a large fraction of the abstracts assessed to have no position on AGW actually rejected AGW. The table above illustrates that this is clearly not the case. Although a large fraction of the abstracts assessed to have no position or to be uncertain about AGW actually (according to the authors) endorse AGW, very few reject AGW.
Essentially what Cook et al. find is that if one assesses the abstracts only (using volunteers) 97.1% of those that take a position on AGW, endorse AGW. If one ask authors to self-rate their papers then 97.2% of the papers that took a position on AGW, endorse AGW. The results are consistent and this indicates, to me at least, that one can use the abstracts to determine the level of consensus in the literature with regards to AGW. Richard Tol’s second draft still finishes with an acknowledgement that the result is probably correct but that the Cook et al. paper is “flawed” and the results “unfounded”. So, they get consistent results whether or not they use volunteer ratings of the abstracts or author self-ratings and the result is consistent with what Richard Tol would expect and with earlier work. This, however, is still not good enough for Richard who thinks they should have done it differently, but can’t be bothered – or doesn’t have the time – to do it himself.
Another thing that Richard Tol complains about is that the “authors choose not to share the full data”. Firstly, the Cook et al. paper is only about a week old. Maybe they have explicitly refused, but maybe Richard just needs to give them some time. There is another issue though. A reasonable expectation of any study is reproducibility. This, as a minimum, would tend to require access to the raw data and access to any information required so as to analyse the raw data. For the Cook et al. study, this is clearly available. The search terms, search restrictions and database are known. The categories for assessing the abstracts are also known. Clearly, this study could be reproduced. What Richard Tol clearly wants is all the data from their analysis. Is this reasonable? Maybe, but if you’ve just committed a great deal of time, effort and money into a project, should you be expected to simply hand all your work over to someone else? It’s not obviously a reasonable expectation. If it is a reasonable expectation, it might also be reasonable that the person asking for the data be someone who plans to assess it objectively and in an unbiased way. Given that Richard Tol has already written two drafts of a paper claiming that the study is “flawed”, the conclusions “unfounded” and that the authors of the Cook et al. paper are “incompetent”, this seems unlikely. Why should Cook et al. be expected to hand all their work over to someone who is more than likely simply going to use it to confirm their existing biases.