Last month I wrote a response to a paper written by Dr. Andrew Lawrence, Dr. Benjamin Sovacool, and Dr. Andrew Stirling. If you haven’t read that, here is a link to that analysis and response. Since then, I have been in contact with some of the authors, and last week I received word that the authors have decided to withdraw their paper, partially based on my analysis of their paper. The full situation has also been written about in Retraction Watch.
Dr. Lawrence, Dr. Sovacool, and Dr. Stirling have written an “Authorial Statement of Article Withdrawal” and have also written a response to my analysis. It should be noted that throughout this process, Dr. Sovacool has been very open and accessible, and I’d like to thank him and the other authors for taking the time to review their work, and admit when a mistake has been made. Here is their response.
Response to Nicholas Thompson
We thank Nicholas Thompson (NT) for his serious engagement with our paper. Below, we respond to the specific issues raised. We also refer to the statement of retraction posted in the journal Climate Policy. The bottom line is that, NT has in his main criticism done an elegant job of deconstructive analysis. We commend and congratulate him on it.
Problem 1: The math is incorrect.
Unfortunately, this is a valid criticism, and on two counts. Very embarrassingly, mistakes were made both in data input and analysis. One column of data was mistranscribed. In addition, some percentage changes were not calculated correctly with respect to the stated baseline but with respect to an earlier date. When these mistakes are corrected, NT is correct that these particular data do indeed show that nuclear states did make the greatest absolute emissions reductions compared to all other groups. These errors should never have happened. A more detailed background is given in the statement of retraction. We are proactively correcting reports.
We also plan on undertaking a new analysis with the correctly transcribed data, to be completed and shared soon.
Problem 2: The method is incorrect and Problem 5 no attempt to account for population.
As acknowledged above, the method was unfortunately implemented in a flawed way. But, contrary to NT’s assertion, it is not self-evident that the method itself was intrinsically deficient. Based as it is on comparisons across qualitatively different institutional and jurisdictional settings, it is not clear that any single given weighting factor (e.g.: per capita emissions; emissions per GDP; or emissions attributable to specific policies) would of itself necessarily constitute an unequivocal improvement in the analysis. One specific dimension of variation could be made more explicit, but only at the expense of obfuscating others. The purpose of this analysis was simply to perform an initial test on a widely received assumption. The possibility of further elaborations does not invalidate the initial design.
Problem 3: There is no justification for the starting year of analysis.
The decision to use 2005 as the baseline year was made in recognition of the highly dynamic nature of change in this field. To use an earlier year would have diluted the effects of more recent trends (and been unfavourable to the stated findings). To use a more recent year would have emphasized the most recent developments (and been more favourable to the stated findings). So the chosen framing is not arbitrary or contrived. It is rather that a more complete analysis would systematically display sensitivities to these divergent framings.
Problem 4: Statistical significance.
At no stage was our analysis intended or claimed to rest on statistical arguments. As this present exchange illustrates, the underlying categories for any analysis of this kind are too ambiguous and volatile to warrant undue confidence in any particular aggregated statistical constructs. To assert this would be misleading. Instead, our contribution was intended to challenge a widespread assumption about the supposed climate benefits of nuclear power. Again, we must acknowledge that the test was in this case poorly implemented.
In summary, NT is correct that our article contains flaws that invalidate the analysis. This is by far the most important issue in this exchange. On other aspects of NTs critique, the position is more ambivalent. But it is certainly the case that analysis of this kind can be improved in ways prompted by NTs critique and discussed above, in order to yield more robust conclusions.
With respect to any implication of intentional bias towards renewable energy at the expense of nuclear power, we should like to stress that the errors described above were the result of honest mistakes. We hope that our unqualified acknowledgement of these errors here – and our proactive voluntary retraction of the article – insufficient to show that we are entirely open both to criticism and to different perspectives.
I do want to push back on a few things in this response:
1) As my original analysis showed, this methodology is somewhat arbitrary, and depending upon the time window that is chosen, will lead to different conclusions. Additionally, giving equal weight to extremely small and large countries can skew the data in different ways. If there is a new analysis down with the same methodology, I have concerns that it will fall prey to the same issues that came up in this paper.
But more importantly,
2) The authors don’t seem to deny that they have a bias against nuclear energy. I completely believe that any mistakes that were made in this paper were honest mistakes. But in this response, they state that their, “…contribution was intended to challenge a widespread assumption about the supposed climate benefits of nuclear power.”
In this sentence, the authors seem to be saying that it is their belief that nuclear does not have climate benefits, or at least that was what they were trying to prove.
Most Life Cycle Analyses (LCA) of nuclear energy show that nuclear energy has relatively low life cycle emissions of greenhouse gases (meaning that over the life of that facility, from mining the materials, building the facility, operating the facility, to decommissioning it and disposition of waste, the total carbon emitted per energy produced is low). One of the most comprehensive studies of this was done by the National Renewables Energy Laboratory in the US, where hundreds of these LCA studies were reviewed, and the results were that nuclear emits about the same amount of greenhouse gases over its life, per power produced, as wind power does (median values for nuclear were 12-13 g CO2eq/kWh, wind was 11-12 g CO2eq/kWh, solar PV was 44 g CO2eq/kWh – for comparison, coal was 979 gCO2eq/kWh).
Nuclear energy, when done well, does produce low carbon energy – this isn’t an assumption, and the climate benefits are real, not “supposed”.
There is some great social science research about how people’s beliefs can actually impact their quantitative reasoning, and this even effects people who are very skilled at math (it can actually effect them worse!). This result was found by Dan Kahan, Ellen Peters, Erica Dawson, and Paul Slovic in 2013 (for a simple explanation, go here). This is why I completely believe these were honest mistakes in the Lawrence, Sovacool, and Stirling paper – because the authors of the paper probably found a result that seemed to reinforce an idea they had (that nuclear does not have climate benefits), and did not question the results.
All scientists should start any project by looking for an answer to a question, and not looking for information that backs up a previous belief.
9 thoughts on “Response from Lawrence, Sovacool, and Stirling and Statement of Article Withdrawal”
In my opinion, their retraction statement at the journal fails to properly acknowledge your contribution. They say “This mistake had also been identified in a blog, but had remained unnoticed by BS and AS until that point.” They ought to have cited your blog posts. In addition, their statement implies that they had independently noticed the error, when it is evident that you had noticed it first.
LikeLiked by 1 person
They noticed my blog straight away. Possibly because I told the journal editors. I can’t see how they could’ve missed Nic’s blog.
The fact that this paper made it past “peer review” and had to be deconstructed by independent analysis like Mr. Thompson demonstrates the review system be, at best, an utter failure, or, more, likely, worse – – infected with bias against nuclear energy such that any paper that “feels” like it must be right because its conclusions slander nuclear energy is given, at best, a lazy review.
LikeLiked by 1 person
In my experience, people who do peer review are looking for methodological errors and errors in logic. It’s pretty rare that a reviewer would actually take the time to sift through the numbers/data. This doesn’t mean that peer review is broken, although it could have been better this time. Part of the scientific process is reviewing and critiquing works after they have been published, which is what happened here.
LikeLiked by 2 people
If only I could believe that. IMHO: Their EU source data is in use all over the place, in energy discussions. Everyone fit to peer review the paper would’ve known SSL’s calculations were wrong. I knew immediately. On publication, I assumed they must’ve used some bizarre calculation, or, had even, cherry picked from some data unavailable to me. I, personally, only publish numeric results on my blog when two independent calculations give the same result. Something I learnt in secondary school. What they did not do.
Their anti-nuclear power bias caught them out. Bias closes one’s eyes to the truth: you question the numbers you disagree with but instantly accept numbers you’re biased in favour of. The editors should’ve given their paper a more rigorous peer review. I have to assume the peer reviewers were all biased against nuclear power too. Nothing else make sense.
Pingback: Advocacy by Fact-Checking. This can work | Meredith Angwin | Author & Nuclear Energy Advocacy Consultant
Aw, this was an incredibly good post. Taking the time and actual effort to
create a great article… but what can I say… I put
things off a whole lot and don’t manage to get anything done.
Are you familiar with other critiques on another blog of Sovacool’s earlier work (which probably should also have been retracted)?
LikeLiked by 1 person
Pingback: Advocacy by Fact-Checking. This can work. | Meredith Angwin