In the wake of a few high-profile errors found in the most recent Intergovernmental Panel on Climate Change assessment report, the organization asked the InterAcademy Council, a coalition of national science organizations, to examine its structure and procedures in order to identify potential weaknesses. The IAC's report came in today, and it more or less indicates that the IPCC has been a victim of its own success. Because so many people, from policy makers to critics, pay attention to the IPCC's reports, the IAC suggests that fundamental reforms are needed to improve the transparency and rigor of the organization.

The IPCC's troubles began with a disclosure that one of the sections in its massive Fourth Assessment Report, the Summary for Policymakers, contained some inaccurate information regarding the likely fate of Himalayan glaciers, suggesting they were melting at an unrealistic rate. That seems to have opened the floodgates, and a variety of claimed inaccuracies (some spurious) were published, as was an attack on the group's chairman, Rajendra Pachauri. These issues helped prompt the IPCC to request an evaluation of its organization and process. For that, it turned to the IAC, which has a membership that includes the national science academies of many nations, including the US' National Academy of Sciences and UK's Royal Society.

Princeton's Hal Shapiro, who chaired the IAC's review committee, introduced the report at a press conference in New York earlier today. He repeatedly made it clear (often to the disappointment of some reporters present) that the committee's task wasn't to re-review the science present in the IPCC report; instead, the focus was on understanding how the IPCC's own review process could be improved, and how the organization could maintain its credibility in the face of both justifiable scrutiny and spurious accusations.

Shapiro praised the organization, saying, "overall, it has served the world well." But its success has resulted in public scrutiny that the IPCC's original structure, which has remained intact through four major reports, simply isn't capable of handling. To provide fast, year-round responses to potential errors, the report calls for the IPCC to put an executive committee in place, headed by an executive director chosen from among the group's senior scientists. To prevent any of the executive committee members from being seen as the face of the IPCC (much as Pachauri is now), the IAC recommends that all of its senior staff serve in those positions for a single assessment.

That sort of approach should extend throughout the organization. The IPCC currently has no rules for handling conflicts of interest or guidelines for determining who within the organization can claim to speak on its behalf. The committee urges that policies for both of these issues be put in place in time for the next IPCC assessment report.

The same goes for the group's review process. The most recent report, AR4, drew roughly 90,000 comments, and Shapiro said that many of these were more editorial than critiques of the science. To manage this flood of critiques, the report recommends that the IPCC "adopt a more targeted and effective process for responding to reviewer comments." This would involve having the Review Editors sift through the flood of input for the valid and significant points. This would do away with the current policy, which requires the authors of a given section to respond to all comments that are submitted.

The final area that could use reform is how uncertainty used in the report is handled. Shapiro highlighted how, in the summary for policymakers, the authors expressed high confidence in predictions that were later found to be erroneous, and in a few cases confidence levels were assigned to statements that he termed "unfalsifiable." The report calls for the use of quantitative probabilities only when the evidence is very well understood, and for a consistent use of a qualitative scale for documents intended for policymakers.

Overall, it's difficult to find fault with any of these recommendations, or the general justification for their adoption—the IPCC has clearly outgrown the process and structure that got it through the first few assessment reports. Will the changes be put in place in time for the next report, AR5?

The IPCC's Pachauri held a press conference to discuss just that question, but failed to address some key points. Since he and many other senior IPCC staff members shepherded AR4 to completion, the IAC's report would seem to call for him to step down. Instead, Pachauri indicated he'd push for the adoption of the IAC's recommendations, but wouldn't say whether he'd step down even if they weren't taken up wholesale. The questions from the media present quickly bogged down that press conference into a discussion of the IPCC's science (in contrast to the first, where Hal Shapiro kept matters on topic).

The end result is that the IPCC appears to now have an excellent roadmap for reforming its governance and process, and its leadership appears to be interested in implementing them. Unfortunately, as the press conference demonstrated, any implementation will take place against a backdrop of criticism, some of it spurious, as well as copious amounts of mistrust.