When melting ice disappears from the arctic, it exposes more of the ocean’s dark surface, which absorbs the sun’s warming rays. The water heats up and more ice melts, the cause and effect feeding each other in a example of a phenomenon known as climate feedback.

It’s an appropriate name for a group that’s attempting to slow some of the runaway misinformation about climate change, by doing what scientists do with their published work: review it. To achieve this, Climate Feedback–less an organization at this point than an amorphous gathering of climate scientists, oceanographers, and atmosperic physicists–is making use of a browser plugin from the nonprofit Hypothes.is to annotate climate journalism on the Web. Readers with the plugin, or with a link created through it, can read an article while simultaneously reading comments and citations from a cadre of experts. Click on the headline, and you’ll see an overall rating, based on the article’s accuracy, fairness, and adherence to evidence.

“We are trying to bring more scientific point of view on what is said about climate change,” says Emmanuel Vincent, a climate scientist at the University of California, Merced’s Center for Climate Communication and the group’s ringleader. “Climate change has been taken a little bit outside of the realm of science.”

About 25 scientists are listed as contributors, and more can apply as long as they’re actively publishing climate research. Because the plugin is still in development, a small group of experimental users is providing input to Climate Feedback on how to best make use of it.

The group’s first target was a piece by Steve Koonin, a theoretical physicist and former BP scientist who now heads NYU’s Center for Urban Science and Progress, that appeared in the Wall Street Journal’s Saturday Essay section in December. Koonin argued that it’s too early to shape climate global policy because the specifics of the science are not settled. Six Climate Feedback scientists gave it a rating of only half a point out of a possible four, placing it somewhere between “poor” and “very poor” in its scientific merit.

We’ve seen some pretty serious misrepresentation of climate science in certain news outlets.

The organization gave a similar review to another Wall Street Journal column that claimed “climate-change alarmists” are ignoring a wealth of climate data that “are actually encouraging,” to the detriment of us all. (The writer of that piece, Danish author and analyst Bjorn Lomborg, has been accused of having links to the Koch Brothers, who are notorious for funding misinformation around climate science. But that’s another story.)

Sign up for CJR 's daily email

As longtime purveyors of information and holders of esteemed editorial ethics, journalists are used to being the ones setting the rules on accuracy. Writers and editors make decisions every day about appropriate sources, fair representation, and whether the “o” in Lomborg’s first name is really an “ø” (not according to his website). But with the democratization of the Web, and the downfall of the paid fact-checker, scientists may be able to step in and provide a much-needed check on unfairly warped truths–if they stick to what they know.

The group recently, for example, evaluated an article by Mashable’s Science Editor Andrew Freedman, giving it good marks overall. The biggest quibble with the piece wasn’t with anything Freedman wrote, but with comments one of his sources–a government climate scientist–made. “I could see how some journalists might think it’s a bit unfair to be held responsible for something their source said, especially if you put it into context by saying other sources say otherwise,” Freedman says.

Vincent says the ultimate goal isn’t to fact-check, but to foster more scientific thinking in journalists and ultimately build more communication between the two parties. That certainly happened with Lomborg, who accused the group of acting as “opinion police” when one of the annotators didn’t back up a statement with facts–which then prompted a long response from Vincent in the comments section of Forbes.

In fact, Climate Feedback could function as an improvement over the comments section, a much-hated part of journalism that’s replaced the loved and largely lost tradition of full-on fact-checking. For New York Times climate reporter Justin Gillis, whose article on 2014’s record-breaking heat received high praise from Climate Feedback, the idea of annotation recalled a practice from his years at The Miami Herald in the 1980s, when editors would send daily feedback forms to experts asking for critiques of the paper’s stories. “We’ve seen some pretty serious misrepresentation of climate science in certain news outlets,” Gillis says. “I would hope those outlets would take the comments seriously.”

That may be the critical part. There’s no obligation for journalists to pay any attention to what Climate Feedback–or any other commenter or letter-writer–has to say. That call still lies with editors. But Vincent hopes his tactic will encourage both journalists and scientists to be more critical in their work, and get them to listen to each other (while also informing the reader). “It’s not just one way,” Vincent says. “It’s not just scientists talking to journalists and telling them what is to be done. It’s also a process of learning, a process of interaction.”

Has America ever needed a media watchdog more than now? Help us by joining CJR today

Laura Dattaro is a science journalist, writer, and producer based in New York.