Artificial intelligence is luring science into dangerous waters. To make scientific publishing more efficient, commercial publishers now rely more and more on editorial software systems. These are beginning to transform peer review from interaction between humans into interaction between humans and AI. We should think twice before allowing autonomous AI systems to decide what research warrants publication.

WIRED OPINION About Janne I. Hukkinen (@JIHukkinen) is professor of environmental policy at University of Helsinki, Finland, and editor of Ecological Economics.

Peer review is the backbone of science. Researchers across disciplines trust that exposing one's research to criticism by fellow researchers improves quality and contributes to scientific advancement. As with any professional procedure, the system has its flaws. But it's still widely accepted as the best available quality control for science.

From its birth in the late 1600s, scientific publishing has grown to an estimated 28,000 scholarly journals generating more than 1.8 million articles per year in 2014. The glut in publishing has raised quality concerns, so publishers have reacted by introducing editorial management software systems. These speed up the processing of manuscripts from submission to review to publication. And they do amazing things.

I have hands-on experience with the Evise system, which Elsevier piloted in 2015 in the environmental journal Ecological Economics, which I edit. When an author submits a manuscript, Evise links with plagiarism-checking software. It suggests reviewers on the basis of the content of the manuscript or allows the journal editor to choose reviewers. And it communicates with another software program called Scopus to check the profile, scientific performance, and conflicts of interest of reviewers. During review, Evise automatically handles all correspondence between editors, reviewers, and authors. It reminds reviewers to review a paper, un-invites them if they don't respond, and invites alternate reviewers when necessary. After the editor's decision, it sends decision letters to authors and thanks the reviewers.

Whether Evise is already an AI system is a matter of perspective. John McCarthy, one of the pioneers of AI research, famously noted: "As soon as it works, no one calls it AI anymore." Evise has emergent AI features, as it can check for plagiarism and suggest reviewers. Yet decisions about quality, significance, and novelty are still in the hands of human editors.

What really matters for knowledge production, however, is who governs and regulates the use of AI. An ever-larger share of scientific publishing is in the hands of commercial publishers, with the top five most prolific publishers (Elsevier, Taylor & Francis, Springer, Wiley-Blackwell, and Sage) accounting for more than 50 percent of all papers in 2013; all use editorial software systems.

Peer review in a commercial publishing house such as Elsevier takes place under stringent efficiency requirements. Of the roughly 1,500 manuscripts submitted annually to Elsevier's Ecological Economics, about 250 are published. Each editor agrees to process up to 100 manuscripts per year, with a target time of 2 months for processing a manuscript from submission to final decision.

Without thoughtful corporate governance and public regulation, there's a danger of a runaway process here. The demand for efficiency and the promise of computerized editorial management push peer review more and more toward AI. Elsevier's human organization defends the technology aggressively. During piloting, Evise malfunctioned chronically and began acting like Hal from 2001: A Space Odyssey, sending terse letters unbeknownst to the editors. Despite pleas to abandon the system because trusted reviewers were rapidly dropping out, Elsevier resisted. They instead formed a team to tackle the problems, with some success.

Imagine for a moment that publishers continue their business-as-usual and embrace AI in peer review. AI performance will increase precisely where human editors today invest most of their time: choosing reviewers and judging whether to publish a manuscript. I don't see why learning algorithms couldn't manage the entire review from submission to decision by drawing on publishers' databases of reviewer profiles, analyzing past streams of comments by reviewers and editors, and recognizing the patterns of change in a manuscript from submission to final editorial decision. What's more, disconnecting humans from peer review would ease the tension between the academics who want open access and the commercial publishers who are resisting it.

But AI could be a powerful assistant in peer review. Institutional rules would need to specify what AI must, may, and may not do. Designing rules for AI is not that different from designing rules for humans. Put simply, AI is a uniquely rich and efficient system of intelligence built from collective human intelligence. It's important to understand AI this way, because setting limits to human behavior is likely to be more acceptable than regulating new technology.

The scenarios raise critical questions for the future of science. How will the writing of scientific articles change when authors know their text will be evaluated by AI instead of humans? AI combines superb reasoning with lesser capacity to consider values, such as ranking two scientifically equal texts on the basis of their social relevance. A peer-review system driven by AI might offer benefits, like an increase in objectivity, since a computer is judging the science rather than a complicated human, as well as drawbacks, as the AI might not be able to divine what's new knowledge. Either way AI will change how authors game peer review.

Why should humans bother with authoring? Researchers and publishers have developed instructions for how to get published. Following instructions is something AI excels in. Could not one AI system craft scientific manuscripts for quality check-up by another AI system?

And whose new knowledge does peer review by AI produce? Unhindered, AI can refine new knowledge without human intervention. Given the subtle differences between AI and human intelligence, it's reasonable to expect that an autonomous AI system will generate its own criteria for judging what passes as new knowledge.

New knowledge which humans no longer experience as something they themselves have produced would shake the foundations of human culture.