The editor-in-chief of an academic journal has resigned after his publication accepted a hoax article.

The Open Information Science Journal failed to spot that the incomprehensible computer-generated paper was a fake. This was despite heavy hints from its authors, who claimed they were from the Centre for Research in Applied Phrenology – which forms the acronym Crap.

The journal, which claims to subject every paper to the scrutiny of other academics, so-called "peer review", accepted the paper.

Philip Davis, a graduate student at Cornell University in New York, who was behind the hoax, said he wanted to test the editorial standards of the journal's publisher, Bentham Science Publishers.

Davis had received unsolicited emails from Bentham asking him to submit papers to some of its 200+ journals that cover a wide range of subject matter from neuroscience to engineering.

If their papers are accepted, academics pay a fee in return for Bentham publishing the papers online. They can then be viewed by other academics for free.

Davis, with the help of Kent Anderson, a member of the publishing team at the New England Journal of Medicine, created the hoax computer science paper. The pair submitted their paper, Deconstructing Access Points, under false names. Four months later, they were told it had been accepted and the fee to have it published was $800 (almost £500).

Davis then withdrew the paper and revealed it as a hoax. Bambang Parmanto has since stepped down as editor-in-chief of the Open Information Science Journal. Parmanto told New Scientist that he never saw the paper.

Mahmood Alam, Bentham's director of publications, told New Scientist: "In this particular case, we were aware that the article submitted was a hoax and we tried to find out the identity of the individual by pretending the article had been accepted for publication when in fact it was not." Davis told the magazine that he had not been directly contacted.

The hoax has triggered a debate about "open access" journals, some of which charge academics fees to publish their papers and allow readers access to research without subscription. Anderson said: "It's almost an inevitability that you might have several publishers tempted to take advantage of this relatively easy money."

Alex Williamson, a former publishing director of the British Medical Journal – partly open access and partly run on subscriptions – said: "There is a whole range in the quality of journals. Some that are open access are extremely good. There are a lot of awful ones, and these are probably more likely to be open access journals. Any idiot can start a journal on the web."