The decision to pull a children’s book last week over the “false impression” it gives of slavery was the wrong one, according to free speech campaigners in America, who believe the move will lead authors to shy away from “taking on racially sensitive … topics for fear of public outcry and reprisals”.

A Birthday Cake for George Washington, by author Ramin Ganeshram and illustrator Vanessa Brantley-Newton, is narrated by the daughter of George Washington’s slave, Hercules, who is preparing a cake for the president’s birthday. In a review, School Library Journal criticised the “dangerously rosy impression of the relationship between slaves and slave owners” that it could give to young readers. It said that the “light tone” of the text and the “colourful, cartoon-style” illustrations “convey a feeling of joyfulness that contrasts starkly with the reality of slave life”, while criticism of the book grew under the hashtag #slaverywithasmile.

Last week publisher Scholastic bowed to pressure and withdrew the book from sale, saying that “while we have great respect for the integrity and scholarship of the author, illustrator and editor, we believe that, without more historical background on the evils of slavery than this book for younger children can provide, the book may give a false impression of the reality of the lives of slaves and therefore should be withdrawn”.

Campaigners hailed the decision as a victory, but the National Coalition Against Censorship and the PEN American Center have now released a statement criticising Scholastic’s move. “While reasonable people can disagree about the book’s historical or literary merit, Scholastic’s decision to pull it in response to controversy is a shocking and nearly unprecedented case of self-censorship,” said NCAC executive director Joan Bertin.

In their statement, which was also endorsed by the First Amendment Committee of the American Society of Journalists and Authors, the NCAC and PEN said they would “express no opinion as to the merits of the book” itself.

But they called its removal from sale troubling, along with the withdrawal last year of another children’s picture book, A Fine Dessert, also criticised for its images of smiling slaves.

“Those who value free speech as an essential human right and a necessary precondition for social change should be alarmed whenever books are removed from circulation because they are controversial,” said the statement.

“While it is perfectly valid for critics to dispute a book’s historical accuracy and literary merit, the appropriate response is not to withdraw the volume and deprive readers of a chance to evaluate the book and the controversy for themselves. In the case of A Birthday Cake for George Washington, a book is gone that generated important discussions about how our nation creates, perceives and perpetuates narratives about slavery and slave ownership.”

They predicted that pulling the book “is likely to have a chilling effect, leading authors and illustrators to hesitate in taking on racially sensitive or politically controversial topics for fear of public outcry and reprisals”, saying that “pulling books out of circulation simply because they cause controversy is the wrong decision”.