Jonathan Ross has withdrawn from hosting science fiction's biggest awards ceremony, the Hugos, after authors and fans were outraged at the news that he would be presenting the prizes at the World Science Fiction Convention in London this summer.

Organisers of the convention announced on Saturday morning that they were "very excited" that the television host, who also writes comics, is a science fiction fan and is married to former Hugo winner Jane Goldman, would be giving out the Hugo Awards in August. Protests followed on Twitter, centring around the offence which has been taken at some of Ross's jokes in the past. Points picked up on by the science-fiction community include comments he made to Gwyneth Paltrow and Andrew Sachs; one of the convention's organisers announced her resignation from the committee over the decision to have Ross as host.

"You know, I've really enjoyed knowing that, were I to be nominated for a Hugo, the host wouldn't see me and make fat jokes," wrote the Hugo award-winning author Seanan McGuire. "Like, that thought has actually crossed my mind, when shopping for Hugo dresses. 'The host won't mock me.' Thanks, @loncon3, for taking that small bit of comfort and reassurance away from me."

Fan comments ranged from "A white man famous for humour denigrating women? I'm the opposite of excited", to "is this some kind of sick joke?". Science-fiction author Alastair Reynolds called Ross "a hugely controversial, polarising figure", while his fellow novelist Charles Stross wrote on his blog that Ross was "a very bad choice for Hugo toastmaster", particularly while science-fiction fandom "is in the process of cleaning house".

"Worldcon should be safe space for fans, and inviting a high profile media personality who has been targeted by the tabloids is going to cause collateral damage, even if nothing happens, simply by making many fans feel less safe," wrote Stross. "If Ross is toastmaster, I can predict that at least one major Hugo nominee/past winner [McGuire] who was planning to be there won't be present at the ceremony, because Ross has past form for using women with weight issues as the butt of his humour. She says she doesn't feel safe, and I believe her: I wouldn't want to be there in her shoes ... I don't like seeing my friends mocked, so I probably won't be there either."

Ross started out engaging with his attackers on Twitter. "Pay £115 for #loncon3 and then find out @wossy is hosting the #HugoAwards. Good to know women and other minorities need not attend, then," wrote one reader. "Absurd," replied Ross. "I'll happily buy the ticket off you and give it someone less stupid."

But as the outrage grew, a little over seven hours after he was announced as host of the Hugos – an event which he was planning to do for free – Ross withdrew, "in response to some who would rather I weren't there".

"Have a lovely convention," he tweeted. "If people genuinely believe I would upset them or those they care about then I'd rather not spoil their night ... I agreed because I love sf. And because Neil Gaiman asked me."

Although some were pleased at his decision, others mourned the move.

"Here's the thing," wrote Rich Johnston at comics site Bleeding Cool . "I've seen Jonathan Ross present both the National Comics Awards and the Eisner Awards. And he got it. Because he is one of the community. He's a fan first and foremost, and he is now a repeat published sci-fi writer to boot. Well, it looks like the Eisners can continue to rely on Jonathan Ross to present their awards without competition."