The story of the seduction of a lesbian by an ageing stage actor, which includes an eye-watering scene with a green dildo, has won Philip Roth the dubious honour of a place on the shortlist for the Literary Review's bad sex in fiction award.

Roth can comfort himself with the fact that a roll call of literary fiction's great and good, from Booker winner John Banville to acclaimed Israeli novelist Amos Oz, Goncourt winner Jonathan Littell and Whitbread winner Paul Theroux, have made it into the line-up for this year's bad sex prize, set up by Auberon Waugh to "draw attention to the crude, tasteless, often perfunctory use of redundant passages of sexual description in the modern novel, and to discourage it".

On a shortlist of 10, singer Nick Cave was picked for his second novel The Death of Bunny Munro, about a sex-obsessed door-to-door salesman. "Frankly we would have been offended if he wasn't shortlisted," said Anna Frame at his publisher Canongate.

The Pulitzer prize-winning Roth makes the line-up for The Humbling, in which the ageing actor Simon converts Pegeen, a lesbian, to heterosexuality. The Literary Review singled out a scene in which Simon and Pegeen pick up a girl from a bar and convince her to take part in a threesome. Simon looks on as Pegeen uses her green dildo to great effect. "This was not soft porn. This was no longer two unclothed women caressing and kissing on a bed. There was something primitive about it now, this woman-on-woman violence, as though in the room filled with shadows, Pegeen were a magical composite of shaman, acrobat, and animal. It was as if she were wearing a mask on her genitals, a weird totem mask, that made her into what she was not and was not supposed to be," writes Roth. "There was something dangerous about it. His heart thumped with excitement – the god Pan looking on from a distance with his spying, lascivious gaze."

'Roth is very anxious about his description of sex," said Jonathan Beckman at the Literary Review of the extract. "Why write of a scene that repeatedly features a green dildo, 'this was not soft porn', unless you're worried that it might be taken as such - in this case, with sentences like 'then she crouched above Tracy, brushing Tracy's lips and nipples with her mouth and fondling her breasts...', the worry seems justified. But it's the overcompensation that qualifies this passage for the award – the totems and shamans are an attempt to convince us that Roth's leering is actually giving some vital anthropological insight."

Sanjida O'Connell is the only woman to make the Bad Sex shortlist, selected for The Naked Name of Love, about a young Jesuit priest who is taught how to love by a gifted shaman woman on the eastern steppes of Mongolia.

Beckman said the line-up for the 17th annual Bad Sex prize was "very strong, with a good mix of well-known writers and others who are less well-known". Also shortlisted are the first novel from Independent film critic Anthony Quinn, set at the outbreak of the first world war, Simon Van Booy's love-themed short story collection Love Begins in Winter, and 23-year-old Richard Milward's novel Ten Storey Love Song. Comprising just one paragraph and replete with graphic sex scenes, Milward's second novel follows the story of Bobby the Artist as he becomes a star and then sinks into drug-induced psychosis.

Milward, who accepted the prize in 2007 on behalf of the late Norman Mailer, said he "would have been upset" if he hadn't been shortlisted this year. "I've been there before and I'll be there again .. There's so much bad sex in my book that this is a nice accolade," he said.

"Some authors spend five pages describing a walk in the park but when it comes to sex they'll just do two sentences - 'she rolled off him'. Sex is exciting stuff - it can be vey dirty and smelly, but you've just got to get stuck in, and I'm not afraid of doing that."

The winner of the award, a plaster foot, will be announced on 30 November at London's In & Out club. Last year's prize was won by Rachel Johnson for her novel Shire Hell, in which at one point the heroine makes a "grab, to put him, now angrily slapping against both our bellies, inside". Previous winners include Sebastian Faulks, AA Gill and Giles Coren, while last year's ceremony also saw John Updike given a lifetime achievement prize after four consecutive nominations.

The shortlist

Paul Theroux for A Dead Hand

Nick Cave for The Death of Bunny Munro

Philip Roth for The Humbling

Jonathan Littell for The Kindly Ones

Amos Oz for Rhyming Life and Death

John Banville for The Infinities

Anthony Quinn for The Rescue Man

Simon Van Booy for Love Begins in Winter

Sanjida O'Connell for The Naked Name of Love

Richard Milward for Ten Storey Love Song