At book readings, Stephen King sometimes tells a story about his "only preproduction discussion" for the 1980 film adaptation of The Shining. At seven in the morning, King was shaving in the bathroom when his wife ran in to tell him there was a call from London, it was Stanley Kubrick. Just the mention of the director's name was shock enough that when King went to the phone, he had a line of blood running down one cheek and the other was still white with foam. The first thing Kubrick said – and it's worth noting that King's growly impersonation makes him sound like a swamp creature – was: "I think stories of the supernatural are fundamentally optimistic, don't you? If there are ghosts then that means we survive death." King asked him about hell, how did that fit in? There was a long pause, then: "I don't believe in hell."

King tells this anecdote like a horror story, which I guess it is, since he famously hates Kubrick's adaptation of his book. We imagine Kubrick as the monster, a corona of wild hair around his pale face, laughing as he drags the novel into the depths of soulless art cinema. The director is turned into a character from a Stephen King novel as payback for having turned The Shining into a Stanley Kubrick film.

Six months ago, I was invited to curate the Page to Screen film festival, in Bridport, Dorset, which is dedicated to films adapted from books. I was thrilled to take part and was surprised, when I started looking, at just how many films start life in print – six of the nine nominees for best picture at this year's Oscars, for example. Many of the films we've chosen gleefully cheat on the books that inspired them. In Clueless, for example, Jane Austen's Emma becomes Cher Horowitz, Georgian Highbury becomes contemporary Beverly Hills, the horse-drawn carriage is a drop-top four by four, while all the bitching and romantic mix-ups remain pretty much the same, just with different slang. King has a neat theory about how an adapter's voice can force itself on a story. His version of The Shining "ends with the hotel burning, and [Kubrick's] with the hotel freezing" because he is a "warm and gooey" person while Kubrick was "the coldest guy in the universe".

One of the films we're showing is based on Barry Gifford's novel Wild at Heart, about a highly sexed, loved-up couple, Sailor and Lula. At the end of the book, once Sailor has completed another long stint in jail, Lula is waiting for him with their son, who has never met his father. After giving his son some life advice, Sailor walks away from his new family, feeling that he will only end up getting them into more trouble. Lula lets him go.

In Chris Rodley's book-length interview with David Lynch, who adapted Wild at Heart for the screen, the director says this ending "honestly didn't feel real, considering the way they felt about each other. When Sam Goldwyn [Jr] saw an early draft of the script he said, 'I hate this ending,' and I said – it just popped out of me – 'I hate that ending, too!' And so he said, 'Why don't you change it?' and I said, 'I'm going to change it, doggone it!'" Once he got started, Lynch changed a lot. He cranked up the ultra-violence, added a strand of screwy humour and, most significantly, draped the story of The Wizard of Oz on top. This is where it enters Lynchland. At the end of the film, while Sailor is lying in the middle of the road, knocked out, with his nose smashed, the good witch visits him in a globe of pink light to tell him to go back to his family. The witch with her magic wand may as well be David Lynch, the adapter, descending from the sky to give the story a happy ending.

Changing key plot points in a well-loved book can be risky. Hell hath no fury like a hardcore fan scorned. Often, the most hardcore fans of all are the authors. Anthony Burgess disowned both the film and the novel of A Clockwork Orange after Kubrick's version didn't allow Alex, the protagonist, redemption from his violent instincts. In 1995, a year after his novel Forrest Gump had been sanitised for the screen, Winston Groom published Gump and Co, a sequel, which began with: "Let me say this: Everybody makes mistakes ... But take my word for it – don't never let anybody make a movie of your life story." Richard Matheson's novel I Am Legend has been filmed four times, and never to his satisfaction. After the most recent adaptation, he was quoted as saying: "I don't know why Hollywood is fascinated by my book when they never care to film it as I wrote it." I can understand why authors might be touchy about the sacredness of their text but, more often than not, it's worth ignoring them. Most great film adaptations have built their success on ignoring, or at least shunting, key elements of the source text.

In some ways, Mike Nichols' film of The Graduate is a very straight adaptation. The script is taken almost entirely from Charles Webb's excellent novel, which itself is sparely written and led by dialogue. According to both script and book, Benjamin Braddock, the protagonist, is a 20-year-old "track star" and "the kind of guy who has trouble keeping the ladies at a distance." On this basis, it makes sense that Nichols wanted to cast Robert Redford for the lead.

But the role also calls for Braddock to struggle with communication, find relationships difficult and see his parents' poolside LA life as grotesque. When Redford auditioned, being tall and dashing, Nichols wasn't convinced. Instead, the director called in a young stage actor who he had seen playing a transvestite in an off-Broadway play. Not many people believed Dustin Hoffman would be right for the role, least of all Dustin Hoffman. "I always felt that I had been miscast," he has said. "I expected to be fired, so the discomfort I felt was not just being new to movies. The discomfort was feeling that they'd made a mistake casting me."

The Graduate, particularly its use of music and Hoffman's performance, inspired Richard Ayoade's adaptation of my novel, Submarine. Hoffman plays Benjamin Braddock in constant physical distress, swallowing his lines, using "an often inflected delivery," as Ayoade puts it. "When you read the book, he sounds more sarcastic and snarky, closer to Holden Caulfield," he says, "but with Dustin Hoffman it feels genuinely rabbit-in-the-headlights."

For me, the book's version of Benjamin Braddock is a good-looking sociopath almost on the Patrick Bateman scale, compared to the film's uneasy non-hero. Comedies of awkwardness – from Rushmore to The Office to Curb Your Enthusiasm – couldn't exist without Hoffman's performance. He allows us to feel for Braddock, despite his selfish, privileged existence.

The Graduate is not unusual in softening its lead character for the screen. Sometimes film-makers are accused of doing this out of commercial concerns, to get a lower certificate or a wider demographic. That might sometimes be true – Forrest Gump, I'm looking at you – but there's also a difference in the way a film is consumed. "When you read, you half become the main character, there's a grey area between your voice and the one on the page," Ayoade says, "but in a film, when you see someone on screen who is obviously not you, there's a physical otherness – you sort of decide whether you enjoy being in the company of that person."

That physical otherness only increases when a lead actor is a star, someone we know outside the film as hugely famous, wealthy and with a complex love life. How much more work do we ask of an audience to emotionally connect with a character played by Tom Cruise? We are less forgiving of characters on screen, too. As Ayoade puts it: "I think there is still a puritanical streak in most films. Wickedness must be punished." As a rule, all actions on the big screen – wicked or otherwise – receive their appropriate reaction. Film-makers are not so easily allowed to use the literary author's get-out-of-jail-free card: letting the reader decide.

In American Psycho, the novel, we are kept aware that Patrick Bateman's murders could all be in his imagination. The ferocious sex-horror is either disturbing because it's real or because it's his fantasy. We only go along with the book's violence because there are the safety valves of unreliability and chapter-long digressions about Whitney Houston. In the film, we meet a considerably mellower grade of psychopath. This seems like a sensible decision since, in film, ambiguity is harder to sustain. Christian Bale's character is never as deranged or as complex as the book's. At the end of the film, we are offered a way out – was it all in his head? – but that moment feels like a cheat rather than a real question.

Alfred Hitchcock's 1950 film, Stage Fright, was criticised for what became known as its "lying flashback" – a long flashback about a murder that we later learn is untrue. Audiences felt tricked, and the film didn't do well. Speaking to François Truffaut, Hitchcock said: "In movies, people never object if a man is shown telling a lie. And it's also acceptable, when a character tells a story about the past, for the flashback to show it as if it were taking place in the present. So why is it that we can't tell a lie through a flashback?"

Even when a film is as aggressively ambiguous and nonsensical as Kubrick's The Shining, audiences still want an objective truth. Perhaps it's the physicality of film – the idea that seeing is believing. The documentary Room 237 is a showcase for the myriad ways in which people have stretched reality to try to find clear meaning in The Shining. Most commentators begin by saying that when they first saw the film they didn't enjoy it, it didn't make sense. Only later did they grasp its true message: it's an apology for Kubrick's faking the moon landings or a metaphor for the genocide of native Americans or a vehicle for sexy subliminal messages. In The Shining, the camera's depth of focus adds to the sense that there must be a grand directorial plan at work: every object burns with significance. And as I watched Room 237, I believed each conspiracy theory in turn.

It's one of the pleasures of watching a story jump between mediums – trying to second guess the logic, or otherwise, behind each decision. At the festival in Bridport, we're showing The Shining in a disused hotel on a cliff top. Jan Harlan, Kubrick's right-hand man, will be there to answer questions. At least then we can finally find out which of our crackpot theories is true.

In some cases, even the source text is unsure of itself. For his second film, Ayoade is working with Avi Korine to adapt The Double by Dostoevsky (the obvious next step after working on one of my novels). It's a story about a man who is disturbed to find that his more charismatic doppelganger works in the same office as him. Ayoade tells me what's unusual about The Double is that Dostoevsky never wrote an ending he was satisfied with. I asked if that would not make life easier, since it removed any issue of the author's infallibility? "Yes," Ayoade replied, "we'll help Fyodor get it right for once."