New research shows works by writers such as Charles Dickens and Téa Obreht sharpen our ability to understand others' emotions – more than thrillers or romance novels

Have you ever felt that reading a good book makes you better able to connect with your fellow human beings? If so, the results of a new scientific study back you up, but only if your reading material is literary fiction – pulp fiction or non-fiction will not do.

Psychologists David Comer Kidd and Emanuele Castano, at the New School for Social Research in New York, have proved that reading literary fiction enhances the ability to detect and understand other people's emotions, a crucial skill in navigating complex social relationships.

In a series of five experiments, 1,000 participants were randomly assigned texts to read, either extracts of popular fiction such as bestseller Danielle Steel's The Sins of the Mother and Gone Girl by Gillian Flynn, or more literary texts, such as Orange-winner The Tiger's Wife by Téa Obreht, Don DeLillo's "The Runner", from his collection The Angel Esmeralda, or work by Anton Chekhov.

The pair then used a variety of Theory of Mind techniques to measure how accurately the participants could identify emotions in others. Scores were consistently higher for those who had read literary fiction than for those with popular fiction or non-fiction texts.

"What great writers do is to turn you into the writer. In literary fiction, the incompleteness of the characters turns your mind to trying to understand the minds of others," said Kidd.

Kidd and Castano, who have published their paper in Science, make a similar distinction between "writerly" writing and "readerly" writing to that made by Roland Barthes in his book on literary theory, The Pleasure of the Text. Mindful of the difficulties of determining what is literary fiction and what is not, certain of the literary extracts were chosen from the PEN/O Henry prize 2012 winners' anthology and the US National book awards finalists.

"Some writing is what you call 'writerly', you fill in the gaps and participate, and some is 'readerly', and you're entertained. We tend to see 'readerly' more in genre fiction like adventure, romance and thrillers, where the author dictates your experience as a reader. Literary [writerly] fiction lets you go into a new environment and you have to find your own way," Kidd said.

Transferring the experience of reading fiction into real-world situations was a natural leap, Kidd argued, because "the same psychological processes are used to navigate fiction and real relationships. Fiction is not just a simulator of a social experience, it is a social experience."

Not all psychologists agreed with Kidd and Castano's use of Theory of Mind techniques. Philip Davies, a professor of psychological sciences at Liverpool University, whose work with the Reader Organisation connects prisoners with literature, said they were "a bit odd".

"Testing people's ability to read faces is a bit odd. The thing about novels is that they give you a view of an inner world that's not on show. Often what you learn from novels is to be a bit baffled … a novel tells you not to judge," Davies said.

"In Great Expectations, Pip is embarrassed by Joe, because he's crude and Pip is on the way up. Reading it, you ask yourself, what is it like to be Pip and what's it like to be Joe? Would I behave better than Pip in his situation? It's the spaces which emerge between the two characters where empathy occurs."

The five experiments used a combination of four different Theory of Mind tests: reading the mind in the eyes (RMET), the diagnostic analysis of non-verbal accuracy test (DANV), the positive affect negative affect scale (PANAS) and the Yoni test.

However, although Castano and Kidd proved that literary fiction improves social empathy, at least by some measures, they were not prepared to nail their colours to the mast when it came to using the results to determine whether a piece of writing is worthy of being called literary.

"These are aesthetic and stylistic concerns which as psychologists we can't and don't want to make judgments about," said Kidd. "Neither do we argue that people should only read literary fiction; it's just that only literary fiction seems to improve Theory of Mind in the short-term. There are likely benefits of reading popular fiction – certainly entertainment. We just did not measure them."