Between the decline of the traditional bookshop and the internet wrecking our concentration, many thought the novel was on its last legs. With all the reports of writers facing penury and the dwindling of literary fiction as a dominant narrative form, you might assume that authors, fighting over a shrinking pot of coin, would be pushed to write pedestrian pieces and avoid risk. But is that what's been happening?

Well, yes. But not only that.

Risks are being taken. More than this, they're being rewarded. With the growth of independent publishing presses, prizes and the rising number of literary journals, there's been something of a tide change in the profile of innovative novels. Challenging writing is not only being produced, it's finding an audience, winning awards, sometimes being advertised on the sides of buses.

"I think we partly have Stella Rimington's disastrous chairmanship of the 2011 Booker to thank," says Sam Jordison, co-director of Galley Beggar Press. "We're enjoying the backlash against the idea that books should be judged mainly on 'readability' … Rightly or wrongly, that year's Booker prize galvanised people into shouting about the rewards and delights of challenging fiction."

Tim Parnell, founder of the Goldsmiths prize, also points to the 2011 Booker prize as an important moment for inventive fiction: "Something has definitely changed in the mood music of literary debate since the 2011 Booker, when Stella Rimington famously championed readability," says Parnell. "Both the Folio and Goldsmiths prizes partly grew out of the debate generated by Rimington's comment and I've been heartened and pleasantly surprised by the very positive responses to the Goldsmiths prize's focus on innovation, daring and experiment."

Eimear McBride's success in the Goldsmiths and Baileys prize with A Girl Is a Half-formed Thing, alongside the centenary anniversary of the publication of Dubliners, means that there's also been attention paid to the grandmaster of challenging fiction, James Joyce. Is McBride's novel, told in a Joycean stream of concsiousness, the latest indication of a resurgence of modernist techniques?

"That style certainly has its advocates – most notably Tom McCarthy and Will Self – but to treat new experiments as revivalism is to risk condemning them to obscurity and academicism," say Benjamin Eastham and Jacques Testard, editors of literary journal The White Review. "Readers don't want something that merely rehashes outmoded styles, but are interested in how authors react to – and distill – the times in which we live. The best new books are written in a form appropriate to our lives as we experience them now, not as we did 100 years ago."

Perhaps the taste for inventiveness stems not so much from reaching back into modernism, but more from the desire to find something representative of the physically detached, digitally connected way most of us communicate, just as Joyce was compelled to find a new way to express the rapidly changing face of the early 20th century. This isn't about highbrow mimicry, but finding the best way possible to write about today's reality.

Of course, today's novels also have today's forms to contend with. Games are becoming more sophisticated in the way they tell stories, while box sets and Netflix continue to scratch the general narrative itch for a large part of the population. Booker prize-winner Eleanor Catton has said that The Luminaries was strongly influenced by box-set TV drama (which in its own way is influenced by the serial novels of the 19th century). But while some novelists will tend to converge with new forms, others will seek divergence.

As the introduction of photography affected the role of painting, or as film altered the place of novels in the 20th century, new narrative forms are inevitably affecting the place and purpose of the novel. As the basic need for storytelling is catered for to a large extent by TV dramas, it may be that readers turn to the novel for something else, something new.

You could argue that this identity crisis about the novel's purpose is driving a desire in writers to express something other than a straightforward story, as well as a taste in readers for something that demands they think in unexpected ways. So the novel's need to justify itself in our current age is a good thing. It keeps it moving. It keeps it alive. In the end it's not about obscure writing, it's about writers working in ways that keep words alive.