Radical change is certainly producing some alarming symptoms – but much of the doomsayers' evidence is anecdotal, and it's possible to read a much happier story

This time last year, I was metaphorically invited to the only party I've ever wanted to be seen at. My first novel, The English Monster, was picked up by an agent, and then by a publisher, Simon and Schuster. It hits the streets in March 2012.

I've made it, I thought to myself as I clutched my invite to the most exclusive set of all. I'm going to be a published author.

So imagine my surprise - nay, dismay - to discover that publishing's streets were not paved with gold, but stalked by the anxious, the gloomy, the suicidal. "Publishing's dead!" shouted men in sackcloth on Bloomsbury street corners. I had arrived at the party, but the coats were being handed out, the drink had dried up and the hostess had collapsed.

So I asked myself (somewhat desperately, positively naively): are things really that bad? What is the actual state of book publishing in Britain? Can writers really only look forward to a life of penury? Or should I stick my head in the sand, if only to deaden the sound of commissioning editors weeping into their lattes?

We're doomed ...

If you don't believe that people are worried, you need only look to the Guardian's own recent debate at the Edinburgh international book festival, called, efficiently and apocalyptically, "The End of Books?". One of the contributors was writer Ewan Morrison who, in a piece on theguardian.com/books after the event, expounded his view that the printed book will go within 25 years, as readers turn more and more to ebooks. What's worse, these ebooks will collapse in value, because that is what today's younger consumers want, as demonstrated by the online shift to free news. Publishers are no longer paying advances to authors, or if they are these advances are a fraction of what they were. And all the time the relentless combination of pirating, retail competition and the demands of younger consumers means that the price of every piece of content – a song, a film, a book – trends towards zero.

We are, in summary and to paraphrase a certain Scottish member of the Home Guard, doomed.

Better than you'd think

But hang on a minute. Anecdotally, that's a pretty awe-inspiring collection of proofs. But the plural of anecdote is not data. What is the data telling us?

According to Nielsen BookScan, the publishing industry standard for book sales data, book sales are pretty healthy, with one significant proviso which I'll come to. Ten years ago in 2001, 162m books were sold in Britain. Ten years later – a decade in which the internet bloomed, online gaming exploded, television channels proliferated, digital piracy rampaged and, latterly, recession gloomed – 229m books sold. So, a 42% increase in the number of books sold over the last 10 years.

But wait, say the gloomy. What about the cash? Haven't publishers been forced by avaricious retail giants into a fearsome downward spiral? Discounting has sharpened, but not as much as you'd think. The standard discount on the recommended retail price of a book in 2001 was already at 17.6%. In 2010 it was 26.7%. We'll return to this later.

Even with this discounting, last year UK consumer publishing drew in sales of £1.7bn, up 36% on 2001. Adult fiction saw an increase of 44%, to £476m; and young adult and children's fiction, realm of all those pesky copiers and pirateers and downloaders, saw sales more than double to £325m.

So why the very, very deep uncertainty and the gloom? Because 2011 is the year this may all change. Here's the proviso on the sales figures I mentioned. These numbers above do not include any ebook sales at all. Nielsen BookScan hasn't yet finalised its tracking of ebooks, and the year to date has seen a drop in printed book sales against 2010. But again, not as much as you'd think. Up to the week ending 13 August, overall sales were down almost 6% on 2010 in volume terms, and just over 4% in value.

Ebooks: death or glory?

The question – the defining question – is whether that gap is being filled by ebooks. David Walter, research and development analyst at Nielsen BookScan, told me that the 2011 decline was at least "partially" down to the transition to ebooks, and also mentioned the general economic climate and the reduction in the number of retail booksellers. But there are no numbers against that. Not to put too fine a point on it, we just don't know. So can we perceive yet what impact ebooks may be having?

We must look to the US for the early signs. Ewan Morrison states in his piece that "Barnes and Noble claims it now sells three times as many digital books as all formats of physical books combined." Well, not quite. That figure is for online sales through bn.com only. In its most recent quarterly sales report, B&N reported an overall increase in sales at bn.com of over 50%. For the year, Barnes and Noble's total sales across all its business were up 20% to a record $7bn. But Barnes and Noble is still losing money ($59m in the fourth quarter), for the good reason that it's struggling to compete with the new, very big, very scary kid on the block: Amazon.

Ah, yes. Amazon. The boogeyman. A company now worth almost $90bn (£55bn). If you're an independent bookseller, Amazon must look like a cold, relentless stealth bomber casting its shadow over the pavement outside. But to the publisher and the writer, don't things in Amazonia look rather different?

For one thing, people are buying more and more books in Amazonia, and more and more of them are on Amazon's ebook platform the Kindle. In May this year, Amazon announced that, for the first time, it was selling more Kindle versions of books than paperback and hardbacks combined, and (here's the thing that doesn't get quoted so often) sales of print books were still increasing.

Amazon also announced that, in the year to May 2011, it had seen the fastest year-on-year growth rate for its US books business, when expressed in volume and in dollars. This included books in all formats, print and digital. In the UK, less than one year after opening its UK Kindle store, Amazon.co.uk is selling more Kindle books than hardcover books. And again, this is while hardcover sales continue to grow.

Let's not be naive. Any retail channel that ends up being dominated by one player will end up squeezing its producers; just ask a farmer. But Amazon is, right now, giving people what they want: competitive pricing, rapid delivery, massive choice, good customer service. And it's selling books. A lot of books.

The rush to zero

So, what about discounting? Amazon is undercutting, goes the cry, selling cheap and devaluing the product. And don't get us started on Tesco …

Is this true? The discounting has increased, no doubt; but the average cost to the consumer of an adult fiction book in 2010 is only 30p less than in 2001. That figure will be higher when inflation is accounted for, but it's not slashed-and-burned; it means a fiction book still sold for £6.11 in 2010, on average.

There is a deeper, much more existential concern: that, basically, all readers are ultimately freeloaders and want to get books for free, and that the transition to digital devices will see an explosion in piracy and a collapse in pricing. The evidence for this is … well, I'm not sure what the evidence is, to be frank. Newspapers, it is said, are being destroyed because of people's appetite for free news. And we all know what happened to music, don't we? Those cockamamie teenagers ruined everything by downloading the stuff illegally.

But where is the evidence that this will happen in the same way with books? One reason the music industry got so badly hit was that it took the devil's own time putting a viable digital distribution mechanism in place; then along came iTunes and, lo and behold, people download less music illegally where they have the tools to download it legally. It is certainly true that rock stars are no longer going to be buying up chunks of the home counties, but wasn't that in itself an anomaly that lasted barely two decades? New music acts are still being signed, new music is still being produced: arguably more of it, or a greater variety, than ever before.

Meanwhile, in Amazonia, Kindle versions of new books are outselling hardback versions - at similar prices. So is there not another view: that people are paying relatively high amounts for books a year before their paperback release, because they want them quickly on their digital devices? That convenience trumps pricing and format every time? There are significant and important complaints about the agreements established between major publishers and Amazon over the pricing of ebooks, and this will no doubt go through significant changes (although you won't get any publisher to discuss ebook pricing with you). For now, people are voting with their wallets. They're buying books.

So the data, at least, shows that book sales are in pretty good health, with the proviso that, in 2011, the data is out of step with the buying habits and we won't know the true picture for a while, although early indicators from out of the US indicate that things look pretty good.

The impecunious author

So what about the other side of the coin? What impact is this change likely to have on authors? Ewan Morrison argues that author advances have collapsed:

"With the era of digital publishing and digital distribution, the age of author advances is coming to an end … The Bookseller claimed in 2009 that 'Publishers are cutting author advances by as much as 80% in the UK'. A popular catchphrase among agents, when discussing advances, meanwhile, is '10K is the new 50K'. And as one literary editor recently put it: 'The days of publishing an author, as opposed to publishing a book, seem to be over.'"

Remember, though: the plural of anecdote is not data. Agreements between authors and publishers are confidential things; any evidence for a decline in advances is entirely anecdotal. That said, things do seem to have changed. Fewer, bigger advances are gravitating towards books which spark debate, which generate conversation, which (and this surprised me) tend towards the more literary end of the spectrum, where books stay in print for longer and sell copies over years, not weeks. Meanwhile advances for genre and commercial fiction do seem to have fallen back from the highs of the 1980s and 1990s.

According to Kate Pool, deputy general secretary at the Society of Authors: "The average advance probably has gone down. The number of commercially marginal books which are no longer commissioned/accepted by publishers when offered on spec, has gone up."

On the other hand, authors are not seeing a sudden collapse in their incomes. The Society of Authors did a survey in 2000 that showed the average annual figure was £16,600; only 5% of authors earned over £75,000; 75% earned less than £20,000. A more recent survey, done by the Authors Licensing and Collecting Society, came up with very similar figures.

So where does this sense of authors being squeezed come from? It could simply be a sign that publishing, as an industry, is becoming more commercial, more competitive, more efficient. You may not like that. You probably don't. There is a profound queasiness which breaks out at the conjunction of art and business. But the pressure is definitely there. As Maxine Hitchcock, editorial director at my publisher Simon and Schuster puts it: "You've got to publish harder and more nimbly than ever before."

There is another pressure on writer's incomes. It seems that there are more writers to go around. Last month, membership of the Society of Authors passed 9,000 people for the first time since the Society was formed in 1884. There has been a steady increase in the number of book titles published in the UK, from almost 110,000 in 2001 to just over 150,000 in 2010. More surprising, perhaps, is the Nielsen Bookscan data on the number of new publishers each year in the UK and Ireland. What this actually records is new entities applying for ISBN records in each year. In 2001, there were 2,248 such new entities. In 2010, there were 3,151 of them. Nielsen Bookscan has this quite interesting thing to say about that increase: "The year-on-year increase between 2001 and 2010 shows that last year's figure is the highest in this period and can be explained by the fact that many new authors continue to publish their work under their own publishing name."

And I'll bet that there are more titles available today from more authors than at any other time in history. So, even if people were buying as many books today as they were a decade ago, the average writer's income would be falling. Now, that may not be good for the average writer – but it might be a good thing for society as a whole.

Onwards to a glorious future?

What does all this data add up to? Hardly an industry in its death throes, so one must ask why there are so many long faces about the place. Let's not be naive. These are times of massive change, and change is never, ever comfortable. The retail sector worries publishers and authors alike; in the past year, publishers have lost Woolworth, Borders and British Bookshops as sales channels and, as Kate Pool from the Society of Authors says: "The increasing dominance of Amazon (as retailer, increasingly as publisher, as owner of the Kindle, etc) is potentially very worrying."

This, combined with the emergence of digital technology, creates enormous uncertainty. It's a fact that the transition to digital devices will mean greater efficiencies and more focus on cost and, overall, a rather less generous publishing industry than before; a rather colder-hearted, fiercer one. The old world is fading, the new world isn't yet in focus. When newspapers and music faced this moment, there was a significant tendency to become hugely angry that the old world in which we were all so comfortable was being "swept away". It's almost impossible for someone who has spent decades working in a calm, creative environment not to be enraged by the sight of American technology companies tipping everything on its head.

But let's not overdo things. Let's not lose sight of the data we have, and let's not invent data when we only have anecdotes. And finally, let's not forget the wonders this new world opens up. Being able to download a book to read instantaneously wherever you are is a thing of wonder, after all (and there is some anecdotal suggestion that people are coming back to books via new digital platforms).

For authors, the chance to reach out to readers, instantly and effectively, is changing the way titles are marketed and delivers a glorious independence that comes with having your own digital presence to curate and to shape. There are new creative opportunities offered by interactive technologies. There is the chance to play in a world where books and stories can be either the private, cherished experience of old or a public, shared conversation with other readers from across the world.

So yes, the party's still on. It's not quite the same party, the drink's a good deal cheaper and we've got crisps, not caviar. But there are more people invited, and some of them look pretty groovy. I'll not get my coat just yet.

Lloyd Shepherd's debut novel, The English Monster, is published in March 2012. He hopes there will still be people around to buy it.