After thinking about that for a while, I came up with a short essay which you can find below the cut (tidied up a little to remove some annoying snags from the original) ...

This question relies on so many implicit underlying assumptions that I feel some dissection is in order before I can try to answer it.

The first unquestioned assumption is the post-18th century Enlightenment concept of progress. This, if anything, is the ideological bedrock underlying "ideas" SF — that Things Can Get Better. Historically, almost all civilisations prior to the Enlightenment ran on the mythology of a distant golden age in the past, which bequeathed us a bunch of moral precepts and firm knowledge about how the world works which we poor degenerates living in the debased relics of a higher civilisation should turn to for guidance. The very concept that we are actually discovering how the universe works, and improving our lives, was a revolutionary rupture with the past — and one that took a long time to sprout any kind of literary or artistic shoots.

The second assumption is that science fiction has ever primarily been a genre of big ideas. I'm not at all sure that this is the case. Certainly fiction with big ideas has found a home within SF, but that's not the same thing at all! For almost all of its history, most SF has been pulp adventure fiction, conceived and written as escapism — lest we forget, Damon Knight's original characterisation of space opera as horse opera (the Western) with blasters instead of six-guns and space ships instead of horses still holds water. The big ideas are, if anything, secondary, not to mention exhibiting a tendency to date badly and carry sinister ideological overtones (as William Gibson so brilliantly skewered in his short story "The Gernsback Continuum").

Calls for more big ideas in SF are generally a political cri de coeur. We might equally imagine a similar essay in the context of mid-sixties Soviet fiction, calling for more fiction about tractors and breakthroughs in agricultural genomics. Whether earnest depictions of young people in space suits wrangling asteroids and bringing home the carbonaceous chondrites actually make sense is another matter entirely; I'm inclined to think that it's a rather implausible future, unless the earnest young people are robots. Tinned apes don't survive exposure to vacuum and cosmic radiation very well, after all.

But. I have my nagging doubts. Because, despite my cynical pose, I am more than a little sympathetic to Stephenson's project, because I share his axiomatic belief in the loose constellation of post-Enlightenment values that brought us this idea of progress and constant improvement. If only because when you stop moving you're dead, and reverting to a late palaeolithic lifestyle looks like it would be a drag, and that's the most likely alternative long-term future for our species if we burn all the coal and oil, wreck the climate, and turn our back on the Enlightenment's ideological values.

In recent decades SF has been spinning its wheels. In fact, in the past 30 years the only truly challenging new concepts to come along were cyberpunk and the singularity. Both of which amount to different attempts within the genre to accommodate the first-order implications of computers and networking as the defining technology of the near future (as opposed to rockets! for! everyone! a la "Space Family Stone") — cyberpunk was the sociological/post new wave SF modelling of a future derived from the 1970s and 1980s weltanschauung, and the singularity was the chew-toy of those members of the hard SF brigade who actually understood computers. There were other movements, true, and possibly more visible to onlookers; urban fantasy and its hybrid offspring (by way of genre romance) the paranormal romance: steampunk (both first, second, and the current third wave varieties): and a huge bloom in alternate history/counterfactuals as much of the rigorous world-building effort that had formerly gone into the near-future space SF field turned sideways and looked for other outlets. But none of these seem to engage with the future in the way hard SF supposedly did in the 1940s to 1960s. What we call "hard SF" today mostly isn't hard, and isn't SF: it's fantasy with nanotech replicators instead of pixie dust and spaceships instead of dragons. Explorations of Singularity teeter dangerously on the precipice of a tumble into Christian apocalyptic eschatology, and in any event beg far too many questions about the nature of intelligence to make a convincing stab at artificial intelligence.

In fact, those people who are doing the "big visionary ideas about the future" SF are mostly doing so in a vacuum of critical appreciation. Greg Egan's wonderful clockwork constructions out of the raw stuff of quantum mechanics, visualising entirely different types of universe, fall on the deaf ears of critics who are looking for depth of characterisation, and don't realize that in his SF the structure of the universe is the character. On Hannu Rajaniemi's brilliant "The Quantum Thief" — I have yet to see a single review that even notices the fact that this is the first hard SF novel to examine the impact of quantum cryptography on human society. (That's a huge idea, but none of the reviewers even noticed it!) And there, over in a corner, is Bruce Sterling, blazing a lonely pioneering trail into the future. Chairman Bruce played out cyberpunk before most of us ever heard of it, invented the New Space Opera in "Schismatrix" (which looked as if nobody appreciated it for a couple of decades), co-wrote the most interesting hard-SF steampunk novel of all, and got into global climate change in the early 90s. He's currently about ten years ahead of the curve. If SF was about big innovative visions, he'd need to build an extension to house all his Hugo awards.

So what's at the root of this problem? Why are the innovative and rigorously extrapolated visions of the future so thin on the ground and so comprehensively ignored?

I'd put it down to us mistaking Sense of Wonder for Innovation. We used to read SF to get the heady high of a big vision, the "eyeball kick" as Rudy Rucker describes it, of seeing something brain-warpingly different and new for the first time. But today you don't need to read SF to get a sense of wonder high: you can just browse "New Scientist". We're living in the frickin' 21st century. Killer robot drones are assassinating people in the hills of Afghanistan. Our civilisation has been invaded and conquered by the hive intelligences of multinational corporations, directed by the new aristocracy of the 0.1%. There are space probes in orbit around Saturn and en route to Pluto. Surgeons are carrying out face transplants. I have more computing power and data storage in my office than probably the entire world had in 1980. (Definitely than in 1970.) We're carrying out this Mind Meld via the internet, and if that isn't a 1980s cyberpunk vision that's imploded into the present, warts and all, I don't know what is. Seriously: to the extent that mainstream literary fiction is about the perfect microscopic anatomization of everyday mundane life, a true and accurate mainstream literary novel today ought to read like a masterpiece of cyberpunk dystopian SF.

We people of the SF-reading ghetto have stumbled blinking into the future, and our dirty little secret is that we don't much like it. And so we retreat into the comfort zones of brass goggles and zeppelins (hey, weren't airships big in the 1910s-1930s? Why, then, are they such a powerful signifier for Victorian-era alternate fictions?), of sexy vampire-run nightclubs and starship-riding knights-errant. Opening the pages of a modern near-future SF novel now invites a neck-chillingly cold draft of wind from the world we're trying to escape, rather than a warm narcotic vision of a better place and time.

And so I conclude: we will not inspire anyone with grand visions of a viable future through the medium of escapism. If we want to write inspirational literature with grand visions we need to dive into to the literary mainstream (which is finally rediscovering fabulism) and, adding a light admixture of Enlightenment ideology along the way, start writing the equivalent of those earnest and plausible hyper-realistic tales of Progress through cotton-planting on the shores of the Aral sea.

But do you really want us to do that? I don't think so. In fact, the traditional response of traditional-minded SF readers to the rigorous exercise of extrapolative vision tends to be denial, disorientation, and distaste. So let me pose for you a different question, which has been exercising me for some time: If SF's core message (to the extent that it ever had one) is obsolete, what do we do next?

Any answers gratefully received.