The Singularity is a conceit of modern science fiction: a place inside vast computers where whole universes are simulated whose reality is every bit as sharp and instantaneous as the physical world we inhabit. Books like Charlie Stross's Singularity Sky and the Matrix movie trilogy have done a great job of representing such alternative, computer-calculated realities.

Science fiction writer Rudy Rucker, however, has taken issue with the whole idea, arguing that the universe is already "calculating" reality far better than any software could manage.

After all, he says, the rock that you just dropped on your foot did a brilliant job of solving the physics of its velocity and direction, taking into account every conceivable variable from wind shear to the minute irregularities in its surface. Your nerve endings calculated, in flawless real-time, the screech of pain that was sent up your nerves into your brain, which in turn calculated precisely how much it all hurt and which cuss words flew involuntarily from your lips.

Most problems don't require such accurate solutions, so we can usefully approximate answers with computers – whether it's predicting the weather or angling the flaps on the wings of a jumbo jet to bring it in for a landing.

But the tiny approximations and fudge factors in those calculations add up. While we're pretty good at predicting the weather a day or two ahead, we are hopeless at predicting the weather a year from now. In order to get an accurate forecast, we'd need a whole solar system's worth of minute measurements and the computer with which to crunch them.

Total accuracy

In fact, argues Rucker, we have one such computer – the solar system itself – and it predicts the weather at the rate of one second per second, with total and complete accuracy. Storing representations of the solar system is inherently less efficient than the real thing, he says. To get a solar system's worth of accuracy out of a computer, your best bet is to just leave the solar system as it is and wait until the outcome you're curious about has arrived.

This conclusion flies in the face of the traditional approach to problem-solving: collect data, compute, collect more data, compute. But that approach gets you nowhere.

At a certain point, data gathered to predict the weather overwhelms your capacity to add it to your calculations efficiently, resulting in ever-longer runtimes that give less accurate predictions. It's better to crunch the data needed to calculate tomorrow's weather in 10 minutes (and refine your guess twice an hour) than to shovel so much data into the hopper that you don't get tomorrow's forecast until next week.

The sweet spot lies somewhere between gathering too much information and gathering too little – and the secret to hitting that spot is intelligent, discriminating data-acquisition.

Take London: cover every square inch of the city with CCTVs and you'll get so much information that you'll never make any sense of it. Scotland Yard says that CCTVs help solve fewer than 3% of all crimes, while a study in San Francisco found that at best, criminals simply move out of camera range, while at worst they assume no one is watching.

Similarly, if you take fingerprints from every person who applies for a visa – or worse still, from every person in Britain who has to carry one of the proposed new biometric cards – you will fill the databases with chaff that slows down searches, generates endless false matches, and threatens everyone in the database with the worst kind of identity theft.

Needles in a haystack

The problem of sifting through vast amounts of data was highlighted by the US 9/11 Commission, which concluded that the American intelligence community knew in advance that the attacks on the World Trade Center and Pentagon were in the offing, they just didn't know they knew it. The pieces were all there for anyone who knew to look for them, needles buried in a haystack of irrelevancies.

The answer in both America and Britain has been to collect more haystacks: useless, indiscriminately acquired information on

people who've done nothing to arouse suspicion. We even inveigle our citizens to become amateur curtain-twitchers and pecksniffs, demanding that they report "suspicious" activity to the authorities.

Between DNA databases, mandatory fingerprinting for visa seekers, CCTV carpet-bombing, and Oyster card data, we've never collected more "security" information than we do today. But does this really make us secure? Is it possible to know too much?

Outside of dramatically failed experiments like the Soviet Union and East Germany, policing has never been a business of gathering data on every single person and arresting the guilty ones. This doesn't catch guilty people, it ensnares the innocent and acts as a kind of monetary black hole, absorbing all the cash we can toss into it, growing larger and more voracious by the day.

Too much data ruins the investigation, every time.