Within a decade, computers will be able to plough through scientific data looking for patterns and connections – then tell scientists what they should do next

Move over scientists – computers will be asking the questions from now on. They will trawl the millions of scientific papers on the web and suggest new hypotheses for humans to test, according to an article in tomorrow's issue of Science.

Scientists are drowning in data. Whether it's high-speed genome sequencing, simulating the early universe or testing complex mathematical proofs, there are often more numbers to crunch than there are people to crunch them. But help is on the way in the form of "automatic hypothesis generation", argue James Evans and Andrey Rzhetsky of the University of Chicago.

"Computer programs increasingly are able to integrate published knowledge with experimental data, search for patterns and logical relations, and enable new hypotheses to emerge with little human intervention," they write. "We predict that within a decade, even more powerful tools will enable automated, high-volume hypothesis generation to guide high-throughput experiments in biomedicine, chemistry, physics, and even the social sciences."

Evans foresees a time when computers crawl the millions of scientific papers online, linking and analysing data and concepts, then suggesting new hypotheses to test. "Wouldn't it make more sense to extract information from the huge corpus of previous research and put it together [to form new hypotheses]?" he told me.

For more than 20 years cosmologists have been using computers to test models designed by people. "The new twist here is that the computer can be given information and told to find its own model explaining the data or the connections between different pieces of data, after some 'ground rules' are set by the user," said Carlton Baugh of the Institute for Computational Cosmology at Durham University, who uses whole armies of computers to run "massively parallel" calculations of how different structures form in the universe.

"With pattern finding, the computer is asked to uncover a connection or relation defined by the user within a dataset. With hypothesis generation, the computer has more flexibility to come up with different patterns to test."

Automatic hypothesis generation may also prove invaluable in genetics.

Increasingly, geneticists can conduct studies from their computers. In genome-wide association studies, they compare all the genes of people who have a disease with those of a healthy control group to find mutations related to the disease. Studies like this have identified risk genes for type 2 diabetes and breast cancer, for example.

There are millions of human DNA sequences stored in online databases such as GenBank, and dealing with this information has spawned a new discipline called bioinformatics, which involves applying statistics and computer science to biological problems. Automatic hypothesis generation could take it one step further.

Dawn Field, head of the molecular evolution and bioinformatics group at the Centre for Ecology and Hydrology in Oxford, says Evans and Rzhetsky are on the right track. "This will become more and more possible in the future. We are just experiencing the beginnings of this field of endeavour."

Some argue that new knowledge will emerge by mechanically applying algorithms to find patterns in large datasets. But pattern-finding without knowing the theoretical context of a field has potential pitfalls, write Evans and Rhetsky. They compare it to the task of an explorer in an unfamiliar jungle without a guide: "With no sense of what is already known about the environment or its perils, [the explorer] is likely to misclassify what she sees – fearing the intimidating but harmless snake; ignoring the tiny lethal frog."

Dr Dietrich Rebholz-Schuhmann of the European Bioinformatics Insititute told me the question was whether a computer can generate hypotheses that can be validated easily. "A computer can propose experiments in combination with a hypothesis in such a way that the hypothesis can be validated in the experiment. This is an important step, but still far away from what humans do."

Automatic hypothesis generation may also have a role in identifying bridges between disciplines. "[Finding] amazing links between facts coming from different disciplines is the most exciting possibility," said Field. "It is hard for working scientists to have more than a shallow knowledge of subjects not in their direct area of expertise and yet there are often many 'low hanging fruits' at the intersection between two fields just ready for the picking."

Within a given field of scientific enquiry, write Evans and Rzhetsky, unpublished connections are likely to represent "negative knowledge" – ideas considered implausible by scientists in that field. Between fields, however, those unpublished connections might turn out to represent unanswered questions. Automatic hypothesis generation is one way of finding those unanswered questions, they argue.

Linking disparate fields of research automatically will be no easy task, however, not least because different terms mean different things to different scientists. Semantic integration – speaking a technical language that all the fields have in common – is important.

"There's no question that semantic integration is a major challenge," said Evans. "But looking at associations between terms also ends up being an opportunity."

Of course there are pitfalls with the automated approach. Computers could find a promising pattern that leads to nothing, for example, or even suggest blind avenues. "There is still a need for a scientist with a basic understanding of the problem to design the framework of such experiments and to interpret the results," said Baugh.

Large-scale computation of this sort is already being applied to problems of "systems thinking".

"This is when you have enough experts all thinking together across disciplines that the 'big picture' emerges and you can solve 'big problems'," said Field. "You can unravel a long trail of causality. X causes Y causes Z etc ... This is especially hard to do unless you have cross-discipline thinking."

She said efforts to tackle global issues such as climate change and the long-term conservation of biodiversity could benefit from this approach.

If computers could start to pull together global patterns and trends, then make predictions or or suggest solutions, said Field, "it would be amazing".

Watch this cyber space.

Do you deal with large datasets? Do you share Evans and Rhzetsky's optimism for automatic hypothesis generation? Post your comments below ...