"Wolfram|Alpha isn't sure what to do with your input." I've been seeing that phrase a lot lately, as I spent a bit of this weekend trying to figure out what the self-styled computational knowledge engine was good for. The conclusion I'm reaching, at least in part based on the frequent appearance of that text, is that the answer will wind up being "not a whole lot," at least not if you've got a general interest in science, as I do. That's not because I have a fear of math or expect that the folks behind it will never overcome some of its current limitations. Instead, I think that Wolfram's approach with this new tool is such that Alpha will largely be limited to producing information that, on a fundamental level, is trivial.

Before trashing the service, though, I should spend a bit of time on its good points. The natural language parser appears to be quite impressive; if it has an answer for something, it seems to be able to take you to that answer using a variety of linguistic approaches. The graphing production is both first class and fast, and there are definitely times that the pictures it produces are worth a thousand words. For a variety of items, the unit conversions are the same: on target and informative.

That's not to say it's designed to handle anything you throw at it, as it's important not to have unrealistic expectations for the service. This is something that fundamentally deals with numbers, not concepts. So, for example, plugging something like "RNA world" into it is a nonsensical thing to do, and it's no surprise that doing so elicits another case of "Wolfram|Alpha isn't sure what to do with your input."

So, in essence, Wolfram's asking you to outsource all the sanity-checking that should be done when obtaining information—does it come from a reliable source? Is it in keeping with other figures that are available? You're just supposed to trust that the folks that populated the databases behind Alpha have done their job well.

The avoidance of concepts isn't complete, though. If you plug in the RNA base "uracil," for example, it recognizes that uridine (the form that's hooked up to ribose) is likely to be what you're after. Alpha also helpfully offers to treat it like a word, which gives you a one-sentence description of its base-pairing behavior when part of RNA. But Wolfram's tendency to treat everything like a value rears its ugly head here. The fact that biologists abbreviate it as U causes Alpha to lump it in a synonym network with Uranium and atomic number 92, which are completely unrelated. Beyond that, the entry has some useful reference values like melting point and density, but I'd generally trust Wikipedia to get those sorts of trivial details right.

There are a number of rough spots that may be possible to fix or at least improve. If, for example, I was interested in "global bioethanol production," I'd get the dreaded "isn't sure what to do with your input" message. Does that mean it doesn't recognize bioethanol? That it does but doesn't have global figures? That it's not sure what to make of my use of "production" in this context? Currently, Alpha gives me no clue. There's also no way to find out if a given term ever does appear without checking back every month in the hope that someone's decided to parse a source that contains it.

Another problem became apparent when I plugged in "25 Gigatons carbon dioxide," which produced a small section of comparisons. None of these were precisely the figure I was interested in, namely a comparison with the US automotive output of carbon dioxide, and Alpha offers no way to search for other comparisons. Again, this can be fixed in an update, but the problem of useful comparisons was small change compared to the bigger issue the page revealed: a problem of accurate comparisons.

Data on my search result page indicated that, in 2003, global human activity led to 27 Gigatons of carbon dioxide emissions. But it also indicated that, in 2000, the US produced 40 Gigatons during cement production alone. One of these has got to be wrong, and Alpha gives you absolutely no way of finding out which; the page doesn't have a link to the source of its data.

Even where it ostensibly does link sources, the links aren't very informative. Returning to ethanol (which I'm prone to do), it's possible to follow links that eventually lead to Mathematica's list of ChemicalData sources. That lists about 25 items as "among" the principal sources, but gives you no clue as to what information on a given page came from where. So, for example, even if you find that a page has data from a usually reliable source like the CIA's World Factbook, there's no way of knowing which data on the page comes from there.

So, in essence, Wolfram's asking you to outsource all the sanity-checking that should be done when obtaining information—does it come from a reliable source? Is it in keeping with other figures that are available? You're just supposed to trust that the folks that populated the databases behind Alpha have done their job well. If they haven't, it's up to you to figure that out.

Given my initial experiences with Alpha, there seem to be at least three things that make it less useful than a general Web search. For one, Alpha is entirely dependent upon the ability of Wolfram's employees to draw in data and, unless Alpha earns the company a GDP-sized income, said employees will always be outpaced by the production of not only new data, but by new data storage formats. That necessarily means some searches will fail, which brings us to problem number two: it's not obvious that there's a way to make them fail usefully. A bad Web search typically brings up results that help you refine your search terms; a bad Alpha search returns nothing, and it's not clear that there's an easy way to fix that.

But the biggest issue is that, in the process of creating the data store behind Alpha, all the context behind a number—who produced it, what were their methods, how was the raw data obtained, is the number actually relevant for a given analysis, etc.—is stripped (or, if it's there, you can't tell from the results). And, for most things other than trivial figures like density and melting point, the source and other contextual information is every bit as important as the number itself. This is why I said at the start that Alpha is only good for trivial uses.

If I can ever figure out which one of the numbers Alpha fed me on carbon emissions is right, though, I'd be happy to have Wolfram produce a nice graph of it for me.