Wolfram Alpha – the good, the bad, the ugly

May 17th, 2009

The most interesting search engine to be launched in years just went public: Wolfram Alpha

Wolfram Alpha sells itself as a ‘computational knowledge engine’. And it’s a very nice proof of concept. But as it stands there’s some things about it that are quite poorly developed, and some things that should be removed immediately.

Search of US GDP to get an idea of what it can do. You get a nice output of the value per year, automatically converted into $A in my case.

But some of the data here is badly out of date. For instance:

WALPH-1.png

GDP per capita 16th in the world? There are a few countries ahead of the US, but it shouldn’t be that many. Wikipedia (which looks right, but I haven’t double checked) says that the US should be more like 4th to 6th. It looks like it’s a combination of counting things as countries that economists don’t normally include, but also a product of outdated data.

Let’s try another example – seach for my name. A bit US centric, but that’s only to be expected I suppose. But here’s the neatest bit: if someone’s name is Robert, how old are they likely to be:

WALPH-2.png

The system has pretty sharp limits. For instance, at the moment I’m interested in Australia’s GDP growth. The basic search works fine, but the most recent growth rate they have is 2005. Which makes that search completely useless – what’s important is now, not 4 years ago. And even the US only has growth rates from 2005. That’s bad, bordering on dangerous for people who don’t check the fine print.

But finally, moving on to the ugly. If you search for a stock you get a lot of data. Try Google as a search on Wolfram Alpha. Mostly a nice summary of market data on the company, fine. But down towards the bottom of the page you get this:

WALPH-3.png

There is no way to describe this but dangerous and irresponsible. This projection will change every time you open the page, so don’t expect to see the results each time. The engine takes a well understood random walk approach, and produces projections using no information other than the historical variations in the stock.

But that’s a very silly thing to do. Sure, over history stocks and other real values seem to follow something that looks like a random walk. But that doesn’t mean that any individual random walk is of value. Or even a set of 5. This forecast has as much value as me asking my cat to walk across a sheet of paper with muddy paws, and then using the positions of the marks to project the stock. Soothsayers looking at entrails have more credibility than this, because they’re at least using their own judgement and intuition subconsciously.

Any informed observer won’t get hurt by this, but what about someone who isn’t that literate in this sort of analysis? What if the random odds give five paths that all point well up? And what if they then go and put their life savings in the stock as well.

If this sort of analysis is going to be included on the Wolfram Alpha search engine then it needs to be accompanied by far more prominent warnings about the use of this sort of analysis, as opposed to their current totally inadequate disclaimer.