A few days ago I started a tool called BSET – Blackbox Search engine Testing tool to evaluate how good Bing really is. If you watch the stats on the page, its clear which search engine is being consistently picked as the winner.
The results were collected from 518 unique source IP addresses (some were just NATs from larger organizations). 251 users just executed 1 query each. 111 users executed 2 queries and rest executed more than that.
A total of 808 results were submitted just for “standard web search” category and of that 44% of the submissions were in favor of Google. 32% of them were for Yahoo. Only about 28% results went for Microsoft’s new search engine “Bing”.
Between Google and Yahoo, a user is 15% more likely to pick Google than Yahoo. Between Google and Bing, a user will pick Google 21% more frequently than Bing.
The results may not be staggering for folks who have been following search engine trends over the last few weeks, but for me, to see the results from this random test is surprising considering the amount of money Microsoft plans to pump into Bing’s advertisement. I wish I had done this test before Bing was launched to find out how different MSN is from Bing…
So why is Google better ?
Since search results were pulled using published search APIs from the search providers and because these search APIs may not always show the same results which users see on the real search page, it could be argued that these results may be inaccurate.
Another problem I noticed is that different search engines behave differently when there are spelling errors in search. For example look at the results for “steven hakwing” ( was looking for Stephen Hawking) on the 3 search engines
Bing – Bing tells you that you could have spelt is wrong, and shows results for “steven hawking” instead.
Yahoo – Yahoo warns me that I should probably correct my spelling to “Stephen Hawking” but shows the search results for “steven hakwing”
Google – Google suggests that I could be looking for “Steven Hawking”, but actually shows me results for “Stephen Hawking” which is what I really wanted.
Since I didn’t use spell-sugession APIs to correct the search terms before it was submitted, it could be argued that my tests are biased towards google which does auto-correction. But as an end-user, I could argue that that I want to see what I intended to type and not what I actually typed. I think the ability to predict what users are thinking is is one of the core reasons why Google has a lead over other search engines.
And as for Bing’s cash-back plan, a friend of mine said that he’d be happy to use Bing to buy something.. as soon as he figures out what he really wants on Google or Yahoo.
I welcome your comments or feedback, especially if you have ideas on how I could improve the tests.
