BSET SearchEngine relevance test results
A few days ago I started a tool called BSET â€“ Blackbox Search engine Testing tool to evaluate how good Bing really is. If you watch the stats on the page, its clear which search engine is being consistently picked as the winner.
The results were collected from 518 unique source IP addresses (some were just NATs from larger organizations). 251 users just executed 1 query each. 111 users executed 2 queries and rest executed more than that.
A total of 808 results were submitted just for â€œstandard web searchâ€ category and of that 44% of the submissions were in favor of Google. 32% of them were for Yahoo. Only about 28% results went for Microsoftâ€™s new search engine â€œBingâ€.
The results may not be staggering for folks who have been following search engine trends over the last few weeks, but for me, to see the results from this random test is surprising considering the amount of money Microsoft plans to pump into Bingâ€™s advertisement. I wish I had done this test before Bing was launched to find out how different MSN is from Bingâ€¦
So why is Google better ?
Since search results were pulled using published search APIs from the search providers and because these search APIs may not always show the same results which users see on the real search page, it could be argued that these results may be inaccurate.
Another problem I noticed is that different search engines behave differently when there are spelling errors in search. For example look at the results for â€œsteven hakwingâ€ ( was looking for Stephen Hawking) on the 3 search engines
Bing - Bing tells you that you could have spelt is wrong, and shows results for â€œsteven hawkingâ€ instead.
Yahoo â€“ Yahoo warns me that I should probably correct my spelling to â€œStephen Hawkingâ€ but shows the search results for â€œsteven hakwingâ€
Google â€“ Google suggests that I could be looking for â€œSteven Hawkingâ€, but actually shows me results for â€œStephen Hawkingâ€ which is what I really wanted.
Since I didnâ€™t use spell-sugession APIs to correct the search terms before it was submitted, it could be argued that my tests are biased towards google which does auto-correction. But as an end-user, I could argue that that I want to see what I intended to type and not what I actually typed. I think the ability to predict what users are thinking is is one of the core reasons why Google has a lead over other search engines.
And as for Bingâ€™s cash-back plan, a friend of mine said that heâ€™d be happy to use Bing to buy something.. as soon as he figures out what he really wants on Google or Yahoo.
I welcome your comments or feedback, especially if you have ideas on how I could improve the tests.