More Research: Precision

Now that we’ve seen how little relative recall tells us about the merits of a search engine, let’s move on to something that is an excellent measure of search engine effectiveness: precision.

Precision is the number of relevant results retrieved by the search engine.  To evaluate this, I used the same method as in the Usmani article, giving the results a score from 0-2:

    • Relevant: 2 points
    • Less Relevant: 1 point
    • Page contains useful links: 1/2 point
    • Irrelevant, or error message such as 404 not found: 0 points

I decided only to analyze the first 40 results, whereas Usamni et al. analyzed 100 results.  This was due to two factors.  First, I was the only one doing this research and so there was a time concern.  Second, research shows that most searchers will only look at the first page of results.  40 results represents about 4 pages, which I felt was a good range to analyze precision.  I think it would be very rare for a searcher to go beyond the fourth page.

To analyze each result, I put myself in the position of a typical undergraduate student looking for an answer.  I ranked each result based on how well it would answer my question.  For example, if I was searching the term “APA style guide,” I put myself in the role of a student looking for how to cite sources using the APA style.  Pages which had examples of APA style were thus given a score of 2.  Here are the full results:

precision graph

The results are fairly mixed.  For one-word and simple-multi word searches (Q1.1 to Q6.2) Bing generally fares better.  For complex searches (Q1.3 to Q6.3), Google is more precise by a slight margin.  Except for Q3.1 (“textbooks”), the results of each search engine were very similar.

What you might notice is that there is an overall trend in these results.  I noticed it too, and this graph proves it:

Trends in precision

Trends in precision

 

The precision tends to go up as the search gets more complex.  This shouldn’t come as a surprise: most of us know that the more specific your search query is, the better our results will be.  My research presents good, hard evidence that this is true!

What I can say generally about this, having analyzed over 1,400 webpages, is that the results were very similar and I could get a very good answer to my search query within the first three results.  So if you want to get technical about it, Bing is better for simple one-word and multi-word searches, while Google does better for complex searches.  However, internet searching is not technical.  Searchers use natural language terms, and are looking for the quick-and-dirty answer.  What I can conclude then, is that Bing and Google are both excellent search engines when it comes to recall.

Advertisements


Categories: Bing vs. Google, Project Details, Research

1 reply

Trackbacks

  1. Search Terms: Mine vs. Theirs « My Month with Bing

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: