Now that we’ve seen how little relative recall tells us about the merits of a search engine, let’s move on to something that is an excellent measure of search engine effectiveness: precision.
Precision is the number of relevant results retrieved by the search engine. To evaluate this, I used the same method as in the Usmani article, giving the results a score from 0-2:
- Relevant: 2 points
- Less Relevant: 1 point
- Page contains useful links: 1/2 point
- Irrelevant, or error message such as 404 not found: 0 points
I decided only to analyze the first 40 results, whereas Usamni et al. analyzed 100 results. This was due to two factors. First, I was the only one doing this research and so there was a time concern. Second, research shows that most searchers will only look at the first page of results. 40 results represents about 4 pages, which I felt was a good range to analyze precision. I think it would be very rare for a searcher to go beyond the fourth page.
To analyze each result, I put myself in the position of a typical undergraduate student looking for an answer. I ranked each result based on how well it would answer my question. For example, if I was searching the term “APA style guide,” I put myself in the role of a student looking for how to cite sources using the APA style. Pages which had examples of APA style were thus given a score of 2. Here are the full results:
The results are fairly mixed. For one-word and simple-multi word searches (Q1.1 to Q6.2) Bing generally fares better. For complex searches (Q1.3 to Q6.3), Google is more precise by a slight margin. Except for Q3.1 (“textbooks”), the results of each search engine were very similar.
What you might notice is that there is an overall trend in these results. I noticed it too, and this graph proves it:
The precision tends to go up as the search gets more complex. This shouldn’t come as a surprise: most of us know that the more specific your search query is, the better our results will be. My research presents good, hard evidence that this is true!
What I can say generally about this, having analyzed over 1,400 webpages, is that the results were very similar and I could get a very good answer to my search query within the first three results. So if you want to get technical about it, Bing is better for simple one-word and multi-word searches, while Google does better for complex searches. However, internet searching is not technical. Searchers use natural language terms, and are looking for the quick-and-dirty answer. What I can conclude then, is that Bing and Google are both excellent search engines when it comes to recall.