Earlier this week, Microsoft released a study that shows that being in the Top10 is simply not good enough anymore. The paper, “Eye tracking in MSN Search: Investigating snippet length, target position and task typesâ€, examines where MSN users look when viewing search results, and if the length of the descriptive text, or “snippets†found beneath search results influences click rates.
Assigning a series of search related tasks to six 3-person test groups, the researchers found that the test subjects, “… seem to exhibit an implicit trust for the rank generated by the search engine…â€. They also found that the length of information snippets shown beneath search results effects click rates, depending on the type of task the research subject was assigned. The published report is full of stats and charts and scientific explanations but it boils down to this.
There were two types of search-task assigned to the test groups. The first was called an informational search, in which a subject was told to find something specific. The second type of search-task was called a navigational search, in which a subject is told to find their way from point A to point Z.
Examples of informational search tasks include finding the address for the Newark Airport and the length of the Las Vegas Monorail. Navigational search tasks included finding the homepage of the “Pinewood†software company and Yahoo! People Search.
When following both types of tasks, the subjects tended to trust the Top5 results. They tended to see those results faster and spent more time pursuing information through them. Search results that appeared lower in lists, often below the fold of the page tended to receive less attention and thus, less clicks. The researchers also found that the subjects scored more accurate results performing navigation related search tasks than they did when looking for information.
The orange line in the chart below demonstrates how long it took the research subjects’ eyes to glance (or arrive) at, a particular search result. The green bars note how long test subjects looked at (or fixated on) each result.
The researchers also found that the descriptive text below search results, the snippet, had an effect on which references were chosen by the subjects.
When trying to get to a certain page, navigation search, test subjects could refer to the URL displayed below the information snippet. That made finding the correct results much easier. When searching for information however, subjects tended to rely on information found in the snippet. They also relied on the position the search engine algorithm assigned the document. In other words, even if a more relevant document might be found in positions 6 – 10, the test subjects tended to choose to extract information from documents with well written snippets ranked 1 – 5.
“Further eye movement analysis showed that people mainly look at the results on top of the list and this explains the uniform decreases in click accuracy for both navigational and informational search. The analysis also suggests that the large decrease in performance for informational search was likely due to the strong confidence on search engine performance even though people know that target results at lower positions can be relevant. People are more likely to degrade their own judgment of target results and trust the ranking determined by the search engine.” (pg. 10)
One interesting finding showed that, when presented with longer information snippets, test subjects tended to not note the URL of the page, possibly because they gathered enough information to make a decision before seeing the URL.
The report concludes with a conundrum for search engine designers. The snippet length has an effect on search. Long snippets make information search easier while making navigation search more difficult. Conversely, short information snippets made finding the URL easier during the navigation search-task tests but didn’t provide enough data for informational search-task tests.
The paper is worth the read, even if one has to read certain passages a few times to cut through the density of the language. With a bunch of charts, and a lengthy body of cited research to troll through, MSN’s search results eyetracking study teaches a lot about search-user behaviours.
Oh… by the way… The researchers did have a suggested fix for the short-long snippet conundrum. Place the URL directly beneath the reference link.
Have a question and need quick answers? 
