SEO Blog

Metamend SEO specialist Colin Cochrane posted an interesting find on his blog Saturday. While collecting materials for work on one of his personal sites, Colin attempted to extract material he had previously bookmarked at del.icio.us. Using a Firefox add-on that had its User-Agent set to Googlebot, Colin searched for the specific file he wanted but instead was greeted with a 404-error page.

After running through the list of mistakes he might have made and eliminating each of them, he remembered he had reset the User-Agent to Googlebot when researching if another site was cloaking or not. In other words, the search he was conducting on del.icio.us looked as if it was being performed by Googlebot.

After resetting the User-Agent to default and resubmitting his search-query, del.icio.us delivered the results Colin originally expected. Puzzled by the experience, Colin checked the robots.txt files in the del.icio.us source code and found it was disallowing bots from all major search engines, including that of its owner, Yahoo.

“Puzzled by this, I took a look at del.icio.us’ robots.txt and found that it was disallowing Googlebot, Slurp, Teoma, and msnbot for the following:

Disallow: /inbox
Disallow: /subscriptions
Disallow: /network
Disallow: /search
Disallow: /post
Disallow: /login
Disallow: /rss