The Root Cause

Well aware of the problems that IE6 and IE7 created for web developers, the IE8 development team took a hard look at the root causes of these problems. Unsurprisingly the core group of issues began with IE6; more specifically with the introduction of the DOCTYPE switch. The DOCTYPE switch was a method used by IE to alter the box model that is used in calculating the dimension and relative position of elements in a web document, depending on whether the page declared a DOCTYPE. If a web page declared a valid DOCTYPE it would be rendered in “standards mode” using the box model outlined in the CSS2 specification, otherwise the page would use the rather creative interpretation of the CSS2 specification employed by IE5.5. The intention of this behaviour was to avoid breaking the countless sites that were designed for IE5.5 (yes, it’s interpretation of CSS2 was that creative), while allowing developers to “opt-in” to the new standards mode.

Non-Standard Standards

The standards mode introduced in IE6 brought Internet Explorer’s implementation of CSS2 closer to the actual W3C specification, but introduced a new class of CSS peculiarities, many of which were hard to identify and would often result in drastic inconsistencies in web pages when compared to other browsers such as Firefox, Opera and Safari. This rose to a multitude of well-documented IE6 “CSS hacks” and the trend of creating a distinct IE6-only style-sheet which led to popular, somewhat tongue-in-cheek, belief that one had to design a web page twice: once for IE6 and once for everything else. IE6 had tipped the balance in favour of backwards compatibility.

Breaking Changes

When IE7 came along the use of CSS had become the norm and the adoption of web standards was continuing to spread, so IE7 was developed with more standards-centric approach, resulting in the first version of Internet Explorer that was in reaching distance of true standards compliance. However, this improved support came with a price, as many of the web pages that utilized the IE6-specific CSS workarounds were suddenly broken in this latest version of IE. Now the balance was tipped towards interoperability, at the expense of backwards-compatibility.

The Perfect Balance?

With IE8 well into development, things are looking good for this next major version of Microsoft’s flagship browser. With Internet Explorer 8 recently passing the Acid2 test, the IE8 development team proved that they mean business when it comes to improving IE’s standard’s compliance. They also learned from the problems that arose from the last two releases of Internet Explorer, and working together with The Web Standards Project, have come up with as solution that could provide that perfect balance between backwards-compatibility and interoperability with web standards.

The solution comes in the form of an IE8 specific meta declaration that allows a web developer to specify what rendering engine IE8 should use for a given page. This declaration will specify the exact version of Internet Explorer that the page is targeting, and IE8 will display the page using the rendering engine from that version.

meta http-equiv="X-UA-Compatible" content="IE=8" />

meta http-equiv="X-UA-Compatible" content="IE=7" />

The ‘Question’ related to the rash of bizarre search referrals that MSN was spreading around earlier this year.

The question was pretty much ‘where the heck is all of this search traffic coming from suddenly?’. Okay, suddenly meaning towards the end of August of this year, just like everyone else. So for an SEO Firm getting hits for the keyword ’search’, this could appear pretty flattering. Did I do something magical to suddenly send the MSNLive Bot into heat? Did someone hijack a PPC campain and we’re going to get a nasty bill from the folks at MSN Adcenter? After playing with those ideas for three to four seconds, it became brutally obvious that this was just bogus bot traffic masking itself with a search referral, thus triggering an entry into our Search Metrics. If I didn’t know any better I could possibly call that cloaking… The plot thickens. After spending some time reflecting on our Enquisite data a few items still didn’t ad up.

1. Enquisite was tagging this traffic as arriving from MSN Live. No one could recreate a human click from this engine using ’search’.
2. It was always the same keyword.
3. The same keyword hit dozens of alternate URL’s. One would think that specific traffic, though not very focused, would generally arrive at a single related page. (same phrase, same search engine)

The geo tagging was returning an ‘unresolved’ so I grabbed the raw logs and compared the referring strings. Here are the bits that matter.

• 65.55.165.51 (Ip of the ’searcher’)
• http://search.live.com/results.aspx?q=search&mrt=en-us&FORM=LIVSOP (Appears to be the referring SERP)
• http://www.metamend.com/what-is-cloaking.html (Target of ’search result’ - Irony?)

This is one of many examples. All were from the same Class A IP block. Have a look for yourself ARIN - 65.55.165.13. So if you caught the Owner there everything would quickly ad up. So then the question becomes “Why is MSN hammering my site with bogus search referrals which are completely skewing my metrics tools and tracking systems. As you can see from the screen above, we are still receiving traffic at time of writing.

Once I found the SERP string I started searching for other victims online and came across several informative posts including this rather enjoyable Post with an equally thoughtful title.

Early this month (I did mention how late I am getting to this right?) MSN posted on it’s brand new Live Search Webmaster Center Blog about this whole topic. They’ve done a pretty lousy job of responding to this epidemic of bad traffic, but hopefully it will result in some warfare in the SE marketshare. Anyhow all paths on this topic, up until the MSN post, ended at this discussion at Webmaster World. They’ve stated that these issues should be disappearing however we’re still seeing the traffic 2 weeks after their announcement so we’ll see how that goes.

Later.

]]> http://www.metamend.com/blog/2007/12/18/live-search-webmaster-center-a-question-answered/feed/ http://www.metamend.com/blog/2007/11/22/google-webmaster-tools-verification-errors/ http://www.metamend.com/blog/2007/11/22/google-webmaster-tools-verification-errors/#comments Thu, 22 Nov 2007 22:52:31 +0000 Alex Hlinski SEOTech Talk googlewebmaster tools http://www.metamend.com/blog/2007/11/22/google-webmaster-tools-verification-errors/ Google Webmaster Tools seems to be having some problems today with its web site verification process. Many of the sites we have listed in Webmaster Tools are no longer “verified” and the problem is reported as being due to “server timeout” and “domain name” errors. This is in spite of being able to navigate to the verification file using the link provided in the “Verify a Site” section.

This outlines the importance of frequently logging into Webmaster Tools to keep up to date on your sites search health and not assuming that this useful tool is “set and forget”. Google seems to be having periodic website verification and Sitemaps errors of late, has anyone else has noticed this?

The website concerned has around 4000+ article pages indexed, with each article page consisting of templated boilerplate sections and the main article text pulled from a HTML include file. Unfortunately on this particular day the directory containing the article include text files was accidentally deleted from the web server. Every article page subsequently served reported an ASP sever execute error as the article text include file directory was missing. The article include directory was only restored from a backup around 12 hours later.

Within several hours of the problem being reported it was obvious that the search engines had taken notice. According to the Enquisite search metrics data for the site, there was a definite and almost immediate reduction in search referrals. As it turned out, there was around 50% less search engine referrals that day compared to the preceding and following days. This demonstrated that the article pages had a temporary reduction in ranking, resulting in the loss of search traffic for that particular day. So what happened from a search engine point of view?

We can assume that if the search spiders visited that day they noticed that the article pages lacked content and provided no value. It is also possible that the search engines took into account the users’ probable immediate and quick bounce from every article back to the search results page. Either of those explanations could result in the temporary loss of ranking witnessed for the article pages. What is certain is that the search engines reacted quickly to the problem by punishing each articles’ ranking which resulted in a devastating loss of traffic.

Live Search obviously has a mountain to climb. Their index has been the brunt of criticism for far a good long time now. So much so that was witness a presenter as SES San Jose in ‘06 refer to their search index irrelevant and ‘who cares’ -ouch. Still, top positioning doesn’t mean that you’ll generate a ton of leads as there just aren’t enough MSN searchers at the moment. Even though IE ships with Live as the ‘default search privider’ it’s clear that it’s time for MSN Search to put up or… well. buy Yahoo! (I love that rumour) Hopefully this new buzz will encourage a few of us to wander back to the MSN for a while in hopes that a meaningful algorithm update is also in the works.

As long as you provide a HTML alternative of the same content for the spiders, and non-JavaScript enabled browsers, then the search engines will not have a problem with your Ajax pages. This will also allow you to reach an audience who have accessibility issues, such as the visually impaired or those using mobile devices.

As part of our initial examination of a client site our SEOs turn off JavaScript whilst browsing or use the Lynx text only browser to determine if all of the content and links can be seen by the spiders. Although I find Lynx to be quite useful for browser accessibility issues, it does not give a true representation of how a search engine spider will see a web page. By not displaying linked images that have a blank or missing alt attribute, Lynx can give the mistaken impression that some links are not spiderable. To get the most realistic impression, there are many “Search Engine Spider Simulators” that will show the text content and links that are visible to a spider after all of the HTML and JavaScript is removed from the page source code.

Although Ajax facilitates powerfully interactive web sites, the same old problems occur for the search engines as with Flash or JavaScript presented content. “Web 2.0″ may be the latest buzz word in web site design, but having already dealt with Flash and JavaScript for some time, SEOs know what is required.

Do you have a multitude of sites that you administer through Google Webmaster tools? We all know it can sometimes be difficult to keep all the sitemap.xml files up-to-date and uploaded to all those seperate web-server FTP accounts, especially if you do not have any sort of automated system to do it for you.

In the past you could only store a Sitemap on its respective host, but this restriction has now been lifted. As long as you can ‘prove’ you control the host, by the verification process in Webmaster tools, you can store the Sitemap on any other host that you control. This means that you could store all the individual Sitemaps on your own server or even have URLs for multiple websites contained within one Sitemap file.
There is one caveat however, in that this will only work for Google at the moment, although they do assure us that they are talking to the other search engines that support the Sitemap Protocol.
I would imagine that the simplest implementation would take the form of a change to the Sitemap declaration within the robots.txt file. As the site ‘owner’ would legitimately have access to the robots.txt anyway, this would neatly resolve the security issue that Google was concerned about. Check out the original Google Sitemap Cross Submissions post.

You can now control the country association of your content on a per-domain, per-sub-domain or a per-directory level. It’s as simple as adding a new website entry to your dashboard for the domain, sub-domain or directory that contains the international content, then using the “set geographic target” interface from the “tools” menu to specify the country.

Although this will work well for sites that determine their international pages using a separate domain or directory structure, it will not help dynamic sites that use a URL parameter for this purpose. Perhaps this is a future addition that we can look forward to. For more information read the Country Associations for your Content post from the Google Blog.

Conversely, Google seems to be doing the opposite and reducing the amount of competing pages. As an example here’s some data for the search phrase “search engine optimization”.

October 2007 - Google: 19,800,000 MSN: 41,644,219
September 2007 - Google: 38,200,000 MSN: 2,200,250

It seems that Google is refining its index to produce more relevant results; Is MSN bloating its index or just trying to play catch up? For those who have also picked this up, please share your thoughts.

All programming languages have semantic and syntactic rules that must be followed if the program is to run correctly, and HTML script is no exception. Unlike ’strict’ programming language compilers, web browsers can be very ‘loose’ in their interpretation of HTML, leading many web developers to take short cuts and ignore HTML code compliance. From the very beginning of the WWW, the W3C has defined the HTML script language and provides the W3C Validator to aid developers in creating compliant code.

HTML validation can be helpful to SEOs as it can outline code errors, missing alt attributes, incorrect and elements and tags and character encoding issues. A web page that successfully validates will have a better chance of displaying consistently across compliant web browsers. For the search engines, processing time and resources will be saved when extracting markup and reduces the chance that content will be stripped out along with the markup.

Code bloat is a double whammy, causing an increase in page load time for users and increasing the time for SE spider crawls across a whole site. Spiders only have a certain amount of time on a site during a crawl and anything that can be done to reduce the amount of code per page will increase the number of pages that can be indexed during this time.

There are a few simple steps that can be undertaken to make sure your code and resulting page size is as small as possible.

1. Remove your JavaScript code from the main HTML source into an external include file.
2. Externalise CSS in the same way (where appropriate).
3. Use CSS to format content instead of HTML font tags, table elements and other markup.
4. To reduce page load time on dynamic websites, reduce the amount of queries to the database per page and reduce the amount of records returned during each query.

The direct results of these methods in reduction of page size can be seen using the following example from Google Webmaster Tools. If we examine the crawl rate data for a website that has around 3500 pages, each of which originally contained many hundreds of lines of JavaScript code and large amounts of CSS declarations. The pages also had multiple sections and incorrectly defined elements. After removal of all of the unnecessary code and correcting the HTML syntax errors, the pages validated and were around 20% of their original Kb size.

As can be seen in the graph below, the time spent downloading a page has reduced (red), and consequently the GoogleBot has been able to crawl more pages per day (blue).

Alt attributes for images are mandatory within the W3C HTML specification and are a Priority 1 checkpoint in the W3C’s Web Content Accessibility Guidelines (WCAG) . Alt attributes are essential in conveying the ‘meaning’ of images for the visually impaired through their use of talking browsers, screen readers or text only browsers. The most important consideration for SEOs is that Google strongly recommends the use of alt attributes for images.

From Google Webmaster Help

“If your site contains elements that aren’t crawlable by search engines (such as Flash, Javascript, or images)….

…. you should consider visitors to your site who are unable to view these elements as well. For instance:

* Provide alt text that describes images for visitors with screen readers or images turned off in their browsers.

Including substantially different content in the alternate element may cause Google to take action on the site.”

Alt attributes should be present within every image element in the HTML code and should describe the content of the image. For spacer images use the null alt text, alt=”", as these images offer no useful information to the user. If the image is active or performs a function, within an anchor link for example, then the alt text should describe the function. If the image is not active, but conveys information, then the alt text should contain the same information. For SEOs the alt attribute is useful as it can contain relevant keywords, however be careful as keyword stuffing alt attributes will have a very negative effect (note the Google excerpt above). Optimization of images is increasingly becoming an important factor and will have an effect within image searches, universal search (news article images, organic image results) and product search (Froogle / Google Products).

The alt attribute is one of the more neglected aspects of web site design and it will be interesting to see if web designers and developers take note of the outcome of the Target lawsuit. Depending on the success of this class action, the California courts may end up saving SEOs a considerable amount of time and energy.

Resources

W3C Validator - Check HTML validation and discover missing alt attributes on your pages.

Use the excellent Web developer browser plugin to outline images that are missing the alt attribute as you browse your pages; (not for Internet Explorer)

An accessibility checking tool will outline a web pages’ issues with accessibility, privacy and quality.