Search Engine Optimization FAQ
Search Engines 101

At Metamend, we understand the subject of Search Engine Optimization (SEO) technology is a new territory for many. We hope these FAQ pages become a valuable source of information for you. If there are any questions not answered directly by this FAQ, please contact us via our Feedback Form.

Specific FAQ Topics:

Read our SEO Services pamphlet [PDF] and see for yourself what our clients our saying about us!

Search FAQ by keyword:

Search Engines 101:

Search Engines 101:

What is a "search engine" and what does it do? top
A search engine directs visitors to a website based on the terms of reference indexed for that site and the search query used. There are many different types of search engines, and the subject matter is fairly complex.

What is a "spider?" top
A spider is an extremely useful software routine that follows links between pages. Search engines use spiders to analyze a website's keywords, validating them against that site's content.

What are "meta tags?" top
Meta tags are lines of code embedded into web pages read by search engines. These tags may contain keywords, descriptions, site author information, copyright information, site titles and more.

Will good page design affect the rating of my website? top
Yes. Badly designed pages with broken links, spelling errors, or slow load times, may be ignored by some spiders. Check out our FAQ Section: Website Design Tips for more information regarding site design.

How do search engines work, and how do I use them effectively? top
Check out our FAQ section: How to Effectively Use The Search Engines for more information.

How do search engines work from a query perspective? top
Search engines work by maintaining large databases of information assembled by spiders. When you type in keywords or phrases in a search, you're entering parameters the engine compares to its database. The most relevant sites are near the top of a returned Search Engine Results Page, or SERP.

Some of the main parameters engines use to predict relevance are based on a site's textual content, the frequency of search terms within each page, and the amount of pages containing the search terms. This is why it's important to use as much relevant textual content as possible when building a site. The more textual content on a site, the more our Metamengine and the spiders have to work with when trying to categorize it.

What's the difference between a "directory" and a search engine? top
A directory is organized by subjects similar to a book's table of contents. Directories tend to have a small abstract for each of the entries to help you determine if the listings are appropriate to you. Directories typically have human editors ensuring high quality of content, but they tend to have a smaller searchable database.

A search engine is similar to a book's index. When using a search engine, you're looking for specific keywords within the search engine's database. Search engines tend to be automated, ensuring there's a large database to search through, but they lack human editors for quality control.

How can I tell if my site is registered with specific search engines? top
Look for your website's URL in the different search engines. For example, if you want to figure out if is in Google, go to and enter "" in the search box. Only sites with the string "" in Google's database will be returned.

What is the "robots.txt" file? top
Your website should have a file in the main web root called "robots.txt." This file is used to communicate with search engine spiders (a.k.a. "robots") telling them what they cannot have access to.

*Please note: this file is only a suggestion to the spiders, and not a way to secure directories and files from being publicly seen. For that, you have to ensure the proper access permissions are set for your files.

Here are some examples of the robots.txt file. *Please note the difference between the "Include all" and the "Exclude all" examples.

Standard Example: (Include all)
User-agent: *

In this example, you have invited the spiders to look through every file in your web directory structure.

Standard Example: (Exclude all)
User-agent: *
Disallow: /

In this example, you have requested your entire web directory structure not be spidered!! Be careful here! With one character you can cause the spiders to ignore you.

Two central problems accompanying robots.txt files:
  1. Not all search engines comply: Although all the major search engines claim to adhere to the Robot Exclusion Standard, some webmasters have complained of engines indexing pages from their "robots.txt" files (most of the time the major search engines comply).

  2. Competitors may see your hidden files: Robots.txt files are by no means secret, and not meant to secure files. Since they're always located in the root directory, your competitors can access yours simply by typing in the URL. You may want to password-protect any secret files mentioned in your robots.txt file.

If you have any questions that are not answered by this FAQ page, please contact us via our Feedback Form.

Please read our License Agreement for more information regarding accounting procedures.