Define: Spider

Spider: Many search engines use programs called spiders to index web sites. Spiders follow hyperlinks and gather textual and meta information for use in the search engine databases. Spiders may also rate the content being indexed to help the search engine determine relevancy levels to a search.