So you enter a search term into a search engine like Google, Yahoo!, or MSN. What happens next? Read on to find out… 

Search engines have a short list of critical operations that allows them to provide relevant web results when searchers use their system to find information.

1. Crawling the Web

Search engines run automated programs, called “bots” or “spiders” that use the hyperlink structure of the web to “crawl” the pages and documents that make up the World Wide Web. Estimates are that of the approximately 20 billion existing pages, search engines have crawled between 8 and 10 billion.

2. Indexing Documents

Once a page has been crawled, its contents can be “indexed” - stored in a giant database of documents that makes up a search engine’s “index”. This index needs to be tightly managed, so that your search requests (which require the search engines to search and sort billions of documents) can be completed in fractions of a second. Continue Reading…