Understanding Search Engine Speed Bumps & Walls
In Part One of our “How Search Engines Work” series, we showed you how search engines go to work the nanosecond you submit a query. But search engines aren’t perfect machines, and some very basic navigation errors can hinder or entirely prevent search engines from reaching your website’s content.
As search engine spiders crawl the Web, they rely on the architecture of links to find new Web pages and revisit those that may have changed. Complex links and deeply buried site content often create “speed bumps” in the search engines’ process. And data that cannot be accessed at all by spider-friendly links qualify as “walls.”
Possible “Speed Bumps” for SE Spiders:
- URLs with 2+ dynamic parameters –take this beast for example http://www.url.com/page.php?id=4&CK=34rr&User=%Tom%. Spiders may be reluctant to crawl complex URLs like this because they often result in errors with non-human visitors.
- Pages with more than 100 unique links to other pages on the site. Spiders may not follow each one.
- Pages buried more than 3 clicks/links from the home page of a website. Unless there are many other external links pointing to them, spiders will often ignore deep pages.
- Pages requiring a “Session ID” or Cookie to enable navigation. Spiders may not be able to retain these elements the way your browser can.
- Pages that are split into “frames” can hinder crawling and cause confusion about which pages to rank in the results.
Possible “Walls” for SE Spiders:
- Pages accessible only via a submit button
- Pages requiring a drop down menu to access them
- Documents accessible only via a search box
- Documents blocked purposefully
- Pages requiring a login
- Pages that re-direct before showing content. Search engines call this cloaking or bait-and-switch and may actually ban sites that use this tactic.
The key to ensuring that your site’s content is fully crawlable is to provide direct, HTML links to each page that you want the search engine spiders to index. Remember that if a page cannot be accessed from the home page (where most spiders are likely to start their crawl) it is likely that it will not be indexed by the search engines. A sitemap can be of tremendous help for this purpose.
Once you’ve eliminated your site’s speed bumps and walls, check back here for Part Three of our Search Engine series where we examine the two biggest factors in your search rankings: popularity and relevancy.
Share This