Web crawlers, bots, spiders, indexers, ants and worms - a whole host of terms have been applied to the computer programs that browse the Internet for the search engines, collecting and storing information for searchers to use. The search engine spiders are critical to the online world since they open up websites as well as new objects to web traffic from across the planet. Search engines are the superhighways of the Internet and their spiders lead traffic through the many routes, the twists and turns of a huge and still expanding World Wide Web.

Since search engines play a key role in connecting people and information on the Web, a thorough study of how they collect their information calls for a look at the search engine spiders. Its sheer vastness and the constant increase in the information available make browsing the Net an almost insurmountable challenge for the spiders. Google, Yahoo! and other search engines use different types of spiders to combat the problems posed by the flux of the web. Search engine spiders are computer programs that create a copy of every visited page and store it in a database, so that a search engine can index the data and provide fast, comprehensive searches.

Enhancing the overall crawlability of a website is a primary objective of search engine optimization. There are some SEO tricks that must be used to ensure that search engine spiders crawl through your website regularly and thoroughly to index information, thus helping drive traffic towards it.

• Create an XML site map list of all the URLs that you want the spiders to crawl


• Use Cascading Style Sheets (CSS) for navigation elements will provide better control over formatting and changing web pages.


• Internal linking should be comprehensive and your allow entire website to be navigated through


• Ensure that URLs are simple and search engine friendly, free from extra attributes

Along with the recommendations, there are also a few pitfalls that an SEO company would do well to avoid. Here are some issues that will jeopardize your website's crawl rate.

• Excessive use of Flash, especially links embedded in Flash can make these uncrawlable


• Use of session id must be restricted to cookies and avoid using them in URLs at all costs


• Incorrect use of the "robots.txt" file can create issues with search engine spiders

Search engines optimization can be performed by both on-site developmental changes as well as off-site techniques. Making a website crawlable is the initial, crucial step of the on-site process and must be conducted rigorously across the entire website.