Have You Been Crawled Lately?

Have you ever searched for a website on a search engine like Google only to find outdated data as one of the top results? For example, you have recently changed or removed one of the pages on your website, but when you conduct a Google search, the edited content appears unchanged in the results. What gets more confusing is when you search for the same content directly from your website and it's all current. What's up with that?

You're not alone with this query. This is a common challenge experienced by users that have recently made changes to their website content. It is not an error, it simply means that your website has not recently been crawled. In basic terms, search engines like Google periodically use cyber robots to crawl through the pages of your website and store content for search engine optimization. Google, for example, equates their "Googlebots" to spiders that cypher through content and index bits and pieces of information that will enhance searches. The information that is indexed by these crawl intervals becomes the most up to date information to appear in the list of search results. There is no exact timeline for these crawls to adhere to; rather, they occur at varying intervals.

The frequency of crawling is based on a variety of factors such as the speed of your site, the amount of up to date content you have, and the website traffic or popularity, just to name a few. Any verified website owner has access to review the frequency of crawling and these statistics are readily available for you at any time. Found in your Google Webmaster Tools, Crawl Statistics will provide you with the information you need to identify what area of your website needs improvement to be more frequently indexed.