Two Types Of Google Website Crawling

byvrodriguez73

250 followers

4 years ago

Google's Two Types Of Website Crawling

I recently read a good article on how Google crawls sites. Basically, there are two types of crawling methods Google uses.

One is a "discovering new content" crawl and the other is refreshing an existing one. In other words refreshing content that's already been published. Google's Search Advocate, Mr. John Mueller explains how this works.

An SEO professional asked Mr. Mueller how often do Googlebots crawl his site. He goes on to say that when he published regularly it got crawled often but when he published fewer articles it didn't get crawled as much. He was worried whether this was a bad thing.

Mr. Mueller responded assuring him it was fine. He also went on to say "That can happen. It's not so much that we crawl a website, but we crawl individual pages of a website. We have two types of crawling roughly.

"One is a discovery crawl where we try to discover new pages on your website. And the other is a refresh crawl where we update existing pages that we know about."

Crawl frequency can vary for the whole site, it also can vary by individual web pages. An example of this is if your homepage is updated more often than other pages you'll see more Googlebot activity on that page.

Mr. Mueller explains it like this, he said that they would refresh crawl the homepage once a day or every couple of hours. If they found new links on the homepage then they would crawl those with the discovery crawl as well.

This is why you'll always see a mixture of discovery and refresh crawling. Every day you'll see some type of baseline crawling.

If they see individual pages don't change often then they know they don't need to do much crawling, Googlebot is able to recognize these patterns.

Another example is if your website was a news website and you update your site every hour, they know to crawl your site every hour but if you only update every month then Googlebot know to only crawl once a month.

Mueller explains that it's not a sign of quality or a sign of ranking, it's purely from a technical point of view. So if you notice Google is visiting your site more or less often, don't worry all it could be is that Google crawled your site but discovered new content.

I thought it was a really interesting and informative article about how Google really crawls, I definately wanted to share this with you guys. I hope it helps you understand Google's crawling methods a little more if you didn't already.

Virginia