Google Will Drop Any Support For crawl-delay, nofollow, and noindex in robots.txt
Hi
Google said that they are going to stop unofficially supporting the noindex, nofollow and crawl-delay directives within robots.txt files. According to Search Engine Roundtable.
This is what they also said and I quote it here:
"In short, if you mention crawl-delay, nofollow, and noindex in your robots.txt file - Google on September 1, 2019 will stop honoring it. They currently do honor some of those implementations, even though they are "unsupported and unpublished rules" but will stop doing so on September 1, 2019."
This came as a shock to me as I never heard about it before. Apparently Google is saying for years that they will stop with it.
What will happen to our pages That we don't want to be crawled and followed? Will it be out in the open for everybody to see?
What do you think about it?
Recent Comments
8
See more comments
This is referring directly to the robots.txt file, which many people use to control these functions. The reality is, you will still be able to leverage and utilize many different functions if you don't want Google to index your content.
The two most common will be:
(1) Noindex in robots meta tags (also can be found within the All in One and Yoast Functions. This directive is supported both in the HTTP response headers and in HTML. It will remove the URL form index when crawling.
Search Console Remove URL tool. You will be able to remove content from Google's search results directly from your Google Search Console account.
So this will actually be a moot issue, it will lead to some work for those with comprehensive and sophisticated robots.txt files that are used for managing these, but in terms of most websites that are handling through the robots noindex meta tags or anyone using Google Search Console, it will be a non issue.
Thank goodness you answered this Kyle. It had me worried.jim
Thanks, Kyle for your information. Much appreciated.
Laura
Thanks for clearing it up, Kyle, I was a bit worried. :)
so basically our privacy policy is just the same deal within aiseo as in your training?