Google Will Drop Any Support For crawl-delay, nofollow, and noindex in robots.txt

Last Update: July 02, 2019

Hi

Google said that they are going to stop unofficially supporting the noindex, nofollow and crawl-delay directives within robots.txt files. According to Search Engine Roundtable.

This is what they also said and I quote it here:

"In short, if you mention crawl-delay, nofollow, and noindex in your robots.txt file - Google on September 1, 2019 will stop honoring it. They currently do honor some of those implementations, even though they are "unsupported and unpublished rules" but will stop doing so on September 1, 2019."

This came as a shock to me as I never heard about it before. Apparently Google is saying for years that they will stop with it.

What will happen to our pages That we don't want to be crawled and followed? Will it be out in the open for everybody to see?

What do you think about it?

Join the Discussion
Write something…
Recent messages
Kyle Premium Plus Featured Comment
This is referring directly to the robots.txt file, which many people use to control these functions. The reality is, you will still be able to leverage and utilize many different functions if you don't want Google to index your content.

The two most common will be:

(1) Noindex in robots meta tags (also can be found within the All in One and Yoast Functions. This directive is supported both in the HTTP response headers and in HTML. It will remove the URL form index when crawling.

Search Console Remove URL tool. You will be able to remove content from Google's search results directly from your Google Search Console account.

So this will actually be a moot issue, it will lead to some work for those with comprehensive and sophisticated robots.txt files that are used for managing these, but in terms of most websites that are handling through the robots noindex meta tags or anyone using Google Search Console, it will be a non issue.
Reply
JKulk1 Premium
Thank goodness you answered this Kyle. It had me worried.jim
Reply
LauraFuller Premium
Thanks, Kyle for your information. Much appreciated.

Laura
Reply
Nick-at-WA Premium
Thanks for clearing it up, Kyle, I was a bit worried. :)
Reply
MozMary Premium Plus
so basically our privacy policy is just the same deal within aiseo as in your training?
Reply
davehayes Premium
Kyle gives a great answer
Reply
LauraFuller Premium
Thank you for bringing this up. Now we have a clear picture of what to expect.

Laura
Reply
Nick-at-WA Premium
Just thought I had to do it...
Reply
Top