Google Will Drop Any Support For crawl-delay, nofollow, and noindex in robots.txt
Google said that they are going to stop unofficially supporting the noindex, nofollow and crawl-delay directives within robots.txt files. According to Search Engine Roundtable.
This is what they also said and I quote it here:
"In short, if you mention crawl-delay, nofollow, and noindex in your robots.txt file - Google on September 1, 2019 will stop honoring it. They currently do honor some of those implementations, even though they are "unsupported and unpublished rules" but will stop doing so on September 1, 2019."
This came as a shock to me as I never heard about it before. Apparently Google is saying for years that they will stop with it.
What will happen to our pages That we don't want to be crawled and followed? Will it be out in the open for everybody to see?
What do you think about it?