Google Will Drop Any Support For crawl-delay, nofollow, and noindex in robots.txt
Published on July 2, 2019
Published on Wealthy Affiliate — a platform for building real online businesses with modern training and AI.
Hi
Google said that they are going to stop unofficially supporting the noindex, nofollow and crawl-delay directives within robots.txt files. According to Search Engine Roundtable.
This is what they also said and I quote it here:
Ready to put this into action?
Start your free journey today — no credit card required.
"In short, if you mention crawl-delay, nofollow, and noindex in your robots.txt file - Google on September 1, 2019 will stop honoring it. They currently do honor some of those implementations, even though they are "unsupported and unpublished rules" but will stop doing so on September 1, 2019."
This came as a shock to me as I never heard about it before. Apparently Google is saying for years that they will stop with it.
What will happen to our pages That we don't want to be crawled and followed? Will it be out in the open for everybody to see?
What do you think about it?
Share this insight
This conversation is happening inside the community.
Join free to continue it.The Internet Changed. Now It Is Time to Build Differently.
If this article resonated, the next step is learning how to apply it. Inside Wealthy Affiliate, we break this down into practical steps you can use to build a real online business.
No credit card. Instant access.