Google Will Drop Any Support For crawl-delay, nofollow, and noindex in robots.txt

blog cover image
12
8
1.3K followers

Hi

Google said that they are going to stop unofficially supporting the noindex, nofollow and crawl-delay directives within robots.txt files. According to Search Engine Roundtable.

This is what they also said and I quote it here:

"In short, if you mention crawl-delay, nofollow, and noindex in your robots.txt file - Google on September 1, 2019 will stop honoring it. They currently do honor some of those implementations, even though they are "unsupported and unpublished rules" but will stop doing so on September 1, 2019."

This came as a shock to me as I never heard about it before. Apparently Google is saying for years that they will stop with it.

What will happen to our pages That we don't want to be crawled and followed? Will it be out in the open for everybody to see?

What do you think about it?

12
8

Join FREE & Launch Your Business!

Exclusive Bonus - Offer Ends at Midnight Today

00

Hours

:

00

Minutes

:

00

Seconds

2,000 AI Credits Worth $10 USD

Build a Logo + Website That Attracts Customers

400 Credits

Discover Hot Niches with AI Market Research

100 Credits

Create SEO Content That Ranks & Converts

800 Credits

Find Affiliate Offers Up to $500/Sale

10 Credits

Access a Community of 2.9M+ Members

By continuing, you agree to our Terms of Service and Privacy Policy
No credit card required

Recent Comments

8

Featured Comment

This is referring directly to the robots.txt file, which many people use to control these functions. The reality is, you will still be able to leverage and utilize many different functions if you don't want Google to index your content.

The two most common will be:

(1) Noindex in robots meta tags (also can be found within the All in One and Yoast Functions. This directive is supported both in the HTTP response headers and in HTML. It will remove the URL form index when crawling.

Search Console Remove URL tool. You will be able to remove content from Google's search results directly from your Google Search Console account.

So this will actually be a moot issue, it will lead to some work for those with comprehensive and sophisticated robots.txt files that are used for managing these, but in terms of most websites that are handling through the robots noindex meta tags or anyone using Google Search Console, it will be a non issue.

7

Thank goodness you answered this Kyle. It had me worried.jim

Thanks, Kyle for your information. Much appreciated.

Laura

Thanks for clearing it up, Kyle, I was a bit worried. :)

1

so basically our privacy policy is just the same deal within aiseo as in your training?

Kyle gives a great answer

Thank you for bringing this up. Now we have a clear picture of what to expect.

Laura

1

Just thought I had to do it...

1

See more comments

Join FREE & Launch Your Business!

Exclusive Bonus - Offer Ends at Midnight Today

00

Hours

:

00

Minutes

:

00

Seconds

2,000 AI Credits Worth $10 USD

Build a Logo + Website That Attracts Customers

400 Credits

Discover Hot Niches with AI Market Research

100 Credits

Create SEO Content That Ranks & Converts

800 Credits

Find Affiliate Offers Up to $500/Sale

10 Credits

Access a Community of 2.9M+ Members

By continuing, you agree to our Terms of Service and Privacy Policy
No credit card required

2.9M+

Members

190+

Countries Served

20+

Years Online

50K+

Success Stories

The world's most successful affiliate marketing training platform. Join 2.9M+ entrepreneurs building their online business with expert training, tools, and support.

© 2005-2025 Wealthy Affiliate
All rights reserved worldwide.

🔒 Trusted by Millions Worldwide

Since 2005, Wealthy Affiliate has been the go-to platform for entrepreneurs looking to build successful online businesses. With industry-leading security, 99.9% uptime, and a proven track record of success, you're in safe hands.