How to make Bing crawl you site faster
Published on October 1, 2014
Published on Wealthy Affiliate — a platform for building real online businesses with modern training and AI.
Crawl Control
The Crawl Control feature in the Configure My Site Section allows you to exert control over the speed at which Bingbot makes requests for pages and resources on your website. You can tell Bingbot to crawl your site faster or slower than the normal crawl rate for each of the 24 hours in the day. This way you can to limit Bingbot activity when your visitors are on your site and allow us more bandwidth during quieter hours.
Ready to put this into action?
Start your free journey today — no credit card required.
| Note |
|---|
| If we find a crawl-delay: directive in your robots.txt file then it will take always precedence over the information from this feature. |
How to Set a Crawl Pattern Using a Preset
The easiest way to set a crawl pattern that optimizes your bandwidth for your visitors is by using the When do you receive most traffic during the local time day drop-down menu. This drop-down menu provides a number of preset options which you can use to indicate the busiest time on your site based on your local time of day. When selecting any of the presets, we will automatically optimize the crawl pattern by telling Bingbot to crawl more slowly during those hours and faster during the other hours. All you need to do to store the setting is clicking Save at the bottom of the page.
The following example shows a site that chose the 9AM - 5AM local time preset as the busiest time for the site, resulting in an inverse crawl pattern that allows faster crawl speeds for Bingbot outside of those hours:
How to Set Custom Crawl Pattern
Alternatively, you can use your mouse to create an entirely custom hourly crawl pattern by moving over the cubes holding the left mouse button down and dragging it to the desired number of quadrants for the timeline denoted on the x-axis. More blue quadrants indicate more visits from Bingbot (faster crawl), fewer quadrants mean the opposite (slower crawl). The default is 5 quadrants for each time slice. As a result, 5 blue quadrants represent the algorithmically determined baseline crawl rate for your site. Moving them up or down increases or decreases sets a higher or lower crawl rate based on the baseline crawl rate that was determined automatically for your site. When your happy with your pattern, simply click Save to store the setting.
Share this insight
This conversation is happening inside the community.
Join free to continue it.The Internet Changed. Now It Is Time to Build Differently.
If this article resonated, the next step is learning how to apply it. Inside Wealthy Affiliate, we break this down into practical steps you can use to build a real online business.
No credit card. Instant access.