In a previous article, Using a robots.txt File, we touched briefly on setting a crawl-delay through a directive in the robots.txt file. While some search crawlers do recognize that directive, Google's search crawler (Googlebot) does not. Adjusting the Googlebot crawl rate is handled within Google's Webmaster Tools.

Googlebot crawling your site is normally considered a good thing. Issues can arise if crawls become excessive and the server your account is negatively affected. Fortunately, by reducing the crawl rate in Google Webmaster Tools, you can reduce the impact of Googlebot on the server.

NOTE: There are three important points to keep in mind when setting the crawl rate in Google Webmaster Tools:

  1. Setting the crawl rate in Google Webmaster Tools sets the speed of Googlebot requests during each crawl process. It does not affect how often Googlebot comes back to crawl your site or how deeply your site structure is crawled.
  2. You can only set the crawl rate for the main domain or a subdomain (yourdomain.com or subdomain.yourdomain.com). You cannot set a separate crawl rate for a subfolder (yourdomain.com/subfolder).
  3. When you set a new crawl rate in Google Webmaster Tools, it will only be valid for 90 days. To keep your reduced crawl rate, log back into Google Webmaster Tools and adjust your crawl rate if it has been more than 90 days since you last adjusted it.


Changing Your Crawl Rate in Google Webmaster Tools

  1. Begin by logging into your Webmaster Tools account.
  2. In the home page, select your site. If you have more than one site, select the site you want to change the crawl rate for.

    google-crawl-delay-select-site
  3. On the left side of the next page, under Site configuration, click on Settings.

    google-crawl-delay-settings
  4. Under Crawl rate, select "Set custom crawl rate and adjust the slider to a slower crawl rate.

    google-crawl-rate
  5. Click the Save button to save your new crawl rate. As you can see in the image below, we reduced our crawl rate by 33% in this example.

    google-crawl-rate-reduced

If you have any questions or need any assistance regarding reducing your GOOGLE crawl rate, please feel free to submit a question in our Support Center.

Did you find this article helpful?

We value your feedback!

Why was this article not helpful? (Check all that apply)
The article is too difficult or too technical to follow.
There is a step or detail missing from the instructions.
The information is incorrect or out-of-date.
It does not resolve the question/problem I have.
How did you find this article?
Please tell us how we can improve our Help Center:
Email Address
Optional, but our team may contact you for more information.
Did you find this article helpful?

Post a Comment

Name:
Email Address:
Comment:
Are you a bot?
Submit

Please note: Your name and comment will be displayed, but we will not show your email address.

Related Questions

Here are a few questions related to this article that our customers have asked:
Ooops! It looks like there are no questions about this page.
Would you like to ask a question about this page? If so, click the button below!

Help Center Search

Current Customers

Email: support@WebHostingHub.com Ticket: Submit a Support Ticket
Call: 877-595-4HUB (4482)
757-416-6627 (Intl.)
Chat: Click To Chat Now

Ask the Community

Get help with your questions from our community of like-minded hosting users and Web Hosting Hub Staff.

Not a Customer?

Get web hosting from a company that is here to help.