In a previous article, Using a robots.txt File
, we touched briefly on setting a crawl-delay through a directive in the robots.txt file. While some search crawlers do recognize that directive, Google's search crawler (Googlebot) does not. Adjusting the Googlebot crawl rate is handled within Google's Webmaster Tools
Googlebot crawling your site is normally considered a good thing. Issues can arise if crawls become excessive and the server your account is negatively affected. Fortunately, by reducing the crawl rate in Google Webmaster Tools, you can reduce the impact of Googlebot on the server.
There are three important points to keep in mind when setting the crawl rate in Google Webmaster Tools:
- Setting the crawl rate in Google Webmaster Tools sets the speed of Googlebot requests during each crawl process. It does not affect how often Googlebot comes back to crawl your site or how deeply your site structure is crawled.
- You can only set the crawl rate for the main domain or a subdomain (yourdomain.com or subdomain.yourdomain.com). You cannot set a separate crawl rate for a subfolder (yourdomain.com/subfolder).
- When you set a new crawl rate in Google Webmaster Tools, it will only be valid for 90 days. To keep your reduced crawl rate, log back into Google Webmaster Tools and adjust your crawl rate if it has been more than 90 days since you last adjusted it.
Changing Your Crawl Rate in Google Webmaster Tools
- Begin by logging into your Webmaster Tools account
- In the home page, select your site. If you have more than one site, select the site you want to change the crawl rate for.
- On the left side of the next page, under Site configuration, click on "Settings."
- Under Crawl rate, select "Set custom crawl rate" and adjust the slider to a slower crawl rate.
- Click the "Save" button to save your new crawl rate. As you can see in the image below, we reduced our crawl rate by 33% in this example.
If you have any questions or need any assistance regarding reducing your Google crawl rate, please feel free to submit a question in our Support Center.