The robots.txt file is a file you can add to your document root to help minimize the impact of "bots" crawling your site. While it is not a guarantee or foolproof method to manage how all bots crawl your site, it can help mitigate the impact of frequent crawls by certain search bots.

Limitations of robots.txt

An important point to keep in mind when using robots.txt is that it should be considered a set of suggestions, not a set of rules that all bots must follow. While most reputable bots will follow the instructions of a robots.txt file, scraper bots and malware bots (along with other bad bots) will most likely ignore your robots.txt file. So, if you have sensitive information that you wish to in your account, you will want to utilize password protection, not robots.txt to protect that data.

Creating a robots.txt File

You can create your robots.txt file either through the File Manager in cPanel or any text editor program on your local computer such as Notepad for Windows, TextEdit on Mac or vi or emacs on Linux. When saving the file, make sure it is saved with exactly the filename and extension robots.txt

What goes into a robots.txt File

You can specify Disallow rules for a specific User-agent (such as googlebot) or specify all bots by using the wild card "*"  (without the quotes).

Here are a few examples:

This disallows all bots for the entire site:

User-agent: *
Disallow: /

While the following directive is considered nonstandard, several major crawlers support this as a directive to limit how often (in seconds) a bot should crawl your site:

User-agent: *
Crawl-delay: 10

For more information regarding User-agents and Disallow statements used in robots.txt, please see:

robots.txt documentation from robotstxt.org

More information can also be found about robots.txt in Goolge Webmaster Tools:

Google Webmaster Tools and robots.txt

Did you find this article helpful?

We value your feedback!

Why was this article not helpful? (Check all that apply)
The article is too difficult or too technical to follow.
There is a step or detail missing from the instructions.
The information is incorrect or out-of-date.
It does not resolve the question/problem I have.
How did you find this article?
Please tell us how we can improve our Help Center:
Email Address
Optional, but our team may contact you for more information.
Did you find this article helpful?

Post a Comment

Name:
Email Address:
Comment:
Are you a bot?
Submit

Please note: Your name and comment will be displayed, but we will not show your email address.

News / Announcements

Update to SSL Certificates - Certificate Warnings
1969-12-31 11:00 am EST
Hits: 648

Related Questions

Here are a few questions related to this article that our customers have asked:
Ooops! It looks like there are no questions about this page.
Would you like to ask a question about this page? If so, click the button below!

Help Center Search

Current Customers

Email: support@WebHostingHub.com Ticket: Submit a Support Ticket
Call: 877-595-4HUB (4482)
757-416-6627 (Intl.)
Chat: Click To Chat Now

Ask the Community

Get help with your questions from our community of like-minded hosting users and Web Hosting Hub Staff.

Not a Customer?

Get web hosting from a company that is here to help.