Quite often there will be an occasion where a search engine is prevented from indexing a website due to an improperly configured ROBOTS.TXT file.  Although a Robots file is used to block or limit search engines from indexing a website it can also prevent a website from being listed on a search engine.  The following article explains the ROBOTS.TXT file and how to find and edit it, as well as other resource links and articles on its use or configuration.

Correcting the Robots.txt from Blocking all websites crawlers


The ROBOTS.TXT is a file that is typically found at the root of the website.  This text file is easily edited with a text editor like notepad in Windows or TextEdit on a Mac.  Here is a common example of an incorrectly set ROBOTS.TXT file:

User-agent: *
Disallow: /

This setting is basically blocking ALL robots due to the "*" following the USER-AGENT line.  The "/" after DISALLOW means that no pages can be visited by a search engine crawler.  

If you are trying to get your website listed on Google or other search engines, then you should remove the "*" from the User-agenta and also the "/" that allows the search engine to scan your website.  Here are the steps to editing the ROBOTS.TXT file:

  1. Login to the Cpanel.
  2. Open File Manager and go to the root directory of your website.
  3. The ROBOTS.TXT file should be in the same location as the index file (the first page) of your website. 
  4. Click on EDITOR or CODE EDIT in the top menu after highlighting the text file in order to make any necessary changes.
  5. Click SAVE CHANGES in the top right hand corner in order to save your entries.

 

Useful ROBOTS.TXT

In spite of this common problem, the ROBOTS.TXT file is a very useful file because it can help keep robots from hitting the website excessively and thus causing excessive resource usage.  Here is a list of resources that you should use when checking ROBOTS.TXT or looking for more information for its use:

Did you find this article helpful?

We value your feedback!

Why was this article not helpful? (Check all that apply)
The article is too difficult or too technical to follow.
There is a step or detail missing from the instructions.
The information is incorrect or out-of-date.
It does not resolve the question/problem I have.
How did you find this article?
Please tell us how we can improve this article:
Email Address
Name

new! - Enter your name and email address above and we will post your feedback in the comments on this page!

Did you find this article helpful?

Comments

n/a Points
2014-11-21 2:15 am

In my file manager, the ROBOTS.TXT is in the public_html folder which is under the /home/******9 root file. Is this incorrect? (my website is ****************.com)

Thank you,

Larissa

Staff
12,339 Points
2014-11-21 2:38 am
Hello Larissa,

Thank you for contacting us today. Yes, /home/******9/public_html is the correct folder location, since your index.php file is there.

Since this is a public forum, I have marked out your account specific information for security purposes.

If you have any further questions, feel free to post them below.

Thank you,
John-Paul
n/a Points
2015-01-10 7:11 am

google shows all my pages as protected by robots.txt even though I can find no robots.text in any directory .. very frustrating .. may have to find a new host..

Staff
16,266 Points
2015-01-11 5:39 pm
Hello Bruce,

The robots.txt directory would be in the root folder for each domain. For your primary domain, it would be in the public_html folder. Any other robots.txt file would have to have been added by you as we do not add them upon creation of an addon domain. If you cannot see one in your account, then the mistake is on Google and not our server.

As an aside, that file is pretty much useless at this point, most search engines ignore it and Google wants you to put your options in your WebMaster tools account as opposed to the robots.txt file.

Kindest Regards,
Scott M

Post a Comment

Name:
Email Address:
Phone Number:
Comment:
Submit

Please note: Your name and comment will be displayed, but we will not show your email address.

Related Questions

Here are a few questions related to this article that our customers have asked:
Ooops! It looks like there are no questions about this page.
Would you like to ask a question about this page? If so, click the button below!
Need More Help?

Help Center Search

Current Customers

Email: support@WebHostingHub.com Ticket: Submit a Support Ticket
Call: 877-595-4HUB (4482)
757-416-6627 (Intl.)
Chat: Click To Chat Now

Ask the Community

Get help with your questions from our community of like-minded hosting users and Web Hosting Hub Staff.

Not a Customer?

Get web hosting from a company that is here to help.
}