Quite often there will be an occasion where a search engine is prevented from indexing a website due to an improperly configured ROBOTS.TXT file. Although a Robots file is used to block or limit search engines from indexing a website it can also prevent a website from being listed on a search engine. The following article explains the ROBOTS.TXT file and how to find and edit it, as well as other resource links and articles on its use or configuration.
The ROBOTS.TXT is a file that is typically found at the root of the website. This text file is easily edited with a text editor like notepad in Windows or TextEdit on a Mac. Here is a common example of an incorrectly set ROBOTS.TXT file:
User-agent: *
Disallow: /
This setting is basically blocking ALL robots due to the "*" following the USER-AGENT line. The "/" after DISALLOW means that no pages can be visited by a search engine crawler.
If you are trying to get your website listed on Google or other search engines, then you should remove the "*" from the User-agenta and also the "/" that allows the search engine to scan your website. Here are the steps to editing the ROBOTS.TXT file:
In spite of this common problem, the ROBOTS.TXT file is a very useful file because it can help keep robots from hitting the website excessively and thus causing excessive resource usage. Here is a list of resources that you should use when checking ROBOTS.TXT or looking for more information for its use:
n/a Points
|
2014-11-21 2:15 am
In my file manager, the ROBOTS.TXT is in the public_html folder which is under the /home/******9 root file. Is this incorrect? (my website is ****************.com) Thank you, Larissa |
Staff 12,339 Points
|
2014-11-21 2:38 am
Hello Larissa,
Thank you for contacting us today. Yes, /home/******9/public_html is the correct folder location, since your index.php file is there. Since this is a public forum, I have marked out your account specific information for security purposes. If you have any further questions, feel free to post them below. Thank you, John-Paul |
n/a Points
|
2015-01-10 7:11 am
google shows all my pages as protected by robots.txt even though I can find no robots.text in any directory .. very frustrating .. may have to find a new host.. |
![]() Staff 16,266 Points
|
2015-01-11 5:39 pm
Hello Bruce,
The robots.txt directory would be in the root folder for each domain. For your primary domain, it would be in the public_html folder. Any other robots.txt file would have to have been added by you as we do not add them upon creation of an addon domain. If you cannot see one in your account, then the mistake is on Google and not our server. As an aside, that file is pretty much useless at this point, most search engines ignore it and Google wants you to put your options in your WebMaster tools account as opposed to the robots.txt file. Kindest Regards, Scott M |
Email: | support@WebHostingHub.com | Ticket: | Submit a Support Ticket |
---|---|---|---|
Call: | 757-416-6627 | Chat: | Click To Chat Now |
We value your feedback!
There is a step or detail missing from the instructions.
The information is incorrect or out-of-date.
It does not resolve the question/problem I have.
new! - Enter your name and email address above and we will post your feedback in the comments on this page!