Quite often there will be an occasion where a search engine is prevented from indexing a website due to an improperly configured ROBOTS.TXT file.  Although a Robots file is used to block or limit search engines from indexing a website it can also prevent a website from being listed on a search engine.  The following article explains the ROBOTS.TXT file and how to find and edit it, as well as other resource links and articles on its use or configuration.

Correcting the Robots.txt from Blocking all websites crawlers


The ROBOTS.TXT is a file that is typically found at the root of the website.  This text file is easily edited with a text editor like notepad in Windows or TextEdit on a Mac.  Here is a common example of an incorrectly set ROBOTS.TXT file:

User-agent: *
Disallow: /

This setting is basically blocking ALL robots due to the "*" following the USER-AGENT line.  The "/" after DISALLOW means that no pages can be visited by a search engine crawler.  

If you are trying to get your website listed on Google or other search engines, then you should remove the "*" from the User-agenta and also the "/" that allows the search engine to scan your website.  Here are the steps to editing the ROBOTS.TXT file:

  1. Login to the Cpanel.
  2. Open File Manager and go to the root directory of your website.
  3. The ROBOTS.TXT file should be in the same location as the index file (the first page) of your website. 
  4. Click on EDITOR or CODE EDIT in the top menu after highlighting the text file in order to make any necessary changes.
  5. Click SAVE CHANGES in the top right hand corner in order to save your entries.

 

Useful ROBOTS.TXT

In spite of this common problem, the ROBOTS.TXT file is a very useful file because it can help keep robots from hitting the website excessively and thus causing excessive resource usage.  Here is a list of resources that you should use when checking ROBOTS.TXT or looking for more information for its use:

Did you find this article helpful?

We value your feedback!

Why was this article not helpful? (Check all that apply)
The article is too difficult or too technical to follow.
There is a step or detail missing from the instructions.
The information is incorrect or out-of-date.
It does not resolve the question/problem I have.
How did you find this article?
Please tell us how we can improve our Help Center:
Email Address
Optional, but our team may contact you for more information.
Did you find this article helpful?

Post a Comment

Name:
Email Address:
Comment:
Are you a bot?
Submit

Please note: Your name and comment will be displayed, but we will not show your email address.

News / Announcements

Update to SSL Certificates - Certificate Warnings
1969-12-31 11:00 am EST
Hits: 624
What is the Heartbleed bug?
1969-12-31 11:00 am EST
Hits: 249

Related Questions

Here are a few questions related to this article that our customers have asked:
Ooops! It looks like there are no questions about this page.
Would you like to ask a question about this page? If so, click the button below!

Help Center Search

Current Customers

Email: support@WebHostingHub.com Ticket: Submit a Support Ticket
Call: 877-595-4HUB (4482)
757-416-6627 (Intl.)
Chat: Click To Chat Now

Ask the Community

Get help with your questions from our community of like-minded hosting users and Web Hosting Hub Staff.

Not a Customer?

Get web hosting from a company that is here to help.