Quite often there will be an occasion where a search engine is prevented from indexing a website due to an improperly configured ROBOTS.TXT file. Although a Robots file is used to block or limit search engines from indexing a website it can also prevent a website from being listed on a search engine. The following article explains the ROBOTS.TXT file and how to find and edit it, as well as other resource links and articles on its use or configuration.
Correcting the Robots.txt from Blocking all websites crawlers
The ROBOTS.TXT is a file that is typically found at the root of the website. This text file is easily edited with a text editor like notepad in Windows or TextEdit on a Mac. Here is a common example of an incorrectly set ROBOTS.TXT file:
This setting is basically blocking ALL robots due to the "*" following the USER-AGENT line. The "/" after DISALLOW means that no pages can be visited by a search engine crawler.
If you are trying to get your website listed on Google or other search engines, then you should remove the "*" from the User-agenta and also the "/" that allows the search engine to scan your website. Here are the steps to editing the ROBOTS.TXT file:
- Login to the Cpanel.
- Open File Manager and go to the root directory of your website.
- The ROBOTS.TXT file should be in the same location as the index file (the first page) of your website.
- Click on EDITOR or CODE EDIT in the top menu after highlighting the text file in order to make any necessary changes.
- Click SAVE CHANGES in the top right hand corner in order to save your entries.
In spite of this common problem, the ROBOTS.TXT file is a very useful file because it can help keep robots from hitting the website excessively and thus causing excessive resource usage. Here is a list of resources that you should use when checking ROBOTS.TXT or looking for more information for its use: