It is often a good idea to  restrict search engine spiders or crawlers from accessing certain parts of your website.  This is done with a file called a ROBOTS.TXT file.  Fortunately, Prestashop 1.5 does this for you.  The following tutorial will walk you through the process of getting the file created.

How to create a robots.txt file in Prestashop 1.5

  1. Login to the Administration Panel as an Administrator
  2. Hover over the menu bar where it says PREFERENCES, and then click on SEO & URLS in the drop down menu.  You will see an existing list of URLs.  Scroll down further until you see ROBOTS FILE GENERATION. The following screenshot indicates the button to click that will generate the ROBOTS.TXT file:


    prestashop15-robotsfile


  3. The information outlined in blue is important as it lets you know where the file is required to be located.  By default, in Web Host Hub's servers, the location of the file is in the PUBLIC_HTML folder if the website is the primary domain.  If it is an addon domain, then the path to the ROBOTS file will look like this:

    /public_html/add-on-domain-name.com

    If you are not hosting with Web Hosting Hub, please consult your Technical Support or documentation about the location of your files.
  4. Click on the GENERATE ROBOTS.TXT FILE button and the ROBOTS file will automatcially be created. 
  5. The ROBOTS.TXT file will generate the necessary information to limit access to search engine spiders and crawlers.  Here's the top of the file:


    # robots.txt automaticaly generated by PrestaShop e-commerce open-source solution
    # http://www.prestashop.com - http://www.prestashop.com/forums
    # This file is to prevent the crawling and indexing of certain parts
    # of your site by web crawlers and spiders run by sites like Yahoo!
    # and Google. By telling these "robots" where not to go on your site,
    # you save bandwidth and server resources.
    # For more information about the robots.txt standard, see:
    # http://www.robotstxt.org/wc/robots.html

The rest of the file includes the entries to keep the robot from going where it doesn't need to go.  The most important part to take from this is that limiting the access helps to save BANDWIDTH and SERVER RESOURCES.

# robots.txt automaticaly generated by PrestaShop e-commerce open-source solution
# http://www.prestashop.com - http://www.prestashop.com/forums
# This file is to prevent the crawling and indexing of certain parts
# of your site by web crawlers and spiders run by sites like Yahoo!
# and Google. By telling these "robots" where not to go on your site,
# you save bandwidth and server resources.
# For more information about the robots.txt standard, see:
# http://www.robotstxt.org/wc/robots.html
Did you find this article helpful?

We value your feedback!

Why was this article not helpful? (Check all that apply)
The article is too difficult or too technical to follow.
There is a step or detail missing from the instructions.
The information is incorrect or out-of-date.
It does not resolve the question/problem I have.
How did you find this article?
Please tell us how we can improve this article:
Email Address
Name

new! - Enter your name and email address above and we will post your feedback in the comments on this page!

Did you find this article helpful?

Comments

n/a Points
2014-06-06 4:41 pm

I´m working on Prestashop 1.6. Because I´m adding a shop to an existing page I have put prestashop into a subfolder = shop.  I know the generated robots.txt file has to be moved to the root. But do I have to change the robots.txt lines in it to work from the root or does all these * in the file mean, that it will work from the root? From the generated file.

# Private pages

Disallow: /*controller=addresses

 

# Directories

Disallow: */classes/Disallow: */config/

Kind regards

Thomas

Staff
13,576 Points
2014-06-06 7:18 pm
Hello Thomas,

You just need to make sure that any specific paths are correct. The general references such as the ones you've listed above should work fine. Check out our tutorial called Using Robots.txt for more specific information on the commands. It also provides links for more detail information on the use of the robots.txt file.

I hope that helps to answer your question. Please let us know if you require further assistance.

Regards,
Arnel C.
n/a Points
2014-06-06 7:31 pm

Hi Arnel.

Thank you for your quick responce.

ok, so the * simply is a message for all robots?

The lines you saw in the first message is generated by prestashop.

 

So I will have to change the path to

 

# Private pages

Disallow: /shop/controller=addresses  (where to place the * )

 

# Directories  (where to place the *)

Disallow: /shop/classes/

Disallow: /shop/config/

 

Best Thomas

Staff
13,576 Points
2014-06-06 7:35 pm
Hello Thomas,

That is correct. The * indicates any robot (also called user agent).

Regards,
Arnel C.

Post a Comment

Name:
Email Address:
Phone Number:
Comment:
Submit

Please note: Your name and comment will be displayed, but we will not show your email address.

Related Questions

Here are a few questions related to this article that our customers have asked:
Ooops! It looks like there are no questions about this page.
Would you like to ask a question about this page? If so, click the button below!

Help Center Search

Current Customers

Email: support@WebHostingHub.com Ticket: Submit a Support Ticket
Call: 877-595-4HUB (4482)
757-416-6627 (Intl.)
Chat: Click To Chat Now

Ask the Community

Get help with your questions from our community of like-minded hosting users and Web Hosting Hub Staff.

Not a Customer?

Get web hosting from a company that is here to help.