In our Using a robots.txt File we went over how to minimize "bots" crawling your entire site and/or directories within your site. In this article we'll go over how to discourage "bot" traffic from an individual file (instead of an entire directory).

The following is an example of the "no follow, no index" code you can use in an individual page:

<html>
<head>
<title>...</title>
<META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">
</head>

When using the <META> tag, keep the following in mind:

  • The tag is not guaranteed to block all robots from scanning your page. While most robots will respect the instructions provided, some robots will ignore the instructions and continue scanning the file. This is usually the case with robots that are looking for exploits or to harvest emails, etc.
  • Notice the NOFOLLOW directive in the 4th line. That directive only pertains to links on the page where this code is placed. It will have no affect on links on other pages.
  • The NOFOLLOW directive used in this tag is not the same as the attribute rel="nofollow" used in <href> tags. The NOFOLLOW directive used in this <META> tag tells robots they are not to follow links from the page containing this tag. The rel="nofollow" attribute is an attribute created by Google to prevent page rank from being passed on through links.

How to Write a ROBOTS META tag

Just like any other <META> tag, the <META NAME="ROBOTS"> tag should appear between the opening and closing <HEAD> tags in the HTML code of your page.

Let's look at the example tag again:

<META NAME="ROBOTS" CONTENT="NOINDEX, FOLLOW">

  • Always use "ROBOTS" for the "NAME" attribute
  • There are four different values you can use for the "CONTENT" attribute:
    • "INDEX" = Search engines can show the page in search results
    • "NOINDEX" = Search engines should NOT include the page in search results
    • "FOLLOW" = Robots may follow links from the page to other pages
    • "NOFOLLOW" = Robots should NOT follow links from the page to other pages

While you can use multiple comma separated values for the "CONTENT" attribute, only certain combinations make sense and should actually be used. For instance, if no <META NAME="ROBOTS"> tag appears, the default behavior is "INDEX, FOLLOW" so it is not necessary to explicitly state that particular combination.

That leaves us with the following combinations:

<META NAME="ROBOTS" CONTENT="NOINDEX, FOLLOW">
<META NAME="ROBOTS" CONTENT="INDEX, NOFOLLOW">
<META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">

Keep in mind that this tag is best suited if you only have one or a few pages you want to discourage "bots" from crawling as the tag will need to be inserted on every page you want it to apply to. If you have many files or entire directories you want to control "bot" traffic on, the robots.txt file is a better option.

Did you find this article helpful?

We value your feedback!

Why was this article not helpful? (Check all that apply)
The article is too difficult or too technical to follow.
There is a step or detail missing from the instructions.
The information is incorrect or out-of-date.
It does not resolve the question/problem I have.
How did you find this article?
Please tell us how we can improve our Help Center:
Email Address
Optional, but our team may contact you for more information.
Did you find this article helpful?

Post a Comment

Name:
Email Address:
Comment:
Are you a bot?
Submit

Please note: Your name and comment will be displayed, but we will not show your email address.

News / Announcements

Update to SSL Certificates - Certificate Warnings
1969-12-31 11:00 am EST
Hits: 645

Related Questions

Here are a few questions related to this article that our customers have asked:
Ooops! It looks like there are no questions about this page.
Would you like to ask a question about this page? If so, click the button below!

Help Center Search

Current Customers

Email: support@WebHostingHub.com Ticket: Submit a Support Ticket
Call: 877-595-4HUB (4482)
757-416-6627 (Intl.)
Chat: Click To Chat Now

Ask the Community

Get help with your questions from our community of like-minded hosting users and Web Hosting Hub Staff.

Not a Customer?

Get web hosting from a company that is here to help.