The Differences Between Meta NoIndex, NoFollow and Robots.txt File

A few weeks ago SEO Zombie wrote an interesting post on the flow of internal link juice.  The post itself goes into ways to prevent duplicate content issues on your blog if you’re running WordPress. However, what I found most helpful was the explanation from Matt Cutts on the differences between the NoIndex tag, NoFollow tag and the Robots.txt file. So if you’re confused at all about what the three of these do and when you need to use them – this should help clear some things up.
According to SEO Zombie, Matt Cutts explains them as the following:
Robots.txt:
Pages that are included in a Robots.txt file tell the search engines NOT to crawl these pages.  However pages in a Robots.txt file can still accrue PageRank and can be indexed in search results, says Matt Cutts.
Why you should use a Robots.txt file:  I like to exclude images and misc. files on my site from being crawled. I don’t want the search engines to waste time on those pages over more important pages of my site – Therefore, I add them in the Robots.txt file so they’re not crawled over more important pages. So if you’re trying to stop the flow of PageRank you’ll need to do more than just add a page to the Robots.txt file…
NoIndex:
The NoIndex tag means the search engines can crawl the page and give it PageRank, however the search engines are not to index the page, and it will not show up in the search results. Again, a page with the NoIndex tag can accumulate PageRank, because the links are still followed outwards from a NoIndex page.
Why you should use a meta NoIndex tag: I would use this tag only if I for sure did not want my page to be indexed within the search engines, but I do want that page to accrue PageRank and pass that on to other pages of my site. For example, shopping cart pages that are dynamically driven or a contact us form you may not want to be indexed.
NoFollow:
Finally, a page with a NoFollow tag tells the search engines that yes, this page can be crawled, but don’t show this page at all in Google’s Index, and don’t follow any outgoing links, and no PageRank flows from that page.
Why you should use a meta NoFollow tag:  Again, if there are more important pages of your site that you would rather the search engines assign PageRank to and index, then you would want to use the NoFollow tag. Most people use the NoFollow and the NoIndex tags together so that a page like a shopping cart page that is dynamically driven isn’t  indexed or assigned any PageRank. The more pages you have on your site, the more your PageRank is distributed between all of them. By eliminating miscellaneous pages from being indexed or passing on PageRank, the more PageRank your important pages get.
It’s also important to know the difference between the NoFollow meta tag and a rel=”nofollow” link tag. Using a ‘nofollow’ tag on a link will only prevent PageRank from flowing through that link. But all other links on a page will pass on PageRank.  Of course if you add the NoFollow meta tag onto the whole page, it prevents all links on that page from passing on PageRank.
Using these tags really provide you the opportunity to control what the search engines do with certain pages of your site, and allow you to sculpt where you want your PageRank to flow.  Just remember that the more important pages of your site that will benefit the user the best are the pages that you want the search engines to focus more time and attention on, not the miscellaneous pages of your site.