Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5

What is robot.txt file and what are the benefits of using it..?

What is robot.txt file and what are the benefits of using it..?

robot is the mechanical machine
Solar PCU

In vascular plants, the root is the organ of a plant that typically lies below the surface of the soil. Roots can also be aerial or aerating, that is growing up above the ground or especially above water. Furthermore, a stem normally occurring below ground is not exceptional either (see rhizome). Therefore, the root is best defined as the non-leaf, non-nodes bearing parts of the plant's body. However, important internal structural differences between stems and roots exist

Maharaja Express

robots.txt does not prevent Google crawling those pages, it stops them being indexed. Your better off writing your own legal pages for seo purposes

Robots.txt is a text file that is inserted into the website to contain instructions for search engine robots. The text file contains lists of webpages that are allowed and disallowed from Google crawling.

Cheap VPS | $1 VPS Hosting
Cheap Dedicated Servers | Free IPMI


Use of Robots.txt - The most common usage of Robots.txt is to ban crawlers from visiting private folders or content that gives them no additional information.

Robots.txt Allowing Access to Specific Crawlers.
Allow everything apart from certain patterns of URLs.

Robots.txt files that help Google and other search engines are crawling and indexing your site. you can use for allowed and disallowed to crawling according to you.

The purpose of a robots.txt file is to tell search engines not to crawl the contents of specific directories or even individual pages within a website. ... However there are a couple of reasons why all websites would benefit from having one. The first thing search engines look for is the robots.txt file.


Food Franchise | Food franchise opportunity | Food Franchise India

A robots.txt file is a file at the root of your site that indicates those parts of your site you don’t want to be accessed by search engine crawlers. The file uses the Robots Exclusion Standard, which is a protocol with a small set of commands that can be used to indicate access to your site by section and by specific kinds of web crawlers (such as mobile crawlers vs desktop crawlers).

Thanks to all of you whom shares this effective information for new users.

Possibly Related Threads...
Thread / Author Replies Views Last Post

Forum Jump:

Users browsing this thread: 1 Guest(s)