What is robots.txt File ?????
It is a text file that gets merged with the websites to instruct search engine robots how to crawl and index pages on their website i.e. it tells the crawler that which of the page of the website is accessible to them and which is not. It is also known as Robots Exclusion Protocol (REP).
The location of a robots.txt is very important. It must be in the main directory because otherwise search engines will not be able to find it. They don't search the whole site for a robots.txt file. Instead they look first in the main directory and if they don't find it there, they simply assume that this site doesn't have a robots.txt file and therefore they index everything they find along the way.Following example illustrates that how robots.txt works.....
Block all web crawlers from all content.
user-agent : *
Disallow : /
It will block all the web crawlers from all of the content on your site.
Block a specific web crawler from a specific folder.
user-agent : Google bot
Disallow : /no-google/
It will block the specific web crawler Google bot from a specific folder /no-google/ .
To allow all robots complete access
user-agent : *
Disallow :
It will allow the all robots complete access.
To allow a single robot
user-agent : Google
Disallow :
The /robots.txt file is a publicly available file. Anyone can see what sections of your site you don't want robots to use.
Always remember to use lowercase for the filename : "robots.txt" not "Robots.TXT".
