
What is robots.txt File ?????
It is a text file that gets merged with the websites to instruct search engine robots how to crawl and index pages on their website i.e. it tells the crawler that which of the page of the website is accessible to them and which is not. It is also known as Robots Exclusion Protocol (REP).
The location of a robots.txt is very important. It must be in the main directory because otherwise search engines will not be able...