It is the standard protocol or procedure that prohibits or disallows the to crawl on the particular area of the web site. In other words it prevents spiders to access all areas of web site. Robot helps search engines for the documentation & categorizing the web sites.
|