Welcome to Webcosmo Webmaster Forum. We are a laid back yet well-maintained Webmaster Forum offering help and resources on SEO, marketing, online business, web design, hosting and other webmaster related topics.
Robots.txt is one of the important element in ON page optimization. Robots.txt file is not for the user. Google crawler will not crawl the pages which are added in the robots.txt file. Mostly payment gateway page, money transaction page, admin page are added in the robots.txt file to avoid the crawl.
Robots.txt is the common name of a text file that is uploaded to a Web site's root directory and linked in the HTML code of the Web site. The robots.txt file is used to provide instructions to the Web site to Web robots and spiders. Web authors can use robots.txt to keep cooperating Web robots from accessing all or parts of a Web site that you want to keep private.