Thread: what is robots.txt file?

Robots.txt is a simple text (not html) file you put on your website root directory to tell search robots which pages you would like them not to visit. By defining a few rules in this text file, you can instruct robots to not crawl certain files, directories within your site, or at all.

Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol.

Already I submitted robot.txt file but when I took some report using tools its shows "that i failed to submit the robot.txt file" What does it mean?

it helps u in stopping website being indexed by search engines

Hello Friends,

It is great when search engines frequently visit your site and index your content but often there are cases when indexing parts of your online content is not what you want. For instance, if you have two versions of a page (one for viewing in the browser and one for printing), you'd rather have the printing version excluded from crawling, otherwise you risk being imposed a duplicate content penalty. Also, if you happen to have sensitive data on your site that you do not want the world to see, you will also prefer that search engines do not index these pages (although in this case the only sure way for not indexing sensitive data is to keep it offline on a separate machine).

Thanks

thanks for the sharing the knowledge of the related this

"Robot txt" is txt file of our website. you can control Google spider..

Robots.txt is used for optioning the robots to your website.

A file which is used to inform the search engine spider which pages on a site should not be indexed. This file sits in your site’s root directory on the web server. (Alternatively, you can do a similar thing by placing tags in the header section of your HTML for search engine robots/spiders to read.)

Hello Friends,

Web Robots (also known as Web Wanderers, Crawlers, or Spiders), are programs that traverse the Web automatically. Search engines such as Google use them to index the web content, spammers use them to scan for email addresses, and they have many other uses.

Thanks

Robot.txt files tell your instructions to a search engine robot. it depends on what you what it to do. if you dont have the robot.txt file then when a robot visits your site and if it doesnt find the file, it will start to visit all the pages and contents of your site. these files are like walls which helps in preventing some files like java scripts, some images or any other files from indexing. it is also important to make a note that it is not a method that prevents search engines from crawling your site. so it is important to create a robot.txt file. if you need this file to be created you can get it from TheWebPole.com

Robot.txt files are simple text files which are used to tell the search engine which files to crawl and which not.

robots.txt is look like one type of entrance gate for search engine, where search engine find which file and folder are not include in index.

robots.txt define rules on website to allow or disallow to the search engine.

Robot.txt is a text file which contains rules for search engine robots,It instruct them which part of the website they should crawl and indexed and which one not.

Robots.txt file is a set of instructions that tell search engine robots which pages of your site to be crawled and indexed. In most cases, your site is consist of many files or folders i.e. admin folders, cgi-bin, image folder, which are not relevant to the search engines. Robots.txt helps tell spiders what is useful and public for sharing in the search engine indexes and what is not. It should also be noted that not all search spiders will follow your instructions left in the robots.txt file. In addition, a poorly done robots.txt file can stop the search spiders from crawling and indexing your website properly.