Thread: what is robots.txt file ?

Is any one please tell me that what is robots.txt file??? and how we implement in my site?? Is it necessary for any site???

Robots text file is the file where you can allow or disallow user agent means robots or spiders to crawl or not to crawl that web page which you restricted. It is not necessary but good to have.

Robots.txt file also known as Robots Exclusion Protocol or Robot Exclusion Standard..Here is a blog post about robots.txt, you can learn here anything you want to know about it and how to apply it to a site..
http://www.premiumseosolutions.com.a...t-file-in-seo/

Thanks for giving nice reply other suggestions are welcome.......

With the help of robots.txt, your website will be crawled by any search engine instantly.

Hi,
Robot.txt file is make for stop indexing the web page from a search engine spider.

Robot.txt file is where you can allow and disallow your website and you can disallow ur folder/page etc.

ITs not mandatory you can use it while you want to hide something from robots else its fine not to have it

Robots.txt file is a must have file in each website as we can restrict web spider crawling any of particular restricted web page in our website. It is helpful particularly when renovation is going on in our website.

i didn't know about Robots txt file.. thanks for telling me about it.

Robots.txt file helps you to control search engine bots in indexing the pages of your site. You can prevent some of the pages being crawled by search engine bots.

If you want all spiders to crawl your site. Add this coding in a notepad and name it as "robots.txt" and upload it in your root folder.

User-agent: *
Allow:/

Robot.txt files tell your instructions to a search engine robot. it depends on what you what it to do. if you dont have the robot.txt file then when a robot visits your site and if it doesnt find the file, it will start to visit all the pages and contents of your site. these files are like walls which helps in preventing some files like java scripts, some images or any other files from indexing. it is also important to make a note that it is not a method that prevents search engines from crawling your site. so it is important to create a robot.txt file. if you need this file to be created you can get it from TheWebPole.com

Robots.txt is a text file which instruct the search engines crawlers that which links are indexed and which are not.

Precisely, robot.txt file contains the information regarding the files that are to be left out from being crawled by the search engine spiders. The link provided will provide a better insight about what robot.txt is. http://en.wikipedia.org/wiki/Robot.txt

A notepad generated file having extension with .txt, is used for a website to restrict or allow a search engine bot to follow/index and page or not.