Thread: what is robot.txt file?

i want to know about robot.txt file...

can anyone provide me the answer in details.....

Robot.txt is the file on server which tells the crawler where to crawl or not.

In simple word Robot.txt is a text file you put on your site to tell search robots which pages would like them not to visit.

Originally Posted by Openxcell

In simple word Robot.txt is a text file you put on your site to tell search robots which pages would like them not to visit.

Yes, I completely agree with you. That's what i wanted to say.

Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do.

robot.txt is a text file which is created at the time of creating a website. This name is case sensitive maintain all the letters to be lower case.
How to create a robots.txt file

[ user-agent:*disallow:/example-folder.exampl.html]

User-agent:*- means that the instruction is for all the search engine crawlers
Disallow: instruction the crawler not to index the file/ example-folder/example.html

Hi dear friend.
Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol. It works ..

Robot.txt is the file on server which tells the crawler where to crawl or not. i also agreed with you.