Thread: what is robots.txt file?

hi...

can anyone provide me the answer about robots.txt file..

Share the answer...

Thanks in advance..

Originally Posted by kylismith

hi...

can anyone provide me the answer about robots.txt file..

Share the answer...

Thanks in advance..

A ‘robots.txt’ file helps you tell the search engine spiders which files to ignore (or alternatively) which files to crawl. It also helps Search Engines to locate the Sitemap of the website and hence crawl the entire website in depth... helping in your rankings and traffic.

robot.txt file is simple text file where some instruction to search engine which have to crawl and which file to ignore so according to instruction Google ignore those files or folder

Search engine bot will follow instruction from robot.txt file ignore those folder or according to their instructions

This is example of robot.txt file

User-agent: *


Disallow: /images/

Disallow: /temp/

Disallow: /cgi-bin/

Robots.txt is a text file you simaply put on your site to tell search robots which page you would like them not to visit.It is very important to clarify that robots.txt is not a way from preventing search engine from crawling your sites.

robot.txt file for Search Engine it's give help you for allowing and diss allowing some page and directory to search engine...

If you want to not index any page of your website then you can tell google by robot.txt file to not index the page.

from this file we can allow/disallow google robot to crawl page or not

Yes i agree with you.

Originally Posted by BrainPulse

A ‘robots.txt’ file helps you tell the search engine spiders which files to ignore (or alternatively) which files to crawl. It also helps Search Engines to locate the Sitemap of the website and hence crawl the entire website in depth... helping in your rankings and traffic.

Hello Friends,

Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do. It is important to clarify that robots.txt is not a way from preventing search engines from crawling your site (i.e. it is not a firewall, or a kind of password protection) and the fact that you put a robots.txt file is something like putting a note “Please, do not enter” on an unlocked door – e.g. you cannot prevent thieves from coming in but the good guys will not open to door and enter. That is why we say that if you have really sen sitive data, it is too naïve to rely on robots.txt to protect it from being indexed and displayed in search results.

Thanks

There is a hidden, relentless force that permeates the web and its billions of web pages and files, unbeknownst to the majority of us sentient beings. I'm talking about search engine crawlers and robots here. Every day hundreds of them go out and scour the web, whether it's Google trying to index the entire web, or a spam bot collecting any email address it could find for less than honorable intentions. As site owners, what little control we have over what robots are allowed to do when they visit our sites exist in a magical little file called "robots.txt."
"Robots.txt" is a regular text file that through its name, has special meaning to the majority of "honorable" robots on the web. By defining a few rules in this text file, you can instruct robots to not crawl and index certain files, directories within your site, or at all. For example, you may not want Google to crawl the /images directory of your site, as it's both meaningless to you and a waste of your site's bandwidth. "Robots.txt" lets you tell Google just that.

Robots.txt is a controller of robots.

Originally pasted by magic 99
Search engine bot will follow instruction from robot.txt file ignore those folder or according to their instructions

This is example of robot.txt file

User-agent: *


Disallow: /images/

Disallow: /temp/

Disallow: /cgi-bin/

Yes this file help to what to show on the website or what to not.

It's a file which helps the google bot to know about which of your blog pages are need to be crawl and which are not. If you dont want some content to be indexed, you can use disallow option in robots.txt file.

A robots.txt file on a website will function as a request that specified robots ignore specified files or directories in their search.this is called The Robots Exclusion Protocol.