Welcome to Webcosmo Webmaster Forum. We are a laid back yet well-maintained Webmaster Forum offering help and resources on SEO, marketing, online business, web design, hosting and other webmaster related topics.
Web site house owners use the /robots.txt file to present directions concerning their web site to net robots; this is often known as The Robots Exclusion Protocol. The "User-agent: *" means that this section applies to all or any robots. The "Disallow: /" tells the mechanism that it shouldn't visit any pages on the positioning.
Web site house owners use the /robots.txt file to present directions concerning their website to internet robots; this is often referred to as The Robots Exclusion Protocol. The "User-agent: *" suggests that this section applies to any or all robots. The "Disallow: /" tells the mechanism that it mustn't visit any pages on the positioning.
Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do. It is important to clarify that robots.txt is not a way from preventing search engines from crawling your site.
The Robots.txt file is used to instruct the search engine to automatically reach which pages you want it to search for and then index the page. Most websites also have directories and files that do not need the robots of the search engine to visit. So creating a robots.txt file can help you in SEO.