Thread: Is the robots.txt file necessary for seo?

You use a Robots.txt file to tell search robots which files to ignore (or alternatively) which files to crawl. It also helps Search Engines to locate the Sitemap of the website and hence crawl the entire website in depth... helping in your rankings and traffic.

I always think it's good practice to have a robots.txt file. You can use it to disallow any directories that you don't want crawled. I always put a line in the robots.txt to the sitemap also.

Originally Posted by Roald

Yes, it is necessary in SEO because when search engine crawler comes to your site, first it look for a file that robots.txt. Robots.txt directs the search engine spider, which pages of your website should be indexed and which one should be ignored.

Hey, thanks a lot for the info.

Originally Posted by serena85

It is if you want to remove some sub-pages

Its not for removing the pages. It's to instruct spiders not to crawl your website, folder or any page

Robots.txt file is one of the most important part of On page optimization as those file and folders which you don't want to get it cached by Google or any of search engines can be declared with the help of robots.txt file.

Originally Posted by Roald

Yes, it is necessary in SEO because when search engine crawler comes to your site, first it look for a file that robots.txt. Robots.txt directs the search engine spider, which pages of your website should be indexed and which one should be ignored.

Definitely it is necessary for seo. You are totally right.

Robots.txt is not mandatory. But having a robots.txt file can help you to control search engine bots in crawling the pages of your site. You can block some pages of your site which need not to be indexed by search engine bots for privacy reasons.

What you said is true but even though its not necessary if you don't want crawl any pages than you can instruct robots over there but if you don't want to hide any page or folder from robots and you have not included robots.txt spiders will think you don't want to hide anything and they will access all your pages

Yes Robot.txt file is very important I use to block some pages to indexing

Yes. Robots.txt file is helpful to block any images/page/directory from being index by search engine.

By implementing this robots.txt in your site, you can decide which page should not get index and which page should get crawl by the google....

robots.txt in very important to business or company sites, you can use this file to restrict search engines to index your confidential part of you website