Results 1 to 8 of 8
  1. #1
    ashy40 is offline Sophomore
    Join Date
    Sep 2011
    Posts
    65
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Feedback Score
    0

    Default what is robot.txt file?

    i want to know about robot.txt file...

    can anyone provide me the answer in details.....

  2. #2
    alina albert is offline Junior
    Join Date
    Mar 2011
    Posts
    342
    Thanks
    0
    Thanked 3 Times in 3 Posts
    Feedback Score
    0

    Default

    Robot.txt is the file on server which tells the crawler where to crawl or not.

  3. #3
    Openxcell is offline Senior
    Join Date
    Apr 2011
    Location
    NY, USA
    Posts
    590
    Thanks
    0
    Thanked 2 Times in 2 Posts
    Feedback Score
    0

    Default

    In simple word Robot.txt is a text file you put on your site to tell search robots which pages would like them not to visit.

  4. #4
    ujwalagawas is offline Junior
    Join Date
    Aug 2011
    Posts
    348
    Thanks
    18
    Thanked 2 Times in 2 Posts
    Feedback Score
    0

    Default

    Quote Originally Posted by Openxcell View Post
    In simple word Robot.txt is a text file you put on your site to tell search robots which pages would like them not to visit.
    Yes, I completely agree with you. That's what i wanted to say.

  5. #5
    HostDone's Avatar
    HostDone is offline Freshman
    Join Date
    Aug 2011
    Posts
    20
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Feedback Score
    0

    Default

    Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do.

  6. #6
    sofiaisabella's Avatar
    sofiaisabella is offline Sophomore
    Join Date
    Jul 2011
    Location
    Australia
    Posts
    79
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Feedback Score
    0

    Default

    robot.txt is a text file which is created at the time of creating a website. This name is case sensitive maintain all the letters to be lower case.
    How to create a robots.txt file

    [ user-agent:*disallow:/example-folder.exampl.html]

    User-agent:*- means that the instruction is for all the search engine crawlers
    Disallow: instruction the crawler not to index the file/ example-folder/example.html


  7. #7
    GETFREEDUM's Avatar
    GETFREEDUM is offline Sophomore
    Join Date
    Sep 2011
    Posts
    62
    Thanks
    0
    Thanked 2 Times in 2 Posts
    Feedback Score
    0

    Default

    Hi dear friend.
    Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol. It works ..

  8. #8
    seoinheritx is offline Sophomore
    Join Date
    Sep 2011
    Location
    USA
    Posts
    52
    Thanks
    0
    Thanked 1 Time in 1 Post
    Feedback Score
    0

    Default

    Robot.txt is the file on server which tells the crawler where to crawl or not. i also agreed with you.

Similar Threads

  1. free forex robot
    By lbrandon74 in forum General Talk
    Replies: 3
    Last Post: 08-26-2011, 01:15 PM
  2. How to create robot.txt File ?
    By rajeevmehtaa in forum Search Engine Optimization
    Replies: 10
    Last Post: 08-09-2011, 09:25 AM
  3. How to create Robot.txt
    By krishnapmk in forum Google
    Replies: 9
    Last Post: 04-20-2011, 02:37 PM
  4. How to find a pdf file is normal file or shared review file
    By sunshine in forum JavaScript & AJAX
    Replies: 0
    Last Post: 01-04-2010, 11:21 AM
  5. Robot With Rat Brain
    By yawez.com in forum General Talk
    Replies: 2
    Last Post: 12-26-2008, 03:29 AM

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Partners: BOSTON WEB DEVELOPER, LLC   |   WEBCOSMO CLASSIFIEDS