Issue 118940 - Tune robots.txt for openoffice.org website
Summary: Tune robots.txt for openoffice.org website
Status: CONFIRMED
Alias: None
Product: Infrastructure
Classification: Infrastructure
Component: Website general issues (show other issues)
Version: current
Hardware: All All
: P3 Normal (vote)
Target Milestone: ---
Assignee: AOO issues mailing list
QA Contact:
URL: http://www.openoffice.org/robots.txt
Keywords:
Depends on:
Blocks:
 
Reported: 2012-02-20 23:06 UTC by Rob Weir
Modified: 2017-05-20 10:45 UTC (History)
1 user (show)

See Also:
Issue Type: TASK
Latest Confirmation in: ---
Developer Difficulty: Easy

Attachments
Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this issue.
Description Rob Weir 2012-02-20 23:06:59 UTC 
We currently have:

==============
User-agent: daumoa
Disallow: /

User-Agent: Sosospider
Disallow: /

User-Agent: Snoopy
Disallow: /

User-Agent: *
Crawl-delay: 5
Disallow: /nonav/
Disallow: /welcome/
Disallow: /white_papers/
==============

Some of this doesn't make sense.  For example, there is nothing at /nonav, /welcome seems to have a redirect issue that we should just fix.  And /white-papers/ has reasonable content that we should not be hiding from index spiders.
Comment 1 Marcus 2013-09-11 20:53:51 UTC 
When we have done our first browser tests for MSIE versions with "www.browsershots.org" I've updated the robots.txt already. IMHO this task is done and I would close this issue as fixed.
Comment 2 Marcus 2017-05-20 10:45:16 UTC 
Reset the assignee to the default "issues@openoffice.apache.org".