Issue 104244 - Modify robots.txt to allow Google to index the issue tracker
Summary: Modify robots.txt to allow Google to index the issue tracker
Status: CLOSED FIXED
Alias: None
Product: Infrastructure
Classification: Infrastructure
Component: Bugzilla (show other issues)
Version: current
Hardware: Unknown All
: P3 Trivial (vote)
Target Milestone: ---
Assignee: stx123
QA Contact: issues@www
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2009-08-15 01:25 UTC by jasonspiro
Modified: 2017-05-20 10:28 UTC (History)
1 user (show)

See Also:
Issue Type: ENHANCEMENT
Latest Confirmation in: ---
Developer Difficulty: ---

Attachments
Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this issue.
Description jasonspiro 2009-08-15 01:25:25 UTC 
Bugzilla's built-in search lacks PageRank, stemming[1], and many other features.

I hear bugzilla 3.0 is faster (usually runs on mod_perl) so performance is no
longer such an issue.

Would you be so kind as to do an experiment?

1. use the Google Webmaster Tools at https://www.google.com/webmasters/tools to
reduce Google's crawl speed on the site to "Slow"
2. add the following to the top of http://bugzilla.mozilla.org/robots.txt:
     User-agent: Googlebot
     Allow: /issues/
3. see if everyone's happier now.

^  [1].  For example, searching for the word "crash" will also find the words
"crashes" and "crashing".
Comment 1 stx123 2009-08-21 15:05:01 UTC 
Btw have you seen that Mozilla's bugzilla blocks robots too...

I changed the setting according to your idea for
qa.openoffice.org/issues

Let's see if anyone is happier now
Comment 2 stx123 2009-11-24 17:53:22 UTC 
From time to time we see some strange crawling activities. But googlebot seems
to work fine...