Issue 122235 - Connection fails with "502 Error reading from remote server" every day for 1/2 hour
Summary: Connection fails with "502 Error reading from remote server" every day for 1/...
Status: CLOSED FIXED_WITHOUT_CODE
Alias: None
Product: Infrastructure
Classification: Infrastructure
Component: Bugzilla (show other issues)
Version: current
Hardware: All All
: P3 Major (vote)
Target Milestone: ---
Assignee: Rainer Bielefeld
QA Contact:
URL: https://issues.apache.org/ooo/
Keywords:
Depends on:
Blocks:
 
Reported: 2013-05-05 08:21 UTC by Rainer Bielefeld
Modified: 2016-04-07 08:28 UTC (History)
5 users (show)

See Also:
Issue Type: DEFECT
Latest Confirmation in: ---
Developer Difficulty: ---


Attachments

Note You need to log in before you can comment on or make changes to this issue.
Description Rainer Bielefeld 2013-05-05 08:21:53 UTC
Bugzilla is not reliable enough for effectual work, I estimate 5% ... 10% of my connections will be not successful, but end with 

Proxy Error 502

The proxy server received an invalid response from an upstream server.
The proxy server could not handle the request GET /ooo/show_bug.cgi.

Reason: Error reading from remote server
Comment 1 Raphael Bircher 2013-08-13 03:43:17 UTC
Yes, everytime in the morning between 9 and 11 Clock European Timezone. I beleve this is the backup system.
Comment 2 Rainer Bielefeld 2013-10-03 08:09:14 UTC
CONFIRMED due to comment 1

@Rob:
Still a problem, more or less 7:45 UTC every day all Apache Bugzillas for particular projects (Jira, AOO, SpamAssassin) are unavailable for a while. 
<https://issues.apache.org/> seems not affected.
Comment 3 Andrea Pescetti 2013-11-03 11:37:49 UTC
Probably it's enough to find out what cron jobs run on that server and schedule the Bugzilla backups a couple hours earlier, so that it will be less annoying to Europeans. Or find a way to run them so that they don't cause an interruption of service at all!
Comment 4 Rainer Bielefeld 2014-01-19 15:56:32 UTC
Well, I'll take this one and will discuss that problem with Apache Infrastructure team.
Comment 5 Andrea Pescetti 2014-01-19 16:20:59 UTC
For the record: I setup a bot 2 months ago that will retrieve a Bugzilla page every 10 minutes. Here is the record of all failed requests. I never got the 502 error (explanations below). Hours are Central European Time (GMT+1).

      0 2013-11-16 01:00
      0 2013-11-19 01:00
  15991 2013-11-26 01:00
      0 2013-12-02 01:00
      0 2013-12-02 16:50
      0 2013-12-04 22:20
      0 2013-12-04 22:30
      0 2013-12-04 22:40
      0 2013-12-04 22:50
      0 2013-12-07 01:00
      0 2013-12-19 01:00
      0 2013-12-24 01:00
      0 2013-12-29 01:00
      0 2014-01-05 01:00
  15991 2014-01-13 01:00
  15991 2014-01-19 01:00

So: there is a brief interruption on 4 December and occasional truncated/empty pages at midnight GMT.

This means that probably we should check authenticated access, since anonymous access (my test) is almost always working. I confirm that, in the same time frame, I've seen the 502 error as authenticated user in the European early morning.
Comment 6 Andrea Pescetti 2014-01-19 16:26:44 UTC
I didn't write it explicitly above, but the fact that anonymous access has virtually no interruptions is easily explained by caching.
Comment 7 Rainer Bielefeld 2014-01-19 16:36:52 UTC
> Here is the record of all failed requests. I never
> got the 502 error (explanations below). 

So indeed that bot access must be different to "real life" access. For me that problem appears precise as by a swisswatch every morning. 

If there are access statistics of course it might be some improvement to move backup or whatever to a minimum access time. But that would be a poor solution - Were is the Wikipedia minimum access time where they bother users with downtimes?
Comment 8 Rainer Bielefeld 2014-01-19 17:39:08 UTC
Good news: Tony Stevenson told me that it's indeed a backup issue and that he is planning some changes (move SQL DB from the local host to the central DB server) what should solve the problem.
Comment 9 Rainer Bielefeld 2014-01-23 09:17:50 UTC
At least the visible problem with every morning blocked BZ has been solved since SQL DB has been moved. 

So I close this one.