Server Error Reports

[gnif]

Hi All,

This thread is to create a central place for reporting server errors instead of having a multitude of threads for such things.

Please do NOT reply to this thread unless you have been instructed to by an error message.

This is NOT for general forum usage issues, this thread is for server outages, configuration problems, etc.

Please Note: I provide gratuitous server support to Dave as a way to support this great community, as such my support is best effort. Please be patient, I am subscribed to this thread and will be notified of your posts.

If any of the below hold true DO NOT post a report.

• Check the next post under this one to see if we have announced an expected outage, or that we are working on the current outage.
• Check the last post to see if someone else has already very recently reported the outage/error, we would like to avoid 'me too' posts as they pull us away from investigating the issue.
• Sometimes the error can be cached after we have announced a fix for the issue, please ensure you clear your browser cache and reload the suspect page before reporting.
• Some errors (5xx errors) will generally affect the entire website, if you find a single page is reporting this error, it is very likely the error is cached, please clear your cache.

If none of the above hold true, please post a detailed error report, the more information the better. Things that help are:

• The time the error occurred as close to the second as possible. To expedite matters please ensure you convert your time to GMT (UTC).
  
• The browser you use, chome/firefox/ie... opera
• The URL of the page you were accessing
• If it was submitting a form such as the forum search, updating your profile, etc... please let us know what exactly you were doing that triggered the error

As is the same with EE, the more detail about what led up to the issue, the higher the odds are of finding the cause and correcting it.

[gnif]

Current Status: OK

29/08/2017: We are performing some changes to how backups are handled which is causing some slowness, please be patient while we get this sorted out.
30/08/2017: The new backup system is in place, but will be running concurrently with the old backup system while it's performance is evaluated.
03/10/2017: The server was restarted to lift some network restrictions that HostGator had in place that were causing issues with the new backup system. Not sure why a restart was required but this is what we were advised.
11/11/2017: We are now on a new backup scheme but are still experiencing the 502 issues, but much less frequently now. There are core dumps from PHP appearing that need to be debugged when I get some time to do so.

[MK14]

EDIT: already my second attempt at making this report. The first just hanged forever, then said error. (Sorry, I should have noted what the error said).

I've just got and seen the new 502 error message. Which instructed me to write a message here.

Over the last day or so, the EEVblog website, although it works fine about 90% of page loads. Seems to take ages to load, perhaps 10% of the time. Especially when making posts or editing.

I have also had problems with other websites, so the problem could be at my end. About 10% of websites are problematic and take ages to load, then just fail or hang forever.

EDIT: More detailed report:

Time was about 5:13 PM 3/8/17 UK time.

The page I was accessing was:
https://www.eevblog.com/forum/unreadreplies/

The web browser I was using: Google Chrome with Adblock Plus.

It said:

502 - Bad Gateway
We are having server side difficulties and are working to resolve these issues as fast as we can.

If you are able please report this error to this thread. But before posting please read the instructions carefully!

Thank you for your patience,
Dave & gnif

EDIT3:
When a page has an intermediate error message, BEFORE getting to the page. It looks like this:

This site can’t be reached

www.eevblog.com’s server DNS address could not be found.
Search Google for eevblog forum chat public transport hell people
ERR_NAME_NOT_RESOLVED

Which MAY be an occasional problem at MY end. Sorry. I have had to change (or consider changing) the default DNS server, previously. Because my ISP one, can be problematic.

[gnif]

The 502 is certainly server side and not caused by your client. I have noted a few pages with slow load issues I am still looking into, one of which is loading a profile page. In future please post the time in GMT so I don't have to guess at your timezone and try to figure this out (not one of my strong points).

[tautech]

Blinking forum just apparently ate a post that took me 30 minutes to put together. 
The display came up with a Cloudfare-EEVblog server page with a red cross on the EEVblog server.
~10am Sydney time.
Chrome

BUT it did post. 
https://www.eevblog.com/forum/testgear/owon-xds3202e-vs-siglent-sds1202x-e/msg1276504/#msg1276504

So false alarm but it did give me a sick feeling.
Back to normal now.

[Ian.M]

Its the first time I've seen your new Bad Gateway page (since you fixed the session data problem.)

502 - Bad Gateway

We are having server side difficulties and are working to resolve these issues as fast as we can.

If you are able please report this error to this thread. But before posting please read the instructions carefully!

Thank you for your patience,
Dave & gnif

It was at 28/08/2017 16:09:13 UTC while refreshing https://www.eevblog.com/forum/unread/

Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.17pre) Gecko K-Meleon/1.6.0

The page reloaded correctly a couple of minutes later.

HTH.

[Richard Crowley]

Same here in Oregon, USA 2017-08-29 @ 4:56 UTC
New error message, and then loaded OK a couple minutes later.

[gnif]

Hi All,

Sorry for the delay in this update, I have setup a new backup system on the server to resolve these issues but it's initial sync has been problematic. Once the intial backup has been completed the old backup will be turned off resolving the recent errors.

For the technical minded:

• Industry standard backup solutution that provides incremental as well as full backups
• Databases are hot copied using Percona XtraDB rather then relying on a slow dump from MySQL
• Backups are perfomed using a pull model to a remote secure server in an undeclosed location
• Said server is on a private VPN both to protect comms, and to avoid needing to give the server a public IP

[Ian.M]

It dit it again, and reload didn't help.  Same browser as before. Time 30/08/2017 08:43:47 UTC.   I *ASSUME* you haven't been able to turn the old backup off yet.

Edit after seeing reply by Gnif:   Sorry!  In my defence 08:43:47 UTC is 09:43:47 GMT which is clearly before 10:00 GMT.   If you want us to ignore errors from 09:30 GMT to 10:30 GMT please say so explicitly.
Edit2: Correction after seeing reply by Salbayeng.  Sorry about the brain-fart re: GMT/BST  *NOT* adding a new post because I don't have another error to report.

[gnif]

It dit it again, and reload didn't help.  Same browser as before. Time 30/08/2017 08:43:47 UTC.   I *ASSUME* you haven't been able to turn the old backup off yet.

• Check the next post under this one to see if we have announced an expected outage, or that we are working on the current outage.

[salbayeng]

@Ian.M ?que? 

08:43:47 UTC is 09:43:47 GMT

Both UTC and GMT are the same for most purposes (GMT does not have summer time adjustments, so local time in the UK will be different by an hour in summer).
One adjusts for leap seconds as they occur, the other adjusts for leap seconds at the end of the year.
The UTC time standard is physically located at a naval observatory in the USA.

GPS time is nearly the same, but doesn't correct for leapseconds, it is currently about 20secs away from UTC (however all GPS receivers know the difference and will tell you UTC if asked).

[edavid]

502 error at 2017-08-30 22:21:34 UTC

Firefox

https://www.eevblog.com/forum/unread/

(but it was affecting all pages)

[gnif]

502 error at 2017-08-30 22:21:34 UTC

Firefox

https://www.eevblog.com/forum/unread/

(but it was affecting all pages)

Again, check the status post, you can clearly see it is 'WiP" and there is additional information stating the details of the work in progress.

[edavid]

Again, check the status post, you can clearly see it is 'WiP" and there is additional information stating the details of the work in progress.

I don't understand, are you saying we should not report errors if the status says "OK - WiP"?

[gnif]

Again, check the status post, you can clearly see it is 'WiP" and there is additional information stating the details of the work in progress.

I don't understand, are you saying we should not report errors if the status says "OK - WiP"?

If any of the below hold true DO NOT post a report.

• Check the next post under this one to see if we have announced an expected outage, or that we are working on the current outage.

You experienced an outage, but the status states we are working on it and there are dated status updates below it.

[edavid]

You experienced an outage, but the status states we are working on it and there are dated status updates below it.

Maybe I'm dense, but that was not at all clear to me.  Why would I think that the fact that you are working on the backup system means I shouldn't report a 502 error?

(Or maybe you could change the 502 error message when you don't want reports.)

[gnif]

You experienced an outage, but the status states we are working on it and there are dated status updates below it.

Maybe I'm dense, but that was not at all clear to me.  Why would I think that the fact that you are working on the backup system means I shouldn't report a 502 error?

(Or maybe you could change the 502 error message when you don't want reports.)

Sorry, I will try to be clearer in the status updates from here on in

[RGB255_0_0]

You know about the 10AM issues...

FYI, I just had a 502 at 5:08PM BST.

[gnif]

You know about the 10AM issues...

FYI, I just had a 502 at 5:08PM BST.

Report Status: Outages are to be expected, reports are not needed at this time.

[McBryce]

502 error at 13:00 German time.

McBryce.

[Cerebus]

502 for "https://www.eevblog.com/forum/blog/?action=post2;start=100" at 17:15:30 GMT.

Also got a "the last post from your IP address" rate limiting message 2+ minutes after posting this message.

[ChrisLX200]

502 - Bad Gateway  10:55pm GMT

[gnif]

Just a little update, we are still working on fixing a known issue (actually, the same issue). I have been away and have not had the opportunity to perform some changes that are required to proceed with the fix. I should be able to free up time in the next few days and work with Dave to schedule some downtime required to put the new system into effect.

[guenthert]

502 just happened here (Berlin/Germany) around 9:40 zulu.

[nugglix]

Same for Hamburg/Germany.