Author Topic: Server Error Reports  (Read 719103 times)

0 Members and 2 Guests are viewing this topic.

Offline gnifTopic starter

  • Administrator
  • *****
  • Posts: 1671
  • Country: au
Server Error Reports
« on: July 30, 2017, 11:53:21 pm »
Hi All,

This thread is to create a central place for reporting server errors instead of having a multitude of threads for such things.

Please do NOT reply to this thread unless you have been instructed to by an error message.

This is NOT for general forum usage issues, this thread is for server outages, configuration problems, etc.

Please Note: I provide gratuitous server support to Dave as a way to support this great community, as such my support is best effort. Please be patient, I am subscribed to this thread and will be notified of your posts.

If any of the below hold true DO NOT post a report.
  • Check the next post under this one to see if we have announced an expected outage, or that we are working on the current outage.
  • Check the last post to see if someone else has already very recently reported the outage/error, we would like to avoid 'me too' posts as they pull us away from investigating the issue.
  • Sometimes the error can be cached after we have announced a fix for the issue, please ensure you clear your browser cache and reload the suspect page before reporting.
  • Some errors (5xx errors) will generally affect the entire website, if you find a single page is reporting this error, it is very likely the error is cached, please clear your cache.
If none of the above hold true, please post a detailed error report, the more information the better. Things that help are:
  • The time the error occurred as close to the second as possible. To expedite matters please ensure you convert your time to GMT (UTC).
  • The browser you use, chome/firefox/ie... opera
  • The URL of the page you were accessing
  • If it was submitting a form such as the forum search, updating your profile, etc... please let us know what exactly you were doing that triggered the error

As is the same with EE, the more detail about what led up to the issue, the higher the odds are of finding the cause and correcting it.
« Last Edit: August 04, 2017, 05:24:18 am by gnif »
 
The following users thanked this post: SeanB

Offline gnifTopic starter

  • Administrator
  • *****
  • Posts: 1671
  • Country: au
Current Server Status
« Reply #1 on: July 30, 2017, 11:54:14 pm »
Current Status: OK

29/08/2017: We are performing some changes to how backups are handled which is causing some slowness, please be patient while we get this sorted out.
30/08/2017: The new backup system is in place, but will be running concurrently with the old backup system while it's performance is evaluated.
03/10/2017: The server was restarted to lift some network restrictions that HostGator had in place that were causing issues with the new backup system. Not sure why a restart was required but this is what we were advised.
11/11/2017: We are now on a new backup scheme but are still experiencing the 502 issues, but much less frequently now. There are core dumps from PHP appearing that need to be debugged when I get some time to do so.
« Last Edit: November 10, 2017, 11:11:37 pm by gnif »
 

Offline MK14

  • Super Contributor
  • ***
  • Posts: 4527
  • Country: gb
Re: Server Error Reports
« Reply #2 on: August 03, 2017, 04:13:49 pm »
EDIT: already my second attempt at making this report. The first just hanged forever, then said error. (Sorry, I should have noted what the error said).

I've just got and seen the new 502 error message. Which instructed me to write a message here.

Over the last day or so, the EEVblog website, although it works fine about 90% of page loads. Seems to take ages to load, perhaps 10% of the time. Especially when making posts or editing.

I have also had problems with other websites, so the problem could be at my end. About 10% of websites are problematic and take ages to load, then just fail or hang forever.

EDIT: More detailed report:

Time was about 5:13 PM 3/8/17 UK time.

The page I was accessing was:
https://www.eevblog.com/forum/unreadreplies/

The web browser I was using: Google Chrome with Adblock Plus.

It said:

Quote
502 - Bad Gateway
We are having server side difficulties and are working to resolve these issues as fast as we can.

If you are able please report this error to this thread. But before posting please read the instructions carefully!

Thank you for your patience,
Dave & gnif

EDIT3:
When a page has an intermediate error message, BEFORE getting to the page. It looks like this:
Quote
This site can’t be reached

www.eevblog.com’s server DNS address could not be found.
Search Google for eevblog forum chat public transport hell people
ERR_NAME_NOT_RESOLVED

Which MAY be an occasional problem at MY end. Sorry. I have had to change (or consider changing) the default DNS server, previously. Because my ISP one, can be problematic.
« Last Edit: August 03, 2017, 05:47:40 pm by MK14 »
 

Offline gnifTopic starter

  • Administrator
  • *****
  • Posts: 1671
  • Country: au
Re: Server Error Reports
« Reply #3 on: August 04, 2017, 05:21:38 am »
The 502 is certainly server side and not caused by your client. I have noted a few pages with slow load issues I am still looking into, one of which is loading a profile page. In future please post the time in GMT so I don't have to guess at your timezone and try to figure this out (not one of my strong points).
 
The following users thanked this post: MK14

Offline tautech

  • Super Contributor
  • ***
  • Posts: 28061
  • Country: nz
  • Taupaki Technologies Ltd. Siglent Distributor NZ.
    • Taupaki Technologies Ltd.
Re: Server Error Reports
« Reply #4 on: August 09, 2017, 12:16:34 am »
Blinking forum just apparently ate a post that took me 30 minutes to put together.  :rant:
The display came up with a Cloudfare-EEVblog server page with a red cross on the EEVblog server.
~10am Sydney time.
Chrome

BUT it did post.  :-//
https://www.eevblog.com/forum/testgear/owon-xds3202e-vs-siglent-sds1202x-e/msg1276504/#msg1276504

So false alarm but it did give me a sick feeling.
Back to normal now.
« Last Edit: August 09, 2017, 12:18:08 am by tautech »
Avid Rabid Hobbyist
Siglent Youtube channel: https://www.youtube.com/@SiglentVideo/videos
 

Online Ian.M

  • Super Contributor
  • ***
  • Posts: 12753
Re: Server Error Reports
« Reply #5 on: August 28, 2017, 04:17:54 pm »
Its the first time I've seen your new Bad Gateway page (since you fixed the session data problem.)
Quote
502 - Bad Gateway

We are having server side difficulties and are working to resolve these issues as fast as we can.

If you are able please report this error to this thread. But before posting please read the instructions carefully!

Thank you for your patience,
Dave & gnif
It was at 28/08/2017 16:09:13 UTC while refreshing https://www.eevblog.com/forum/unread/

Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.17pre) Gecko K-Meleon/1.6.0

The page reloaded correctly a couple of minutes later.

HTH. ;)
 

Offline Richard Crowley

  • Super Contributor
  • ***
  • Posts: 4317
  • Country: us
  • KJ7YLK
Re: Server Error Reports
« Reply #6 on: August 29, 2017, 05:00:27 am »
Same here in Oregon, USA 2017-08-29 @ 4:56 UTC
New error message, and then loaded OK a couple minutes later.
 

Offline gnifTopic starter

  • Administrator
  • *****
  • Posts: 1671
  • Country: au
Re: Server Error Reports
« Reply #7 on: August 29, 2017, 11:11:38 am »
Hi All,

Sorry for the delay in this update, I have setup a new backup system on the server to resolve these issues but it's initial sync has been problematic. Once the intial backup has been completed the old backup will be turned off resolving the recent errors.

For the technical minded:
  • Industry standard backup solutution that provides incremental as well as full backups
  • Databases are hot copied using Percona XtraDB rather then relying on a slow dump from MySQL
  • Backups are perfomed using a pull model to a remote secure server in an undeclosed location
  • Said server is on a private VPN both to protect comms, and to avoid needing to give the server a public IP
 

Online Ian.M

  • Super Contributor
  • ***
  • Posts: 12753
Re: Server Error Reports
« Reply #8 on: August 30, 2017, 08:48:38 am »
It dit it again, and reload didn't help.  Same browser as before. Time 30/08/2017 08:43:47 UTC.   I *ASSUME* you haven't been able to turn the old backup off yet.

Edit after seeing reply by Gnif:   Sorry!  In my defence 08:43:47 UTC is 09:43:47 GMT which is clearly before 10:00 GMT.   If you want us to ignore errors from 09:30 GMT to 10:30 GMT please say so explicitly.
Edit2: Correction after seeing reply by Salbayeng.  Sorry about the brain-fart re: GMT/BST  *NOT* adding a new post because I don't have another error to report.
« Last Edit: August 30, 2017, 09:15:45 pm by Ian.M »
 

Offline gnifTopic starter

  • Administrator
  • *****
  • Posts: 1671
  • Country: au
Re: Server Error Reports
« Reply #9 on: August 30, 2017, 08:49:52 am »
It dit it again, and reload didn't help.  Same browser as before. Time 30/08/2017 08:43:47 UTC.   I *ASSUME* you haven't been able to turn the old backup off yet.

  • Check the next post under this one to see if we have announced an expected outage, or that we are working on the current outage.
 

Offline salbayeng

  • Frequent Contributor
  • **
  • Posts: 296
  • Country: au
Re: Server Error Reports
« Reply #10 on: August 30, 2017, 08:38:48 pm »
@Ian.M ?que? 
Quote
08:43:47 UTC is 09:43:47 GMT
Both UTC and GMT are the same for most purposes (GMT does not have summer time adjustments, so local time in the UK will be different by an hour in summer).
One adjusts for leap seconds as they occur, the other adjusts for leap seconds at the end of the year.
The UTC time standard is physically located at a naval observatory in the USA.

GPS time is nearly the same, but doesn't correct for leapseconds, it is currently about 20secs away from UTC (however all GPS receivers know the difference and will tell you UTC if asked).
 

Offline edavid

  • Super Contributor
  • ***
  • Posts: 3375
  • Country: us
Re: Server Error Reports
« Reply #11 on: August 30, 2017, 10:30:06 pm »
502 error at 2017-08-30 22:21:34 UTC

Firefox

https://www.eevblog.com/forum/unread/

(but it was affecting all pages)
« Last Edit: August 30, 2017, 10:32:00 pm by edavid »
 

Offline gnifTopic starter

  • Administrator
  • *****
  • Posts: 1671
  • Country: au
Re: Server Error Reports
« Reply #12 on: August 30, 2017, 11:42:23 pm »
502 error at 2017-08-30 22:21:34 UTC

Firefox

https://www.eevblog.com/forum/unread/

(but it was affecting all pages)

Again, check the status post, you can clearly see it is 'WiP" and there is additional information stating the details of the work in progress.
 

Offline edavid

  • Super Contributor
  • ***
  • Posts: 3375
  • Country: us
Re: Server Error Reports
« Reply #13 on: August 31, 2017, 12:26:25 am »
Again, check the status post, you can clearly see it is 'WiP" and there is additional information stating the details of the work in progress.

I don't understand, are you saying we should not report errors if the status says "OK - WiP"?
 

Offline gnifTopic starter

  • Administrator
  • *****
  • Posts: 1671
  • Country: au
Re: Server Error Reports
« Reply #14 on: August 31, 2017, 12:51:32 am »
Again, check the status post, you can clearly see it is 'WiP" and there is additional information stating the details of the work in progress.

I don't understand, are you saying we should not report errors if the status says "OK - WiP"?

Quote from: gnif
If any of the below hold true DO NOT post a report.
  • Check the next post under this one to see if we have announced an expected outage, or that we are working on the current outage.

You experienced an outage, but the status states we are working on it and there are dated status updates below it.
 

Offline edavid

  • Super Contributor
  • ***
  • Posts: 3375
  • Country: us
Re: Server Error Reports
« Reply #15 on: August 31, 2017, 12:55:52 am »
You experienced an outage, but the status states we are working on it and there are dated status updates below it.

Maybe I'm dense, but that was not at all clear to me.  Why would I think that the fact that you are working on the backup system means I shouldn't report a 502 error?

(Or maybe you could change the 502 error message when you don't want reports.)
 

Offline gnifTopic starter

  • Administrator
  • *****
  • Posts: 1671
  • Country: au
Re: Server Error Reports
« Reply #16 on: August 31, 2017, 01:11:28 am »
You experienced an outage, but the status states we are working on it and there are dated status updates below it.

Maybe I'm dense, but that was not at all clear to me.  Why would I think that the fact that you are working on the backup system means I shouldn't report a 502 error?

(Or maybe you could change the 502 error message when you don't want reports.)

Sorry, I will try to be clearer in the status updates from here on in :)
 
The following users thanked this post: Ian.M

Offline RGB255_0_0

  • Frequent Contributor
  • **
  • Posts: 772
  • Country: gb
Re: Server Error Reports
« Reply #17 on: August 31, 2017, 04:11:27 pm »
You know about the 10AM issues...

FYI, I just had a 502 at 5:08PM BST.
Your toaster just set fire to an African child over TCP.
 

Offline gnifTopic starter

  • Administrator
  • *****
  • Posts: 1671
  • Country: au
Re: Server Error Reports
« Reply #18 on: August 31, 2017, 04:12:39 pm »
You know about the 10AM issues...

FYI, I just had a 502 at 5:08PM BST.

Quote from: gnif
Report Status: Outages are to be expected, reports are not needed at this time.
 

Offline McBryce

  • Super Contributor
  • ***
  • Posts: 2678
  • Country: de
Re: Server Error Reports
« Reply #19 on: September 04, 2017, 11:28:31 am »
502 error at 13:00 German time.

McBryce.
30 Years making cars more difficult to repair.
 

Offline Cerebus

  • Super Contributor
  • ***
  • Posts: 10576
  • Country: gb
Re: Server Error Reports
« Reply #20 on: September 28, 2017, 04:16:13 pm »
502 for "https://www.eevblog.com/forum/blog/?action=post2;start=100" at 17:15:30 GMT.

Also got a "the last post from your IP address" rate limiting message 2+ minutes after posting this message.
« Last Edit: September 28, 2017, 04:21:57 pm by Cerebus »
Anybody got a syringe I can use to squeeze the magic smoke back into this?
 

Offline ChrisLX200

  • Supporter
  • ****
  • Posts: 458
  • Country: gb
Re: Server Error Reports
« Reply #21 on: September 29, 2017, 09:59:20 pm »
502 - Bad Gateway  10:55pm GMT
 

Offline gnifTopic starter

  • Administrator
  • *****
  • Posts: 1671
  • Country: au
Re: Server Error Reports
« Reply #22 on: October 02, 2017, 09:04:17 am »
Just a little update, we are still working on fixing a known issue (actually, the same issue). I have been away and have not had the opportunity to perform some changes that are required to proceed with the fix. I should be able to free up time in the next few days and work with Dave to schedule some downtime required to put the new system into effect.
 

Online guenthert

  • Frequent Contributor
  • **
  • Posts: 702
  • Country: de
Re: Server Error Reports
« Reply #23 on: October 09, 2017, 09:48:44 am »
502 just happened here (Berlin/Germany) around 9:40 zulu.
 

Offline nugglix

  • Regular Contributor
  • *
  • Posts: 209
  • Country: de
Re: Server Error Reports
« Reply #24 on: October 09, 2017, 09:57:55 am »
Same for Hamburg/Germany.
 


Share me

Digg  Facebook  SlashDot  Delicious  Technorati  Twitter  Google  Yahoo
Smf