Server Error Reports

[gnif]

The major work has been completed but the site will still have some slowness while I remove the old now deprecated gluster storage. Once this is completed we should be back to normal, but faster

Edit: work has been completed, there should not be any more interruptions to services.

[gnif]

Heading this off before people report it. There was an unexpected power interruption at the Data Centre that brought the servers down that are hosting this website. Things are fully back online, and we are seeking information from the DC as to what the reason for the outage was.

[gnif]

Just a heads up.

In the next 30 minutes or so there will be some disruptions while issues from the outage today are rectified. I will as always, post here when this work is complete.

Edit: Still in progress. sorry for the long outage, we are still having issues with data consistency across the cluster. We had the failed drive replaced and the fault moved to another disk, so we have been carefully bringing things back into a consistent state before allowing the DC to mess around with things futher.

[ifrythings]

Just a heads up.

In the next 30 minutes or so there will be some disruptions while issues from the outage today are rectified. I will as always, post here when this work is complete.

Edit: Still in progress. sorry for the long outage, we are still having issues with data consistency across the cluster. We had the failed drive replaced and the fault moved to another disk, so we have been carefully bringing things back into a consistent state before allowing the DC to mess around with things futher.

Would this cause the 502 bad gateway error and random “file not found” error that will load after several refreshes?

[gnif]

Yes, one of the servers in the cluster was having issues with accessing content on the disk.

[gnif]

Services have returned to normal.

[gnif]

Seems there was some kind of outage at the data center again, both of our servers were rebooted. Services are back online, but may be a bit slow while the servers caches warm back up.

[Simmed]

FYI encountered 502 and 504 screen about 1 - 2hrs ago

[gnif]

Yes, we are still not fully operational but working on it.

[Kean]

I am getting the occasional "file not found" error, and some other weirdness (error thanking posts, missing images).

I hope that whatever caused the problem, it doesn't take too much time out of your weekend.

[hexbus]

I am getting the occasional "file not found" error, and some other weirdness (error thanking posts, missing images).

I hope that whatever caused the problem, it doesn't take too much time out of your weekend.

Same here.  I had to refresh several times while trying to attach and post just now.

[PA0PBZ]

If I remember correctly there is a load balancing system with 2(?) servers and it looks like one of the servers is not playing nicely which gives the 'file not found' message. So refreshing the page until you land on the good server solves it for now. I'm sure gnif will be able to solve it once they are awake over there.

[Simmed]

haha this is so bad
i have to login 5x to get to this post 
and the post = "file missing"
and the post was successful after 2nd try

[EEVblog]

Test.
Yeah, file not found images missing etc.

[gnif]

Things should be back to normal shortly, the second node had a failure with the clustered storage and was removed from the storage group. The re-sync has completed and now I am just in the process of bringing the other services on the server back up and online.

[gnif]

All services have been returned to normal.

[EEVblog]

TEST.
Everything seems fine now 
What was the exact initial cause?

[gnif]

What was the exact initial cause?

I will email you on this as it's a security risk to expose details of your configuration