Server Error Reports

[gnif]

Heading this off before people report it. There was an unexpected power interruption at the Data Centre that brought the servers down that are hosting this website. Things are fully back online, and we are seeking information from the DC as to what the reason for the outage was.

[gnif]

Just a heads up.

In the next 30 minutes or so there will be some disruptions while issues from the outage today are rectified. I will as always, post here when this work is complete.

Edit: Still in progress. sorry for the long outage, we are still having issues with data consistency across the cluster. We had the failed drive replaced and the fault moved to another disk, so we have been carefully bringing things back into a consistent state before allowing the DC to mess around with things futher.

[ifrythings]

Just a heads up.

In the next 30 minutes or so there will be some disruptions while issues from the outage today are rectified. I will as always, post here when this work is complete.

Edit: Still in progress. sorry for the long outage, we are still having issues with data consistency across the cluster. We had the failed drive replaced and the fault moved to another disk, so we have been carefully bringing things back into a consistent state before allowing the DC to mess around with things futher.

Would this cause the 502 bad gateway error and random “file not found” error that will load after several refreshes?

[gnif]

Yes, one of the servers in the cluster was having issues with accessing content on the disk.

[gnif]

Services have returned to normal.

[gnif]

Seems there was some kind of outage at the data center again, both of our servers were rebooted. Services are back online, but may be a bit slow while the servers caches warm back up.

[Simmed]

FYI encountered 502 and 504 screen about 1 - 2hrs ago

[gnif]

Yes, we are still not fully operational but working on it.

[Kean]

I am getting the occasional "file not found" error, and some other weirdness (error thanking posts, missing images).

I hope that whatever caused the problem, it doesn't take too much time out of your weekend.

[hexbus]

I am getting the occasional "file not found" error, and some other weirdness (error thanking posts, missing images).

I hope that whatever caused the problem, it doesn't take too much time out of your weekend.

Same here.  I had to refresh several times while trying to attach and post just now.

[PA0PBZ]

If I remember correctly there is a load balancing system with 2(?) servers and it looks like one of the servers is not playing nicely which gives the 'file not found' message. So refreshing the page until you land on the good server solves it for now. I'm sure gnif will be able to solve it once they are awake over there.

[EEVblog]

Test.
Yeah, file not found images missing etc.

[gnif]

Things should be back to normal shortly, the second node had a failure with the clustered storage and was removed from the storage group. The re-sync has completed and now I am just in the process of bringing the other services on the server back up and online.

[gnif]

All services have been returned to normal.

[EEVblog]

TEST.
Everything seems fine now 
What was the exact initial cause?

[gnif]

What was the exact initial cause?

I will email you on this as it's a security risk to expose details of your configuration

[MK14]

Perhaps 10% .. 50% of the time (but it can be fine, for many minutes on the trot), maybe on 10+ occasions, in the last few minutes (longer now, since I edited the post), I'm getting database errors, on this forum.
E.g. 'Show new replies to your posts.', but I'm seeing it in other place(s) as well.

    EEVblog Electronics Community Forum


Database Error
Please try again. If you come back to this error screen, report the error to an administrator.
Back

[Simmed]

when click forum home orange button
it displays error (randomly)
the normal forum page does not appear

Please try again. If you come back to this error screen, report the error to an administrator.

[themadhippy]

Same here as wot MK14 is getting

[Xena E]

when click forum home orange button
it displays error (randomly)
the normal forum page does not appear

Please try again. If you come back to this error screen, report the error to an administrator.

+1

[AVGresponding]

Yep.

They need to feed the hamsters running on the generator wheels more

[gnif]

I'm looking into it

[gnif]

Seems there was a substantial traffic spike that may have been the cause, unfortunately due to a configuration error the logs during that period are not very useful (reverse proxy IP logged instead of real IP). The traffic looks quite unusual and may have been either a DoS attack, or some super aggressive crawler. We will simply have to wait and see if it occurs again.

[EEVblog]

The traffic looks quite unusual and may have been either a DoS attack, or some super aggressive crawler. We will simply have to wait and see if it occurs again.

Another new AI bot sucking up everything perhaps?

[gnif]

That's my guess