Author Topic: Server Error Reports  (Read 756091 times)

0 Members and 2 Guests are viewing this topic.

Offline gnifTopic starter

  • Administrator
  • *****
  • Posts: 1691
  • Country: au
Re: Server Error Reports
« Reply #700 on: January 12, 2024, 01:33:43 pm »
Forums are back up (obviously) however attachments are still being copied over, so expect broken attachments until they have all moved over.
 
The following users thanked this post: EEVblog

Offline EEVblog

  • Administrator
  • *****
  • Posts: 38019
  • Country: au
    • EEVblog
Re: Server Error Reports
« Reply #701 on: January 12, 2024, 02:55:44 pm »
Forums are back up (obviously) however attachments are still being copied over, so expect broken attachments until they have all moved over.

Thanks.
Forum seems crazy quick!
But it was always slick.
 

Offline gnifTopic starter

  • Administrator
  • *****
  • Posts: 1691
  • Country: au
Re: Server Error Reports
« Reply #702 on: January 12, 2024, 10:39:53 pm »
Yes, there is certainly a huge performance boost here, IO load on average has dropped from 90+% to around 5-10%. DRBD + GFS2 is so much faster I can actually now count how many attachment files are on disk (for the record, over 810,000 files in one directory (SMF :palm:), and because it's so much less disruptive now to manage this directory we can start to look at migrating this into a proper directory structure.

Note the work is still not complete though, there is still a few things to shuffle around and there may be some minor outages lasting a few minutes while the work is completed.
 
The following users thanked this post: EEVblog, Kean

Offline gnifTopic starter

  • Administrator
  • *****
  • Posts: 1691
  • Country: au
Re: Server Error Reports
« Reply #703 on: January 13, 2024, 12:28:34 am »
The major work has been completed but the site will still have some slowness while I remove the old now deprecated gluster storage. Once this is completed we should be back to normal, but faster ;)

Edit: work has been completed, there should not be any more interruptions to services.
« Last Edit: January 13, 2024, 12:49:55 am by gnif »
 
The following users thanked this post: EEVblog, PA0PBZ, kjpye, PlainName, Andy Watson, Kean, HwAoRrDk, Nominal Animal, pcprogrammer

Offline gnifTopic starter

  • Administrator
  • *****
  • Posts: 1691
  • Country: au
Re: Server Error Reports
« Reply #704 on: February 06, 2024, 04:36:59 am »
Heading this off before people report it. There was an unexpected power interruption at the Data Centre that brought the servers down that are hosting this website. Things are fully back online, and we are seeking information from the DC as to what the reason for the outage was.
 
The following users thanked this post: EEVblog, Ed.Kloonk, SeanB, Kean

Offline gnifTopic starter

  • Administrator
  • *****
  • Posts: 1691
  • Country: au
Re: Server Error Reports
« Reply #705 on: February 06, 2024, 09:53:50 am »
Just a heads up.

In the next 30 minutes or so there will be some disruptions while issues from the outage today are rectified. I will as always, post here when this work is complete.

Edit: Still in progress. sorry for the long outage, we are still having issues with data consistency across the cluster. We had the failed drive replaced and the fault moved to another disk, so we have been carefully bringing things back into a consistent state before allowing the DC to mess around with things futher.
« Last Edit: February 06, 2024, 04:53:49 pm by gnif »
 

Offline ifrythings

  • Contributor
  • Posts: 14
  • Country: ca
Re: Server Error Reports
« Reply #706 on: February 06, 2024, 05:11:18 pm »
Just a heads up.

In the next 30 minutes or so there will be some disruptions while issues from the outage today are rectified. I will as always, post here when this work is complete.

Edit: Still in progress. sorry for the long outage, we are still having issues with data consistency across the cluster. We had the failed drive replaced and the fault moved to another disk, so we have been carefully bringing things back into a consistent state before allowing the DC to mess around with things futher.

Would this cause the 502 bad gateway error and random “file not found” error that will load after several refreshes?
 

Offline gnifTopic starter

  • Administrator
  • *****
  • Posts: 1691
  • Country: au
Re: Server Error Reports
« Reply #707 on: February 06, 2024, 05:13:45 pm »
Yes, one of the servers in the cluster was having issues with accessing content on the disk.
 
The following users thanked this post: The Soulman, Nominal Animal

Offline gnifTopic starter

  • Administrator
  • *****
  • Posts: 1691
  • Country: au
Re: Server Error Reports
« Reply #708 on: February 07, 2024, 02:49:26 am »
Services have returned to normal.
 
The following users thanked this post: EEVblog, IanB, xrunner, Kean

Offline gnifTopic starter

  • Administrator
  • *****
  • Posts: 1691
  • Country: au
Re: Server Error Reports
« Reply #709 on: April 22, 2024, 01:29:53 am »
Seems there was some kind of outage at the data center again, both of our servers were rebooted. Services are back online, but may be a bit slow while the servers caches warm back up.
 
The following users thanked this post: EEVblog, Kean, Brumby, MK14


Share me

Digg  Facebook  SlashDot  Delicious  Technorati  Twitter  Google  Yahoo
Smf