Forum is on MySQL. Should be on PostgreSQL for maximum robustness.

[StuUK]

Just read this thread..

Oh boy! another this database vs that database discussion.... Absolutely nothing wrong with using MySQL and obviously the amount of (or should I say lack of) downtime clearly demonstrates no disernable value vs the effort of swapping databases, why would you unless it's grinding to a halt or falling over every day, which it clearly isn't and even then the database may not be the issue.

The only real issue I can see is the reference to special symbols/characters which could be resolved by changing the character set on a number of tables to UTF etc which is no big deal.

And the idea that swapping to PostgreSQL gives maximum robustness clearly demonstrates a naivety in respect to IT systems. There are many many components that contribute to robustness, the database is just one of them and a single instance of poorly configured PostgreSQL is no more robust than a single instance of a poorly configured MySQL. There is more chance of an Amazon region failure or instance outage causing an issue than the MySQL database suddenly corrupting itself.

I could just as easily say throw out this piece of forum software for another piece of forum software to achieve more robustness...

 

[kcbrown]

Well, anyway, sorry for bringing up this suggestion, guys.  I thought it would be a reasonable suggestion, given my rather horrid experience with MySQL's MyISAM tables and the opposite experience I've had with PostgreSQL, and the claim by Simple Machines that PostgreSQL is supported by SMF 2.0...

 

[timb]

Well, anyway, sorry for bringing up this suggestion, guys.  I thought it would be a reasonable suggestion, given my rather horrid experience with MySQL's MyISAM tables and the opposite experience I've had with PostgreSQL, and the claim by Simple Machines that PostgreSQL is supported by SMF 2.0...

 

It's your own fault for even suggesting such a thing! What do you think this is, some kind of *suggestion sub-forum* or something?

Oh, wait...

[kcbrown]

LOL.  Reminds me of that famous movie quote.  "Hey, you can't fight in here!  This is the War Room!"    


(Sent with Tapatalk, so apologies for the lackluster formatting)

[EEVblog]

No matter how you look at it it cannot be denied that MyIsam tables have inherent flaws which lead to loss of data so it makes sense to use a table structure which is more robust and thus improve the forum availability.

The forum already has like 99.99% uptime, it's fine, relax.

[nctnico]

99.99% is still over to 3.5 days of outage per year!

[StuUK]

99.99% is still over to 3.5 days of outage per year!

Seriously, have you any idea about just how good that is in the real IT world..... most service providers never achieve it even for critical systems....

[nctnico]

99.99% is still over to 3.5 days of outage per year!

Seriously, have you any idea about just how good that is in the real IT world..... most service providers never achieve it even for critical systems....

I have 'some' experience with system administration but stopped doing that a long time ago that because it was utterly boring. Anyway: 3.5 days unscheduled downtime per year? Not on my watch! If you maintain systems used for trading stocks and options with a daily trading value of a couple of billion euro you'll have to get things running smoothly and reliable. I also used to co-own a webdesign company and we had way better availability than 99.99%! Maybe 3 or 4 hours per year downtime.

[EEVblog]

99.99% is still over to 3.5 days of outage per year!

Err, your math is several orders out, it's 0.0365 days (0.8 hours)
Ok, so it's maybe 99.9% (0.365 days a year), even that is still pretty good and a figure that most host providers will guarantee.
And those failures are almost always because of power or other dedicated server failure. If it is a database table corruption (happens maybe 1 in 10 times the server hiccups), then it will stay down until I or gnif notice it and fix it (i.e we are sleeping).
Again, a database table corruption has never happened just on it's own, it's always the result of hardware or something else going down on the server.

[MatthewMorgan]

Also, who's to say the new environment won't have it's own foibles that require even more attention than the current one?

Yes. Just getting the current forum and eevblog server running as smooth as it does has taken years of tweaking many minor things. Many issues you don't know about until it becomes a problem and you investigate it.
To think you can "just change the database to InnoDB" and that's that is quite unlikely not a realistic way it would unfold.

Making things more cacheable would reduce server load,
Maybe if possible show all logged out users a html version that is refused when you come to a maximum of once per minute. (cache)

[EEVblog]

I also used to co-own a webdesign company and we had way better availability than 99.99%! Maybe 3 or 4 hours per year downtime.

That's what the EEVblog server basically gets, a handful of real server hours per year of server downtime. As I said, the extra downtime is the result of not having someone respond immediately to fix database problems. There have also been a few cloudflare issues, but these are not related to the server.
My host have staff to fix actual server problems swiftly when they happen, but they do not monitor this forum or any other running databases.

[EEVblog]

Making things more cacheable would reduce server load,
Maybe if possible show all logged out users a html version that is refused when you come to a maximum of once per minute. (cache)

The server goes through Cloudflare, and has caching of various sorts, about as good as you can get without going to a multiple distributed server database structure thingo (I don't know the correct term). The server has very little real load, currently:
0.49 0.47 0.36 according to WHM

[StuUK]

99.99% is still over to 3.5 days of outage per year!

Seriously, have you any idea about just how good that is in the real IT world..... most service providers never achieve it even for critical systems....

I have 'some' experience with system administration but stopped doing a long time ago that because it was utterly boring. Anyway: 3.5 days unscheduled downtime per year? Not on my watch! If you maintain systems used for trading stocks and options with a daily trading value of a couple of billion euro you'll have to get things running smoothly and reliable. I also used to co-own a webdesign company and we had way better availability than 99.99%! Maybe 3 or 4 hours per year downtime.

Apart from the dodgy maths, trading systems are looking at 5 9's (or better) or zero point of failure and spend the money to achieve it... hardly think Dave needs to worry about that kind of expenditure for his forum and irrespective of what Dave does with his software config he's still reliant upon the Amazon cloud infrastructure which is significantly more likely to cause problems than ISAM tables.....

[nctnico]

99.99% is still over to 3.5 days of outage per year!

Err, your math is several orders out, it's 0.0365 days (0.8 hours)

You are right! But all in all the EEVblog is out for at least several days if you add it all up so that is why I didn't detect the error.

[EEVblog]

Apart from the dodgy maths, trading systems are looking at 5 9's (or better) or zero point of failure and spend the money to achieve it... hardly think Dave needs to worry about that kind of expenditure for his forum and irrespective of what Dave does with his software config he's still reliant upon the Amazon cloud infrastructure which is significantly more likely to cause problems than ISAM tables.....

I don't use the amazon cloud, I have a dedicated Xeon server box in a bunker somewhere in Texas.

[EEVblog]

You are right! But all in all the EEVblog is out for at least several days if you add it all up so that is why I didn't detect the error.

Shit happens, you can't easily and cheaply avoid it.

[SeanB]

Most of the server downtime generally is fixed fast, or at least within 2 hours from my perspective and times of use. All in all the biggest outages are mostly Cloudflare shitting itself locally, or a local interruption which drops connectivity. Migrating or moving is something I think Dave really will only do as a last resort, though I do hope he has at least tested his disaster recovery by at least trying to restore a full backup to a test server, probably easiest to do using a few small PC units running on a closed network, one as simple DNS, pop and NTP to emulate the real internet, the restore server and then a machine to test from, so he can restore the full configuration and emulate the cloudflare requests and such, and see if the restored server reacts correctly.

No good having a backup of the data if the configuration is all back to blank or the default sets, it is a massive pain to get it all the same again afterwards.

[StuUK]

Apart from the dodgy maths, trading systems are looking at 5 9's (or better) or zero point of failure and spend the money to achieve it... hardly think Dave needs to worry about that kind of expenditure for his forum and irrespective of what Dave does with his software config he's still reliant upon the Amazon cloud infrastructure which is significantly more likely to cause problems than ISAM tables.....

I don't use the amazon cloud, I have a dedicated Xeon server box in a bunker somewhere in Texas.

Apologies, must have misread something somewhere about you using Amazon... which probably explains why you've got such good uptime

[EEVblog]

though I do hope he has at least tested his disaster recovery by at least trying to restore a full backup to a test server, probably easiest to do using a few small PC units running on a closed network, one as simple DNS, pop and NTP to emulate the real internet, the restore server and then a machine to test from, so he can restore the full configuration and emulate the cloudflare requests and such, and see if the restored server reacts correctly.
No good having a backup of the data if the configuration is all back to blank or the default sets, it is a massive pain to get it all the same again afterwards.

The backups are standard WHM backup requests of the entire server files + databases, stored both locally on the server on a separate drive, and also Amazon S3.
So if the WHM backup system doesn't work, then I suspect there will be millions of other systems that don't work either. No I have not tried to restore the 25GB backup file.
Que everyone saying how I'm doing my backups wrong in 3.. 2.. 1..

[EEVblog]

Most of the server downtime generally is fixed fast, or at least within 2 hours from my perspective and times of use. All in all the biggest outages are mostly Cloudflare shitting itself locally, or a local interruption which drops connectivity.

Correct.

[RGB255_0_0]

99.99% is still over to 3.5 days of outage per year!

Seriously, have you any idea about just how good that is in the real IT world..... most service providers never achieve it even for critical systems....

That would not class as mission-critical. Heck, if your Internet was down for more than 12 hours you'd be on the phone to your service provider. With RAID redundancy, hot-swappable hardware, clusters across different countries and UPS 3 days is far too long, even over a major holiday like Christmas.

[RGB255_0_0]

Making things more cacheable would reduce server load,
Maybe if possible show all logged out users a html version that is refused when you come to a maximum of once per minute. (cache)

The server goes through Cloudflare, and has caching of various sorts, about as good as you can get without going to a multiple distributed server database structure thingo (I don't know the correct term). The server has very little real load, currently:

Not really wise to disclose that info if it's actually yours. There could be disclosed bugs not patched

[SeanB]

though I do hope he has at least tested his disaster recovery by at least trying to restore a full backup to a test server, probably easiest to do using a few small PC units running on a closed network, one as simple DNS, pop and NTP to emulate the real internet, the restore server and then a machine to test from, so he can restore the full configuration and emulate the cloudflare requests and such, and see if the restored server reacts correctly.
No good having a backup of the data if the configuration is all back to blank or the default sets, it is a massive pain to get it all the same again afterwards.

The backups are standard WHM backup requests of the entire server files + databases, stored both locally on the server on a separate drive, and also Amazon S3.
So if the WHM backup system doesn't work, then I suspect there will be millions of other systems that don't work either. No I have not tried to restore the 25GB backup file.
Que everyone saying how I'm doing my backups wrong in 3.. 2.. 1..

No, that is a good backup. The whole kitchen sink is good, though hopefully there is more than a single copy, so you can at least have the last week, month just in case. If you do Amazon Glacier then the tape store will be cheap, and you can simply delete the earliest one if you are hitting the limits of the price you want to pay.

At work we do a similar, DLT drive to copy the RAID to tape, with a 2 week tape rotation to an off site secure storage, plus the server warns if a drive goes failed, when there is a mad scramble to drop in a new one fast to do a slow rebuild. Luckily load is very low for the most part, most usage is the actual backup itself which might get the jet turbines to move off the lowest settings for speed.

[nctnico]

though I do hope he has at least tested his disaster recovery by at least trying to restore a full backup to a test server, probably easiest to do using a few small PC units running on a closed network, one as simple DNS, pop and NTP to emulate the real internet, the restore server and then a machine to test from, so he can restore the full configuration and emulate the cloudflare requests and such, and see if the restored server reacts correctly.
No good having a backup of the data if the configuration is all back to blank or the default sets, it is a massive pain to get it all the same again afterwards.

The backups are standard WHM backup requests of the entire server files + databases, stored both locally on the server on a separate drive, and also Amazon S3.
So if the WHM backup system doesn't work, then I suspect there will be millions of other systems that don't work either. No I have not tried to restore the 25GB backup file.
Que everyone saying how I'm doing my backups wrong in 3.. 2.. 1..

I just assume there is some history build in. Say you backup a completely non-working database... At one of my employers I lost about 2 weeks of work due to the server crashing and a wrongly implemented backup scheme (on tape) which had a hole in the history. Guess where my work dissapeared into.

[Brumby]

I wasn't going to say anything, but nctnico raised the topic.  I know this is not going to be enthusiastically received, but it applies to anyone who does backups .....

Do you know your backups will have everything you need to recover?  (Not just the things you thought of.)

The only way to really know is to take your backups and restore onto a blank machine.  If you haven't done that exercise when it isn't required, how do you know it will work when it IS?


I'm not jumping on anyone's head and I'm not casting any aspersions on Dave.  This is just a question - and one that opens up the much bigger topic of Disaster Recovery Planning.