Server Backup Upgrade

[EEVblog]

Not that anyone needs to know, but some people might be interested.
I will be switching the forum backup from a daily automated cpanel based backup to Amazon S3, to an incremental system handled by gnif/hostfission.
It will also be possible to restore individual files with ease.
There is apparently still a database load causing an website response issue when the current cpanel backup starts daily, this new service will fix that.

FYI, we have never had to restore the server or any part of it from a backup in the last 8+ years.

[Rerouter]

8+ years of getting lucky , I've actually had my backup system fail 3 times during the life of my primary drives, (i consider a drive failure on raid 5 a failure until its rebuilt)

[anishkgt]

8+ years uptime. That is a good.


Sent from my iPhone using Tapatalk

[EEVblog]

FYI, we have never had to restore the server or any part of it from a backup in the last 8+ years.

As long as you know that you can. That's the important thing. Test it if you haven't. Before committing to it.

It was a standard cpanel backup with all database and server files etc. Never tested the complete thing, but if standard cPanel backup doesn't work then millions of websites would be in the same boat.
If it came down to it then it could all be restored manually.

I don' know how to test gnif's new backup system, in fact I personally won't have access to it (for security reasons), it's a paid service that Hostfission offer.

[bd139]

if standard cPanel backup doesn't work then millions of websites would be in the same boat.

This is a scary assumption having had to recover the steaming wreck of a cPanel frontend ISP that was compromised here in the UK. cPanel aren't exactly known for their quality control to start with or support when something does go bang.

If you want to test it, get someone to restore the whole thing randomly at the worst possible time for the hoster (think 3AM) onto another machine and make sure it's how you think it's supposed to be.

The ONLY way to test a backup is do a full restore.

[EEVblog]

if standard cPanel backup doesn't work then millions of websites would be in the same boat.

This is a scary assumption having had to recover the steaming wreck of a cPanel frontend ISP that was compromised here in the UK. cPanel aren't exactly known for their quality control to start with or support when something does go bang.

If you want to test it, get someone to restore the whole thing randomly at the worst possible time for the hoster (think 3AM) onto another machine and make sure it's how you think it's supposed to be.

The ONLY way to test a backup is do a full restore.

It's not scary or risky, it's just a matter of how much time and effort it takes to recover.
Even if the cPanel restore didn't work automatically, I have the databases (the most important thing), and all the server files. Even with my feeble penguin skills I've always been capable of eventually restoring the entire site.

Cue the hundred different people with 100 different way on how to do backups "properly" 

[dr.diesel]

Curious, how big is the DB now?

[EEVblog]

Curious, how big is the DB now?

1-2GB or something like that.

[CJay]

Nah, the only way to back up 'properly' is the way that works.

All solutions are different, the only things that matter is that you can get it back and that you do a sanity check (of the backup, not necessarily yourself) occasionally by restoring bits of it and checking they make sense.

All the rest is personal taste.

[Brumby]

Cue the hundred different people with 100 different way on how to do backups "properly" 

I won't be one.  Between you and GNIF I think it's going to be adequately covered.


I will just say - thank you for providing the facility and the service we have.

[tszaboo]

Nah, if it works, why poke it. I guess most of the server downtime, data loss, and other undesirable things happen, because people like tinkering around them.

Software guy: Hey this new version of software does this thing, lets update to it! Why is it not working anymore?
Hardware guy: Hey, there is a new version of software. How boring. Software keeps on running for 8 years.

Good job not messing with it, I wish you another 8 years of uninterrupted service!

[bd139]

Most of the upgrades we do are "holy shit, there's a hole the size of the moon in this" or "something is breaking because Broadcom have shitty drivers". Everything else is not important.

if standard cPanel backup doesn't work then millions of websites would be in the same boat.

This is a scary assumption having had to recover the steaming wreck of a cPanel frontend ISP that was compromised here in the UK. cPanel aren't exactly known for their quality control to start with or support when something does go bang.

If you want to test it, get someone to restore the whole thing randomly at the worst possible time for the hoster (think 3AM) onto another machine and make sure it's how you think it's supposed to be.

The ONLY way to test a backup is do a full restore.

It's not scary or risky, it's just a matter of how much time and effort it takes to recover.
Even if the cPanel restore didn't work automatically, I have the databases (the most important thing), and all the server files. Even with my feeble penguin skills I've always been capable of eventually restoring the entire site.

Cue the hundred different people with 100 different way on how to do backups "properly" 

If you've got that you're sorted. The rest is just a headache you can fix on the fly. It's not like we're going to fly to Australia and murder you for it being down for a few hours

(I have had death threats from clients who want to make sure I can fix their cock up)

[CJay]

(I have had death threats from clients who want to make sure I can fix their cock up)

I've had barristers try and make me guarantee systems won't ever crash again, I can handle death threats (most of them sit down, shut up and behave when they realise I'm close to 6' tall with a 48" chest) legal action scares me.

[bd139]

I've been sued before. It's not all that bad. Fire up the solicitor and let them work it out.  Cost me about £15k in total which was then immediately billed to the client to continue their contract which they couldn't refuse because there was a "payment while under dispute" clause. Mwuhahaha.

I have a Limited Company though which is kind of like a protected no claims bonus

I was sued for non delivery because they changed the requirements three times during the fixed term contract and delivery slipped and I said they'd need to pay more.

They no longer exist

[StillTrying]

Curious, how big is the DB now?

1-2GB or something like that.

TB ?

[Monkeh]

Curious, how big is the DB now?

1-2GB or something like that.

TB ?

It's essentially text, it's not going to be TB.

[dr.diesel]

And picts now, though not sure how those are stored, could be linked or blobs.

[Monkeh]

And picts now, though not sure how those are stored, could be linked or blobs.

Forgot about those, I'd figure they're stored flat.

[EEVblog]

And picts now, though not sure how those are stored, could be linked or blobs.

Images are stored in the SMF forum file structure in an attachments directory, they are not in the database.

[Brumby]

For those wondering, the word "Database" can be taken in two ways.

The first is to refer to every piece of data, including images, that a system needs to function.  The further up the management chain, the more likely you are to find someone who thinks this way.

Within the industry, especially at the "coal face" a Database is a specific part of that collection, which is what Dave is referring to.  It is, essentially, just pure text.  Typically it would be a relational database with queries extracting information using SQL.  Data can also be inserted and updated.

The image files will be in a separate directory structure - which will look familiar to anyone who has worked with any sort of a file manager.

The database will have the name and path (text) that points to the image file.

SMF then builds the page HTHL from the database and include the URLs for the images.  Your browser then renders the page with everything in place.

[gnif]

The backup solution that we are offering to Dave is a industry standard designed for recovery from complete system failure. It is performing both full and incremental backups over a private VPN to an undisclosed location. It is currently in use across our entire infrastructure and that of nearly all of our client base. We have not yet had to use it to perform a bare metal recovery, but in simulated failures we are able to recover a server from bare metal without any issue.

As for the database, we are using a module that invokes Percona's MySQL backup tool which is capable of taking a hot copy of the database without causing any service interruptions.

cPanel's backup system has several issues:

1) It uses mysqldump to dump the databases, which will lock tables to ensure a consistent dump. Unfortunately when you get a database as large and busy as Dave's, that lock time is too long and causes a backlog of requests eventually hitting the MySQL max connection limit.
2) The backup scheme is a simple compressed tar archive, creation of which is quite CPU expensive and can take several hours.
3) It doesn't backup the entire server, only the accounts on the server. Restoration from bare metal is not possible with cPanel's backup, it needs to be bootstrapped with a cPanel install first, not to mention all the customizations we have made to get this to perform as well as it does.
4) Recovery of a single file requires decompressing the entire archive (yes, even if you tell tar to just extract one file, it needs to decompress up to that point to find it). If the archive is old and only exists on a remote server such as Amazon S3 it requires transferring the entire archive to a server to work on it first.
5) cPanel works on a push model for backup, meaning that if an attacker gained access to the server they could use the cPanel access details to the remote site (Amazon S3) and delete the remote backups too.

The backup solution we offer deals with all of these issues and more.

Curious, how big is the DB now?

1-2GB or something like that.

I think Dave is listing the compressed size, this is where it is at current.

Code: [Select]

# du -hs /var/lib/mysql/
9.0G mysql/

# du -hs /home/REDACTED/
43G /home/REDACTED/