504 Gateway Time-out

[Monkeh]

I don't trust them from a security standpoint. Nor do I trust them to be doing their jobs right.

That of absolutely no help to me what so ever.

My sincerest apologies for having an opinion.

If they provide unencrypted FTP access, they've failed completely already.

They don't, it has nothing to do with FTP.

Fair enough, that was me making 4am assumptions.

Well, I just got the 504 error message again.
This is now beyond ridiculous.
I cannot believe that both capnel and WHM backups cause a dedicated server to lock up and refuse to respond every time it does a backup. If that was the case, millions of websites would be down.
This same server used to work just fine using HostGators own WHM automated backup to a 2nd disk, and also using siteautobackup.com
I suspect it's only since the server was switched over to 64bit and all the performance tweeks have been made that the problem has arisen. I could be wrong, but it most certainly used to work just fine with no reports of 504 gateway errors.
 

Millions of websites don't attempt to make a 7GB copy of themselves to another host every day. It's not a good way to go about things.

[EEVblog]

Millions of websites don't attempt to make a 7GB copy of themselves to another host every day. It's not a good way to go about things.

So what's your easy to use no fuss foolproof solution?

[GeoffS]

wilfred: I only run cPanel and not WHM.
There's no facility for scheduling a backup in cPanel, Any backups you run from cPanel are created as gzipped tar files on the server. You can download these to your own server or cPanel can create them on another cPanel hosted system.
Database backups are created as text files containing all the SQL statements to recreate the database from scratch, no facility for partial restore unless you know enough to edit the saved data.

This is based on my own cPanel hosting on a shared server, might be different for dedicated servers

[EEVblog]

As I follow this thread I have attempted to find out a bit more about cPanel and WHM. It seems to me that cPanel can do, but not schedule, backups and WHM provides the facility to create a backup schedule. I am wondering how the cPanel backups were scheduled before WHM was attempted.

I'm using an external service siteautobackup.com that initiates initiates a scheduled backup and then download the file when it's finished.
It then purges older backups and only keeps however many backup you tell it too in order to fit into the backup space you have paid for.
I think that's a really neat solution, and I'd love to have something similar.

It does the same thing for either Cpanel, WHM, or just the databases. Perhaps I can try just the databases? That would at least give me a back of them daily, and shouldn't take too much space or resources to do that?
Then I could find another (incremental?) solution for the files?

I seemed to get the impression that WHM schedules a cPanel backup and provides various options for frequency and retention of generations but I didn't see that it can do incremental backups. Can cPanel?

No, cpanel is all or nothing.

I can't see how a database can be backed up incrementally from outside the database manager (MySQL). In any case if WHM is just a layer on top of the cPanel backup then wouldn't it still create the same problem of MySQL tables needing to be locked pending the completion of the backup.

cpanel is layer on top of WHM, which is a layer upon the basic linux install.
According to the linux penguins it's evil to use either 

[EEVblog]

Ok, another try with siteautobackup.com
Trying a "partial" cpanel back right now. It is set to backup the databases and also all the home directory files. I can disable the files if needed, but thought I'd try it.
Lets see if the server goes down again...

[Monkeh]

Millions of websites don't attempt to make a 7GB copy of themselves to another host every day. It's not a good way to go about things.

So what's your easy to use no fuss foolproof solution?

There's no such thing, sorry.

[xrunner]

You're not gonna believe this, but it's my duty to report it. It lasted for at least  10 minutes.

504 Gateway Time-out
nginx

[alm]

So on that basis, have you considered setting up a MySQL replication slave server? As I understand it the server can be paused independently of the master and backups performed at any leisurely pace without disrupting the master where updates will continue. Then after backups on the slave are completed the slave is re-enabled and the master can replicate any pending updates to the slave.

That's a fine way of doing non-intrusive backups. Another good way is to do the backup in a transaction (mysqdump --single-transaction), although this requires you to use storage engines with transaction support (eg. InnoDB). There will still be a performance impact, but at least the tables are not locked.

[justanothercanuck]

Does SMF store it's uploads in the database?  7GB seems like a lot, if it's just the text from our posts...  FWIW, phpBB stores its uploads as files, making the database easier to backup. 

[EEVblog]

Does SMF store it's uploads in the database?  7GB seems like a lot, if it's just the text from our posts...  FWIW, phpBB stores its uploads as files, making the database easier to backup. 

No, its stores the uploads in a directory. IIRC the database is about 100MB or so last I looked.
I've had limits set on the uploads for a long time, so thankfully it's only 7GB total for all the stuff, but it does grow daily.

[justanothercanuck]

Of course, thanks for the clarification.

[metalphreak]

Does it put a lock on the files while it does the backup? How good is the connection from your server to the backup service? Just because you have a 100mbit port doesn't mean you have 100mbit bandwidth to every location. The backup could be taking forever at a slow transfer speed causing the 504's for a random length of time?

[GeoffS]

I just had another 504 occur. 11:30 local time.

When I was again able to access the forum a couple of minutes later, no new posts were shown.
I normally use the Show unread posts since last visit button but no posts were visible.

[xrunner]

I had the 504 error off and on for at least 30 - 40 min. plus a couple of "SMF could not connect to the database ..." thrown into the mix.

[EEVblog]

I had the 504 error off and on for at least 30 - 40 min. plus a couple of "SMF could not connect to the database ..." thrown into the mix.

Yep, that was me, sorry.
I have now narrowed it down.
I tried to perform an optimisation of the database table, an option provided from within SMF, and got a 504 error straight away.
But what cases the outage was that I when I tried a database backup (another option provided from within SMF) that immediately locked up the server with a 504 for at least 30 minutes. Something I have don many times in the past without issue.
So it's not the backup the server files that is the problem, it's a matter of the database.

gnif tells me that this is to be expected for a database of my size under cpanel/mysql, and the only solution is a MySQL slave on another server which is then backed up. Although I kinda find this hard to believe, as I've never had this issue before with either the former dedicated server box (32bit one, same host), cloud server, or shared server solutions. It seems to have only been since the move to the 64bit dedicated server that the issue has arisen. And the database hasn't gotten a huge amount larger since then, so that's what makes me think there could be something else at play here.
But then again I know jack about this stuff, just going on my experience.

[EEVblog]

I'm about to tell siteautobackup.com to start a backup of the directory files only and none of the databases, I'm curious to see what happens...

[EEVblog]

I'm about to tell siteautobackup.com to start a backup of the directory files only and none of the databases, I'm curious to see what happens...

Well, the forum still works as fast as ever, so I guess automated file backup is not a problem, so far, backup not complete yet though. But at least it didn't lock it up.

[GeoffS]

From what you've reported, it sounds as if a database dump via (say) mysqldump is done with the tables locked. This would prevent any database access, read or write, until the backup process is finished. There are options that allow the dump program to operate reliably without exclusively locking the database.

Is there any information available as to how siteautobackup handles the database for its backups?

[EEVblog]

From what you've reported, it sounds as if a database dump via (say) mysqldump is done with the tables locked. This would prevent any database access, read or write, until the backup process is finished. There are options that allow the dump program to operate reliably without exclusively locking the database.

If there is a such an option I'm keen to try it.

Is there any information available as to how siteautobackup handles the database for its backups?

Not that I'm aware of.
The same thing seem to to happen in SMF or siteautobackup.

[GeoffS]

Another long 504 error session. Followed by a couple of SMF errors 'unable to connect to database'

[EEVblog]

Another long 504 error session. Followed by a couple of SMF errors 'unable to connect to database'

Yep, sorry, likely me again.
I did a databse backup from within cpanel and the site instantly went 504.
Although I suspect the siteautobackup.com was still running as well.
At least we know what causes it now!

[edavid]

Have you tried MySQLDumper?

[ve7xen]

Perhaps the table type has been changed in the move?

I would suggest dispensing with mysqldump backups unless you have a replication slave.

Shut down mysql to guarantee a consistent file structure, make an lvm snapshot (takes a few seconds at most), start mysql again and then duplicity/rdiff-backup or rsync your snapshot to a remote server. Duplicity is smart enough that you can use pretty much any file storage service, and both are incremental. A few seconds of downtime and a much faster (and less bandwidth intensive) backup process.

You can use something like backupninja to help automate this (example), but it sounds like you're getting to the point where you need someone with actual systems experience to figure out the best way to handle all of this for your setup and implement it properly. It should be possible to do it with zero to minimal downtime and less bandwidth (and time) than your current solution.

[GeoffS]

Without knowing the current configuration and usage of the MySQL database(s), I think we're just guessing when it comes to suggesting better methods of backup.

Standby sites for backup and failover are all well and good but someone has to pay for the necessary resources. 

[EEVblog]

Aside from just getting the backups without disrupting the forum server, how important is recovery time from a complete server loss? Would it be useful to have another forum server running alongside the primary that can be activated by redirecting network traffic in the event of a loss of the primary?

The world won't end if it takes me a day or two to fix it. I'd lose incoming email in that time, but no biggie.

How much of the current server capacity is required to run the forum and how much of it is headroom just to take backups and perform administrative tasks? With a MySQL replication server you can offload the backup overhead (well most of it) that impacts the forum database offsite to a remote server. It may be that you could move back to a pair of smaller servers, since neither has to run both the forum and the backups at the same time.

The current dedicated server is pretty overkill. CPU load for examples borders on trivial.

gnif has figured out it's the persistent connections in the fourm software in combination with the myqsl version and cpanel backup etc. He's currently trying to patch it.
So this is hopefully close to being fixed.