IMPORTANT: Server Changeover

[gnif]

I just saw the database exceed the old max connection limit, this would indicate that it was indeed the cause of the database error problem and it has now been resolved.

[gnif]

Sigh, the outage that just occurred was due to the way the old server was transferred to the new server, the issue has been fixed, sorry for the any inconvenience that it may have caused.

[bingo600]

You mean this one 

Connection Problems
Sorry, SMF was unable to connect to the database. This may be caused by the server being busy. Please try again later.

Thanx for your effort 

/Bingo

[gnif]

You mean this one 

Connection Problems
Sorry, SMF was unable to connect to the database. This may be caused by the server being busy. Please try again later.

Thanx for your effort 

/Bingo

One of them, there was also a big bolded "500" error, and a a few others. If you see any issues as of now, please let me know.
I may not see them for 8 hours or so though as I am going to bed, it is 4AM here in AU.

[firewalker]

Just gave me the same error. I had to reload 5~6 times.

Alexander.

[gnif]

very odd, let me look into it.

[bingo600]

If you see any issues as of now, please let me know.
I may not see them for 8 hours or so though as I am going to bed, it is 4AM here in AU.

Will post here with message & a time , if i encounter any probs.

Sleep tight down there

/Bingo

[gnif]

Thanks.

My additional logging caught the error which was a time-out on the connection, seems the connection time-out was set to 4 seconds :S. I have increased this, again, if errors are noticed please update this thread.

[manicdoc]

Possibly not, but this is something that I would not suggest just turning on as it requires a PHP recompile to enable it. Also APC is better then what cPanel provides (eAccelerator, XCache) and is pretty simple to configure and install and can be done without re-building PHP.

I believe that Simple Machine forum supports these caches if installed. At the moment, just basic caching is turned on in the forum.

Dave.

SMF caching is different, op-code caching is where the compiled interpreted PHP code is stored in RAM or even on disk, and next time the script loads, if the file on disk has not been changed, it uses the pre-compiled one in cache instead of having to re-compile the script again.

Yep, it can make a big difference to the CPU load. I think Dave was referring to memcache(d) support. Be good to check if that is operational whilst you are in there, given its a dedicated machine it should be safe to enable. SMF should use it.

[croberts]

The home page is not loading if I use www.eevblog.com/forum/index.php (the forum home button uses this url).
The home page will load if I use www.eevblog.com/forum/

[bingo600]

The https://www.eevblog.com/forum/index.php

Loads fine here Firefox 19.0 (Linux)  , just tried your url 2 times

/Bingo

[croberts]

I'm using firefox 19.0 (Windows 7). The url https://www.eevblog.com/forum/index.php worked fine for me until today. If I remove index.php it loads fine.

[gnif]

Can you try the broken address and hit refresh while holding shift? I suspect that you have cached the down site as I can not reproduce this problem.

[manicdoc]

Thanks.

My additional logging caught the error which was a time-out on the connection, seems the connection time-out was set to 4 seconds :S. I have increased this, again, if errors are noticed please update this thread.

I'm assuming thats not the wait_timeout or something is really off with the setup. Also I'm presuming its a local port connect and not socket?

Also given the fact that we seem to require a hell of a lot of connections in proportion to the traffic that the forum s/w isn't that well optimized around db usage. I might download a copy myself and have a peek (btw direct email is keith at aykira dot com).

[croberts]

Hello gnif

I cleared the cache and restarted firefox. Now the url works. Thanks for your help.

[gnif]

Thanks.

My additional logging caught the error which was a time-out on the connection, seems the connection time-out was set to 4 seconds :S. I have increased this, again, if errors are noticed please update this thread.

I'm assuming thats not the wait_timeout or something is really off with the setup. Also I'm presuming its a local port connect and not socket?

Also given the fact that we seem to require a hell of a lot of connections in proportion to the traffic that the forum s/w isn't that well optimized around db usage. I might download a copy myself and have a peek (btw direct email is keith at aykira dot com).

It is a local connect via a unix socket, but the issue was resolved as far as I can tell at the moment. The slow query can not be avoided easily, it is when someone searches the forum, I have already investigated added a fulltext index to the posts to improve this, but it will require some code changes to implement it. The database was exceeding 50 connections across 200+ users, which is not bad at all. On average there is 4-10 connections open to the database, but when a few of these searches occur the numbers jump up a bit. The max connections has been increased and since the last round of fixes I have not seen any more instances in the additional logging I added to SMF.

Hello gnif

I cleared the cache and restarted firefox. Now the url works. Thanks for your help.

You're welcome

[manicdoc]

Thanks.

My additional logging caught the error which was a time-out on the connection, seems the connection time-out was set to 4 seconds :S. I have increased this, again, if errors are noticed please update this thread.

I'm assuming thats not the wait_timeout or something is really off with the setup. Also I'm presuming its a local port connect and not socket?

Also given the fact that we seem to require a hell of a lot of connections in proportion to the traffic that the forum s/w isn't that well optimized around db usage. I might download a copy myself and have a peek (btw direct email is keith at aykira dot com).

It is a local connect via a unix socket, but the issue was resolved as far as I can tell at the moment. The slow query can not be avoided easily, it is when someone searches the forum, I have already investigated added a fulltext index to the posts to improve this, but it will require some code changes to implement it. The database was exceeding 50 connections across 200+ users, which is not bad at all. On average there is 4-10 connections open to the database, but when a few of these searches occur the numbers jump up a bit. The max connections has been increased and since the last round of fixes I have not seen any more instances in the additional logging I added to SMF.

Funny, I thought it was exactly the text search when Dave first had the problem (linear search across all rows if memory serves). See if you can get memcache doing its thing - myself I like to keep requests for static content 'away' from the database and let it focus on the truly dynamic none cacheable queries. How many thread instances of mysqld are running?  Also be careful on the text search coding (i.e. guard against stupid hogging searches).

We'll get the speed stripes on this set up yet 

[EEVblog]

One of them, there was also a big bolded "500" error, and a a few others. If you see any issues as of now, please let me know.
I may not see them for 8 hours or so though as I am going to bed, it is 4AM here in AU.

Huge thanks to gnif who worked some incredible penguin magic into the wee hours of the morning to fix the server that HostGator was supposed to professionally install properly. Turned out they didn't do such a professional job.

AND something else potentially BIG has also been discovered.
I won't effect the server in the short or medium term, but it will effect HostGaotr reputation if they do not explain...   

Dave.

[Monkeh]

One of them, there was also a big bolded "500" error, and a a few others. If you see any issues as of now, please let me know.
I may not see them for 8 hours or so though as I am going to bed, it is 4AM here in AU.

Huge thanks to gnif who worked some incredible penguin magic into the wee hours of the morning to fix the server that HostGator was supposed to professionally install properly. Turned out they didn't do such a professional job.

AND something else potentially BIG has also been discovered.
I won't effect the server in the short or medium term, but it will effect HostGaotr reputation if they do not explain...   

Dave.

They're a large company and they use cPanel. Lack of quality is to be expected.

[gnif]

One of them, there was also a big bolded "500" error, and a a few others. If you see any issues as of now, please let me know.
I may not see them for 8 hours or so though as I am going to bed, it is 4AM here in AU.

Huge thanks to gnif who worked some incredible penguin magic into the wee hours of the morning to fix the server that HostGator was supposed to professionally install properly. Turned out they didn't do such a professional job.

AND something else potentially BIG has also been discovered.
I won't effect the server in the short or medium term, but it will effect HostGaotr reputation if they do not explain...   

Dave.

They're a large company and they use cPanel. Lack of quality is to be expected.

You are more then welcome Dave 

cPanel does not imply bad quality, but that is not so much the issue here. I will not announce what it is and leave it with Dave to follow up and give them a chance to resolve it before they get publicly exposed for it.

As for the migration they performed... well... this is why I hate it when people claim to be a 'Linux Professional', so often they end up pulling this crap.

They installed the new server with CentOS6, then performed a rsync of the entire root filesystem minus /boot to the new server, so the machine was running on a later kernel but the rest of the system was on CentOS5. This caused all sorts of stability problems with MySQL & PHP, which last night when the server started to get loaded down with more requests caused the service to completely crash. Also due to the method of sync, a heap of CentOS6 files were left laying around on the system, specifically the MySQL client libraries for 5.5, so even re-building PHP was a problem as the build kept finding the newer libs and trying to use them for MySQL 5.0. The final fix was to upgrade MySQL to 5.5, remove some HostGator custom cPanel addons that prevented PHP from building, and re-build PHP/Apache via easyapache. Sigh, what a mess.


They have confirmed that they did just do a straight copy between servers, very odd though that there was CentOS6 files hanging about from RPMs, perhaps someone made a mistake some time in the past, or cPanel did something strange. I apologise to HostGator for the quick jump to the incorrect conclusion.

The correct way to do this would have been to install cPanel on the new server, on the nice clean CentOS6 install and then use cPanel's backup and restore feature to copy the site to the new server... rsync JUST the web directory and if not using MySQL master/slace, perform a mysqldump and import on the new server to nothing gets lost during the migration, forward data to the new server and hand the keys to the owner. Due to the way they performed it, the new system is only running 32bit still which is also a reason for limited database performance because it can not use as much RAM as it should be able to use.

Anyway, amongst all those changes I also enabled HTTP compression of text based content, as the new server has ample CPU resources to do it, and prevented the server from advertising it's Apache version, modules and PHP from advertising it's version in the headers, mainly for security.

[manicdoc]

One of them, there was also a big bolded "500" error, and a a few others. If you see any issues as of now, please let me know.
I may not see them for 8 hours or so though as I am going to bed, it is 4AM here in AU.

Huge thanks to gnif who worked some incredible penguin magic into the wee hours of the morning to fix the server that HostGator was supposed to professionally install properly. Turned out they didn't do such a professional job.

AND something else potentially BIG has also been discovered.
I won't effect the server in the short or medium term, but it will effect HostGaotr reputation if they do not explain...   

Dave.

They're a large company and they use cPanel. Lack of quality is to be expected.

You are more then welcome Dave 

cPanel does not imply bad quality, but that is not so much the issue here. I will not announce what it is and leave it with Dave to follow up and give them a chance to resolve it before they get publicly exposed for it.

As for the migration they performed... well... this is why I hate it when people claim to be a 'Linux Professional', so often they end up pulling this crap.

They installed the new server with CentOS6, then performed a rsync of the entire root filesystem minus /boot to the new server, so the machine was running on a later kernel but the rest of the system was on CentOS5. This caused all sorts of stability problems with MySQL & PHP, which last night when the server started to get loaded down with more requests caused the service to completely crash. Also due to the method of sync, a heap of CentOS6 files were left laying around on the system, specifically the MySQL client libraries for 5.5, so even re-building PHP was a problem as the build kept finding the newer libs and trying to use them for MySQL 5.0. The final fix was to upgrade MySQL to 5.5, remove some HostGator custom cPanel addons that prevented PHP from building, and re-build PHP/Apache via easyapache. Sigh, what a mess.

The correct way to do this would have been to install cPanel on the new server, on the nice clean CentOS6 install and then use cPanel's backup and restore feature to copy the site to the new server... rsync JUST the web directory and if not using MySQL master/slace, perform a mysqldump and import on the new server to nothing gets lost during the migration, forward data to the new server and hand the keys to the owner. Due to the way they performed it, the new system is only running 32bit still which is also a reason for limited database performance because it can not use as much RAM as it should be able to use.

Anyway, amongst all those changes I also enabled HTTP compression of text based content, as the new server has ample CPU resources to do it, and prevented the server from advertising it's Apache version, modules and PHP from advertising it's version in the headers, mainly for security.

ugh, thats just plain nasty and so wrong.  Especially given cPanel is there to make this painless; even via WHM you can remote pull a site to a new box...    Obviously dealing with dedicated servers is not their 'strength'. Well done in sorting out the true dogs dinner of an upgrade.

I take it you put mod_security in? Also check the cache settings on .js files - got some strange behaviour with the jquery lib on the WP homepage, kept being pulled down on every load.

[Monkeh]

cPanel does not imply bad quality, but that is not so much the issue here.

cPanel is one gigantic security hole waiting to happen (and doing so fairly regularly). It's also one gigantic limitation on what you can do when it's the only avenue of access you have (which, if present, it usually is).

They installed the new server with CentOS6, then performed a rsync of the entire root filesystem minus /boot to the new server, so the machine was running on a later kernel but the rest of the system was on CentOS5. This caused all sorts of stability problems with MySQL & PHP, which last night when the server started to get loaded down with more requests caused the service to completely crash. Also due to the method of sync, a heap of CentOS6 files were left laying around on the system, specifically the MySQL client libraries for 5.5, so even re-building PHP was a problem as the build kept finding the newer libs and trying to use them for MySQL 5.0. The final fix was to upgrade MySQL to 5.5, remove some HostGator custom cPanel addons that prevented PHP from building, and re-build PHP/Apache via easyapache. Sigh, what a mess.

I don't even have words for that. This is why I don't deal with such companies..

[gnif]

ugh, thats just plain nasty and so wrong.  Especially given cPanel is there to make this painless; even via WHM you can remote pull a site to a new box...    Obviously dealing with dedicated servers is not their 'strength'. Well done in sorting out the true dogs dinner of an upgrade.

I take it you put mod_security in? Also check the cache settings on .js files - got some strange behaviour with the jquery lib on the WP homepage, kept being pulled down on every load.

In the rush to get the site back on-line and discovering broken custom config I just disabled mod_sec and their custom crap for the moment, it is top of my list to fix today . I can not replicate the cache issue, chrome is telling me that all the static files are coming from cache, as expected.

cPanel is one gigantic security hole waiting to happen (and doing so fairly regularly). It's also one gigantic limitation on what you can do when it's the only avenue of access you have (which, if present, it usually is).

Agreed, I do not like cPanel either, standard it is very insecure, but fixable if you know how. The issue is that it is sold as a 'you can be your own admin' type package but they do not educate people that there is way more to administration of a Linux server then just what cPanel provides. cPanel does not even come with a firewall. To be honest I could gripe on about cPanel for weeks. Personally I prefer a Debian based server, no management interface at all, Nginx+php-fpm over Apache, and my monitoring solution to keep an eye on the server and keep it up to date.

[manicdoc]

ugh, thats just plain nasty and so wrong.  Especially given cPanel is there to make this painless; even via WHM you can remote pull a site to a new box...    Obviously dealing with dedicated servers is not their 'strength'. Well done in sorting out the true dogs dinner of an upgrade.

I take it you put mod_security in? Also check the cache settings on .js files - got some strange behaviour with the jquery lib on the WP homepage, kept being pulled down on every load.

In the rush to get the site back on-line and discovering broken custom config I just disabled mod_sec and their custom crap for the moment, it is top of my list to fix today . I can not replicate the cache issue, chrome is telling me that all the static files are coming from cache, as expected.

Okay, could have been some temporary weirdness in amongst all the other weirdness... I'll have a good crawl around when I'm back home this evening and see if there is any more weirdness to be found..

[EEVblog]

cPanel does not imply bad quality, but that is not so much the issue here. I will not announce what it is and leave it with Dave to follow up and give them a chance to resolve it before they get publicly exposed for it.

Ok, the issue is that gnif discovered that my "dedicated" server looked to be in fact a virtual server. By virtue of the drive directory name, the driver, and the drive bios all being virtual.

I asked HostGator about this, and their response was:

Thank you for contacting HostGator. Our new dedicated servers are
virtualized so as to provide a better management back end however this
is not shared between anyone else and you have full access to all
resources on the server. The Rescue & Hardware Monitoring KVM layer
allows us to monitor the server for any potential hardware failures
and proactively resolve these issues before they result in any
downtime and/or possible data loss. If you have any other questions
please let us know. We are more than happy to assist you. Thanks!

Which actually sounds reasonable to my untrained ears.
They even said in the original email about the new server, that I posted at the start of this topic:

-- All servers also feature an optional Rescue & Hardware Monitoring KVM layer (which is included by default) allowing HostGator to monitor the server for any potential hardware failures and proactively resolve these issues before they result in any downtime and/or possible data loss."

I'll leave it up to the penguin initiated to determine if HostGator are actually trying to pull a swifty here with their new dedicated servers, or if they have genuine reasons for adding that virtual layer.
I don't mind running the virtual layer, if it means HostGator can monitor and detect problems etc. But I'll be really pissed if one single byte of another user touches my dedicated hard drive or processor! 

Dave.