IMPORTANT: Server Changeover

[gnif]

If they are being honest here, what they have done is install a single virtual machine on the hardware, so you are not sharing the resources with other users, but how can you be sure of that? and there is a performance loss incurred by this due to the VM layer.

It is more likely that they are doing this so that if hardware fails they can just transfer the virtual machine to a new server and not have to mess about with reconfiguration.

The disadvantage is, if their hardware monitoring is not up to scratch (which I doubt it is) and they do not detect an error, you do not have the ability to detect errors yourself either or have a third party monitor the server's hardware as you are inside the virtual box without direct hardware access.

I would demand access to the physical box for two reasons:

1) It is a security threat, you need to be sure that what they have done is not going to compromise your host. One of the main reasons of having a dedicated server is that you have total control so you can secure the machine and configure it according you your requirements.

2) To verify that they are indeed doing what they say they are, it all sounds a bit dodgy to me.

Edit: This is not a KVM by the way... a KVM is a physical feature of the server that provides a virtual keyboard, video and mouse interface that allows you to use the server as if you are physically sitting on it. That means you can reboot and configure the bios over the internet if you want, ore recover from a critical failure, etc.

Edit 2: Also if there is a hardware failure, say one of your disks, they can decide not to replace it for as long as they want (ie, lazy, cost saving etc) and you will never know as the RAID configuration will hide the problem from you. This means that you will loose your redundancy for who knows how long.

Edit 3: Security is a problem here also because you dont know if they are keeping that virtual layer up to date and patched with the latest updates, your VM may be 100% secure, but could be back-doored via the (and I use this term loosely) "Virtual KVM".

[EEVblog]

Edit 2: Also if there is a hardware failure, say one of your disks, they can decide not to replace it for as long as they want (ie, lazy, cost saving etc) and you will never know as the RAID configuration will hide the problem from you. This means that you will loose your redundancy for who knows how long.

Wouldn't that same thing happen with or without the KVM layer?
If a RAID 1 drive fails in a truly dedicated box, the server would keep running without batting an eyelid. I don't even know if I get alerted when such a thing happens?

Dave.

[gnif]

Edit 2: Also if there is a hardware failure, say one of your disks, they can decide not to replace it for as long as they want (ie, lazy, cost saving etc) and you will never know as the RAID configuration will hide the problem from you. This means that you will loose your redundancy for who knows how long.

Wouldn't that same thing happen with or without the KVM layer?
If a drive fails in a truly dedicated box, the server keep running without batting an eyelid. I don't even know if I get alerted when such a thing happens?

Dave.

With the KVM in there, you would never know if they had been lazy and never replaced the drive. With a dedicated server it can be configured to email you alerts on failures such as this, but because you do not have physical hardware access you would never know you were running a degraded array, and you would not be able to chase them to fix it.

[drewtronics]

I have to disagree almost entirely with gnif. If you want to be super paranoid, run your own gear, in your own place with your own money.

However, running in a VM container has lots of benefits. VMware has a feature called vmotion which is automatic migration technology for failing over a vm from one machine to another automatically when the host pc fails. Its magic, and would prevent downtime at a big hosting farm. I'm sure there are comparable solutions from other vendors/packages.

It can be much easier to check on the health of machines, see if something is going haywire, and respond to customer needs with everything virtualized. I monitor and run a couple of racks worth of equipment at work, among other duties, and virtualization makes it nearly painless. I even use machines on the virtualization pool for engineering and development duties. It all works great. In this day and age, there not a terrible amount of overhead lost, except for some RAM.  The security risk is minimal. It is much, much, more likely someone will find an exploit in this forum software than the hypervisor running the VMs.

I think its a good move hostgator's part.

[gnif]

I have to disagree almost entirely with gnif. If you want to be super paranoid, run your own gear, in your own place with your own money.

However, running in a VM container has lots of benefits. VMware has a feature called vmotion which is automatic migration technology for failing over a vm from one machine to another automatically when the host pc fails. Its magic, and would prevent downtime at a big hosting farm. I'm sure there are comparable solutions from other vendors/packages.

They are not using VMware nor are they auto moving failed hosts to new machines.

It can be much easier to check on the health of machines, see if something is going haywire, and respond to customer needs with everything virtualized. I monitor and run a couple of racks worth of equipment at work, among other duties, and virtualization makes it nearly painless. I even use machines on the virtualization pool for engineering and development duties. It all works great. In this day and age, there not a terrible amount of overhead lost, except for some RAM.  The security risk is minimal. It is much, much, more likely someone will find an exploit in this forum software than the hypervisor running the VMs.

How so? what is the difference between them running their monitoring software on the same host as the client's software? I did not say there was a risk in the VM tech being compromised, I said there was a risk of the DC not keeping the software on the physical host up to date, such as SSH, the Kernel, etc.

Edit:

And how do you know how much they are doing to monitor the server? how heavy is their resource consumption? Are they scheduling checks of things during peak times the client may need the servers resources for other tasks? If you want a VPS you get a VPS, this is a glorified VPS, not a dedicated server.

And do not get me wrong, I like VM environments, I often deploy XEN, the issue at hand here is they were not honest about it, and Dave is paying for a dedicated server, not a VM.

[drewtronics]

How so? what is the difference between them running their monitoring software on the same host as the client's software? I did not say there was a risk in the VM tech being compromised, I said there was a risk of the DC not keeping the software on the physical host up to date, such as SSH, the Kernel, etc.

I've only used VMware, so I can't compare much else, but in that case, the Virtualization package and the monitoring software are the same thing. You don't really separate them. If they're trying to kludge together one of the free software virtualization solutions at a commercial hosting service, they deserve the problems they are almost certainly going to have. Citrix, VMware, w/e Microsoft is calling it these days, they're all pretty manageable.

For instance, just yesterday I was doing something else on a VM host box and noticed one of the cores was pegged at 100% constantly from one of the graphs I saw navigating through the software. I talk to the owner of the VM, and he hasn't used it in months.  I shut it down and archived the VM to a file server. It could have crashed, been compromised, who knows. His responsibility that he can figure out later - but the management was crazy simple.

Again, it doesn't seem like you've done this. You don't run SSH or other services on the hypervisor (aka thing running the VMs). Its a super minimal install with the virtualization software -- that's it.   Usually you also put the management interface on a different VLAN that's under well guarded security. Standard practice.

[EEVblog]

A reply from HostGator, including replying to gnifs concerns:

You have a virtual machine running on a dedicated server. There is one VM on your hardware and it's configured to use the entire server.
These commands were run on the hypervisor host:
>Server model: Supermicro X9SCD+-F-BH002
[ulhglive-root@d9435 ~]# dmidecode --type baseboard | grep -A 1 Super
Manufacturer: Supermicro
Product Name: X9SCD+-F-BH002

>Processor: Intel(R) Xeon(R) CPU E3-1265L V2 @ 2.50GHz
[ulhglive-root@d9435 ~]# grep model\ name /proc/cpuinfo | head -n 1
model name   : Intel(R) Xeon(R) CPU E3-1265L V2 @ 2.50GHz

>Memory: Kingston DDR3 1333MHz
[ulhglive-root@d9435 ~]# dmidecode -t 17 | grep -B 1 King
Speed: 1333 MHz
Manufacturer: Kingston

>Hard drive: Western Digital RE4 Serial ATA (WD1003FBYX-01Y7B1) x2
[ulhglive-root@d9435 ~]# smartctl -a /dev/sda | grep Model
Model Family: Western Digital RE4 Serial ATA
Device Model: WDC WD1003FBYX-01Y7B1
[ulhglive-root@d9435 ~]# smartctl -a /dev/sdb | grep Model
Model Family: Western Digital RE4 Serial ATA
Device Model: WDC WD1003FBYX-01Y7B1

>Is anyone else sharing my hard drive, my processor, or anything to do
with my box?
>Is all that hardware truly dedicated to me?
Absolutely, you're on top of a hypervisor but no one else is using your resources, full stop.
virsh is a command line interface to libvirt, and its output here shows that there are no other instances running on the server:
[ulhglive-root@d9435 ~]# virsh list --all
Id Name State
----------------------------------------------------
1 instance-0000330b running

[ulhglive-root@d9435 ~]#

>Does this mean I can chose to opt-out of having this KVM layer?
Unfortunately our use of the word "optional" was a mistake. Our network infrastructure (among other things) does not allow outside access to the bare metal, we do not offer a non-kvm solution at this time.

I also wanted to address some of the points mentioned by "gnif" on your forum.

>It is more likely that they are doing this so that if hardware fails they can just transfer the virtual machine to a new server and not have to mess about with reconfiguration.
This is actually completely true. Previously our OS reload/hardware failure procedure required hours of downtime while we made backups, provisioned new drives/hardware, restored backups, fixed issues, etc. Not only is this not efficient for us, it means that your sites, email, etc are down for hours. With the VM layer we can simply move your VM to a new node and you'll be back online in under an hour (depending of course on the amount of data you have).

>It is a security threat, you need to be sure that what they have done is not going to compromise your host. One of the main reasons of having a dedicated server is that you have total control so you can secure the machine and configure it according you your requirements.
This is not a concern as the bare metal is completely isolated from the outside world.

>Edit: This is not a KVM by the way... a KVM is a physical feature of the server that provides a virtual keyboard, video and mouse interface that allows you to use the server as if you are physically sitting on it.
KVM in this context stands for Kernel Virtual Machine, not Keyboard Video Mouse.

>Edit 2: Also if there is a hardware failure, say one of your disks, they can decide not to replace it for as long as they want (ie, lazy, cost saving etc) and you will never know as the RAID configuration will hide the problem from you. This means that you will loose your redundancy for who knows how long.
This is simply not true. A 2 disk RAID array with a failed disk will see severe performance issues and would be noticed, plus if we were going to do this why would we make it a RAID in the first place?

Hopefully this addresses these concerns, if not I'd be happy to answer any further questions you may have. We appreciate your understanding and patience.

--
Cheers,

Sean C.
Linux Systems Administration Shift Lead

Hats off to Hostgator for clearing it up, and tying in my Twitter mumble to my ticket and reading and responding to the forum 

Dave.

[gnif]

How so? what is the difference between them running their monitoring software on the same host as the client's software? I did not say there was a risk in the VM tech being compromised, I said there was a risk of the DC not keeping the software on the physical host up to date, such as SSH, the Kernel, etc.

I've only used VMware, so I can't compare much else, but in that case, the Virtualization package and the monitoring software are the same thing. You don't really separate them. If they're trying to kludge together one of the free software virtualization solutions at a commercial hosting service, they deserve the problems they are almost certainly going to have. Citrix, VMware, w/e Microsoft is calling it these days, they're all pretty manageable.

For instance, just yesterday I was doing something else on a VM host box and noticed one of the cores was pegged at 100% constantly from one of the graphs I saw navigating through the software. I talk to the owner of the VM, and he hasn't used it in months.  I shut it down and archived the VM to a file server. It could have crashed, been compromised, who knows. His responsibility that he can figure out later - but the management was crazy simple.

Again, it doesn't seem like you've done this. You don't run SSH or other services on the hypervisor (aka thing running the VMs). Its a super minimal install with the virtualization software -- that's it.   Usually you also put the management interface on a different VLAN that's under well guarded security. Standard practice.

I have done this many times before, I think you should go back and re-read my initial post where I clearly stated that they do this to make it easier to move a server between hosts, which is essentially what you did in the example you stated. My concern is that they did not let Dave know, he should have been fully informed as to what they have done with his server.

[gnif]

A reply from HostGator, including replying to gnifs concerns:

A 2 disk RAID array with a failed disk will see severe performance issues and would be noticed, plus if we were going to do this why would we make it a RAID in the first place?

A clone raid array does not always see huge performance benefits, often the controller compares reads from both disks against each other negating the ability to read from both disks at once. It would only be noticed if they were using RAID5 of RAID1+0. The reason why I raised this was because often I have dealt with DCs that refuse to change a failed disk even after multiple angry email's to the DC. Many of them expect you to notice the issue and then you to chase them on it.

Edit: Just looked at the specs of the board... it does not have hardware raid, so unless they have installed a controller into it, it will be using software raid, which I do like and there is nothing wrong with it, but I can say for certain that RAID1 on a mdadm soft raid device does not see more then a few % performance gain. Assuming that this is what they have done.

[gnif]

And to eat my own words...

They installed the new server with CentOS6, then performed a rsync of the entire root filesystem minus /boot to the new server, so the machine was running on a later kernel but the rest of the system was on CentOS5. This caused all sorts of stability problems with MySQL & PHP, which last night when the server started to get loaded down with more requests caused the service to completely crash. Also due to the method of sync, a heap of CentOS6 files were left laying around on the system, specifically the MySQL client libraries for 5.5, so even re-building PHP was a problem as the build kept finding the newer libs and trying to use them for MySQL 5.0. The final fix was to upgrade MySQL to 5.5, remove some HostGator custom cPanel addons that prevented PHP from building, and re-build PHP/Apache via easyapache. Sigh, what a mess.

They have confirmed that they did just do a straight copy between servers, very odd though that there was CentOS6 files hanging about from RPMs, perhaps someone made a mistake some time in the past, or cPanel did something strange. I apologise to HostGator for the quick jump to the incorrect conclusion.

[EEVblog]

Ok, I very happy with HostGators response and am going to stick with the VM.
The server is currently running a 32bit version of CentOS though, so I have asked them to upgrade this to 64bit.
So we could have another little hiccup in next day or two in the server when that is swapped over, but it should be good for speed and the long term.

Dave.

[manicdoc]

I have to disagree almost entirely with gnif. If you want to be super paranoid, run your own gear, in your own place with your own money.

However, running in a VM container has lots of benefits. VMware has a feature called vmotion which is automatic migration technology for failing over a vm from one machine to another automatically when the host pc fails. Its magic, and would prevent downtime at a big hosting farm. I'm sure there are comparable solutions from other vendors/packages.

They are not using VMware nor are they auto moving failed hosts to new machines.

It can be much easier to check on the health of machines, see if something is going haywire, and respond to customer needs with everything virtualized. I monitor and run a couple of racks worth of equipment at work, among other duties, and virtualization makes it nearly painless. I even use machines on the virtualization pool for engineering and development duties. It all works great. In this day and age, there not a terrible amount of overhead lost, except for some RAM.  The security risk is minimal. It is much, much, more likely someone will find an exploit in this forum software than the hypervisor running the VMs.

How so? what is the difference between them running their monitoring software on the same host as the client's software? I did not say there was a risk in the VM tech being compromised, I said there was a risk of the DC not keeping the software on the physical host up to date, such as SSH, the Kernel, etc.

Edit:

And how do you know how much they are doing to monitor the server? how heavy is their resource consumption? Are they scheduling checks of things during peak times the client may need the servers resources for other tasks? If you want a VPS you get a VPS, this is a glorified VPS, not a dedicated server.

And do not get me wrong, I like VM environments, I often deploy XEN, the issue at hand here is they were not honest about it, and Dave is paying for a dedicated server, not a VM.

To me If (big if) they are genuine about what they are doing, then the overhead should be quite minimal, as modern chipsets have VM specific features to keep the speed up. BUT there is no way of knowing if the box is indeed being shared. Also given their rather 'hacked up' way of doing a migration - I wouldn't be too sure the image is actually worth the effort carting around. KVM is another issue.

[amspire]

cPanel is one gigantic security hole waiting to happen (and doing so fairly regularly). It's also one gigantic limitation on what you can do when it's the only avenue of access you have (which, if present, it usually is).

I have never had a problem finding hosting companies offering SSH access as well as CPanel. Also everybody offers FTP access. I have been with probably 6 different hosting companies and I have always had SSH terminal access.

The cheapest host was $2 a month and it still had SSH access. I currently have a $5 a month single domain site with SSH and a $20 a month reseller package which is great as you are the one who controls the SSH to all the domains you host. You also have full control of the DNS for all the domains, and you can set up a temporary new web and FTP site in seconds (you do not need to have a registered domain to set up a temporary site - I have done it without any domain name plenty of times).

I have never found the need to spend over $5/month for a single domain yet, but then again, my sites have never been high traffic. I have always had great technical support from the hosting companies I have chosen.

If you only have CPanel access, you are probably with the wrong hosting company.

Richard.

[gnif]

cPanel is one gigantic security hole waiting to happen (and doing so fairly regularly). It's also one gigantic limitation on what you can do when it's the only avenue of access you have (which, if present, it usually is).

I have never had a problem finding hosting companies offering SSH access as well as CPanel. Also everybody offers FTP access. I have been with probably 6 different hosting companies and I have always had SSH terminal access.

The cheapest host was $2 a month and it still had SSH access. I currently have a $5 a month single domain site with SSH and a $20 a month reseller package which is great as you are the one who controls the SSH to all the domains you host. You also have full control of the DNS for all the domains, and you can set up a temporary new web and FTP site in seconds (you do not need to have a registered domain to set up a temporary site - I have done it without any domain name plenty of times).

I have never found the need to spend over $5/month for a single domain yet, but then again, my sites have never been high traffic. I have always had great technical support from the hosting companies I have chosen.

If you only have CPanel access, you are probably with the wrong hosting company.

Richard.

Yeah, you are referring to purchasing an account on the box, Dave has full control over his server as the site is pretty resource intensive. The SSH access he has to the server is full root level access, not per account.

[EEVblog]

Yeah, you are referring to purchasing an account on the box, Dave has full control over his server as the site is pretty resource intensive. The SSH access he has to the server is full root level access, not per account.

Yep, I have the ability to create cpanel accounts and sell space if I wanted, just like any shared web host.

Dave.

[BravoV]

Connection Problems
Sorry, SMF was unable to connect to the database. This may be caused by the server being busy. Please try again later.

Just fyi, this happened again, post this straight after I managed to get into this thread.

It happened for almost for 7 minutes, and tried 5 times browser refreshes still no access to the forum.

[gnif]

This time we know why, HostGator was very quick in getting the transfer under way, the server load is real high at the moment, this is unavoidable.

[ve7xen]

Edit: This is not a KVM by the way... a KVM is a physical feature of the server that provides a virtual keyboard, video and mouse interface that allows you to use the server as if you are physically sitting on it. That means you can reboot and configure the bios over the internet if you want, ore recover from a critical failure, etc.

KVM is Kernel-based Virtual Machine, the most modern of Linux's virtualization layers. Most distros have moved from Xen to KVM, and aside from large hosts that have existing infrastructure, most new deployments use it, including OpenStack etc. You're confused.

Edit 3: Security is a problem here also because you dont know if they are keeping that virtual layer up to date and patched with the latest updates, your VM may be 100% secure, but could be back-doored via the (and I use this term loosely) "Virtual KVM".

One would hope that that bare metal is on a separate management network not accessible from the Internet. Of course, what one hopes, and how things actually are are quite likely different things.

As a managed service I don't see where you have any place to complain about how they set up their infrastructure. If you're paying for management, that's their responsibility and you shouldn't need to know or care how it's set up, you're trusting them to do that for you. Not many are any good at it, but that is what you're paying for.

Honestly I don't see what the point of Dave paying for a "managed" server is at this point. gnif is doing most of the work, and all they're charging you for seems to be giving you less flexibility and having a third party with easy root access to your box.

I'd say get an unmanaged dedicated server, ditch fscking cPanel (what the heck is the point of that monstrosity on a site hosting just a couple vhosts?? it's far more of a liability than an asset) and pay one of your competent forum members a token sum to maintain it for you. Should save you a ton of money and get you a better setup to boot.

[manicdoc]

Ok, I very happy with HostGators response and am going to stick with the VM.
The server is currently running a 32bit version of CentOS though, so I have asked them to upgrade this to 64bit.
So we could have another little hiccup in next day or two in the server when that is swapped over, but it should be good for speed and the long term.

Dave.

Pretty please (with ice cream on top) make sure memcached is running and used on the new OS, your forum s/w should make good use of it and it will offload a lot of the 'same as' type queries from the database. Your running dedicated, so no security problem to worry about.  Also look at eAccelerator, which should not require a Php recompile to use (its a module).

BTW If you happen to find yourself by the Bull and Bush between 7 and 9am tomorrow I can say hi.

[gnif]

Ok, I very happy with HostGators response and am going to stick with the VM.
The server is currently running a 32bit version of CentOS though, so I have asked them to upgrade this to 64bit.
So we could have another little hiccup in next day or two in the server when that is swapped over, but it should be good for speed and the long term.

Dave.

Pretty please (with ice cream on top) make sure memcached is running and used on the new OS, your forum s/w should make good use of it and it will offload a lot of the 'same as' type queries from the database. Your running dedicated, so no security problem to worry about.  Also look at eAccelerator, which should not require a Php recompile to use (its a module).

BTW If you happen to find yourself by the Bull and Bush between 7 and 9am tomorrow I can say hi.

I will be doing this amongst other things to improve performance, but if we make too many changes at once then when there are issues from the migration it makes it harder to track them down. We will get it onto the new install, let it run for a while and then look at performance improvements.

As for APC/XCache/eAccelerator... they can all be compiled as modules, but cPanel does not support this, the next time an update to PHP is performed it will break the server as it wont compile the modules for the new version during the upgrade... thankfully there are build hooks that this can be done in however, just pointing out that it is not quite as simple as you would imagine. We need to keep cPanel operating without admin intervention so that Dave can manage things on the server in a pinch, this kind of custom job needs to be done carefully.

PS: cPanel supports eAccelerator compiled into PHP, not as a module, but I would prefer to get APC on here as it is what the PHP devs are officially supporting and working on.

[manicdoc]

Ok, I very happy with HostGators response and am going to stick with the VM.
The server is currently running a 32bit version of CentOS though, so I have asked them to upgrade this to 64bit.
So we could have another little hiccup in next day or two in the server when that is swapped over, but it should be good for speed and the long term.

Dave.

Pretty please (with ice cream on top) make sure memcached is running and used on the new OS, your forum s/w should make good use of it and it will offload a lot of the 'same as' type queries from the database. Your running dedicated, so no security problem to worry about.  Also look at eAccelerator, which should not require a Php recompile to use (its a module).

BTW If you happen to find yourself by the Bull and Bush between 7 and 9am tomorrow I can say hi.

I will be doing this amongst other things to improve performance, but if we make too many changes at once then when there are issues from the migration it makes it harder to track them down. We will get it onto the new install, let it run for a while and then look at performance improvements.

As for APC/XCache/eAccelerator... they can all be compiled as modules, but cPanel does not support this, the next time an update to PHP is performed it will break the server as it wont compile the modules for the new version during the upgrade... thankfully there are build hooks that this can be done in however, just pointing out that it is not quite as simple as you would imagine. We need to keep cPanel operating without admin intervention so that Dave can manage things on the server in a pinch, this kind of custom job needs to be done carefully.

PS: cPanel supports eAccelerator compiled into PHP, not as a module, but I would prefer to get APC on here as it is what the PHP devs are officially supporting and working on.

No worries, sounds like cPanel needs a tad more flexibility around this perhaps. BTW what you think of the whole APS package management? Got a client looking to use it for custom builds, seems solid.

[gnif]

No worries, sounds like cPanel needs a tad more flexibility around this perhaps. BTW what you think of the whole APS package management? Got a client looking to use it for custom builds, seems solid.

APS? Do you mean APT?

[BravoV]

Not sure .. just now I was trying to hammer the forum by opening multiple tabs > 10 concurrently at my browser, it does "feel" very responsive, have you applied those fine-tunning ?

[gnif]

Heh, no, you are just seeing the effects of the earlier changes, no more tuning will be done until we are on the new server install.

[manicdoc]

No worries, sounds like cPanel needs a tad more flexibility around this perhaps. BTW what you think of the whole APS package management? Got a client looking to use it for custom builds, seems solid.

APS? Do you mean APT?

http://apsstandard.org/applications

packaging standard - claims to be management suite independent - although plesk are pushing it.