POLL: Should I change servers?

[gnif]

Having said that, though, it would be very unwise to not have someone else who could step in should gnif be indisposed for any reason - or at least a plan of what will need to be done.  Or is this something that has already been addressed?

That has not been addressed.
As long as I have login's then I presume it's likely not hard to find another person at short notice, I've always had many offers of help.

My 0.02$: set this up in advance.

Never a bad idea, I do have people on retainer though that can handle most things.

[gnif]

To be more judgemental I'd want to know how and why the standard HG setup was changed to cause the update incompatibilities HG report. If you NEED these changes then you should change. If you can return to a HG supplied standard setup then you may as well stay if you are otherwise happy.

HG changed their setup and never updated the server's configuration, it was their breakage.

I'd need to know how it came to be moved from a static IP to DHCP.

It's the other way around, HG initially deployed the servers on their network with DHCP, which IMHO is a joke. Seems they found out the hard way this was a bad idea and switched to static IP assignment, just like every other data center on the planet uses. HG failed to mention this to Dave, and they failed to update the server's IP configuration with the static IP addressing details.

The questions I have are why you didn't start the server up on the backup system? Not saying you should have but could you have?  If your backup system is with another hosting vendor is it the same as HG supply? And why the outage was so long?

Dave doesn't have a backup system, I provide server backups, data storage, etc. The outage length was two fold, HG didn't detect that the host had stopped responding, being a "managed service" they should have had an alarm for this. Secondly I can not contact HG on Dave's behalf at this point, we had to wait for Dave to raise the problem with them.

[BravoV]

Specs:
Dual Xeon L5630
  96GB RAM
  1 x 2TB HDD
  1 x 120GB SSD
  1 GBps Port
  30TB Monthly
  5 IPs

So on new site, there is no redundancy on the storage as I see only one HD, and relies on Gnif's remote backup ?

[gnif]

Specs:
Dual Xeon L5630
  96GB RAM
  1 x 2TB HDD
  1 x 120GB SSD
  1 GBps Port
  30TB Monthly
  5 IPs

So on new site, there is no redundancy on the storage as I see only one HD, and relies on Gnif's remote backup ?

This is only one potential configuration, the host would be configured with a proper redundant RAID array. This package deal was a quick quote I copied and pasted in as an example of what could be provided with an alternative supplier of whom I have many clients and trust implicitly.

[Halcyon]

Dave doesn't have a backup system, I provide server backups, data storage, etc.

How large is the dataset? I'm happy to relinquish some storage on my servers at home. Both running FreeNAS, one is a high speed array, the other is a cold storage machine that replicates periodically with the main server. Dave has been here and has seen my "work in progress" server room.

[Avacee]

I voted Yes.

Always support a volunteer (within reason). If gnif will be happier I assume part of this is him calculating lower time requirements long-term and being able to solve issues with less hassle. Always a good way of keeping volunteers volunteering

The current configuration sounds highly modified and if gnif 1.0 was hit by the proverbial bus how much up the creek without a paddle are you compared to a more standard configuration. How quickly can gnif 2.0 understand all those little tweaks that have accumulated over the years?
Whilst you personally may not be as familiar with the tools involved with a more standard configuration any gnif 2.0 will be up to speed much faster. You can learn the essentials at your own pace.

The current hosting company changing things as has been described is a red flag that says "Start your engines on a move"

Moving has a high enough level of probability of longer term peace of mind after the higher up front costs and time spent.

[Halcyon]

The current configuration sounds highly modified and if gnif 1.0 was hit by the proverbial bus how much up the creek without a paddle are you compared to a more standard configuration. How quickly can gnif 2.0 understand all those little tweaks that have accumulated over the years?

Let's hope that never happens! Although it would be kind of neat to have a bit red flip switch in the EEVblog lab labelled "Geoff / No Geoff" for when he is on holidays and something breaks, the "No Geoff" position redirects DNS requests to an alternate server with a replica of the information ;-)

[gnif]

Dave doesn't have a backup system, I provide server backups, data storage, etc.

How large is the dataset? I'm happy to relinquish some storage on my servers at home. Both running FreeNAS, one is a high speed array, the other is a cold storage machine that replicates periodically with the main server. Dave has been here and has seen my "work in progress" server room.

Thanks but the data backup solution in place is already enterprise grade. By backup system I was referring to a hot spare HTTP server.

[Ampera]

The current configuration sounds highly modified and if gnif 1.0 was hit by the proverbial bus how much up the creek without a paddle are you compared to a more standard configuration. How quickly can gnif 2.0 understand all those little tweaks that have accumulated over the years?

Let's hope that never happens! Although it would be kind of neat to have a bit red flip switch in the EEVblog lab labelled "Geoff / No Geoff" for when he is on holidays and something breaks, the "No Geoff" position redirects DNS requests to an alternate server with a replica of the information ;-)

*Break glass in emergency lack of Geoff*

[gnif]

At current it would be quite hard to bring someone up to speed with the configuration of the existing system, it is by no means standard due to the need to work around cPanel. If the server was built without cPanel it would be pretty simple for pretty much any competent server administrator to take over should the worst occur. When I setup systems I follow the best practices of the distribution to ensure ease of management and upgrades into the future, about the only custom stuff would be the monitoring and backup configuration as that is infrastructure dependent.

[tautech]

Dave's pushing to lift his income to the next level and along with that comes cost, there's only one of any of us and only so many hours in the day......well, if we want to keep our wife and family.

There comes a time when it's plain to see we can't 'do it all' like we used to and it's concerning to have to hand over responsibilities to others so that we can devote our time to what really matters....the future, with earned gains from the past compounding AND time to enjoy them.

Get good people and providers around you and move forward, faster, better and with less stress.

We'll put up with anything that's not quite right for a bit.............

[ovnr]

I'd move; HostGator have shown themselves to be grossly incompetent, to be honest. I'd also drop cPanel because it's terrible.

As for Dave's concerns about being able to sort out minor things himself: You can always install phpMyAdmin on the new box too so he can fiddle with the DB, and I expect most other common tasks are a shell script away.


Incidentally, how much stuff does PHP 7 break for you? I noticed that you're running a pretty old MediaWiki install (1.17 vs 1.30); the main website seems to just be a WordPress blog with some plugins, and unless you have a pile of custom plugins WP tends to be pretty OK about updates. Anyway, my point being that I'm sure you can find someone to fix whatever PHP 7 breaks; I could probably do it too, but I'm a bit swamped these days.

[EEVblog]

3) The server is not a true dedicated server, it is a VM and if HG are to be trusted, running on a host with no other VMs.

I have no doubt I have my own box.
I believe I'm now on a "legacy" server box, and perhaps this had something to do with the change that caused the problem.
Yes it's a VM they are using.

As for Dave's management of the things he needs, there are three options.
1) Move to a dedicated server with a completely custom stack, and rely on an admin (myself) to manage it for him.
2) Bring up two servers, a basic low spec server for cPanel/Email/DNS, etc... the other for just the HTTP hosting.
3) Same as 2, but run two VMs on the single server saving the additional cost.

#2 sounds good, but considering that the load on the server is currently quite minimal, we no doubt have plenty of grunt to run two VM's on the one server.
I'd want the ability to set up my own email and FTP stuff etc.

Ideally we'd have completely mirrored backup servers in different locations and providers that can automatically (or manually) be swapped out if there is an issue. Although I'm not sure how that actually works in practice. But really, we aren't running a bank here, it's not the end of the world if the server goes down due to unforeseen circumstances for a few hours or a day every year. Every host eventually gets hit with some form of attack or something else that causes issues.
How many forums and other websites this size do that?

Would we drop Cloudflare or keep it?

With my new Synology RAID, if possible I'd like to set up some form of additional daily backup on my desktop here for piece of mind. I think we currently use about 393GB of disk space, but that includes the OS and all the crap. Even just the databases and website files backed up daily on my RAID would be nice.

[EEVblog]

How large is the dataset? I'm happy to relinquish some storage on my servers at home. Both running FreeNAS, one is a high speed array, the other is a cold storage machine that replicates periodically with the main server. Dave has been here and has seen my "work in progress" server room.

Even Mrs EEVblog has seen the rats nest, don't make me post a photo

[EEVblog]

The questions I have are why you didn't start the server up on the backup system? Not saying you should have but could you have?  If your backup system is with another hosting vendor is it the same as HG supply? And why the outage was so long?

We do not have a backup server we can just switch to instantly. It would take some time to re-implement a new server from a backup.
The outage was so long because I went to sleep thinking it was just another HostGator network glitch or some other transient network thing that happens a few times a year. Woke up to find it still down.
They apparently don't have anything that monitors if my actual website is contactable. As far as they were concerned, the server was up and running just fine.

Maybe you should write down what your actual requirements are, then sort the list. You sound pretty happy with HG, so what are the chances you'll be happier elsewhere?

Hostgater have been nice to me, even so far as escalating a major issue to the CEO for resolution. (a viewer has a high level position there)
And their service is pretty good when you actually ask for it, but yeah, they seem to screw up a lot 
But then, I've had issues with every web host I've ever used.

[Halcyon]

How large is the dataset? I'm happy to relinquish some storage on my servers at home. Both running FreeNAS, one is a high speed array, the other is a cold storage machine that replicates periodically with the main server. Dave has been here and has seen my "work in progress" server room.

Even Mrs EEVblog has seen the rats nest, don't make me post a photo

For your eyes only Dave. You're welcome to come and take a fresh photo once the cabling is finished, THEN you can share. :-) Also, hopefully I'll have some more tear-downy goodness for you by then.

[madires]

And my 2ct The web based server management panels are fine for standard stuff but they can't be used for special configurations. You should be aware that they also increase the attack surface. It's a tradeoff. A hoster who changes the IP address of a managed server without informing the customer of the maintenance a few weeks in advance and not checking if the server runs fine after the change is completely hopeless. Usually the change of the IP address implicates updates in the DNS. Those have to be done in parallel with some TTL tweaking to minimize the impact. Apparently HG caused the outage by the unprofessional handling of the IP address change. I'd tell them this and ask for a refund. I'd also make clear that I will move to another hoster if something like this happens again or they won't give a refund. And don't forget too add that the server runs a forum with a huge community and you publicly share your experience with HG.

[Mr. Scram]

It's various incidents that have occurred in the past. Europe tends to have issues with mentions of previously (or to this day) violent groups like national socialists and communist parties. Not saying that I want to make it a habit of talking about those things, but this is the internet, and a diverse forum with varying topics, being restricted to specific topics just gives my American blood at least a bit of a boil.

NZ has their strange cyber bullying laws which were the talk of the world a few years ago.

I'm not saying that the US has great laws, or that the laws of other nations are terrible, but we tend to be very very relaxed on what is being said over the internet, where other nations, even first world european style nations, tend to not be so forgiving.

As for what gnif has said, if this is the case, I would drop this host right here, and right now without a moment of hesitation. What you move to is a decision best left to people who know what they are doing better than I, and that has been stated in the OP, but I do believe given what I've seen, and what has been shown, your gators are starting to get a bit gnashy.

I'm really not sure what you're talking about in regards to restrictions in the EU. I am not aware of any nation where mentioning groups will get you into trouble.

[trys]

I run some sites, one of which includes a forum running on the same software as here. The forum we run is a bit bigger than this one (about four to six times, in terms of posts, topics, members) and also has image attachments too like here. 

The MySQL database itself is roughly 4Gb.

I'm not going to go into very specific details of the server setup for security reasons, but very broadly we have found high performance with 4 CPU cores and just 8Gb allocated to it.

We own our own kit, co-locate them at a local datacentre, and then vitualize on those servers.

To reduce load, and to distribute content quickly we use Amazon's AWS Cloudfront this is a Content Distribution Network (CDN). This is really easy to set up with this forum software.

For resilience in case of downtime, we use Amazon's AWS Route 53 so we can point the DNS to another copy of the sites. As on AWS you just pay for what you use, all that's needed is the data stored, and then spin up a large instance when this happens. We have a micro instance spinning all the time just with the configs set up on it, and doing little disaster recovery jobs for us.

This last bit is probably the bit that really is handy, because of the short TTL (time to live) of the DNS on Route 53 we really can move so a new site very quickly when things go wrong.

The outgoing mail from the servers are handled by an external email handling organisation. That takes a huge burden off the servers, especially the forum, with all the notifications. This handles bounced emails and the like and the option for people to opt-out of emails and never get bothered by us again.

We also use another company for email whitelisting (which we had to jump through a few hoops to get certified) so that the notification emails get through spam filters as we adhere to good industry practices.

Another good thing of this, is if we do get prolonged downtime, we can still contact people by email.

Incoming emails are handled by a cloud-based ticketing system. This takes a burden off our server as it's not getting hammered by spammy emails. It also allows us to offer support and answer questions even if the site is down.

As far as a "hot spare" then then the lost revenue due to maximum anticipated downtime needs to be accounted for.

A live copy of the database for the forum can be achieved by using two MySQL instances, in a Master-Slave setup. The slave can be on a very low spec machine, it will just catch up when it can.

As an aside, for very short physical downtimes, I've even set up a Raspberry Pi with NGINX that displays a "Down for maintenance" that I plug the relevant ethernet cable into when I'm at the datacentre running off an USB batterypack (because you can't start plugging things in willy-nilly).

As far as legal threats on taking down the site, we identified that too as one of the biggest threats of downtime too. We've reduced this risk by being as above-board as we possibly can.

I do most of the sysadmin, but I do have a contract with a linux support company which I can turn to when I feel out of my depth, or I need a shoulder to cry on (in Linux terms anyway).

If you did want to host in a country that is well placed geographically and is a popular place for websites that my be under higher risk of legal scrutiny, then Iceland has a whole industry set up for this.

SO in summary (phew) you could stay with Hostgator if you wanted to, just add layers above it for added reliability, flexibility and improved performance (which results in increased revenue).

Gnif is doing a great job, not to be gniffed at.

Trys

[gnif]

I run some sites, one of which includes a forum running on the same software as here. The forum we run is a bit bigger than this one (about four to six times, in terms of posts, topics, members) and also has image attachments too like here. 

The MySQL database itself is roughly 4Gb.

Interesting, we are currently at 10GB, what forum are you running?

I'm not going to go into very specific details of the server setup for security reasons, but very broadly we have found high performance with 4 CPU cores and just 8Gb allocated to it.

8 Cores, 32GB RAM with database and sphinx cached in RAM.

To reduce load, and to distribute content quickly we use Amazon's AWS Cloudfront this is a Content Distribution Network (CDN). This is really easy to set up with this forum software.

Bulk transfer is not the issue we face, the load of this is minimal with Nginx, so a CDN doesn't make much sense for us. That said we do use CloudFlare which is caching what it can.

Gnif is doing a great job, not to be gniffed at.

Trys

Thanks mate . cPanel has mostly tied my hands with this configuration, it looks like we will be proceeding with a new configuration. On behalf of Dave & with his blessing I am entering negotiations with the data center to see what they can provide us.

[Ampera]

I'm really not sure what you're talking about in regards to restrictions in the EU. I am not aware of any nation where mentioning groups will get you into trouble.

Germany has a legal ban on supporting or spreading unconstitutional thoughts like national socialism. This same law exists in the UK. It's the reason if you lived in Germany during the early 1990's you couldn't legally buy a copy of Wolfenstein 3D.

[trys]

I run some sites, one of which includes a forum running on the same software as here. The forum we run is a bit bigger than this one (about four to six times, in terms of posts, topics, members) and also has image attachments too like here. 

The MySQL database itself is roughly 4Gb.

Interesting, we are currently at 10GB, what forum are you running?

I've just checked. The data directory (/var/lib/mysql/thedatabaseitself ) is 6.9G not 4G - my mistake. We're running the same forum software as on here (~0.7M topics ~6M posts). Running a mix of MyISAM (220 Tables. 7G) and InnoDB (73 Tables, 1G) on MySQL.

You can reduce the size of your database by using "Log pruning" (Maintenance>Logs). Turn off "Track daily page views (must have stats enabled)" (Configuration>Features and Options). This last one improves performance too.

The CDN takes burden off the server, not just bulk transfer, but repeated round trip server requests. The template itself is loaded in the CDN too, including the avatars, smilies etc. It also improves page load speeds. The Amazon Cloudfront side of things is incredibly cheap - it will cost you less than a couple of Big Mac meals a month.

Trys

[EEVblog]

The Yes's and I don't care's have it.
We'll be changing servers.
gnif is on the job...
If we didn't tell you, you might not even have noticed, hopefully.

[gnif]

The Yes's and I don't care's have it.
We'll be changing servers.
gnif is on the job...
If we didn't tell you, you might not even have noticed, hopefully.

It should be noted that this will take some time too, not only do we have to organize the new servers but I have to schedule the time to configure, deploy and test before migrating to the servers.

[Brumby]

If we didn't tell you, you might not even have noticed, hopefully.

I eagerly wait - to be left wondering...