POLL: Should I change servers?

[EEVblog]

The EEVblog server is currently running on a HostGator dedicated server.
Dual Xeon linux thingo in a basement in Texas somewhere, 1TB RAID1, 32GB, and when operating correctly it handles this forum (which is a big forum) and all the website and other database stuff with ease.
Gnif (https://hostfission.com/ who kindly donates his time to manage this server) has put untold hours into getting this server working correctly and smoothly for this forum and other needs. So apart from the occasional screwup causing some down time, I'm very happy with it, and there are zero complaints from users about speed or anything else.

But for those who have followed along, Hostgator screw up occasionally, most recently last night when they configured an IP without telling me that took the server down for 12 hours until I alerted them it was down. They also changed a bunch of stuff because they saw that the server had been changed "significantly from how they provisioned it". No kidding, it's a dedicated server, we are supposed to tweak it.
But to be fair, it's a fully managed dedicated server that Hostgator offer, so I guess it's no surprise that they played with it, and will do so again.

gnif has asked (again) whether I want to move to another dedicated server he recommends (I won't name them, and PLEASE, I don't want host suggestions), it's based in Utah.

Specs:
Dual Xeon L5630
  96GB RAM
  1 x 2TB HDD
  1 x 120GB SSD
  1 GBps Port
  30TB Monthly
  5 IPs
They also provide IPMI access to the server which means that when outages like this occur gnif is able to directly manage the system without the Data Centers involvement.
He couldn't do anything with the recent HostGator screwup, I had to contact them to reboot the server. (Granted they did it within 10's minutes)
Also, we would ditch cPanel which gnif doesn't like for various reasons I'm sure are valid 

Now normally I am totally adverse to changing web hosts, as IME it is absolutely no guarantee at all that I will have no or fewer problems than Hostgator. The "if it ain't broke, don't fix it" attitude.
And given that there are zero complaints from users about the server, and it only goes down when there is some big screw up, I personally see little value proposition in switching hosts.

As I see it:
Advantages to switching:
- No one will screw with it again, gnif becomes the only manager.
- A bit cheaper (but I don't care about that)
- More resources and potentially faster (although the forum is already lightning quick, so I don't expect to see any real improvement there)
- gnif will be as happy as a pig in mud

Disadvantages
- It will take time and effort
- There will likely be a downtime period and maybe some bugs to fix, either public ones that affect the forum and website, or private ones that only affect my own services (email, ftp etc)
- I will no longer have the knowledge to be able to access it personally, as it won't be running WHM and cPanel which I know fairly well and can actually do stuff with myself (reboots, ftp & email setups, phpMyAdmin database stuff etc). So unless I want to put untold hours into learning some command line thingo, I'll have to rely upon gnif and/or others entirely for anything operational. gnif already handles all the hard stuff, but I like the warm fuzzy knowing I can get into cPanel and WHM and do stuff on my own.

What say you?

[Ampera]

My experience with server hosting is minimal, but more than most I would say. I used to run (don't anymore because of bunny-related issues) two IBM E-Servers (235 and 325) to do simple things like hosting some gameplay servers, and I have also toyed around with Azure and Google Cloud.

I think you should change for a few reasons. For one, I have been in gnif's situation where I know the one wand to flick to make everything not suck is to up and move. The second reason is this isn't the first time they've had problems. I've heard bad things about these gators and their servers. For an enthusiast forum like this one, half a day of downtime isn't drastic, and we can live (not easily, though) without the EEVBlog forum for a while, especially if it means moving hosts.

I personally am VERY weary of any software that promises to automate and simplify configuration for me. I've never used cPanel, but I have used different dashboards and panels, and I would take SSH (or RDP for the NT kernel friends) over some fancy shiny control panel any day. If it means spending a few hours to get it working once, versus taking days to get what should have been five minutes to work, I would rather put the upfront cost in place. It's the same reason why I dislike Ubuntu, and love Arch Linux.

My personal skills are greatest with computer science and mathematics, so this is nearing into my forte, even though I hate websites for the stupidity they all bring. Excited to see what this might bring.

[DimitriP]

Changes should happen for your convenience, not anyone else's.

Trading infrequent "occasional  screw-up" for a mountain of work, more work, some more down-time with no guarantee  for lack of new  "occasional screw-ups" seems like trading one old used car for another old used car.

I like to stick with the devil I know.
If it's excitement you are after, I'm sure you can come up with your own ideas

[tautech]

- gnif will be as happy as a pig in mud

This ^

Honestly, try as gnif might he hasn't been able to get on top of the almost daily micro-outages that have plagued us all for months to the point where we almost expect them and few now bother to even report them.

If gnif is happy to take this on and has the obvious prerequisite server migration experience then JUMP !


Edit
What seems to have bought to a head and Dave mentioned last night's outage in the OP:
https://www.eevblog.com/forum/news/server-outage-12-3-18/

[Jeroen3]

took the server down for 12 hours until I alerted them it was down.

uptimerobot, pingdom.

Changing network settings on any server without asking or warning the owner is not acceptable.
If they do not apologize, or screw up again, you should move.

[tszaboo]

It should be a business decision.
How much money do you loose, when the server is down?
How much extra time do you have to pay to gnif when fixing stuff?
How much does it cost to move?
Is long term satisfaction and reliability an essential factor in your brand?

Only you can answer these questions.

Perhaps you just need another admin at hostgator, who doesnt have two left hands.

[Ampera]

- gnif will be as happy as a pig in mud

This ^

Honestly, try as gnif might he hasn't been able to get on top of the almost daily micro-outages that have plagued us all for months to the point where we almost expect them and few now bother to even report them.

If gnif is happy to take this on and has the obvious prerequisite server migration experience then JUMP !

You gotta know we all love gnif!

[nugglix]

I also voted for YES.
AFAIR this was not the first time they acted stupid aka incompetent.

I bet they sell "professional services" for "professional rates".
Unfortunately they're not up to the task.

A move might cost some extra money now, but given that the forum belongs
to and is a big plus of the EEVBLOG company (eco-system) it might be worth the cost.

And given the move will take place, one might also think about the location of the server.

Edit: would like to see pictures of happy gnif   

[EEVblog]

I personally am VERY weary of any software that promises to automate and simplify configuration for me. I've never used cPanel, but I have used different dashboards and panels, and I would take SSH (or RDP for the NT kernel friends) over some fancy shiny control panel any day.

I've been using cPanel on my web servers for almost 20 years now and have a lot of experience with it, it gets stuff done for me.
I can understand though how it's layer that gets in the way on "real" servers.

[EEVblog]

And given the move will take place, one might also think about the location of the server.

The US based server has been an issue in the past for legal reasons, having been shut down twice because of fraudulent DMCA requests.

[EEVblog]

It should be a business decision.
How much money do you loose, when the server is down?

Nothing really.

How much extra time do you have to pay to gnif when fixing stuff?

He kindly donates his time to maintain this server, although I do buy the backup service from him.

How much does it cost to move?
Is long term satisfaction and reliability an essential factor in your brand?

Like I said before, a change in web host does not guarantee this.

[Ampera]

I personally am VERY weary of any software that promises to automate and simplify configuration for me. I've never used cPanel, but I have used different dashboards and panels, and I would take SSH (or RDP for the NT kernel friends) over some fancy shiny control panel any day.

I've been using cPanel on my web servers for almost 20 years now and have a lot of experience with it, it gets stuff done for me.
I can understand though how it's layer that gets in the way on "real" servers.

I've never even looked at a picture of it myself, nevermind used it, but I am aware it's quite popular. My experience with anything on the HTTP is almost nothing outside of a bare theoretical overview of how it all works together. I could figure it out if I wanted to, but I don't know if I want to as I just hate the multi-language, incredibly messy freakshow that are website and web-apps. I'd take a well coded desktop app over a web app any day myself, but that's just because I don't like dealing with websites.

There is the old saying of if it's not broken, don't fix it, but I do believe the random issues with the form would constitute as a broken, even if you can't track down what the issue is. I had an experience with FreeBSD a few days ago, trying to install nVidia's X11 drivers (so I could use it as a desktop operating system) and no matter what guides I followed, no matter what steps I made, I couldn't for the life of me get it to work over the several hours I poured into it. Sometimes, you can follow all the right steps with computers, and it just still doesn't work. I afterwards installed Arch Linux again, and it worked (and is working) flawlessly. Likewise, sometimes the best solution is to just take it all off and do something else.

[beanflying]

Given the size of the database and number of users you need reliability. Your current server has been glitchy as all get out in the last 6 months at least, last nights host induced error would be the cherry on top of a shit sandwich if it were me. 

[Brumby]

- gnif will be as happy as a pig in mud

This ^

Honestly, try as gnif might he hasn't been able to get on top of the almost daily micro-outages that have plagued us all for months to the point where we almost expect them and few now bother to even report them.

If gnif is happy to take this on and has the obvious prerequisite server migration experience then JUMP !

Seconded.

Having said that, though, it would be very unwise to not have someone else who could step in should gnif be indisposed for any reason - or at least a plan of what will need to be done.  Or is this something that has already been addressed?

[CJay]

And given the move will take place, one might also think about the location of the server.

The US based server has been an issue in the past for legal reasons, having been shut down twice because of fraudulent DMCA requests.

The DMCA thing and being located  in the US opens you up to a pile of potential vexatious legal trouble if people start talking about hacking stuff or make some scurrilous claim about an audiophool product being bullshit perhaps?

Are there no decent CoLo/hosting companies in Aus, UK or some other less litigious place than the US that could be considered?

Other than that, always keep your sysadmin happy for 'we' hold the keys to your happiness.

[3db]

I think if you have confidence in gnif then go with what he recommends.
3DB

[Nusa]

I'd say it comes down to how much you value gnif vs your ability to do it yourself.

Trusting him and taking his advice would be an indirect form of payment to him. On the minus side, if he had a life event that made him unavailable, you might be screwed for a while.

Another version of that would be to formalize your current arrangement and hire his company to handle hosting, which would presumably mean that there would be more than just gnif himself available to help in the case of issues.

[EEVblog]

Having said that, though, it would be very unwise to not have someone else who could step in should gnif be indisposed for any reason - or at least a plan of what will need to be done.  Or is this something that has already been addressed?

That has not been addressed.
As long as I have login's then I presume it's likely not hard to find another person at short notice, I've always had many offers of help.

[Mr. Scram]

The US based server has been an issue in the past for legal reasons, having been shut down twice because of fraudulent DMCA requests.

I wanted to ask you about that, because hosting in the US seems to attract legal issues.  As a foreigner witout a large legal team, being broadsided by surprise is always a possibility. Is there any particular reason to go for a server there?

[Ampera]

Then again, US has nicer freedom of speech laws. You could in theory be held accountable in some countries for having specific elements of speech and "unconstitutional thoughts" in some places. Yes, the US has some bullshit copyright laws, but we trade blows with other nation's stupid laws.

[gnif]

ok, time for me to weigh in

Firstly thanks for the vote of confidence Dave.  Most problems with the website that cause these spot outages come down to one of several things

1) We are not using Apache (and thus cPanel) to host the website already, I have disabled apache and configured Nginx with PHP. The reason for this should be obvious, but for those that are not in the know, Apache is a terrible performer, especially with a website of this size.

2) Portions of the website are preventing the use of PHP 7, as such the server is running two configurations, PHP 7 for the forums, and PHP 5 for the website, shop, wiki, etc. This complicates matters as we have to maintain a custom build of PHP 5 as well as PHP 7 that sits outsides cPanel's realm of control.

3) The server is not a true dedicated server, it is a VM and if HG are to be trusted, running on a host with no other VMs. Even with this 1:1 configuration are bound to be performance losses, not to mention the slow down that the Intel patches bring with them (Meltdown and Spectre) compounded by the overhead of running in the VM. That said I have a hard time believing that this is configured correctly, the performance seems to vary day to day, it really makes me wonder if they are being honest if it is a 1:1 host/vm config, or if the VM is just very very poorly configured.

4) HG seem to think it's ok to just jump on the server and change our kernel, update cpanel, etc. When Dave alerted them to this outage, they didn't just fix the problem (Configure the IP correctly), they took it upon themselves to perform additional unnecessary tasks without first consulting the client (Dave), that broke our customized server configuration.

5) HF had the server configured to use DHCP... this is what broke. Apparently when they moved data centers they switched to static IPs (which is what they should have been using anyway), but we were never notified nor did they "manage" the server to update it. (The dark side of me wonders if they had filtering setup correctly and if it would be possible to configure a rogue DHCP server....)

6) HF failed to detect that the "managed" server was down for hours and it took the client (Dave) prodding them to get it back online. Seriously, how hard is it to just ping the hosts to see if they are alive?

As for Dave's management of the things he needs, there are three options.

1) Move to a dedicated server with a completely custom stack, and rely on an admin (myself) to manage it for him.

2) Bring up two servers, a basic low spec server for cPanel/Email/DNS, etc... the other for just the HTTP hosting.

3) Same as 2, but run two VMs on the single server saving the additional cost.

Personally I would opt for 2 servers, but I would be just as happy with option 3 as I would be able to tune the VMs correctly.

[Mr. Scram]

Then again, US has nicer freedom of speech laws. You could in theory be held accountable in some countries for having specific elements of speech and "unconstitutional thoughts" in some places. Yes, the US has some bullshit copyright laws, but we trade blows with other nation's stupid laws.

Nicer than where? I doubt anyone had plans to host in Venezuela or China.

[Halcyon]

If it directly supports Geoff, then my answer is a definite "yes". If it doesn't, my answer is the "I don't care" one.

[Ampera]

Then again, US has nicer freedom of speech laws. You could in theory be held accountable in some countries for having specific elements of speech and "unconstitutional thoughts" in some places. Yes, the US has some bullshit copyright laws, but we trade blows with other nation's stupid laws.

Nicer than where? I doubt anyone had plans to host in Venezuela or China.

It's various incidents that have occurred in the past. Europe tends to have issues with mentions of previously (or to this day) violent groups like national socialists and communist parties. Not saying that I want to make it a habit of talking about those things, but this is the internet, and a diverse forum with varying topics, being restricted to specific topics just gives my American blood at least a bit of a boil.

NZ has their strange cyber bullying laws which were the talk of the world a few years ago.

I'm not saying that the US has great laws, or that the laws of other nations are terrible, but we tend to be very very relaxed on what is being said over the internet, where other nations, even first world european style nations, tend to not be so forgiving.

As for what gnif has said, if this is the case, I would drop this host right here, and right now without a moment of hesitation. What you move to is a decision best left to people who know what they are doing better than I, and that has been stated in the OP, but I do believe given what I've seen, and what has been shown, your gators are starting to get a bit gnashy.

[Ice-Tea]

Having said that, though, it would be very unwise to not have someone else who could step in should gnif be indisposed for any reason - or at least a plan of what will need to be done.  Or is this something that has already been addressed?

That has not been addressed.
As long as I have login's then I presume it's likely not hard to find another person at short notice, I've always had many offers of help.

My 0.02$: set this up in advance.