POLL: Should I change servers?

[EEVblog]

The EEVblog server is currently running on a HostGator dedicated server.
Dual Xeon linux thingo in a basement in Texas somewhere, 1TB RAID1, 32GB, and when operating correctly it handles this forum (which is a big forum) and all the website and other database stuff with ease.
Gnif (https://hostfission.com/ who kindly donates his time to manage this server) has put untold hours into getting this server working correctly and smoothly for this forum and other needs. So apart from the occasional screwup causing some down time, I'm very happy with it, and there are zero complaints from users about speed or anything else.

But for those who have followed along, Hostgator screw up occasionally, most recently last night when they configured an IP without telling me that took the server down for 12 hours until I alerted them it was down. They also changed a bunch of stuff because they saw that the server had been changed "significantly from how they provisioned it". No kidding, it's a dedicated server, we are supposed to tweak it.
But to be fair, it's a fully managed dedicated server that Hostgator offer, so I guess it's no surprise that they played with it, and will do so again.

gnif has asked (again) whether I want to move to another dedicated server he recommends (I won't name them, and PLEASE, I don't want host suggestions), it's based in Utah.

Specs:
Dual Xeon L5630
  96GB RAM
  1 x 2TB HDD
  1 x 120GB SSD
  1 GBps Port
  30TB Monthly
  5 IPs
They also provide IPMI access to the server which means that when outages like this occur gnif is able to directly manage the system without the Data Centers involvement.
He couldn't do anything with the recent HostGator screwup, I had to contact them to reboot the server. (Granted they did it within 10's minutes)
Also, we would ditch cPanel which gnif doesn't like for various reasons I'm sure are valid 

Now normally I am totally adverse to changing web hosts, as IME it is absolutely no guarantee at all that I will have no or fewer problems than Hostgator. The "if it ain't broke, don't fix it" attitude.
And given that there are zero complaints from users about the server, and it only goes down when there is some big screw up, I personally see little value proposition in switching hosts.

As I see it:
Advantages to switching:
- No one will screw with it again, gnif becomes the only manager.
- A bit cheaper (but I don't care about that)
- More resources and potentially faster (although the forum is already lightning quick, so I don't expect to see any real improvement there)
- gnif will be as happy as a pig in mud

Disadvantages
- It will take time and effort
- There will likely be a downtime period and maybe some bugs to fix, either public ones that affect the forum and website, or private ones that only affect my own services (email, ftp etc)
- I will no longer have the knowledge to be able to access it personally, as it won't be running WHM and cPanel which I know fairly well and can actually do stuff with myself (reboots, ftp & email setups, phpMyAdmin database stuff etc). So unless I want to put untold hours into learning some command line thingo, I'll have to rely upon gnif and/or others entirely for anything operational. gnif already handles all the hard stuff, but I like the warm fuzzy knowing I can get into cPanel and WHM and do stuff on my own.

What say you?

[Ampera]

My experience with server hosting is minimal, but more than most I would say. I used to run (don't anymore because of bunny-related issues) two IBM E-Servers (235 and 325) to do simple things like hosting some gameplay servers, and I have also toyed around with Azure and Google Cloud.

I think you should change for a few reasons. For one, I have been in gnif's situation where I know the one wand to flick to make everything not suck is to up and move. The second reason is this isn't the first time they've had problems. I've heard bad things about these gators and their servers. For an enthusiast forum like this one, half a day of downtime isn't drastic, and we can live (not easily, though) without the EEVBlog forum for a while, especially if it means moving hosts.

I personally am VERY weary of any software that promises to automate and simplify configuration for me. I've never used cPanel, but I have used different dashboards and panels, and I would take SSH (or RDP for the NT kernel friends) over some fancy shiny control panel any day. If it means spending a few hours to get it working once, versus taking days to get what should have been five minutes to work, I would rather put the upfront cost in place. It's the same reason why I dislike Ubuntu, and love Arch Linux.

My personal skills are greatest with computer science and mathematics, so this is nearing into my forte, even though I hate websites for the stupidity they all bring. Excited to see what this might bring.

[DimitriP]

Changes should happen for your convenience, not anyone else's.

Trading infrequent "occasional  screw-up" for a mountain of work, more work, some more down-time with no guarantee  for lack of new  "occasional screw-ups" seems like trading one old used car for another old used car.

I like to stick with the devil I know.
If it's excitement you are after, I'm sure you can come up with your own ideas

[tautech]

- gnif will be as happy as a pig in mud

This ^

Honestly, try as gnif might he hasn't been able to get on top of the almost daily micro-outages that have plagued us all for months to the point where we almost expect them and few now bother to even report them.

If gnif is happy to take this on and has the obvious prerequisite server migration experience then JUMP !


Edit
What seems to have bought to a head and Dave mentioned last night's outage in the OP:
https://www.eevblog.com/forum/news/server-outage-12-3-18/

[Jeroen3]

took the server down for 12 hours until I alerted them it was down.

uptimerobot, pingdom.

Changing network settings on any server without asking or warning the owner is not acceptable.
If they do not apologize, or screw up again, you should move.

[tszaboo]

It should be a business decision.
How much money do you loose, when the server is down?
How much extra time do you have to pay to gnif when fixing stuff?
How much does it cost to move?
Is long term satisfaction and reliability an essential factor in your brand?

Only you can answer these questions.

Perhaps you just need another admin at hostgator, who doesnt have two left hands.

[Ampera]

- gnif will be as happy as a pig in mud

This ^

Honestly, try as gnif might he hasn't been able to get on top of the almost daily micro-outages that have plagued us all for months to the point where we almost expect them and few now bother to even report them.

If gnif is happy to take this on and has the obvious prerequisite server migration experience then JUMP !

You gotta know we all love gnif!

[nugglix]

I also voted for YES.
AFAIR this was not the first time they acted stupid aka incompetent.

I bet they sell "professional services" for "professional rates".
Unfortunately they're not up to the task.

A move might cost some extra money now, but given that the forum belongs
to and is a big plus of the EEVBLOG company (eco-system) it might be worth the cost.

And given the move will take place, one might also think about the location of the server.

Edit: would like to see pictures of happy gnif   

[EEVblog]

I personally am VERY weary of any software that promises to automate and simplify configuration for me. I've never used cPanel, but I have used different dashboards and panels, and I would take SSH (or RDP for the NT kernel friends) over some fancy shiny control panel any day.

I've been using cPanel on my web servers for almost 20 years now and have a lot of experience with it, it gets stuff done for me.
I can understand though how it's layer that gets in the way on "real" servers.

[EEVblog]

And given the move will take place, one might also think about the location of the server.

The US based server has been an issue in the past for legal reasons, having been shut down twice because of fraudulent DMCA requests.

[EEVblog]

It should be a business decision.
How much money do you loose, when the server is down?

Nothing really.

How much extra time do you have to pay to gnif when fixing stuff?

He kindly donates his time to maintain this server, although I do buy the backup service from him.

How much does it cost to move?
Is long term satisfaction and reliability an essential factor in your brand?

Like I said before, a change in web host does not guarantee this.

[Ampera]

I personally am VERY weary of any software that promises to automate and simplify configuration for me. I've never used cPanel, but I have used different dashboards and panels, and I would take SSH (or RDP for the NT kernel friends) over some fancy shiny control panel any day.

I've been using cPanel on my web servers for almost 20 years now and have a lot of experience with it, it gets stuff done for me.
I can understand though how it's layer that gets in the way on "real" servers.

I've never even looked at a picture of it myself, nevermind used it, but I am aware it's quite popular. My experience with anything on the HTTP is almost nothing outside of a bare theoretical overview of how it all works together. I could figure it out if I wanted to, but I don't know if I want to as I just hate the multi-language, incredibly messy freakshow that are website and web-apps. I'd take a well coded desktop app over a web app any day myself, but that's just because I don't like dealing with websites.

There is the old saying of if it's not broken, don't fix it, but I do believe the random issues with the form would constitute as a broken, even if you can't track down what the issue is. I had an experience with FreeBSD a few days ago, trying to install nVidia's X11 drivers (so I could use it as a desktop operating system) and no matter what guides I followed, no matter what steps I made, I couldn't for the life of me get it to work over the several hours I poured into it. Sometimes, you can follow all the right steps with computers, and it just still doesn't work. I afterwards installed Arch Linux again, and it worked (and is working) flawlessly. Likewise, sometimes the best solution is to just take it all off and do something else.

[beanflying]

Given the size of the database and number of users you need reliability. Your current server has been glitchy as all get out in the last 6 months at least, last nights host induced error would be the cherry on top of a shit sandwich if it were me. 

[Brumby]

- gnif will be as happy as a pig in mud

This ^

Honestly, try as gnif might he hasn't been able to get on top of the almost daily micro-outages that have plagued us all for months to the point where we almost expect them and few now bother to even report them.

If gnif is happy to take this on and has the obvious prerequisite server migration experience then JUMP !

Seconded.

Having said that, though, it would be very unwise to not have someone else who could step in should gnif be indisposed for any reason - or at least a plan of what will need to be done.  Or is this something that has already been addressed?

[CJay]

And given the move will take place, one might also think about the location of the server.

The US based server has been an issue in the past for legal reasons, having been shut down twice because of fraudulent DMCA requests.

The DMCA thing and being located  in the US opens you up to a pile of potential vexatious legal trouble if people start talking about hacking stuff or make some scurrilous claim about an audiophool product being bullshit perhaps?

Are there no decent CoLo/hosting companies in Aus, UK or some other less litigious place than the US that could be considered?

Other than that, always keep your sysadmin happy for 'we' hold the keys to your happiness.

[3db]

I think if you have confidence in gnif then go with what he recommends.
3DB

[Nusa]

I'd say it comes down to how much you value gnif vs your ability to do it yourself.

Trusting him and taking his advice would be an indirect form of payment to him. On the minus side, if he had a life event that made him unavailable, you might be screwed for a while.

Another version of that would be to formalize your current arrangement and hire his company to handle hosting, which would presumably mean that there would be more than just gnif himself available to help in the case of issues.

[EEVblog]

Having said that, though, it would be very unwise to not have someone else who could step in should gnif be indisposed for any reason - or at least a plan of what will need to be done.  Or is this something that has already been addressed?

That has not been addressed.
As long as I have login's then I presume it's likely not hard to find another person at short notice, I've always had many offers of help.

[Mr. Scram]

The US based server has been an issue in the past for legal reasons, having been shut down twice because of fraudulent DMCA requests.

I wanted to ask you about that, because hosting in the US seems to attract legal issues.  As a foreigner witout a large legal team, being broadsided by surprise is always a possibility. Is there any particular reason to go for a server there?

[Ampera]

Then again, US has nicer freedom of speech laws. You could in theory be held accountable in some countries for having specific elements of speech and "unconstitutional thoughts" in some places. Yes, the US has some bullshit copyright laws, but we trade blows with other nation's stupid laws.

[gnif]

ok, time for me to weigh in

Firstly thanks for the vote of confidence Dave.  Most problems with the website that cause these spot outages come down to one of several things

1) We are not using Apache (and thus cPanel) to host the website already, I have disabled apache and configured Nginx with PHP. The reason for this should be obvious, but for those that are not in the know, Apache is a terrible performer, especially with a website of this size.

2) Portions of the website are preventing the use of PHP 7, as such the server is running two configurations, PHP 7 for the forums, and PHP 5 for the website, shop, wiki, etc. This complicates matters as we have to maintain a custom build of PHP 5 as well as PHP 7 that sits outsides cPanel's realm of control.

3) The server is not a true dedicated server, it is a VM and if HG are to be trusted, running on a host with no other VMs. Even with this 1:1 configuration are bound to be performance losses, not to mention the slow down that the Intel patches bring with them (Meltdown and Spectre) compounded by the overhead of running in the VM. That said I have a hard time believing that this is configured correctly, the performance seems to vary day to day, it really makes me wonder if they are being honest if it is a 1:1 host/vm config, or if the VM is just very very poorly configured.

4) HG seem to think it's ok to just jump on the server and change our kernel, update cpanel, etc. When Dave alerted them to this outage, they didn't just fix the problem (Configure the IP correctly), they took it upon themselves to perform additional unnecessary tasks without first consulting the client (Dave), that broke our customized server configuration.

5) HF had the server configured to use DHCP... this is what broke. Apparently when they moved data centers they switched to static IPs (which is what they should have been using anyway), but we were never notified nor did they "manage" the server to update it. (The dark side of me wonders if they had filtering setup correctly and if it would be possible to configure a rogue DHCP server....)

6) HF failed to detect that the "managed" server was down for hours and it took the client (Dave) prodding them to get it back online. Seriously, how hard is it to just ping the hosts to see if they are alive?

As for Dave's management of the things he needs, there are three options.

1) Move to a dedicated server with a completely custom stack, and rely on an admin (myself) to manage it for him.

2) Bring up two servers, a basic low spec server for cPanel/Email/DNS, etc... the other for just the HTTP hosting.

3) Same as 2, but run two VMs on the single server saving the additional cost.

Personally I would opt for 2 servers, but I would be just as happy with option 3 as I would be able to tune the VMs correctly.

[Mr. Scram]

Then again, US has nicer freedom of speech laws. You could in theory be held accountable in some countries for having specific elements of speech and "unconstitutional thoughts" in some places. Yes, the US has some bullshit copyright laws, but we trade blows with other nation's stupid laws.

Nicer than where? I doubt anyone had plans to host in Venezuela or China.

[Halcyon]

If it directly supports Geoff, then my answer is a definite "yes". If it doesn't, my answer is the "I don't care" one.

[Ampera]

Then again, US has nicer freedom of speech laws. You could in theory be held accountable in some countries for having specific elements of speech and "unconstitutional thoughts" in some places. Yes, the US has some bullshit copyright laws, but we trade blows with other nation's stupid laws.

Nicer than where? I doubt anyone had plans to host in Venezuela or China.

It's various incidents that have occurred in the past. Europe tends to have issues with mentions of previously (or to this day) violent groups like national socialists and communist parties. Not saying that I want to make it a habit of talking about those things, but this is the internet, and a diverse forum with varying topics, being restricted to specific topics just gives my American blood at least a bit of a boil.

NZ has their strange cyber bullying laws which were the talk of the world a few years ago.

I'm not saying that the US has great laws, or that the laws of other nations are terrible, but we tend to be very very relaxed on what is being said over the internet, where other nations, even first world european style nations, tend to not be so forgiving.

As for what gnif has said, if this is the case, I would drop this host right here, and right now without a moment of hesitation. What you move to is a decision best left to people who know what they are doing better than I, and that has been stated in the OP, but I do believe given what I've seen, and what has been shown, your gators are starting to get a bit gnashy.

[Ice-Tea]

Having said that, though, it would be very unwise to not have someone else who could step in should gnif be indisposed for any reason - or at least a plan of what will need to be done.  Or is this something that has already been addressed?

That has not been addressed.
As long as I have login's then I presume it's likely not hard to find another person at short notice, I've always had many offers of help.

My 0.02$: set this up in advance.

[gnif]

Having said that, though, it would be very unwise to not have someone else who could step in should gnif be indisposed for any reason - or at least a plan of what will need to be done.  Or is this something that has already been addressed?

That has not been addressed.
As long as I have login's then I presume it's likely not hard to find another person at short notice, I've always had many offers of help.

My 0.02$: set this up in advance.

Never a bad idea, I do have people on retainer though that can handle most things.

[gnif]

To be more judgemental I'd want to know how and why the standard HG setup was changed to cause the update incompatibilities HG report. If you NEED these changes then you should change. If you can return to a HG supplied standard setup then you may as well stay if you are otherwise happy.

HG changed their setup and never updated the server's configuration, it was their breakage.

I'd need to know how it came to be moved from a static IP to DHCP.

It's the other way around, HG initially deployed the servers on their network with DHCP, which IMHO is a joke. Seems they found out the hard way this was a bad idea and switched to static IP assignment, just like every other data center on the planet uses. HG failed to mention this to Dave, and they failed to update the server's IP configuration with the static IP addressing details.

The questions I have are why you didn't start the server up on the backup system? Not saying you should have but could you have?  If your backup system is with another hosting vendor is it the same as HG supply? And why the outage was so long?

Dave doesn't have a backup system, I provide server backups, data storage, etc. The outage length was two fold, HG didn't detect that the host had stopped responding, being a "managed service" they should have had an alarm for this. Secondly I can not contact HG on Dave's behalf at this point, we had to wait for Dave to raise the problem with them.

[BravoV]

Specs:
Dual Xeon L5630
  96GB RAM
  1 x 2TB HDD
  1 x 120GB SSD
  1 GBps Port
  30TB Monthly
  5 IPs

So on new site, there is no redundancy on the storage as I see only one HD, and relies on Gnif's remote backup ?

[gnif]

Specs:
Dual Xeon L5630
  96GB RAM
  1 x 2TB HDD
  1 x 120GB SSD
  1 GBps Port
  30TB Monthly
  5 IPs

So on new site, there is no redundancy on the storage as I see only one HD, and relies on Gnif's remote backup ?

This is only one potential configuration, the host would be configured with a proper redundant RAID array. This package deal was a quick quote I copied and pasted in as an example of what could be provided with an alternative supplier of whom I have many clients and trust implicitly.

[Halcyon]

Dave doesn't have a backup system, I provide server backups, data storage, etc.

How large is the dataset? I'm happy to relinquish some storage on my servers at home. Both running FreeNAS, one is a high speed array, the other is a cold storage machine that replicates periodically with the main server. Dave has been here and has seen my "work in progress" server room.

[Avacee]

I voted Yes.

Always support a volunteer (within reason). If gnif will be happier I assume part of this is him calculating lower time requirements long-term and being able to solve issues with less hassle. Always a good way of keeping volunteers volunteering

The current configuration sounds highly modified and if gnif 1.0 was hit by the proverbial bus how much up the creek without a paddle are you compared to a more standard configuration. How quickly can gnif 2.0 understand all those little tweaks that have accumulated over the years?
Whilst you personally may not be as familiar with the tools involved with a more standard configuration any gnif 2.0 will be up to speed much faster. You can learn the essentials at your own pace.

The current hosting company changing things as has been described is a red flag that says "Start your engines on a move"

Moving has a high enough level of probability of longer term peace of mind after the higher up front costs and time spent.

[Halcyon]

The current configuration sounds highly modified and if gnif 1.0 was hit by the proverbial bus how much up the creek without a paddle are you compared to a more standard configuration. How quickly can gnif 2.0 understand all those little tweaks that have accumulated over the years?

Let's hope that never happens! Although it would be kind of neat to have a bit red flip switch in the EEVblog lab labelled "Geoff / No Geoff" for when he is on holidays and something breaks, the "No Geoff" position redirects DNS requests to an alternate server with a replica of the information ;-)

[gnif]

Dave doesn't have a backup system, I provide server backups, data storage, etc.

How large is the dataset? I'm happy to relinquish some storage on my servers at home. Both running FreeNAS, one is a high speed array, the other is a cold storage machine that replicates periodically with the main server. Dave has been here and has seen my "work in progress" server room.

Thanks but the data backup solution in place is already enterprise grade. By backup system I was referring to a hot spare HTTP server.

[Ampera]

The current configuration sounds highly modified and if gnif 1.0 was hit by the proverbial bus how much up the creek without a paddle are you compared to a more standard configuration. How quickly can gnif 2.0 understand all those little tweaks that have accumulated over the years?

Let's hope that never happens! Although it would be kind of neat to have a bit red flip switch in the EEVblog lab labelled "Geoff / No Geoff" for when he is on holidays and something breaks, the "No Geoff" position redirects DNS requests to an alternate server with a replica of the information ;-)

*Break glass in emergency lack of Geoff*

[gnif]

At current it would be quite hard to bring someone up to speed with the configuration of the existing system, it is by no means standard due to the need to work around cPanel. If the server was built without cPanel it would be pretty simple for pretty much any competent server administrator to take over should the worst occur. When I setup systems I follow the best practices of the distribution to ensure ease of management and upgrades into the future, about the only custom stuff would be the monitoring and backup configuration as that is infrastructure dependent.

[tautech]

Dave's pushing to lift his income to the next level and along with that comes cost, there's only one of any of us and only so many hours in the day......well, if we want to keep our wife and family.

There comes a time when it's plain to see we can't 'do it all' like we used to and it's concerning to have to hand over responsibilities to others so that we can devote our time to what really matters....the future, with earned gains from the past compounding AND time to enjoy them.

Get good people and providers around you and move forward, faster, better and with less stress.

We'll put up with anything that's not quite right for a bit.............

[ovnr]

I'd move; HostGator have shown themselves to be grossly incompetent, to be honest. I'd also drop cPanel because it's terrible.

As for Dave's concerns about being able to sort out minor things himself: You can always install phpMyAdmin on the new box too so he can fiddle with the DB, and I expect most other common tasks are a shell script away.


Incidentally, how much stuff does PHP 7 break for you? I noticed that you're running a pretty old MediaWiki install (1.17 vs 1.30); the main website seems to just be a WordPress blog with some plugins, and unless you have a pile of custom plugins WP tends to be pretty OK about updates. Anyway, my point being that I'm sure you can find someone to fix whatever PHP 7 breaks; I could probably do it too, but I'm a bit swamped these days.

[EEVblog]

3) The server is not a true dedicated server, it is a VM and if HG are to be trusted, running on a host with no other VMs.

I have no doubt I have my own box.
I believe I'm now on a "legacy" server box, and perhaps this had something to do with the change that caused the problem.
Yes it's a VM they are using.

As for Dave's management of the things he needs, there are three options.
1) Move to a dedicated server with a completely custom stack, and rely on an admin (myself) to manage it for him.
2) Bring up two servers, a basic low spec server for cPanel/Email/DNS, etc... the other for just the HTTP hosting.
3) Same as 2, but run two VMs on the single server saving the additional cost.

#2 sounds good, but considering that the load on the server is currently quite minimal, we no doubt have plenty of grunt to run two VM's on the one server.
I'd want the ability to set up my own email and FTP stuff etc.

Ideally we'd have completely mirrored backup servers in different locations and providers that can automatically (or manually) be swapped out if there is an issue. Although I'm not sure how that actually works in practice. But really, we aren't running a bank here, it's not the end of the world if the server goes down due to unforeseen circumstances for a few hours or a day every year. Every host eventually gets hit with some form of attack or something else that causes issues.
How many forums and other websites this size do that?

Would we drop Cloudflare or keep it?

With my new Synology RAID, if possible I'd like to set up some form of additional daily backup on my desktop here for piece of mind. I think we currently use about 393GB of disk space, but that includes the OS and all the crap. Even just the databases and website files backed up daily on my RAID would be nice.

[EEVblog]

How large is the dataset? I'm happy to relinquish some storage on my servers at home. Both running FreeNAS, one is a high speed array, the other is a cold storage machine that replicates periodically with the main server. Dave has been here and has seen my "work in progress" server room.

Even Mrs EEVblog has seen the rats nest, don't make me post a photo

[EEVblog]

The questions I have are why you didn't start the server up on the backup system? Not saying you should have but could you have?  If your backup system is with another hosting vendor is it the same as HG supply? And why the outage was so long?

We do not have a backup server we can just switch to instantly. It would take some time to re-implement a new server from a backup.
The outage was so long because I went to sleep thinking it was just another HostGator network glitch or some other transient network thing that happens a few times a year. Woke up to find it still down.
They apparently don't have anything that monitors if my actual website is contactable. As far as they were concerned, the server was up and running just fine.

Maybe you should write down what your actual requirements are, then sort the list. You sound pretty happy with HG, so what are the chances you'll be happier elsewhere?

Hostgater have been nice to me, even so far as escalating a major issue to the CEO for resolution. (a viewer has a high level position there)
And their service is pretty good when you actually ask for it, but yeah, they seem to screw up a lot 
But then, I've had issues with every web host I've ever used.

[Halcyon]

How large is the dataset? I'm happy to relinquish some storage on my servers at home. Both running FreeNAS, one is a high speed array, the other is a cold storage machine that replicates periodically with the main server. Dave has been here and has seen my "work in progress" server room.

Even Mrs EEVblog has seen the rats nest, don't make me post a photo

For your eyes only Dave. You're welcome to come and take a fresh photo once the cabling is finished, THEN you can share. :-) Also, hopefully I'll have some more tear-downy goodness for you by then.

[madires]

And my 2ct The web based server management panels are fine for standard stuff but they can't be used for special configurations. You should be aware that they also increase the attack surface. It's a tradeoff. A hoster who changes the IP address of a managed server without informing the customer of the maintenance a few weeks in advance and not checking if the server runs fine after the change is completely hopeless. Usually the change of the IP address implicates updates in the DNS. Those have to be done in parallel with some TTL tweaking to minimize the impact. Apparently HG caused the outage by the unprofessional handling of the IP address change. I'd tell them this and ask for a refund. I'd also make clear that I will move to another hoster if something like this happens again or they won't give a refund. And don't forget too add that the server runs a forum with a huge community and you publicly share your experience with HG.

[Mr. Scram]

It's various incidents that have occurred in the past. Europe tends to have issues with mentions of previously (or to this day) violent groups like national socialists and communist parties. Not saying that I want to make it a habit of talking about those things, but this is the internet, and a diverse forum with varying topics, being restricted to specific topics just gives my American blood at least a bit of a boil.

NZ has their strange cyber bullying laws which were the talk of the world a few years ago.

I'm not saying that the US has great laws, or that the laws of other nations are terrible, but we tend to be very very relaxed on what is being said over the internet, where other nations, even first world european style nations, tend to not be so forgiving.

As for what gnif has said, if this is the case, I would drop this host right here, and right now without a moment of hesitation. What you move to is a decision best left to people who know what they are doing better than I, and that has been stated in the OP, but I do believe given what I've seen, and what has been shown, your gators are starting to get a bit gnashy.

I'm really not sure what you're talking about in regards to restrictions in the EU. I am not aware of any nation where mentioning groups will get you into trouble.

[trys]

I run some sites, one of which includes a forum running on the same software as here. The forum we run is a bit bigger than this one (about four to six times, in terms of posts, topics, members) and also has image attachments too like here. 

The MySQL database itself is roughly 4Gb.

I'm not going to go into very specific details of the server setup for security reasons, but very broadly we have found high performance with 4 CPU cores and just 8Gb allocated to it.

We own our own kit, co-locate them at a local datacentre, and then vitualize on those servers.

To reduce load, and to distribute content quickly we use Amazon's AWS Cloudfront this is a Content Distribution Network (CDN). This is really easy to set up with this forum software.

For resilience in case of downtime, we use Amazon's AWS Route 53 so we can point the DNS to another copy of the sites. As on AWS you just pay for what you use, all that's needed is the data stored, and then spin up a large instance when this happens. We have a micro instance spinning all the time just with the configs set up on it, and doing little disaster recovery jobs for us.

This last bit is probably the bit that really is handy, because of the short TTL (time to live) of the DNS on Route 53 we really can move so a new site very quickly when things go wrong.

The outgoing mail from the servers are handled by an external email handling organisation. That takes a huge burden off the servers, especially the forum, with all the notifications. This handles bounced emails and the like and the option for people to opt-out of emails and never get bothered by us again.

We also use another company for email whitelisting (which we had to jump through a few hoops to get certified) so that the notification emails get through spam filters as we adhere to good industry practices.

Another good thing of this, is if we do get prolonged downtime, we can still contact people by email.

Incoming emails are handled by a cloud-based ticketing system. This takes a burden off our server as it's not getting hammered by spammy emails. It also allows us to offer support and answer questions even if the site is down.

As far as a "hot spare" then then the lost revenue due to maximum anticipated downtime needs to be accounted for.

A live copy of the database for the forum can be achieved by using two MySQL instances, in a Master-Slave setup. The slave can be on a very low spec machine, it will just catch up when it can.

As an aside, for very short physical downtimes, I've even set up a Raspberry Pi with NGINX that displays a "Down for maintenance" that I plug the relevant ethernet cable into when I'm at the datacentre running off an USB batterypack (because you can't start plugging things in willy-nilly).

As far as legal threats on taking down the site, we identified that too as one of the biggest threats of downtime too. We've reduced this risk by being as above-board as we possibly can.

I do most of the sysadmin, but I do have a contract with a linux support company which I can turn to when I feel out of my depth, or I need a shoulder to cry on (in Linux terms anyway).

If you did want to host in a country that is well placed geographically and is a popular place for websites that my be under higher risk of legal scrutiny, then Iceland has a whole industry set up for this.

SO in summary (phew) you could stay with Hostgator if you wanted to, just add layers above it for added reliability, flexibility and improved performance (which results in increased revenue).

Gnif is doing a great job, not to be gniffed at.

Trys

[gnif]

I run some sites, one of which includes a forum running on the same software as here. The forum we run is a bit bigger than this one (about four to six times, in terms of posts, topics, members) and also has image attachments too like here. 

The MySQL database itself is roughly 4Gb.

Interesting, we are currently at 10GB, what forum are you running?

I'm not going to go into very specific details of the server setup for security reasons, but very broadly we have found high performance with 4 CPU cores and just 8Gb allocated to it.

8 Cores, 32GB RAM with database and sphinx cached in RAM.

To reduce load, and to distribute content quickly we use Amazon's AWS Cloudfront this is a Content Distribution Network (CDN). This is really easy to set up with this forum software.

Bulk transfer is not the issue we face, the load of this is minimal with Nginx, so a CDN doesn't make much sense for us. That said we do use CloudFlare which is caching what it can.

Gnif is doing a great job, not to be gniffed at.

Trys

Thanks mate . cPanel has mostly tied my hands with this configuration, it looks like we will be proceeding with a new configuration. On behalf of Dave & with his blessing I am entering negotiations with the data center to see what they can provide us.

[Ampera]

I'm really not sure what you're talking about in regards to restrictions in the EU. I am not aware of any nation where mentioning groups will get you into trouble.

Germany has a legal ban on supporting or spreading unconstitutional thoughts like national socialism. This same law exists in the UK. It's the reason if you lived in Germany during the early 1990's you couldn't legally buy a copy of Wolfenstein 3D.

[trys]

I run some sites, one of which includes a forum running on the same software as here. The forum we run is a bit bigger than this one (about four to six times, in terms of posts, topics, members) and also has image attachments too like here. 

The MySQL database itself is roughly 4Gb.

Interesting, we are currently at 10GB, what forum are you running?

I've just checked. The data directory (/var/lib/mysql/thedatabaseitself ) is 6.9G not 4G - my mistake. We're running the same forum software as on here (~0.7M topics ~6M posts). Running a mix of MyISAM (220 Tables. 7G) and InnoDB (73 Tables, 1G) on MySQL.

You can reduce the size of your database by using "Log pruning" (Maintenance>Logs). Turn off "Track daily page views (must have stats enabled)" (Configuration>Features and Options). This last one improves performance too.

The CDN takes burden off the server, not just bulk transfer, but repeated round trip server requests. The template itself is loaded in the CDN too, including the avatars, smilies etc. It also improves page load speeds. The Amazon Cloudfront side of things is incredibly cheap - it will cost you less than a couple of Big Mac meals a month.

Trys

[EEVblog]

The Yes's and I don't care's have it.
We'll be changing servers.
gnif is on the job...
If we didn't tell you, you might not even have noticed, hopefully.

[gnif]

The Yes's and I don't care's have it.
We'll be changing servers.
gnif is on the job...
If we didn't tell you, you might not even have noticed, hopefully.

It should be noted that this will take some time too, not only do we have to organize the new servers but I have to schedule the time to configure, deploy and test before migrating to the servers.

[Brumby]

If we didn't tell you, you might not even have noticed, hopefully.

I eagerly wait - to be left wondering...

[tpowell1830]

The Yes's and I don't care's have it.
We'll be changing servers.
gnif is on the job...
If we didn't tell you, you might not even have noticed, hopefully.

Being from Houston, it looks like my forum loads would be lightening fast for the forum as it is, however, the load times vary all over the place from very fast to sickly slow. With that said, changing hosts means that Gnif will be very busy for a few months getting all of the tweaks done. Meanwhile back in OZ, Dave will be intently trying to learn how to access and manage the new configuration. Does this mean fewer YT videos? I guess we will see. Just to be clear, I did not vote because I think this is strictly a business decision that is Dave's and Dave's alone.

[gnif]

The Yes's and I don't care's have it.
We'll be changing servers.
gnif is on the job...
If we didn't tell you, you might not even have noticed, hopefully.

Being from Houston, it looks like my forum loads would be lightening fast for the forum as it is, however, the load times vary all over the place from very fast to sickly slow. With that said, changing hosts means that Gnif will be very busy for a few months getting all of the tweaks done. Meanwhile back in OZ, Dave will be intently trying to learn how to access and manage the new configuration. Does this mean fewer YT videos? I guess we will see. Just to be clear, I did not vote because I think this is strictly a business decision that is Dave's and Dave's alone.

This will not affect Dave's throughput, from his point of view pretty much nothing will change.

[TopLoser]

You were right, whatever they messed up has affected quite a few other people. I’m still seeing dead sites with the same error screen even now.

[Rog520]

FWIW, I have hosted with the (what I assume is the same) "Utah company" for many years now and their service and support is excellent. Granted, my sites have been nowhere near as intensive as eevblog, though.....

[gnif]

FWIW, I have hosted with the (what I assume is the same) "Utah company" for many years now and their service and support is excellent. Granted, my sites have been nowhere near as intensive as eevblog, though.....

If it is the same company, I also have had the same experience with both small and massive scale stuff. Not only has the service been impeccable, but their support has been within minutes. Last week I had a failed disk in a server replaced, it took about 8 minutes from when I sent in the hands on request. This level of service has been consistent over the last 10 years I have had contact with the company.

[hermit]

That GH didn't have monitoring up for their boxes is a bit baffling and tells you what you need to know in terms of 'attention to detail'.  Especially true from a monitoring point when they knew they were making these changes and could potentially break something.  It isn't fun to get those 3AM alerts but that's the business.

[nctnico]

The EEVblog server is currently running on a HostGator dedicated server.
Dual Xeon linux thingo in a basement in Texas somewhere, 1TB RAID1, 32GB, and when operating correctly it handles this forum (which is a big forum) and all the website and other database stuff with ease.
Gnif (https://hostfission.com/ who kindly donates his time to manage this server) has put untold hours into getting this server working correctly and smoothly for this forum and other needs. So apart from the occasional screwup causing some down time, I'm very happy with it, and there are zero complaints from users about speed or anything else.

gnif has asked (again) whether I want to move to another dedicated server he recommends (I won't name them, and PLEASE, I don't want host suggestions), it's based in Utah.

Specs:
Dual Xeon L5630
  96GB RAM
  1 x 2TB HDD
  1 x 120GB SSD
  1 GBps Port
  30TB Monthly
  5 IPs
They also provide IPMI access to the server which means that when outages like this occur gnif is able to directly manage the system without the Data Centers involvement.
He couldn't do anything with the recent HostGator screwup, I had to contact them to reboot the server. (Granted they did it within 10's minutes)
Also, we would ditch cPanel which gnif doesn't like for various reasons I'm sure are valid 

Now normally I am totally adverse to changing web hosts, as IME it is absolutely no guarantee at all that I will have no or fewer problems than Hostgator. The "if it ain't broke, don't fix it" attitude.
And given that there are zero complaints from users about the server, and it only goes down when there is some big screw up, I personally see little value proposition in switching hosts.

As I see it:
Advantages to switching:
- No one will screw with it again, gnif becomes the only manager.
- A bit cheaper (but I don't care about that)
- More resources and potentially faster (although the forum is already lightning quick, so I don't expect to see any real improvement there)
- gnif will be as happy as a pig in mud

Disadvantages
- It will take time and effort
- There will likely be a downtime period and maybe some bugs to fix, either public ones that affect the forum and website, or private ones that only affect my own services (email, ftp etc)
- I will no longer have the knowledge to be able to access it personally, as it won't be running WHM and cPanel which I know fairly well and can actually do stuff with myself (reboots, ftp & email setups, phpMyAdmin database stuff etc). So unless I want to put untold hours into learning some command line thingo, I'll have to rely upon gnif and/or others entirely for anything operational. gnif already handles all the hard stuff, but I like the warm fuzzy knowing I can get into cPanel and WHM and do stuff on my own.

What say you?

What I'm missing here is the question 'what happens if gnif is no longer is able to administer the server?'. IMHO you shouldn't loose the ability to do administrative tasks yourself (or a hired help for that matter) to unburden gnif.

[gnif]

What I'm missing here is the question 'what happens if gnif is no longer is able to administer the server?'. IMHO you shouldn't loose the ability to do administrative tasks yourself (or a hired help for that matter) to unburden gnif.

This was answered, please read through.

[trys]

That GH didn't have monitoring up for their boxes is a bit baffling and tells you what you need to know in terms of 'attention to detail'.  Especially true from a monitoring point when they knew they were making these changes and could potentially break something.  It isn't fun to get those 3AM alerts but that's the business.

Why would they if a customer has a dedicated server (or VM). Application monitoring is down to the customer. They provide power and the network socket and they monitor those. Application monitoring is usually an additional extra.

A monitoring solution searching for a known keyword like "Guest" in the case of this forum would tell you that the webserver is running and that the database is running too. A cheap way of doing it (which would be good enough) would be to use the free uptimerobot.com (you can put check for the existence of a keyword). Anybody can set it up, and you can even set it up to monitor websites that don't even belong to you if you are interested in their reliability for whatever reason.

Even better are the likes of newrelic.com who not only give you server monitoring, but application monitoring and tools to discover what are the bottlenecks. Newrelic is an amazing tool and gives a great insight on how things are performing.

Trys

[gnif]

That GH didn't have monitoring up for their boxes is a bit baffling and tells you what you need to know in terms of 'attention to detail'.  Especially true from a monitoring point when they knew they were making these changes and could potentially break something.  It isn't fun to get those 3AM alerts but that's the business.

Why would they if a customer has a dedicated server (or VM). Application monitoring is down to the customer. They provide power and the network socket and they monitor those. Application monitoring is usually an additional extra.

A monitoring solution searching for a known keyword like "Guest" in the case of this forum would tell you that the webserver is running and that the database is running too. A cheap way of doing it (which would be good enough) would be to use the free uptimerobot.com (you can put check for the existence of a keyword). Anybody can set it up, and you can even set it up to monitor websites that don't even belong to you if you are interested in their reliability for whatever reason.

Even better are the likes of newrelic.com who not only give you server monitoring, but application monitoring and tools to discover what are the bottlenecks. Newrelic is an amazing tool and gives a great insight on how things are performing.

Trys

Nope sorry, they claim to be fully managed which implies at the very least, ping monitoring of the host.. Even OVH perform this on completely unmanaged servers, there is NO excuse in this day and age for a DC to not have that ability for what they claim to be a "managed server".

[james_s]

Given the size of the database and number of users you need reliability. Your current server has been glitchy as all get out in the last 6 months at least, last nights host induced error would be the cherry on top of a shit sandwich if it were me. 

Yeah eevblog has been probably the least reliable website I use, it's possible though that it's the one I use most frequently so whenever a problem does occur I tend to see it.

[trys]

Nope sorry, they claim to be fully managed which implies at the very least, ping monitoring of the host.. Even OVH perform this on completely unmanaged servers, there is NO excuse in this day and age for a DC to not have that ability for what they claim to be a "managed server".

Unfortunately not.

Their managed offering is:

System Monitoring
Maintaining optimal server functionality and health

Proactive: overall network and host health

Reactive: customer environment

Also if they ping a server that has been set up with DHCP they will still get a healthy ping response even if the address is different to what you as the user are using elsewhere to resolve a domain name (in this case cloudflare origin server)

[fourtytwo42]

- gnif will be as happy as a pig in mud

gnif does a brillient job and anything that gets a few tinkerers out of his way gets my vote

[hermit]

Unfortunately not.

Their managed offering is:

System Monitoring
Maintaining optimal server functionality and health

Proactive: overall network and host health

Reactive: customer environment

Also if they ping a server that has been set up with DHCP they will still get a healthy ping response even if the address is different to what you as the user are using elsewhere to resolve a domain name (in this case cloudflare origin server)

So they failed 1 and 2.  Clearly this should have been covered according to this.  Period.  As a managed server it is their responsibility to make sure the machine is up and reachable at a minimum.  They own it. They lease it.  Not much to discuss there.

Check out Nagios.  In addition to a simple ping it can check to see if the host is doing all sorts of stuff like serving up web pages.  It is free.  I've used it and it works well.  Again, this goes back to attention to detail.  It isn't that unusual for a hung machine to respond to pings so that is the MINIMAL check you would expect.  This isn't rocket science.

[trys]

I'm not going to get into semantics here, but no, the server was most probably up and running quite nicely by the sound of it - in as much it was powered up and the operating system was loading and it was responding to ping requests on their network. The chances are the control panel could be accessed too.

The bad thing is though, is the wording of "Proactive: overall network and host health. Reactive: customer environment" is so woolly you could knit a great big wooly jumper out of it, so it's hardly surprising for yourself to see that they have failed, and for me to think they haven't really.

It probably doesn't help either that Cloudflare is pretty pants when a site goes down.

Like I said gnif is doing a great job and I'm sure his arse dropped when he saw that the site was still down. I've been there myself. I feel the pain.

Trys

[gnif]

I'm not going to get into semantics here, but no, the server was most probably up and running quite nicely by the sound of it - in as much it was powered up and the operating system was loading and it was responding to ping requests on their network. The chances are the control panel could be accessed too.

How could it respond to ping?, it has a single Ethernet device, which had lost it's DHCP lease... thus no IP.

[trys]

5) HF had the server configured to use DHCP... this is what broke. Apparently when they moved data centers they switched to static IPs (which is what they should have been using anyway), but we were never notified nor did they "manage" the server to update it. (The dark side of me wonders if they had filtering setup correctly and if it would be possible to configure a rogue DHCP server....)

Ah right. I stand corrected. I'd misinterpreted what the failure mode was there. I took it that a DHCP lease had been dropped and that it was assigned a new static IP address that was not the IP address that you had configured with Cloudflare as the origin server.

[Ampera]

Just to put it out there, I'm a person with a bit of Unix/Linux experience, lots of free time, and the gift of picking stuff up very fast. If you want a quick hand to do something simple, let me know, I'll be happy to help.

[Nusa]

I'm not going to get into semantics here, but no, the server was most probably up and running quite nicely by the sound of it - in as much it was powered up and the operating system was loading and it was responding to ping requests on their network. The chances are the control panel could be accessed too.

How could it respond to ping?, it has a single Ethernet device, which had lost it's DHCP lease... thus no IP.

Actually, I did both a tracert and ping to www.eevblog.com fairly early in the outage and they seemed normal. Felt like a server was there but the host processes were down/broken.

[gnif]

I'm not going to get into semantics here, but no, the server was most probably up and running quite nicely by the sound of it - in as much it was powered up and the operating system was loading and it was responding to ping requests on their network. The chances are the control panel could be accessed too.

How could it respond to ping?, it has a single Ethernet device, which had lost it's DHCP lease... thus no IP.

Actually, I did both a tracert and ping to www.eevblog.com fairly early in the outage and they seemed normal. Felt like a server was there but the host processes were down/broken.

I can confirm without a doubt that the host had lost its IP, there was no breakage of our configuration outside of that. EEVBlog is hosted behind cloudflare, you were pinging their proxies.

[gnif]

I wasn't pinging anything.

I wasn't replying to you

[TiN]

Make your sysadm happy, it will pay off

[trys]

I wasn't pinging anything.

I wasn't replying to you

Whoops. I guessed that.

The server's ip now ends in 143 I'm guessing that wasn't different before the downtime.

[gnif]

I wasn't pinging anything.

I wasn't replying to you

Whoops. I guessed that.

The server's ip now ends in 143 I'm guessing that wasn't different before the downtime.

Correct, the IP has not changed.

[trys]

They really messed up then, didn't they?!

[gnif]

They really messed up then, didn't they?!

Yup, which is why I am not pulling any punches, they should have known better then to rely on DHCP for mission critical infrastructure. For it to fail this bad, their DHCP server had to fail, which makes me wonder how many other servers went offline due to the same fault.

[Ampera]

They really messed up then, didn't they?!

Yup, which is why I am not pulling any punches, they should have known better then to rely on DHCP for mission critical infrastructure. For it to fail this bad, their DHCP server had to fail, which makes me wonder how many other servers went offline due to the same fault.

DHCP is fun to have for the home, but for no bullshit networking, it would be the first thing to go for me.

[capt bullshot]

Just my two cents

I've had some trouble last year with my hoster (hosting my tiny homepage), they had completetly changed the setup and taken away some features - quite a screw up. What HG did sounds a bit like what they did to me (and an unknown number of other customers with simple hosted homepages).
I quit their service, and now I'm running my own tiny server (in my own basement), and no one else (except the free DNS services I'm using) can screw it up. The downside is, I'm now responsible for all the patching and updating stuff on my own, and there's no one else who can maintain it. So if it breaks, I'd have to fix it on my spare time. If the HW fails, it's up to me to repair it. I've no redundancy for email and whatever. I'd rather like to spread the stuff over more servers, but this is all time consuming setup and maintenance and so I don't for now.

So if Gnif is willing to take the workload to maintain the new server, do it. Beware of that single point of failure. That's my opinion.

[hermit]

Just my two cents

I've had some trouble last year with my hoster (hosting my tiny homepage), they had completetly changed the setup and taken away some features - quite a screw up. What HG did sounds a bit like what they did to me (and an unknown number of other customers with simple hosted homepages).
I quit their service, and now I'm running my own tiny server (in my own basement), and no one else (except the free DNS services I'm using) can screw it up. The downside is, I'm now responsible for all the patching and updating stuff on my own, and there's no one else who can maintain it. So if it breaks, I'd have to fix it on my spare time. If the HW fails, it's up to me to repair it. I've no redundancy for email and whatever. I'd rather like to spread the stuff over more servers, but this is all time consuming setup and maintenance and so I don't for now.

So if Gnif is willing to take the workload to maintain the new server, do it. Beware of that single point of failure. That's my opinion.

If your site is small then you could run a vm and then use something like a beaglbone black as a backup device.  You can keep them synced with this.
http://searchdatacenter.techtarget.com/definition/DRBD-Distributed-Replicated-Block-Device
Doesn't solve your time problem but that's the kind of stuff I do and call it fun.

[David Hess]

- gnif will be as happy as a pig in mud

I would maintain a hot local backup on different hardware but if I was relying on gnif, the above would be my upmost consideration.

[Ampera]

- gnif will be as happy as a pig in mud

I would maintain a hot local backup on different hardware but if I was relying on gnif, the above would be my upmost consideration.

Honestly, if it's just 2TB of storage you need, or close to it, I'd throw down the 70 USD (probably around 100 AUD) to grab a bare 2TB drive and keep an all has gone to shit true emergency backup in a closet somewhere.

[Towger]

Utah does not share the same religious views as Dave.  Are there any unique local laws which should be taken into account?

[richnormand]

For what it's worth:

I have been a member of several forums that have grown over the years/decades and at the end they all pretty much all changed to pro type servers, hosting and forum software.
I would go outside the US at this time. I am sure there are competent alternatives with decent protection and security in other locations with sane copywrong and privacy laws.
A pro type approach would also provide mirrors, security and data image backups.
Also a 24/7 team to alert you of issues and take care of stuff.

The hiccups here are noticeable compared to other sites, but considering the growth it is to be expected. Does not mean it should not be addressed with the proper tools.... 

[Ampera]

Utah does not share the same religious views as Dave.  Are there any unique local laws which should be taken into account?

The US has absolutely no laws restricting or dictating the permission of religion anywhere. Laws based on religion are against our constitution and if they are found to exist in some fashion can be struck off the books.

[paulca]

It's up and down like a french whore's knickers.

I'd say find yourself a friendly sys admin rat, feed him some biscuits and coffee and spin up an amazon instance.

[paulca]

Utah does not share the same religious views as Dave.  Are there any unique local laws which should be taken into account?

The US has absolutely no laws restricting or dictating the permission of religion anywhere. Laws based on religion are against our constitution and if they are found to exist in some fashion can be struck off the books.

Or you could just bomb them.

[gnif]

I think this thread has been derailed far enough, please stick to the topic at hand.

[jmelson]

took the server down for 12 hours until I alerted them it was down.

uptimerobot, pingdom.

Changing network settings on any server without asking or warning the owner is not acceptable.
If they do not apologize, or screw up again, you should move.

YES!  You are PAYING for the service, they should run anything other than necessary maintenance by you for approval.    In this case, not a sales business, but can you imagine shutting somebody's web store down for 12 hours?  The owners would be screaming!

I must be crazy, I run a web store out of my basement, and the server is a Dell desktop, running on a cable co. business account.  But, it DOES give me control over the thing.  So, I own my own domain, run my primary DNS, mail server, etc. all by myself.  I just had to update the whole thing to stay compliant with the payment card industry requirements for encryption.   It was a big hassle because the web store software I use needed a number of fixes.  But, it is all back running.

Jon

[paulca]

I must be crazy, I run a web store out of my basement, and the server is a Dell desktop, running on a cable co. business account. 

This works great for small ventures, but it doesn't scale, nor does it provide any resiliency at all.  I've done it for years, but just wait till you get your first "big hit" or someone starts downloading a large file off your server and your internet starts to stutter.  You post a link to a 30Mb project file on a forum and in the next 20 minutes 200 people start downloading it.

I gave up hosting my own public website when bots started downloading media files to index, running my internet connection at 100% 24/7.  I had to routinely ban their IPs to stop them.  That an exploit bots running on windows trying to overflow the IIS buffers... when it was running apache.

I stopped running my own DNS when I was processing about 1000 requests a minute from random internet noise and random bot DOS attacks.

I gave up hosting my own mail server when the amount of spam mail bouncing back out of it and then other servers bouncing the bounces back to me rose to 0.5Gb.  Note that's not 0.5Gb of static mail volume, that was the mail queue size just rejecting the spam.  This actually caused my mail server to get black listed.  I was simply rejecting mail with "Mailbox not found" for 99.9% of the addresses it was receiving at that domain.

I hated getting kicked out of bed on a Saturday morning because my Brother couldn't play online games because my server was eating bandwidth at a high rate because something was indexing a forum I hosted.

etc. etc. etc.

Network and application servers have a tipping point where they go into saturation.  If you can maintain <90% load and <90% bandwidth capacity it all runs smoothly.  If you start peaking out at 101% and 101% it comes crashing down quickly.  Queued requests stack, queued traffic stacks, resource contention and switching results in less performance.  Server comes down to it's knees.

It's all avoidable, but you end up spending way too much time fixing it.

Better to pay someone to do it.  Then when it all gets nasty it's them who has to fix it while you sleep at 2am.

[jmelson]

I must be crazy, I run a web store out of my basement, and the server is a Dell desktop, running on a cable co. business account. 

This works great for small ventures, but it doesn't scale, nor does it provide any resiliency at all.  I've done it for years, but just wait till you get your first "big hit" or someone starts downloading a large file off your server and your internet starts to stutter.  You post a link to a 30Mb project file on a forum and in the next 20 minutes 200 people start downloading it.

Yes, of course, what you say is completely correct.  So far, there has been no such issue, and I don't expect it to show up.  But, for any web site that expects a lot of traffic, then running it out of your hose on a cable modem is NOT going to work.

I gave up hosting my own public website when bots started downloading media files to index, running my internet connection at 100% 24/7.  I had to routinely ban their IPs to stop them.  That an exploit bots running on windows trying to overflow the IIS buffers... when it was running apache.

Yup, every once in a while I have to block Baiduspider in my ip tables, when they start monopolizing my bandwidth.

I gave up hosting my own mail server when the amount of spam mail bouncing back out of it and then other servers bouncing the bounces back to me rose to 0.5Gb.  Note that's not 0.5Gb of static mail volume, that was the mail queue size just rejecting the spam.  This actually caused my mail server to get black listed.  I was simply rejecting mail with "Mailbox not found" for 99.9% of the addresses it was receiving at that domain.

Well, it hasn't gotten to that point yet, but I do get some of these bounces from spam forged with my address.  The spam has actually dropped a fair bit in the last year or so.  Thanks Goodness!

Jon

[paulca]

Well, it hasn't gotten to that point yet, but I do get some of these bounces from spam forged with my address.  The spam has actually dropped a fair bit in the last year or so.  Thanks Goodness!

The way to stop it is the same as you do with a firewall.

Silently ignore it, don't reject it.  Alter the mail server reject rules to just drop the email with no reply.  It stops the "back scatter" loops.  Depending on the mail server you can just terminate the connection if a mail is addressed to an unknown mailbox.

[jasonbrent]

Having said that, though, it would be very unwise to not have someone else who could step in should gnif be indisposed for any reason - or at least a plan of what will need to be done.  Or is this something that has already been addressed?

That has not been addressed.
As long as I have login's then I presume it's likely not hard to find another person at short notice, I've always had many offers of help.

.... a hosting company that goes and changes configuration on a machine I've setup is a hosting company I'd be off of faster than I could "find / -print | cpio -pdmuv | ssh -C...." to a new host.

Trust the admin you've been trusting is my neckbeard advice.

[bitseeker]

The Yes's and I don't care's have it.
We'll be changing servers.
gnif is on the job...
If we didn't tell you, you might not even have noticed, hopefully.

It should be noted that this will take some time too, not only do we have to organize the new servers but I have to schedule the time to configure, deploy and test before migrating to the servers.

The move to a truly dedicated server where you have control over the machine sounds great. GH had a funny notion of what "managed server" means. As for the significant time required for this major change, that's totally understandable and the effort is much appreciated.

[Cerebus]

Actually, I did both a tracert and ping to www.eevblog.com fairly early in the outage and they seemed normal. Felt like a server was there but the host processes were down/broken.

You don't understand the part Cloudflare play in this:

$ dig www.eevblog.com

...

;; ANSWER SECTION:
www.eevblog.com.   125   IN   A   104.31.74.133
www.eevblog.com.   125   IN   A   104.31.75.133

...

$ whois 104.31.74.133
...

NetRange:       104.16.0.0 - 104.31.255.255
CIDR:           104.16.0.0/12
NetName:        CLOUDFLARENET
NetHandle:      NET-104-16-0-0-1
Parent:         NET104 (NET-104-0-0-0-0)
NetType:        Direct Assignment
OriginAS:       AS13335
Organization:   Cloudflare, Inc. (CLOUD14)
RegDate:        2014-03-28
Updated:        2017-02-17
Comment:        All Cloudflare abuse reporting can be done via https://www.cloudflare.com/abuse
Ref:            https://whois.arin.net/rest/net/NET-104-16-0-0-1

You were talking to a Cloudflare proxy, not the actual EEvblog server.

[ez24]

I think gnif should make the decision

[Elasia]

Oh dear god, did i read that right and the site uses hostgator? /puke

Really though this site seems big enough to get hosting donations and use a modern day dual data center high availability setup either active/active or active/passive... or at least a T1 DC Host

[gnif]

Oh dear god, did i read that right and the site uses hostgator? /puke

Really though this site seems big enough to get hosting donations and use a modern day dual data center high availability setup either active/active or active/passive... or at least a T1 DC Host

We are very likely moving to a active/active setup, waiting on final decisions from Dave

[Yellofriend]

Is Hostgator working for you? Their reputation is mixed.

I would probably go for Ramnode and get a dediserver from them. As OS I would go with CentOS, and have everything installed via the high performance https://centminmod.com . If the server must be in Europe I would probably chose https://netcup.de and do the same there.

[ez24]

= If the server must be in Europe I would probably chose https://netcup.de and do the same there.

The server has been selected !  Utah

[Elasia]

= If the server must be in Europe I would probably chose https://netcup.de and do the same there.

The server has been selected !  Utah

Is is being colod with the NSA for maximum bandwidth?

[Elasia]

Oh dear god, did i read that right and the site uses hostgator? /puke

Really though this site seems big enough to get hosting donations and use a modern day dual data center high availability setup either active/active or active/passive... or at least a T1 DC Host

We are very likely moving to a active/active setup, waiting on final decisions from Dave

Thats sweeeeetttt...  just dont park em both in range of L3 on the eastern seaboard LOL

[andybeals]

How about the happy medium? Monitoring on the server. (Why doesn't your current hosting provider do this, tho?) (Sorry, gnif)  Being able to get in and do your thing is good. Having a single point of failure is Very Bad - what if gnif gets hit by a bus after you move?

Sent from my Nexus 6P using Tapatalk

[gnif]

How about the happy medium? Monitoring on the server. (Why doesn't your current hosting provider do this, tho?) (Sorry, gnif)  Being able to get in and do your thing is good. Having a single point of failure is Very Bad - what if gnif gets hit by a bus after you move?

Sent from my Nexus 6P using Tapatalk

Please read through the thread, this has been discussed already.

We are moving to a very standardized setup so that Dave can have someone else on standby should the worst happen to me. The configuration will consist of three servers.

• Stock Standard cPanel Server - For Dave's Convenience, General Hosting, etc.
• Two servers configured with a HTTP stack.

The two servers will very likely be configured using the following (note, this is a quick list off the top of my head at this time):

• Nginx HTTP server + PHP
• MariaDB using Galera to keep the two servers in sync, cPanel server will likely be used as the Arbiter
• Since very little outside of the database changes on disk, `unison` will likely be used to sync the HTTP directories, this simplifies the sync and allows each host to operate independently should the other die, unlike configurations with NFS deployments.
• The two servers will be active/active, keepalived will likely be used to handle automatic failover between the two hosts.
• Server configuration will likely be deployed via Puppet or SaltStack, I have not yet decided. This if done properly will ensure configuration is consistent across both nodes, as well as future proofing for possible growth

This is a pretty common way to configure things, making it possible for any competent admin with fail over hosting experience capable of managing the servers without too much trouble.

[Monkeh]

Can I make a request for the big move? Can we please at long last at least test transitioning the DB to UTF-8 so posts can stop being mangled?

[gnif]

Can I make a request for the big move? Can we please at long last at least test transitioning the DB to UTF-8 so posts can stop being mangled?

The new configuration will give us room to wriggle and test stuff like this, but it wont happen until after this work is completed.

[Monkeh]

Fair enough.

[bitseeker]

The new configuration will give us room to wriggle and test stuff like this,

Excellent!

but it wont happen until after this work is completed.

No worries and worth the wait.

[trys]

How about the happy medium? Monitoring on the server.

I've just noticed gnif's signature.

[Electro Detective]

Aren't there decent hosting server companies here in Australia with decent kit?

Why be at the mercy of a server overseas that can go suss or belly up any time,

or do 'changes' without notice or apology,

and or no one picks up the phone or replies to emails,

and when they do reply it's more or less a "have a nice day"  placebo support...


The U.S. is a politically unstable place at the best of times, why have your bread and butter 'over there' instead of here just in case shtf?

No offence meant to Americans, but would they have their blog server parked in Australia? I don't think so...

Keep it local 

[gnif]

Aren't there decent hosting server companies here in Australia with decent kit?

Why be at the mercy of a server overseas that can go suss or belly up any time,

or do 'changes' without notice or apology,

and or no one picks up the phone or replies to emails,

and when they do reply it's more or less a "have a nice day"  placebo support...


The U.S. is a politically unstable place at the best of times, why have your bread and butter 'over there' instead of here just in case shtf?

No offence meant to Americans, but would they have their blog server parked in Australia? I don't think so...

Keep it local 

This has been discussed to death multiple times in the past,.

• The majority of EEVBlog viewers are in the US
• Bandwidth in Australia is extremely expensive
• The datacenter we are going to use has an excellent reputation of having staff on hand 24/7, it is rare to not have an issue actioned within 5-10 minutes.

[gnif]

Please don't derail this with a political debate.

[amspire]

We are moving to a very standardized setup so that Dave can have someone else on standby should the worst happen to me. The configuration will consist of three servers.

• Stock Standard cPanel Server - For Dave's Convenience, General Hosting, etc.
• Two servers configured with a HTTP stack.

The two servers will very likely be configured using the following (note, this is a quick list off the top of my head at this time):

• Nginx HTTP server + PHP
• MariaDB using Galera to keep the two servers in sync, cPanel server will likely be used as the Arbiter
• Since very little outside of the database changes on disk, `unison` will likely be used to sync the HTTP directories, this simplifies the sync and allows each host to operate independently should the other die, unlike configurations with NFS deployments.
• The two servers will be active/active, keepalived will likely be used to handle automatic failover between the two hosts.
• Server configuration will likely be deployed via Puppet or SaltStack, I have not yet decided. This if done properly will ensure configuration is consistent across both nodes, as well as future proofing for possible growth

This is a pretty common way to configure things, making it possible for any competent admin with fail over hosting experience capable of managing the servers without too much trouble.

This all sounds very good. I was originally worried with the idea that Dave would end up with custom tweaked servers that are only understood by one person, but if you are going to use a tools like Puppet or SaltStack for the configuration, it should be possible to have a documented procedure for quickly rebuilding servers from scratch.

You could make it even easier by using containers such as Docker, but it is one more thing you have to keep going. Containers would probably separate the http/nginx stack, Simple Machines and the database into separate containers with the actual data perhaps with its own location. It does make it extremely easy to backup the actual running containers and to make it trivial to get it running on new PCs/VMs -  even if it is just to have a debugging on a home/office PC that is running exactly the same containers as the Xeon servers.

It could be that Puppet effectively provides a similar speed of recovery and flexability. The actual size of the EEVblog customisation could easily be pretty small, and so the cost of rebuilding may not be significant. I am pretty ignorant of Puppet and SaltStack so I am not sure how over time, you ensure that they can rebuild the server with 100% of the current mods. If Gnif is not there, will rebuilding a server be a task that for the new admin person is a hope-it-works-or-else leap in the dark?

Is MariaDB a bottlekneck at all? If it is handling the load without a problem, then there is probably no need to touch it.

One of the fastest and most stable relational databases for large data is, believe it or not, SQlite3 - provided you have lots of RAM. There is a project http://bedrockdb.com that makes SQlite3 a replicated and geo-redundant dB. It provides MySQL compatibility if required. Bedrock does provide features like automatic fallover to a slave, and automatic recovery back to the master. If DB issues are a bit of a headache, Bedrock is worth a look.

https://twit.tv/shows/floss-weekly/episodes/456

As a matter of interest, how big is the DB and the associated attached files?

Overall though, this is a great move. I think it is time to move to dedicated servers. The redundant server plan sounds great. If there are a few hiccups along the way, so be it.

[gnif]

This all sounds very good. I was originally worried with the idea that Dave would end up with custom tweaked servers that are only understood by one person, but if you are going to use a tools like Puppet or SaltStack for the configuration, it should be possible to have a documented procedure for quickly rebuilding servers from scratch.

This is the plan.

You could make it even easier by using containers such as Docker, but it is one more thing you have to keep going. Containers would probably separate the http/nginx stack, Simple Machines and the database into separate containers with the actual data perhaps with its own location. It does make it extremely easy to backup the actual running containers and to make it trivial to get it running on new PCs/VMs -  even if it is just to have a debugging on a home/office PC that is running exactly the same containers as the Xeon servers.

At this time this would be over-engineering and adds additional unnecessary complexity.

It could be that Puppet effectively provides a similar speed of recovery and flexability. The actual size of the EEVblog customisation could easily be pretty small, and so the cost of rebuilding may not be significant. I am pretty ignorant of Puppet and SaltStack so I am not sure how over time, you ensure that they can rebuild the server with 100% of the current mods. If Gnif is not there, will rebuilding a server be a task that for the new admin person is a hope-it-works-or-else leap in the dark?

A server once bootstrapped can be deployed with puppet in a matter of minutes in a completely automated manner, which is the entire point .

Is MariaDB a bottlekneck at all? If it is handling the load without a problem, then there is probably no need to touch it.

The only changes that will be made here is the replication configuration so we can load balance reads between the servers. SMF was not designed with multiple masters in mind, ie insert ID problems, so writes will have to be directed to a single server, but this will be done with keepalived, not the database.

One of the fastest and most stable relational databases for large data is, believe it or not, SQlite3 - provided you have lots of RAM. There is a project http://bedrockdb.com that makes SQlite3 a replicated and geo-redundant dB. It provides MySQL compatibility if required. Bedrock does provide features like automatic fallover to a slave, and automatic recovery back to the master. If DB issues are a bit of a headache, Bedrock is worth a look.

Changing databases is not an option, we require the feature set of MySQL.

As a matter of interest, how big is the DB and the associated attached files?

On disk the database is currently consuming 10GB, the fileset well over 100GB, but very little data on disk outside of the database changes each day.

[amspire]

If Gnif is not there, will rebuilding a server be a task that for the new admin person is a hope-it-works-or-else leap in the dark?

A server once bootstrapped can be deployed with puppet in a matter of minutes in a completely automated manner, which is the entire point .

If in the course of keeping the server running, you have to update anything, change the settings, add modules or whatever, can you do it first in Puppet and then let Puppet make the changes, or do you have to keep the server and Puppet modified in parallel?

Is MariaDB a bottlekneck at all? If it is handling the load without a problem, then there is probably no need to touch it.

The only changes that will be made here is the replication configuration so we can load balance reads between the servers. SMF was not designed with multiple masters in mind, ie insert ID problems, so writes will have to be directed to a single server, but this will be done with keepalived, not the database.

One of the fastest and most stable relational databases for large data is, believe it or not, SQlite3 - provided you have lots of RAM. There is a project http://bedrockdb.com that makes SQlite3 a replicated and geo-redundant dB. It provides MySQL compatibility if required. Bedrock does provide features like automatic fallover to a slave, and automatic recovery back to the master. If DB issues are a bit of a headache, Bedrock is worth a look.

Changing databases is not an option, we require the feature set of MySQL.

As a matter of interest, how big is the DB and the associated attached files?

On disk the database is currently consuming 10GB, the fileset well over 100GB, but very little data on disk outside of the database changes each day.

Bedrock is MySQL compatible, but I do understand - you have it running on MySQL/MariaDB and so life is easier if you do not change it. It is hard to be 100% compatible.

10GB is not too bad. I was only mentioning Bedrock in case you needed a big speed boost. Bedrock/SQLite3 can hold the whole 10G database in RAM while also writing it to the drives, so it can be exceptionally fast.

I am sure the daily DB changes probably can be synced in seconds, so as long as running Simple Machines off a synced database does not cause stability problems, I do like the plan.

[gnif]

If in the course of keeping the server running, you have to update anything, change the settings, add modules or whatever, can you do it first in Puppet and then let Puppet make the changes, or do you have to keep the server and Puppet modified in parallel?

Puppet applies the changes.

Bedrock is MySQL compatible, but I do understand - you have it running on MySQL/MariaDB and so life is easier if you do not change it. It is hard to be 100% compatible.

10GB is not too bad. I was only mentioning Bedrock in case you needed a big speed boost. Bedrock/SQLite3 can hold the whole 10G database in RAM while also writing it to the drives, so it can be exceptionally fast.

I am sure the daily DB changes probably can be synced in seconds, so as long as running Simple Machines off a synced database does not cause stability problems, I do like the plan.

The DB is not synced daily, every single update is synchronized live, usually in under a millisecond. The delay is in-perceivable.

I think you don't quite get how this will be working. Active/Active means that both servers will be operating in tandem 100% of the time, they will be sharing the load. To make this work both servers need to keep their databases in constant sync, this is absolutely critical for it to operate properly, things such as auto increment IDs would clash otherwise.

As an example of the performance, have a browse of https://forums.realgm.com/boards, this is a similar configuration, but on a database that is 60GB on disk, consisting of nearly 38 million posts across 128,068 members. It absolutely dwarfs the EEV forum, both in data set size and sheer throughput. There is so much traffic on that website that we had to run an internal 10Gbit private LAN for database communications as we were saturating the 2Gbit bonded link the servers had to each other.

The trick to making it perform, keeping the data-set in RAM, just like Bedrock does... MySQL is capable of magic too when configured correctly. We also have enough ram that we can cluster memcached using CouchBase, which is great for performance and reducing database load.

I appreciate the input, but even if Bedrock claimed 100% compatibility it is something that we would have to test extensively over a long period before committing to moving the database across. You also need to note we are not experiencing server/database overload even here at HostGator, most outages are network related, or something screwy with the VM setup that we have (perhaps HG have our storage on a SAN array with inconsistent performance).... All I can say for certain is that we are not on bare metal, and as such we can not confirm any of HG's claims as to how things are configured.

[Zucca]

Last night I had a dream:

May be upgrade and allow more MB and freedom in posting/attaching pictures here? 

Everytime I need to attach a picture here my eyes are rolling up in the sky.

[gnif]

Last night I had a dream:

May be upgrade and allow more MB and freedom in posting/attaching pictures here? 

Everytime I need to attach a picture here my eyes are rolling up in the sky.

We can't lift that restriction, it's not a server limitation but a problem with how SMF handles image thumbnails.

[amspire]

Gnif, I didn't know MySQL could keep data in ram. Interesting to know.

When I mentioned daily sync, the thought in my head was that even a whole day's changes would probably take no time to sync. What I didn't realise is that you could keep the dabases synced within a fraction of a second. Very impressive!

It does seem you have a very solid plan, which is really good for Dave.

Just on the use of Unison, does that mean if one server gets the website hacked, then the hacks are duplicated to the other server?

[gnif]

Just on the use of Unison, does that mean if one server gets the website hacked, then the hacks are duplicated to the other server?

Yes, clustering servers is not for data backup, just like replicating databases doesn't prevent someone running a bad query that drops the data on both servers in a fraction of a second. We will still be running a daily off site backup.

[gnif]

how much more complicated / expensive / etc would it be if dave buys his own hardware to put inside his house to function as a server, assuming there is a low cost way?
and would that possibly be the best way to remove all server issues?

Self-hosting is not only unreliable, but is not an option due to the sheer amount of bandwidth required.

Do the math, leasing servers on a gigabit link with redundant power, fault tolerant connectivity, enterprise grade fire suppression in a HEPA filtered positive pressure environmental controlled environment with 24/7 hands on support and on hand spares at all time for little more then the cost of the monthly power in AU required to run them and guarded physical security/entry/exit to the building

vs.

Self hosting, having to buy 2x the equipment to have spares available, no or very limited power redundancy, on at best a 1Gbit port (if you're very lucky in AU) that costs several hundred dollars a month, times two because you need two for redundancy... etc... etc... etc.

In short, anyone self-hosting something of this scale is just kidding themselves if they think it's better and they are saving money.

[Monkeh]

But it's just a little website, how hard can it be?

People don't see the scale.

[gnif]

I self-host some services, but nothing mission critical, things such as development servers, my personal GitLab server, things that make sense to have locally simply for local bandwidth & convenience reasons. In the past I used to self host much more, but when you go offline due to a small glitch, even something as simple as a rodent/marsupial chewing into some cabling, you quickly realize how frail your connection actually is.

[Cerebus]

..., even something as simple as a rodent/marsupial chewing into some cabling, you quickly realize how frail your connection actually is.

Even if you hand over the money to have guaranteed redundancy on links to you, even from separate suppliers, you can find that they share some point of failure further up the line.

I used to be in the ISP business and to avoid exactly that we used to get physical network maps off fibre and other raw data circuit suppliers (under NDA). Firstly, it was an enormous amount of work, secondly, once you do this exercise you find there are a handful of critical physical points in any nations infrastructure. At the time I was doing this there were two critical physical cable routes in London that more than 90% of the UK's Internet traffic passed through. If your redundant paths went under both those manhole covers you were OK, if they went under the same manhole cover, you were stuffed.

[beanflying]

Last night I had a dream:

May be upgrade and allow more MB and freedom in posting/attaching pictures here? 

Everytime I need to attach a picture here my eyes are rolling up in the sky.

Really bad idea for lots of reasons. Not all the planet has uber broadband. Loading threads with off site images is bad enough now on DSL2.

There are several stable SMF mods that can auto strip images down to size but you can find yourself in a position with a mod (generally built by one person) that doesn't keep compatibility with a patch or major upgrade of the core software.

[gnif]

Last night I had a dream:

May be upgrade and allow more MB and freedom in posting/attaching pictures here? 

Everytime I need to attach a picture here my eyes are rolling up in the sky.

Really bad idea for lots of reasons. Not all the planet has uber broadband. Loading threads with off site images is bad enough now on DSL2.

There are several stable SMF mods that can auto strip images down to size but you can find yourself in a position with a mod (generally built by one person) that doesn't keep compatibility with a patch or major upgrade of the core software.

The main issue isn't the image size, it's how much ram said mods would need to process them. They all assume that they can safely just load the image to create a thumbnail or re sample it to something reasonable. It is very easy to create a 10,000 x 10,000 RGBA PNG that is about 2kb in size, but when loaded into ram will consume 10,000 * 10,000 * 4 = 400000000 bytes (381.47 MB) of RAM. It's a DoS attack waiting to happen, and has happened in the past. A real image resampler should be checking the image size (php: getimagesize) before attempting to load the image, most don't.

[beanflying]

I have used this mod in the past but like I mentioned in my last post 4 years since it was last upgraded. While it will most likely still work on most version 2.0.x SMF most likely will fail at some point.
https://custom.simplemachines.org/mods/index.php?mod=2206

[Brumby]

... if dave buys his own hardware to put inside his house ...

This is all I saw from this post.

A bad idea for a huge number of reasons - only some of which have been mentioned so far ... and I don't know all of them.

Gnif obviously has the expertise as well as having the best interests for the EEVblog.  Not only will I refrain from commenting on how to suck eggs, I'm not going to even suggest what eggs to try.

[Muttley Snickers]

Just a reminder to members that a sometimes overlooked option for communications during an interruption or outage is the EEVblog IRC channel, after Dave's Twitter page this was the first place I went for up to the minute speculation and information in regards to the recent offline event.

[Elasia]

..., even something as simple as a rodent/marsupial chewing into some cabling, you quickly realize how frail your connection actually is.

Even if you hand over the money to have guaranteed redundancy on links to you, even from separate suppliers, you can find that they share some point of failure further up the line.

I used to be in the ISP business and to avoid exactly that we used to get physical network maps off fibre and other raw data circuit suppliers (under NDA). Firstly, it was an enormous amount of work, secondly, once you do this exercise you find there are a handful of critical physical points in any nations infrastructure. At the time I was doing this there were two critical physical cable routes in London that more than 90% of the UK's Internet traffic passed through. If your redundant paths went under both those manhole covers you were OK, if they went under the same manhole cover, you were stuffed.

if you ever get lost in the forest, drop some fiber cable and the nearest backhoe will come and pick you up

State side a lot of choke points follow the old mainline railways and discarded oil and gas pipelines which is kinda neat

Really though... best solution these days, active/active with one on the west and one on the east in T1 grade DCs.  Just have to be able to properly handle network splitting and recombining

[NiHaoMike]

Last night I had a dream:

May be upgrade and allow more MB and freedom in posting/attaching pictures here? 

Everytime I need to attach a picture here my eyes are rolling up in the sky.

Really bad idea for lots of reasons. Not all the planet has uber broadband. Loading threads with off site images is bad enough now on DSL2.

There are several stable SMF mods that can auto strip images down to size but you can find yourself in a position with a mod (generally built by one person) that doesn't keep compatibility with a patch or major upgrade of the core software.

Maybe we should increase the thumbnails to a reasonable size? Like 480p? As it stands right now, the thumbnails are barely even good enough to tell if it is worth opening the full size image.

[gnif]

Last night I had a dream:

May be upgrade and allow more MB and freedom in posting/attaching pictures here? 

Everytime I need to attach a picture here my eyes are rolling up in the sky.

Really bad idea for lots of reasons. Not all the planet has uber broadband. Loading threads with off site images is bad enough now on DSL2.

There are several stable SMF mods that can auto strip images down to size but you can find yourself in a position with a mod (generally built by one person) that doesn't keep compatibility with a patch or major upgrade of the core software.

Maybe we should increase the thumbnails to a reasonable size? Like 480p? As it stands right now, the thumbnails are barely even good enough to tell if it is worth opening the full size image.

Maybe, we need to wait till we are on the new infrastructure for a while so we can collect some metrics to know where we stand.

[tautech]

Last night I had a dream:

May be upgrade and allow more MB and freedom in posting/attaching pictures here? 

Everytime I need to attach a picture here my eyes are rolling up in the sky.

Really bad idea for lots of reasons. Not all the planet has uber broadband. Loading threads with off site images is bad enough now on DSL2.

There are several stable SMF mods that can auto strip images down to size but you can find yourself in a position with a mod (generally built by one person) that doesn't keep compatibility with a patch or major upgrade of the core software.

Maybe we should increase the thumbnails to a reasonable size? Like 480p? As it stands right now, the thumbnails are barely even good enough to tell if it is worth opening the full size image.

Agreed, on 17" or smaller.
Much has to do with a users 'net connection and I always looked at the size before clicking on them.
Now I have a reasonable connection and more importantly NO data cap I click on anything and everything.

Well you might say I'm running a business and I should have a good connection but that's not reality, the landscape here in NZ is changing rapidly and what I have now is cheaper than what I had just a few months ago.

[paulca]

Gnif, I didn't know MySQL could keep data in ram. Interesting to know.

All databases keep data in RAM.  There are multiple layers of caching.  At least:

DBRMS:
Query cache - keeps recent result sets in RAM for instant replay.
Data cache - keeps recently used datasets in RAM for quick query.
OS:
File cache - keeps recently accessed file blocks in RAM.
Hardware:
SATA/SCSI buffers - keeps recent disc reads in DMA buffer RAM.
HD buffers - Keeps recent read sectors in RAM on the physical disk cache.

Of course if the infrastructure is using networked disks or abstracted VM storage discs, SANs and all those other good things there are even more layers of caching.

On Linux writes are not write through unless you force such, writes go to disk cache in memory and then the OS flushes the cache when it pleases or when you ask it to.  If hard transactions are required this can of course be configured in the DB to force synchronous writes.

Often a trap for people with USB memory sticks at home, on linux if you write to the USB drive it appears to happen very, very quickly, but if you pull the memory stick out you will get a surprise that your file isn't actually on it at all.  If you use "umount" or pointy-clicky-draggy-droppy "Eject disc" you will have to wait on the disk cache being synced.  You can enable synchronous writes on a per partition level in fstab though... or just run the "sync" command to sync all discs.

[rrinker]

 I skipped a bunch of pages, but I really like the idea of running VMs - as long as they are properly configured. There are numerous advantages, and properly configured there shouldn't be all that much of a performance hit. Especially when the VM hosting the forum can be bigger than the one it currently runs on. Adding a second, smaller VM to run cPanel and whatnot so Dave still has some visibility and control without having to learn a whole bunch of other server admin stuff he probably has no real interest in is certainly a positive.

[Elasia]

I skipped a bunch of pages, but I really like the idea of running VMs - as long as they are properly configured. There are numerous advantages, and properly configured there shouldn't be all that much of a performance hit. Especially when the VM hosting the forum can be bigger than the one it currently runs on. Adding a second, smaller VM to run cPanel and whatnot so Dave still has some visibility and control without having to learn a whole bunch of other server admin stuff he probably has no real interest in is certainly a positive.

I just installed an emc vxrail not a month ago at work...  once you virtualize there is no going back.. anything else is subpar, only thing for baremetal should be hypervisors and related code stacks, san, firewall, etc etc.  Powershell, PowerCLI, and some perl for good measure.

[Brumby]

What is someone going to hack from this forum?

[Elasia]

What is someone going to hack from this forum?

Valid question but sadly the answer is yes.. its a common form of credential fishing so they can get into other more juicy accounts like your work account and then either look about or use it for a mim attack, or if your really unlucky its a true apt and not a script kiddy and they just lurk there using you as a spring board while staging other things and then they bomb you for extortion once you are worthless to them otherwise.

Always use dedicated passwords for each site/system and get lastpass or similar safe programs to store them all in.  If you like local I use keypass/passwordsafe myself and then less secure ones for browser caching.. lastpass.


The mim attack usually ends up being they want C levels and accounting/purchasing people to trick with invoices and they will even insert themselves right in the middle of a companies processes just so they can insert their own swift codes for wires and then zap... 50k is poof.

[hermit]

What is someone going to hack from this forum?

Valid question but sadly the answer is yes.. <snip>

Honest people have trouble understanding just how devious these folks can be.  That is a big part of the problem.

[Cerebus]

What is someone going to hack from this forum?

The usual, username and passwords that some idiots will have also used for their online banking, email addresses for spam, planting malware for people to inadvertently download as part of the forum's javascript - all the usual stuff.

For the average hacker and flinger of mallware the EEVBlog forum isn't a high-value target, it's just a target in the same way that any online system is. However, as I've said before, for nation-state level hacking here makes a good target. There are a lot of EEs, many EEs work in the 'defence' industry. If you can get an advanced persistent threat into a defence company via some EE you're laughing.

[Elasia]

What is someone going to hack from this forum?

The usual, username and passwords that some idiots will have also used for their online banking, email addresses for spam, planting malware for people to inadvertently download as part of the forum's javascript - all the usual stuff.

For the average hacker and flinger of mallware the EEVBlog forum isn't a high-value target, it's just a target in the same way that any online system is. However, as I've said before, for nation-state level hacking here makes a good target. There are a lot of EEs, many EEs work in the 'defence' industry. If you can get an advanced persistent threat into a defence company via some EE you're laughing.

Happens a lot more than is ever in the press.  I do ITAR work as well and we all get specific training on it.

[paulca]

Always use dedicated passwords for each site/system and get lastpass or similar safe programs to store them all in.  If you like local I use keypass/passwordsafe myself and then less secure ones for browser caching.. lastpass.

It's fine to use the same password for low security services.  Just use more secure ones for stuff like your email or any online shopping stores that have access to your CC/Billing information.

Password managers are just one big basket you put all your eggs into, the are an absolutely prime value target for any hackers or malware.  Not only do they contain all your passwords, but they tell you which site, which user to log in as.  It's like finding someone's wallet with all the credit card and debit cards with the pin number written on the back!

"Oh but they are secure.", Bollox.  If you can open it, so can a malicious program.  Key logger, sniffer, umpteen different vulnerabilities etc. etc.

Use throw away passwords on anything that does not have your full identity or billing information, such as forums.  If they get that password, who really cares.  Put secure passwords on your email, online stores.

Online banking.  If your bank allows you to pick your password, find a better bank.  Mine requires a card reader, two (three?) factor authentication with hardware device to generate a token.  While I can open my account to view on my phone with a finger print scanner if I want to move money outside of my own accounts I need the hardware two factor device to do so.

Also, for things like GMail and Steam make sure and enable their log in guards which block new logins from unknown devices without verifying it was you through other channels such as you phone.

If the hacker gets into your email, you are had.  They can work out where you shop, where you chat, where you bank and they can impresonate you enough to reset most of your passwords to allow them access.

Which reminds me, I was meant to download and dump my GMail box again, so it's not left online in harms way.

[paulca]

As to picking secure passwords that are memorable.  Here are a few times.

Break the password down into multilple blocks.  Select those blocks from lists.  This way they are easy to remember.  Build passwords from a palete.

I use:

A non-nonsensical, non-dictionary pure word.  "Pure" meaning it tells you nothing and contains no information.  Like an old friends nick name or a word you made up as a child.  Make a list of a few of there.  Avoid dictionary words and common names.

A number, one that I use is a video club membership number from way, way back in time.  This number is unrelatable to me and is not recorded anywhere, it's not a date of birth or a house number I have lived at, so it's decoupled and pure.

Some punctuation.  An interesting way to produce this is to just hold down shift while typing a number you will remember. 

Sometimes an incrementor.  However if you are actually using incremental passwords it is often better to put the incrementor somewhere other than the end!  The guy who is claimed to be the origin of the password advice involving changing your password every month and not reusing the last 20 passwords has recently retracted this advice and appologised for it as it makes people use Password1, Password2, Password3 or... it makes them write them down.

[Cerebus]

Password managers are just one big basket you put all your eggs into, the are an absolutely prime value target for any hackers or malware.  Not only do they contain all your passwords, but they tell you which site, which user to log in as.  It's like finding someone's wallet with all the credit card and debit cards with the pin number written on the back!

"Oh but they are secure.", Bollox.  If you can open it, so can a malicious program.  Key logger, sniffer, umpteen different vulnerabilities etc. etc.

A password manager, filled with unguessable, random passwords is a much better bet than trying to keep track of 100s of passwords in your head which will, by dint of needing to be memorable, be much much weaker. Yes, a password manager represents a risk in itself but so does a front door lock, a bank vault or any other item you need to be secure.

Using a password manager is like having a lot of high security locks (highly random passwords) and then keeping all the keys for those in a secure key safe protected by a combination lock. Yes, the safe is a single point of failure but someone has to get to it and break the combination, meantime leaving the individual 'locks' strong in and of themselves until that has happened. Remembering 100s of passwords is like just using cheap low security locks - no one needs to get to the key vault, or break into it, they can just work on the crappy lock in quiet isolation. I know which sounds more secure to me.

Online banking.  If your bank allows you to pick your password, find a better bank. 

What? If your bank doesn't allow you to choose your password, find a better bank. If your bank has had your password, in cleartext, so they can give it to you, or so that they can recreate it at any point, then someone in that bank is in an ideal position to defraud you. The rest, about 2 factor is fine, but if your bank requires you to use a password they supply then you are in a world of trouble; particularity if buried on page 73 of the terms and conditions of the contract you have with them is a clause that says something to the effect "use of your password will be conclusive proof that you did whatever required it and that we have no liability for what happened next".

[paulca]

A password manager, filled with unguessable, random passwords is a much better bet than trying to keep track of 100s of passwords in your head which will, by dint of needing to be memorable, be much much weaker. Yes, a password manager represents a risk in itself but so does a front door lock, a bank vault or any other item you need to be secure.

The problem is that password managers do not just store the password, they usually store the password, the username to use it with and the site to use it on.  That's 3 bits of information and usually all you need to access an account... and then you put all of your accounts together into a convenient list and save it on your computer.

Before they crack open your password file they have no idea what sites you visit.  Once they have access to that file you have a 100% across the board breach.

Here is an analogy.  Would you put a metal box which required a key to open on the outside of you house with all your house, other property, with address cars with location and model, business keys, credit cards including the PIN numbers and all your private financial credentials?  Sure it's secure it's a drill proof, cutting disc proof safe isn't it?  But if they DO get in there your life is 100% stolen almost instantly.  The theif will have access to knick things he could never have dreamed you owned, but you handed it all over to him with the instructions.  All he had to do was break one lock.

What if YOU lose the key to the box?

What? If your bank doesn't allow you to choose your password, find a better bank. If your bank has had your password, in cleartext, so they can give it to you, or so that they can recreate it at any point, then someone in that bank is in an ideal position to defraud you. The rest, about 2 factor is fine, but if your bank requires you to use a password they supply then you are in a world of trouble; particularity if buried on page 73 of the terms and conditions of the contract you have with them is a clause that says something to the effect "use of your password will be conclusive proof that you did whatever required it and that we have no liability for what happened next".

All banks I have been with have provided at least a set of authentication tokens to me.  None have provided the ability for me to choose them.  They do provide a means to request new ones and that requires a whole set of other information I gave them to be verified.  Parts they will send by email, but usually it is followed by either a text message or a physical letter to "activate" the new authentication tokens.  Usually they are in the form of a "identity number" plus a random numeric pass code.  To log in I need the email I used to sign up, plus both the full ID number and random digits from the pass code.  My new bank has the hardware box which requires my bank card and pin code to get the login token which requires my id number and email address.  Yet I cannot change any of that... except the PIN for my card.

When they mail them to me they come on privacy paper, little plastic stickers which can't be read unless you peel them off, showing a "TAMPERED" warning when you do.  THen you have to hold them up to light or against a white background to read them.  They always tell you to memorise then destroy the letter.

[Elasia]

You both have valid points, but what I think both of you have not really touched on and is really the meat and potatoes you are arguing around is.. risk control.

Risk management highly comes into play on this vector and much to what paul says, dont put everything in the same basket you are not willing to risk.  If you got high value accounts, they should be getting higher risk controls to pare down their level of risk.  Take like the bank example, I too use token generators for any account I keep a set level of value in.  Why?  I am not going to risk my golden eggs should I get weed wacked by some jerk off in another land with nothing better to do.

At the same time.. take like this site.. low value but still I dont want to just use any password and if it should get hacked nor do i want a common one floating about.. answer? Lastpass.. its perfectly fit for the level of risk applied and what I will gamble... credit cards? who gives a shit, any decent bank will kill them and return all fraud charges / cancel them.  Now.. debit card? Hell no I dont ever use them and lock em up... why? Its a direct line into my main funds.  Each person must figure out what they are willing to accept. 

Now for the rub... the bulk of people on this planet do not understand risk management.  This to me should be a course taught in high school or collage instead of bull shit people will never use in their career track.  Also along with basic financial management.. checking, savings, etc.

[Cerebus]

ould you put a metal box which required a key to open on the outside of you house with all your house, other property, with address cars with location and model, business keys, credit cards including the PIN numbers and all your private financial credentials?  Sure it's secure it's a drill proof, cutting disc proof safe isn't it?  But if they DO get in there your life is 100% stolen almost instantly.  The theif will have access to knick things he could never have dreamed you owned, but you handed it all over to him with the instructions.  All he had to do was break one lock.

You completely dodge the point that if you don't use a password manager then you must use relatively weak passwords because they must be memorable as must the user identifiers. One of my password stores contains 98 passwords, the other contains 110. They almost all have completely random passwords with 48 bits minimum entropy and most of the user identifiers are also random strings (or email address with random user parts) with a minimum of 48 bits of entropy. That's a lot of entropy, certainly more that anyone normal could remember.

A password manager, on a machine I control, behind a firewall that I control is acceptably secure. If someone can get to that then they can install a keylogger, and at that point no password alone is secure and you're stuffed anyway. So what if they get it all at once, or have to wait a few days to watch me type it. But until and if that happens my userids and passwords out there on machines I can't control are secure because they are random and unique, certainly more so that any I or you could remember.

All the people I know or know of in the infosec business recommend using a password manager, and wherever possible using non-memorable random passwords. Bruce Schneier even once went so far as to say that you should write strong passwords down on paper in preference to using ones that are memorable but weak.

[hermit]

Passwords and managers are kinda outside the thread topic.  Server security is a valid question but @gnif has demonstrated to me that he probably has a better handle on this than most.  Hopefully the people doing the forum software are doing what they can to keep it up to date on software vulnerabilities.

[paulca]

I'd advise keeping your "current account", as current funds.  "To pay the bills this month and provide my disposable income for the month".  Not your main funds.

At the end of each month I move the residual balance to savings.  So the maximum available in my current account is one months salary.  So even after stealing my debit card they go on a shopping spree before I get to cancel it AND the bank refuse to honor their protection of same, I loose at most the remainder of my current months salary.

I have no direct access to savings to withdraw, I have to transfer it to a current account to access it.

[Elasia]

I'd advise keeping your "current account", as current funds.  "To pay the bills this month and provide my disposable income for the month".  Not your main funds.

At the end of each month I move the residual balance to savings.  So the maximum available in my current account is one months salary.  So even after stealing my debit card they go on a shopping spree before I get to cancel it AND the bank refuse to honor their protection of same, I loose at most the remainder of my current months salary.

I have no direct access to savings to withdraw, I have to transfer it to a current account to access it.

Mine is in a very similar scheme, it works well, cant complain

[Elasia]

ould you put a metal box which required a key to open on the outside of you house with all your house, other property, with address cars with location and model, business keys, credit cards including the PIN numbers and all your private financial credentials?  Sure it's secure it's a drill proof, cutting disc proof safe isn't it?  But if they DO get in there your life is 100% stolen almost instantly.  The theif will have access to knick things he could never have dreamed you owned, but you handed it all over to him with the instructions.  All he had to do was break one lock.

You completely dodge the point that if you don't use a password manager then you must use relatively weak passwords because they must be memorable as must the user identifiers. One of my password stores contains 98 passwords, the other contains 110. They almost all have completely random passwords with 48 bits minimum entropy and most of the user identifiers are also random strings (or email address with random user parts) with a minimum of 48 bits of entropy. That's a lot of entropy, certainly more that anyone normal could remember.

A password manager, on a machine I control, behind a firewall that I control is acceptably secure. If someone can get to that then they can install a keylogger, and at that point no password alone is secure and you're stuffed anyway. So what if they get it all at once, or have to wait a few days to watch me type it. But until and if that happens my userids and passwords out there on machines I can't control are secure because they are random and unique, certainly more so that any I or you could remember.

All the people I know or know of in the infosec business recommend using a password manager, and wherever possible using non-memorable random passwords. Bruce Schneier even once went so far as to say that you should write strong passwords down on paper in preference to using ones that are memorable but weak.

Actually that "then you must use relatively weak passwords because they must be memorable as must the user identifiers"  is wrong.  There is a way and newer papers have been coming out on it.  You use random but associable words along with a symbol/number or two mixed in.  This produces very high entropy while still being able to be used by most humans.  Really good read if you search for it.  I used to think like you do..  now this does have a flaw in in big data can really weaken it due to predicting human patterns at large.. kinda like Hunter2 but if you can train people away from those patterns and into using a more random but remember-able state.. it does work wonders.  Least for now.

[Mr. Scram]

Actually that "then you must use relatively weak passwords because they must be memorable as must the user identifiers"  is wrong.  There is a way and newer papers have been coming out on it.  You use random but associable words along with a symbol/number or two mixed in.  This produces very high entropy while still being able to be used by most humans.  Really good read if you search for it.  I used to think like you do..  now this does have a flaw in in big data can really weaken it due to predicting human patterns at large.. kinda like Hunter2 but if you can train people away from those patterns and into using a more random but remember-able state.. it does work wonders.  Least for now.

Passphrases aren't ideal. A dictionary attack will greatly reduce the entropy of one and the fact that people permutate words in fairly predictable ways only exacerbates the issue.

The underlying problem is that people are terrible at being properly random. On paper a passphrase yields a huge entropy, but human behaviour ruins that.

[Zucca]

Any updates?

[gnif]

We should be going live later this week, I have had to give priority to paid work.

[Ampera]

We should be going live later this week, I have had to give priority to paid work.

Excited to see how it all works out. Good work gnif!

[kulla]

Holding thumbs crossed that migration goes well, as I know how migrations works

I know you don't need it but in case you need any help from fellow sysadmin regarding nginx or anything else I'm all yours.

[helius]

You completely dodge the point that if you don't use a password manager then you must use relatively weak passwords because they must be memorable as must the user identifiers. One of my password stores contains 98 passwords, the other contains 110. They almost all have completely random passwords with 48 bits minimum entropy and most of the user identifiers are also random strings (or email address with random user parts) with a minimum of 48 bits of entropy. That's a lot of entropy, certainly more that anyone normal could remember.

"Must" is too strong. Have you never seen one of these?

You don't even have to write down the random letters you use, you can use letters picked (systematically) from a novel or other book. The mind is a critical keystore (it's the only one that requires your volition to unlock it) but it should be used to key into larger sets of data.

[Cerebus]

You completely dodge the point that if you don't use a password manager then you must use relatively weak passwords because they must be memorable as must the user identifiers. One of my password stores contains 98 passwords, the other contains 110. They almost all have completely random passwords with 48 bits minimum entropy and most of the user identifiers are also random strings (or email address with random user parts) with a minimum of 48 bits of entropy. That's a lot of entropy, certainly more that anyone normal could remember.

"Must" is too strong. Have you never seen one of these?

You don't even have to write down the random letters you use, you can use letters picked (systematically) from a novel or other book. The mind is a critical keystore (it's the only one that requires your volition to unlock it) but it should be used to key into larger sets of data.

How strange to post an image of a notebook, but to omit to quote the last paragraph of what I wrote:

All the people I know or know of in the infosec business recommend using a password manager, and wherever possible using non-memorable random passwords. Bruce Schneier even once went so far as to say that you should write strong passwords down on paper in preference to using ones that are memorable but weak.

In case you don't recognise the name, Bruce is possibly the foremost cryptologist regularly writing for consumption by the general technical community. He was one of the submitters, and finalists, for both NIST competitions that led to AES and SHA-3. When someone of that calibre offers advice I'm going to listen to them.

[helius]

I subscribe to the belief that you needn't read everything someone has ever said to understand one sentence. The consequence of this belief is that if you start your message with a falsehood, I will not read it to the end.

[Cerebus]

I subscribe to the belief that you needn't read everything someone has ever said to understand one sentence. The consequence of this belief is that if you start your message with a falsehood, I will not read it to the end.

  Unbelievable ...

[Zucca]

Are we now riding on the new beast?

[Monkeh]

Are we now riding on the new beast?

For some time. Only minor bumps in the road.

[bitseeker]

Excellent!