POLL: Should I change servers?

[Elasia]

Oh dear god, did i read that right and the site uses hostgator? /puke

Really though this site seems big enough to get hosting donations and use a modern day dual data center high availability setup either active/active or active/passive... or at least a T1 DC Host

We are very likely moving to a active/active setup, waiting on final decisions from Dave

Thats sweeeeetttt...  just dont park em both in range of L3 on the eastern seaboard LOL

[andybeals]

How about the happy medium? Monitoring on the server. (Why doesn't your current hosting provider do this, tho?) (Sorry, gnif)  Being able to get in and do your thing is good. Having a single point of failure is Very Bad - what if gnif gets hit by a bus after you move?

Sent from my Nexus 6P using Tapatalk

[gnif]

How about the happy medium? Monitoring on the server. (Why doesn't your current hosting provider do this, tho?) (Sorry, gnif)  Being able to get in and do your thing is good. Having a single point of failure is Very Bad - what if gnif gets hit by a bus after you move?

Sent from my Nexus 6P using Tapatalk

Please read through the thread, this has been discussed already.

We are moving to a very standardized setup so that Dave can have someone else on standby should the worst happen to me. The configuration will consist of three servers.

• Stock Standard cPanel Server - For Dave's Convenience, General Hosting, etc.
• Two servers configured with a HTTP stack.

The two servers will very likely be configured using the following (note, this is a quick list off the top of my head at this time):

• Nginx HTTP server + PHP
• MariaDB using Galera to keep the two servers in sync, cPanel server will likely be used as the Arbiter
• Since very little outside of the database changes on disk, `unison` will likely be used to sync the HTTP directories, this simplifies the sync and allows each host to operate independently should the other die, unlike configurations with NFS deployments.
• The two servers will be active/active, keepalived will likely be used to handle automatic failover between the two hosts.
• Server configuration will likely be deployed via Puppet or SaltStack, I have not yet decided. This if done properly will ensure configuration is consistent across both nodes, as well as future proofing for possible growth

This is a pretty common way to configure things, making it possible for any competent admin with fail over hosting experience capable of managing the servers without too much trouble.

[Monkeh]

Can I make a request for the big move? Can we please at long last at least test transitioning the DB to UTF-8 so posts can stop being mangled?

[gnif]

Can I make a request for the big move? Can we please at long last at least test transitioning the DB to UTF-8 so posts can stop being mangled?

The new configuration will give us room to wriggle and test stuff like this, but it wont happen until after this work is completed.

[Monkeh]

Fair enough.

[bitseeker]

The new configuration will give us room to wriggle and test stuff like this,

Excellent!

but it wont happen until after this work is completed.

No worries and worth the wait.

[trys]

How about the happy medium? Monitoring on the server.

I've just noticed gnif's signature.

[Electro Detective]

Aren't there decent hosting server companies here in Australia with decent kit?

Why be at the mercy of a server overseas that can go suss or belly up any time,

or do 'changes' without notice or apology,

and or no one picks up the phone or replies to emails,

and when they do reply it's more or less a "have a nice day"  placebo support...


The U.S. is a politically unstable place at the best of times, why have your bread and butter 'over there' instead of here just in case shtf?

No offence meant to Americans, but would they have their blog server parked in Australia? I don't think so...

Keep it local 

[gnif]

Aren't there decent hosting server companies here in Australia with decent kit?

Why be at the mercy of a server overseas that can go suss or belly up any time,

or do 'changes' without notice or apology,

and or no one picks up the phone or replies to emails,

and when they do reply it's more or less a "have a nice day"  placebo support...


The U.S. is a politically unstable place at the best of times, why have your bread and butter 'over there' instead of here just in case shtf?

No offence meant to Americans, but would they have their blog server parked in Australia? I don't think so...

Keep it local 

This has been discussed to death multiple times in the past,.

• The majority of EEVBlog viewers are in the US
• Bandwidth in Australia is extremely expensive
• The datacenter we are going to use has an excellent reputation of having staff on hand 24/7, it is rare to not have an issue actioned within 5-10 minutes.

[gnif]

Please don't derail this with a political debate.

[amspire]

We are moving to a very standardized setup so that Dave can have someone else on standby should the worst happen to me. The configuration will consist of three servers.

• Stock Standard cPanel Server - For Dave's Convenience, General Hosting, etc.
• Two servers configured with a HTTP stack.

The two servers will very likely be configured using the following (note, this is a quick list off the top of my head at this time):

• Nginx HTTP server + PHP
• MariaDB using Galera to keep the two servers in sync, cPanel server will likely be used as the Arbiter
• Since very little outside of the database changes on disk, `unison` will likely be used to sync the HTTP directories, this simplifies the sync and allows each host to operate independently should the other die, unlike configurations with NFS deployments.
• The two servers will be active/active, keepalived will likely be used to handle automatic failover between the two hosts.
• Server configuration will likely be deployed via Puppet or SaltStack, I have not yet decided. This if done properly will ensure configuration is consistent across both nodes, as well as future proofing for possible growth

This is a pretty common way to configure things, making it possible for any competent admin with fail over hosting experience capable of managing the servers without too much trouble.

This all sounds very good. I was originally worried with the idea that Dave would end up with custom tweaked servers that are only understood by one person, but if you are going to use a tools like Puppet or SaltStack for the configuration, it should be possible to have a documented procedure for quickly rebuilding servers from scratch.

You could make it even easier by using containers such as Docker, but it is one more thing you have to keep going. Containers would probably separate the http/nginx stack, Simple Machines and the database into separate containers with the actual data perhaps with its own location. It does make it extremely easy to backup the actual running containers and to make it trivial to get it running on new PCs/VMs -  even if it is just to have a debugging on a home/office PC that is running exactly the same containers as the Xeon servers.

It could be that Puppet effectively provides a similar speed of recovery and flexability. The actual size of the EEVblog customisation could easily be pretty small, and so the cost of rebuilding may not be significant. I am pretty ignorant of Puppet and SaltStack so I am not sure how over time, you ensure that they can rebuild the server with 100% of the current mods. If Gnif is not there, will rebuilding a server be a task that for the new admin person is a hope-it-works-or-else leap in the dark?

Is MariaDB a bottlekneck at all? If it is handling the load without a problem, then there is probably no need to touch it.

One of the fastest and most stable relational databases for large data is, believe it or not, SQlite3 - provided you have lots of RAM. There is a project http://bedrockdb.com that makes SQlite3 a replicated and geo-redundant dB. It provides MySQL compatibility if required. Bedrock does provide features like automatic fallover to a slave, and automatic recovery back to the master. If DB issues are a bit of a headache, Bedrock is worth a look.

https://twit.tv/shows/floss-weekly/episodes/456

As a matter of interest, how big is the DB and the associated attached files?

Overall though, this is a great move. I think it is time to move to dedicated servers. The redundant server plan sounds great. If there are a few hiccups along the way, so be it.

[gnif]

This all sounds very good. I was originally worried with the idea that Dave would end up with custom tweaked servers that are only understood by one person, but if you are going to use a tools like Puppet or SaltStack for the configuration, it should be possible to have a documented procedure for quickly rebuilding servers from scratch.

This is the plan.

You could make it even easier by using containers such as Docker, but it is one more thing you have to keep going. Containers would probably separate the http/nginx stack, Simple Machines and the database into separate containers with the actual data perhaps with its own location. It does make it extremely easy to backup the actual running containers and to make it trivial to get it running on new PCs/VMs -  even if it is just to have a debugging on a home/office PC that is running exactly the same containers as the Xeon servers.

At this time this would be over-engineering and adds additional unnecessary complexity.

It could be that Puppet effectively provides a similar speed of recovery and flexability. The actual size of the EEVblog customisation could easily be pretty small, and so the cost of rebuilding may not be significant. I am pretty ignorant of Puppet and SaltStack so I am not sure how over time, you ensure that they can rebuild the server with 100% of the current mods. If Gnif is not there, will rebuilding a server be a task that for the new admin person is a hope-it-works-or-else leap in the dark?

A server once bootstrapped can be deployed with puppet in a matter of minutes in a completely automated manner, which is the entire point .

Is MariaDB a bottlekneck at all? If it is handling the load without a problem, then there is probably no need to touch it.

The only changes that will be made here is the replication configuration so we can load balance reads between the servers. SMF was not designed with multiple masters in mind, ie insert ID problems, so writes will have to be directed to a single server, but this will be done with keepalived, not the database.

One of the fastest and most stable relational databases for large data is, believe it or not, SQlite3 - provided you have lots of RAM. There is a project http://bedrockdb.com that makes SQlite3 a replicated and geo-redundant dB. It provides MySQL compatibility if required. Bedrock does provide features like automatic fallover to a slave, and automatic recovery back to the master. If DB issues are a bit of a headache, Bedrock is worth a look.

Changing databases is not an option, we require the feature set of MySQL.

As a matter of interest, how big is the DB and the associated attached files?

On disk the database is currently consuming 10GB, the fileset well over 100GB, but very little data on disk outside of the database changes each day.

[amspire]

If Gnif is not there, will rebuilding a server be a task that for the new admin person is a hope-it-works-or-else leap in the dark?

A server once bootstrapped can be deployed with puppet in a matter of minutes in a completely automated manner, which is the entire point .

If in the course of keeping the server running, you have to update anything, change the settings, add modules or whatever, can you do it first in Puppet and then let Puppet make the changes, or do you have to keep the server and Puppet modified in parallel?

Is MariaDB a bottlekneck at all? If it is handling the load without a problem, then there is probably no need to touch it.

The only changes that will be made here is the replication configuration so we can load balance reads between the servers. SMF was not designed with multiple masters in mind, ie insert ID problems, so writes will have to be directed to a single server, but this will be done with keepalived, not the database.

One of the fastest and most stable relational databases for large data is, believe it or not, SQlite3 - provided you have lots of RAM. There is a project http://bedrockdb.com that makes SQlite3 a replicated and geo-redundant dB. It provides MySQL compatibility if required. Bedrock does provide features like automatic fallover to a slave, and automatic recovery back to the master. If DB issues are a bit of a headache, Bedrock is worth a look.

Changing databases is not an option, we require the feature set of MySQL.

As a matter of interest, how big is the DB and the associated attached files?

On disk the database is currently consuming 10GB, the fileset well over 100GB, but very little data on disk outside of the database changes each day.

Bedrock is MySQL compatible, but I do understand - you have it running on MySQL/MariaDB and so life is easier if you do not change it. It is hard to be 100% compatible.

10GB is not too bad. I was only mentioning Bedrock in case you needed a big speed boost. Bedrock/SQLite3 can hold the whole 10G database in RAM while also writing it to the drives, so it can be exceptionally fast.

I am sure the daily DB changes probably can be synced in seconds, so as long as running Simple Machines off a synced database does not cause stability problems, I do like the plan.

[gnif]

If in the course of keeping the server running, you have to update anything, change the settings, add modules or whatever, can you do it first in Puppet and then let Puppet make the changes, or do you have to keep the server and Puppet modified in parallel?

Puppet applies the changes.

Bedrock is MySQL compatible, but I do understand - you have it running on MySQL/MariaDB and so life is easier if you do not change it. It is hard to be 100% compatible.

10GB is not too bad. I was only mentioning Bedrock in case you needed a big speed boost. Bedrock/SQLite3 can hold the whole 10G database in RAM while also writing it to the drives, so it can be exceptionally fast.

I am sure the daily DB changes probably can be synced in seconds, so as long as running Simple Machines off a synced database does not cause stability problems, I do like the plan.

The DB is not synced daily, every single update is synchronized live, usually in under a millisecond. The delay is in-perceivable.

I think you don't quite get how this will be working. Active/Active means that both servers will be operating in tandem 100% of the time, they will be sharing the load. To make this work both servers need to keep their databases in constant sync, this is absolutely critical for it to operate properly, things such as auto increment IDs would clash otherwise.

As an example of the performance, have a browse of https://forums.realgm.com/boards, this is a similar configuration, but on a database that is 60GB on disk, consisting of nearly 38 million posts across 128,068 members. It absolutely dwarfs the EEV forum, both in data set size and sheer throughput. There is so much traffic on that website that we had to run an internal 10Gbit private LAN for database communications as we were saturating the 2Gbit bonded link the servers had to each other.

The trick to making it perform, keeping the data-set in RAM, just like Bedrock does... MySQL is capable of magic too when configured correctly. We also have enough ram that we can cluster memcached using CouchBase, which is great for performance and reducing database load.

I appreciate the input, but even if Bedrock claimed 100% compatibility it is something that we would have to test extensively over a long period before committing to moving the database across. You also need to note we are not experiencing server/database overload even here at HostGator, most outages are network related, or something screwy with the VM setup that we have (perhaps HG have our storage on a SAN array with inconsistent performance).... All I can say for certain is that we are not on bare metal, and as such we can not confirm any of HG's claims as to how things are configured.

[Zucca]

Last night I had a dream:

May be upgrade and allow more MB and freedom in posting/attaching pictures here? 

Everytime I need to attach a picture here my eyes are rolling up in the sky.

[gnif]

Last night I had a dream:

May be upgrade and allow more MB and freedom in posting/attaching pictures here? 

Everytime I need to attach a picture here my eyes are rolling up in the sky.

We can't lift that restriction, it's not a server limitation but a problem with how SMF handles image thumbnails.

[amspire]

Gnif, I didn't know MySQL could keep data in ram. Interesting to know.

When I mentioned daily sync, the thought in my head was that even a whole day's changes would probably take no time to sync. What I didn't realise is that you could keep the dabases synced within a fraction of a second. Very impressive!

It does seem you have a very solid plan, which is really good for Dave.

Just on the use of Unison, does that mean if one server gets the website hacked, then the hacks are duplicated to the other server?

[gnif]

Just on the use of Unison, does that mean if one server gets the website hacked, then the hacks are duplicated to the other server?

Yes, clustering servers is not for data backup, just like replicating databases doesn't prevent someone running a bad query that drops the data on both servers in a fraction of a second. We will still be running a daily off site backup.

[gnif]

how much more complicated / expensive / etc would it be if dave buys his own hardware to put inside his house to function as a server, assuming there is a low cost way?
and would that possibly be the best way to remove all server issues?

Self-hosting is not only unreliable, but is not an option due to the sheer amount of bandwidth required.

Do the math, leasing servers on a gigabit link with redundant power, fault tolerant connectivity, enterprise grade fire suppression in a HEPA filtered positive pressure environmental controlled environment with 24/7 hands on support and on hand spares at all time for little more then the cost of the monthly power in AU required to run them and guarded physical security/entry/exit to the building

vs.

Self hosting, having to buy 2x the equipment to have spares available, no or very limited power redundancy, on at best a 1Gbit port (if you're very lucky in AU) that costs several hundred dollars a month, times two because you need two for redundancy... etc... etc... etc.

In short, anyone self-hosting something of this scale is just kidding themselves if they think it's better and they are saving money.

[Monkeh]

But it's just a little website, how hard can it be?

People don't see the scale.

[gnif]

I self-host some services, but nothing mission critical, things such as development servers, my personal GitLab server, things that make sense to have locally simply for local bandwidth & convenience reasons. In the past I used to self host much more, but when you go offline due to a small glitch, even something as simple as a rodent/marsupial chewing into some cabling, you quickly realize how frail your connection actually is.

[Cerebus]

..., even something as simple as a rodent/marsupial chewing into some cabling, you quickly realize how frail your connection actually is.

Even if you hand over the money to have guaranteed redundancy on links to you, even from separate suppliers, you can find that they share some point of failure further up the line.

I used to be in the ISP business and to avoid exactly that we used to get physical network maps off fibre and other raw data circuit suppliers (under NDA). Firstly, it was an enormous amount of work, secondly, once you do this exercise you find there are a handful of critical physical points in any nations infrastructure. At the time I was doing this there were two critical physical cable routes in London that more than 90% of the UK's Internet traffic passed through. If your redundant paths went under both those manhole covers you were OK, if they went under the same manhole cover, you were stuffed.

[beanflying]

Last night I had a dream:

May be upgrade and allow more MB and freedom in posting/attaching pictures here? 

Everytime I need to attach a picture here my eyes are rolling up in the sky.

Really bad idea for lots of reasons. Not all the planet has uber broadband. Loading threads with off site images is bad enough now on DSL2.

There are several stable SMF mods that can auto strip images down to size but you can find yourself in a position with a mod (generally built by one person) that doesn't keep compatibility with a patch or major upgrade of the core software.

[gnif]

Last night I had a dream:

May be upgrade and allow more MB and freedom in posting/attaching pictures here? 

Everytime I need to attach a picture here my eyes are rolling up in the sky.

Really bad idea for lots of reasons. Not all the planet has uber broadband. Loading threads with off site images is bad enough now on DSL2.

There are several stable SMF mods that can auto strip images down to size but you can find yourself in a position with a mod (generally built by one person) that doesn't keep compatibility with a patch or major upgrade of the core software.

The main issue isn't the image size, it's how much ram said mods would need to process them. They all assume that they can safely just load the image to create a thumbnail or re sample it to something reasonable. It is very easy to create a 10,000 x 10,000 RGBA PNG that is about 2kb in size, but when loaded into ram will consume 10,000 * 10,000 * 4 = 400000000 bytes (381.47 MB) of RAM. It's a DoS attack waiting to happen, and has happened in the past. A real image resampler should be checking the image size (php: getimagesize) before attempting to load the image, most don't.