Poll

Should the EEVblog Forum change servers?

Yes
76 (46.9%)
No
13 (8%)
I don't care, just don't screw it up.
73 (45.1%)

Total Members Voted: 162

Author Topic: POLL: Should I change servers?  (Read 7324 times)

0 Members and 1 Guest are viewing this topic.

Offline Elasia

  • Regular Contributor
  • *
  • Posts: 157
  • Country: us
Re: POLL: Should I change servers?
« Reply #100 on: March 13, 2018, 04:19:32 pm »
= If the server must be in Europe I would probably chose https://netcup.de and do the same there.

The server has been selected !  Utah

Is is being colod with the NSA for maximum bandwidth? :D
 

Offline Elasia

  • Regular Contributor
  • *
  • Posts: 157
  • Country: us
Re: POLL: Should I change servers?
« Reply #101 on: March 13, 2018, 04:22:24 pm »
Oh dear god, did i read that right and the site uses hostgator? /puke

Really though this site seems big enough to get hosting donations and use a modern day dual data center high availability setup either active/active or active/passive... or at least a T1 DC Host

We are very likely moving to a active/active setup, waiting on final decisions from Dave :)

Thats sweeeeetttt...  just dont park em both in range of L3 on the eastern seaboard LOL
 

Offline andybeals

  • Newbie
  • Posts: 2
  • Country: us
Re: POLL: Should I change servers?
« Reply #102 on: March 14, 2018, 01:50:38 pm »
How about the happy medium? Monitoring on the server. (Why doesn't your current hosting provider do this, tho?) (Sorry, gnif)  Being able to get in and do your thing is good. Having a single point of failure is Very Bad - what if gnif gets hit by a bus after you move?

Sent from my Nexus 6P using Tapatalk

 

Offline gnif

  • Administrator
  • *****
  • Posts: 1043
  • Country: au
Re: POLL: Should I change servers?
« Reply #103 on: March 14, 2018, 02:05:43 pm »
How about the happy medium? Monitoring on the server. (Why doesn't your current hosting provider do this, tho?) (Sorry, gnif)  Being able to get in and do your thing is good. Having a single point of failure is Very Bad - what if gnif gets hit by a bus after you move?

Sent from my Nexus 6P using Tapatalk

Please read through the thread, this has been discussed already.

We are moving to a very standardized setup so that Dave can have someone else on standby should the worst happen to me. The configuration will consist of three servers.

  • Stock Standard cPanel Server - For Dave's Convenience, General Hosting, etc.
  • Two servers configured with a HTTP stack.

The two servers will very likely be configured using the following (note, this is a quick list off the top of my head at this time):

  • Nginx HTTP server + PHP
  • MariaDB using Galera to keep the two servers in sync, cPanel server will likely be used as the Arbiter
  • Since very little outside of the database changes on disk, `unison` will likely be used to sync the HTTP directories, this simplifies the sync and allows each host to operate independently should the other die, unlike configurations with NFS deployments.
  • The two servers will be active/active, keepalived will likely be used to handle automatic failover between the two hosts.
  • Server configuration will likely be deployed via Puppet or SaltStack, I have not yet decided. This if done properly will ensure configuration is consistent across both nodes, as well as future proofing for possible growth

This is a pretty common way to configure things, making it possible for any competent admin with fail over hosting experience capable of managing the servers without too much trouble.
« Last Edit: March 14, 2018, 02:08:22 pm by gnif »
HostFission - Full Server Monitoring and Management Solutions.
https://hostfission.com/
https://twitter.com/HostFission

I volunteer my time to manage this server, if you would like to support this work I have a patreon here:
https://www.patreon.com/gnif
 
The following users thanked this post: EEVblog, ez24

Offline Monkeh

  • Super Contributor
  • ***
  • Posts: 5607
  • Country: gb
Re: POLL: Should I change servers?
« Reply #104 on: March 14, 2018, 03:22:24 pm »
Can I make a request for the big move? Can we please at long last at least test transitioning the DB to UTF-8 so posts can stop being mangled?
 

Offline gnif

  • Administrator
  • *****
  • Posts: 1043
  • Country: au
Re: POLL: Should I change servers?
« Reply #105 on: March 14, 2018, 03:27:33 pm »
Can I make a request for the big move? Can we please at long last at least test transitioning the DB to UTF-8 so posts can stop being mangled?

The new configuration will give us room to wriggle and test stuff like this, but it wont happen until after this work is completed.
HostFission - Full Server Monitoring and Management Solutions.
https://hostfission.com/
https://twitter.com/HostFission

I volunteer my time to manage this server, if you would like to support this work I have a patreon here:
https://www.patreon.com/gnif
 

Offline Monkeh

  • Super Contributor
  • ***
  • Posts: 5607
  • Country: gb
Re: POLL: Should I change servers?
« Reply #106 on: March 14, 2018, 03:28:20 pm »
Fair enough.
 

Offline bitseeker

  • Super Contributor
  • ***
  • Posts: 6182
  • Country: us
  • Lots of engineer-tweakable parts inside!
Re: POLL: Should I change servers?
« Reply #107 on: March 14, 2018, 03:39:54 pm »
The new configuration will give us room to wriggle and test stuff like this,

Excellent!

Quote
but it wont happen until after this work is completed.

No worries and worth the wait.
You don't acquire TEA. It acquires you.
 

Offline trys

  • Regular Contributor
  • *
  • Posts: 170
  • Country: gb
  • I started with the AC128
    • Trystan's Workbench
Re: POLL: Should I change servers?
« Reply #108 on: March 14, 2018, 07:22:39 pm »
How about the happy medium? Monitoring on the server.

I've just noticed gnif's signature.
 

Online Electro Detective

  • Super Contributor
  • ***
  • Posts: 1585
  • Country: au
Re: POLL: Should I change servers?
« Reply #109 on: March 14, 2018, 09:18:32 pm »
Aren't there decent hosting server companies here in Australia with decent kit?

Why be at the mercy of a server overseas that can go suss or belly up any time,

or do 'changes' without notice or apology,

and or no one picks up the phone or replies to emails,

and when they do reply it's more or less a "have a nice day"  placebo support...


The U.S. is a politically unstable place at the best of times, why have your bread and butter 'over there' instead of here just in case shtf?

No offence meant to Americans, but would they have their blog server parked in Australia? I don't think so...

Keep it local  :-+
 

Offline gnif

  • Administrator
  • *****
  • Posts: 1043
  • Country: au
Re: POLL: Should I change servers?
« Reply #110 on: March 14, 2018, 09:21:23 pm »
Aren't there decent hosting server companies here in Australia with decent kit?

Why be at the mercy of a server overseas that can go suss or belly up any time,

or do 'changes' without notice or apology,

and or no one picks up the phone or replies to emails,

and when they do reply it's more or less a "have a nice day"  placebo support...


The U.S. is a politically unstable place at the best of times, why have your bread and butter 'over there' instead of here just in case shtf?

No offence meant to Americans, but would they have their blog server parked in Australia? I don't think so...

Keep it local  :-+

This has been discussed to death multiple times in the past,.

  • The majority of EEVBlog viewers are in the US
  • Bandwidth in Australia is extremely expensive
  • The datacenter we are going to use has an excellent reputation of having staff on hand 24/7, it is rare to not have an issue actioned within 5-10 minutes.
HostFission - Full Server Monitoring and Management Solutions.
https://hostfission.com/
https://twitter.com/HostFission

I volunteer my time to manage this server, if you would like to support this work I have a patreon here:
https://www.patreon.com/gnif
 
The following users thanked this post: Electro Detective

Offline gnif

  • Administrator
  • *****
  • Posts: 1043
  • Country: au
Re: POLL: Should I change servers?
« Reply #111 on: March 14, 2018, 10:37:43 pm »
Please don't derail this with a political debate.
HostFission - Full Server Monitoring and Management Solutions.
https://hostfission.com/
https://twitter.com/HostFission

I volunteer my time to manage this server, if you would like to support this work I have a patreon here:
https://www.patreon.com/gnif
 

Offline amspire

  • Super Contributor
  • ***
  • Posts: 3633
  • Country: au
Re: POLL: Should I change servers?
« Reply #112 on: March 14, 2018, 10:37:56 pm »

We are moving to a very standardized setup so that Dave can have someone else on standby should the worst happen to me. The configuration will consist of three servers.

  • Stock Standard cPanel Server - For Dave's Convenience, General Hosting, etc.
  • Two servers configured with a HTTP stack.

The two servers will very likely be configured using the following (note, this is a quick list off the top of my head at this time):

  • Nginx HTTP server + PHP
  • MariaDB using Galera to keep the two servers in sync, cPanel server will likely be used as the Arbiter
  • Since very little outside of the database changes on disk, `unison` will likely be used to sync the HTTP directories, this simplifies the sync and allows each host to operate independently should the other die, unlike configurations with NFS deployments.
  • The two servers will be active/active, keepalived will likely be used to handle automatic failover between the two hosts.
  • Server configuration will likely be deployed via Puppet or SaltStack, I have not yet decided. This if done properly will ensure configuration is consistent across both nodes, as well as future proofing for possible growth

This is a pretty common way to configure things, making it possible for any competent admin with fail over hosting experience capable of managing the servers without too much trouble.

This all sounds very good. I was originally worried with the idea that Dave would end up with custom tweaked servers that are only understood by one person, but if you are going to use a tools like Puppet or SaltStack for the configuration, it should be possible to have a documented procedure for quickly rebuilding servers from scratch.

You could make it even easier by using containers such as Docker, but it is one more thing you have to keep going. Containers would probably separate the http/nginx stack, Simple Machines and the database into separate containers with the actual data perhaps with its own location. It does make it extremely easy to backup the actual running containers and to make it trivial to get it running on new PCs/VMs -  even if it is just to have a debugging on a home/office PC that is running exactly the same containers as the Xeon servers.

It could be that Puppet effectively provides a similar speed of recovery and flexability. The actual size of the EEVblog customisation could easily be pretty small, and so the cost of rebuilding may not be significant. I am pretty ignorant of Puppet and SaltStack so I am not sure how over time, you ensure that they can rebuild the server with 100% of the current mods. If Gnif is not there, will rebuilding a server be a task that for the new admin person is a hope-it-works-or-else leap in the dark?

Is MariaDB a bottlekneck at all? If it is handling the load without a problem, then there is probably no need to touch it.

One of the fastest and most stable relational databases for large data is, believe it or not, SQlite3 - provided you have lots of RAM. There is a project http://bedrockdb.com that makes SQlite3 a replicated and geo-redundant dB. It provides MySQL compatibility if required. Bedrock does provide features like automatic fallover to a slave, and automatic recovery back to the master. If DB issues are a bit of a headache, Bedrock is worth a look.

https://twit.tv/shows/floss-weekly/episodes/456

As a matter of interest, how big is the DB and the associated attached files?

Overall though, this is a great move. I think it is time to move to dedicated servers. The redundant server plan sounds great. If there are a few hiccups along the way, so be it.
 

Offline gnif

  • Administrator
  • *****
  • Posts: 1043
  • Country: au
Re: POLL: Should I change servers?
« Reply #113 on: March 14, 2018, 10:45:46 pm »
This all sounds very good. I was originally worried with the idea that Dave would end up with custom tweaked servers that are only understood by one person, but if you are going to use a tools like Puppet or SaltStack for the configuration, it should be possible to have a documented procedure for quickly rebuilding servers from scratch.

This is the plan.

You could make it even easier by using containers such as Docker, but it is one more thing you have to keep going. Containers would probably separate the http/nginx stack, Simple Machines and the database into separate containers with the actual data perhaps with its own location. It does make it extremely easy to backup the actual running containers and to make it trivial to get it running on new PCs/VMs -  even if it is just to have a debugging on a home/office PC that is running exactly the same containers as the Xeon servers.

At this time this would be over-engineering and adds additional unnecessary complexity.

It could be that Puppet effectively provides a similar speed of recovery and flexability. The actual size of the EEVblog customisation could easily be pretty small, and so the cost of rebuilding may not be significant. I am pretty ignorant of Puppet and SaltStack so I am not sure how over time, you ensure that they can rebuild the server with 100% of the current mods. If Gnif is not there, will rebuilding a server be a task that for the new admin person is a hope-it-works-or-else leap in the dark?

A server once bootstrapped can be deployed with puppet in a matter of minutes in a completely automated manner, which is the entire point :).

Is MariaDB a bottlekneck at all? If it is handling the load without a problem, then there is probably no need to touch it.

The only changes that will be made here is the replication configuration so we can load balance reads between the servers. SMF was not designed with multiple masters in mind, ie insert ID problems, so writes will have to be directed to a single server, but this will be done with keepalived, not the database.

One of the fastest and most stable relational databases for large data is, believe it or not, SQlite3 - provided you have lots of RAM. There is a project http://bedrockdb.com that makes SQlite3 a replicated and geo-redundant dB. It provides MySQL compatibility if required. Bedrock does provide features like automatic fallover to a slave, and automatic recovery back to the master. If DB issues are a bit of a headache, Bedrock is worth a look.

Changing databases is not an option, we require the feature set of MySQL.

As a matter of interest, how big is the DB and the associated attached files?

On disk the database is currently consuming 10GB, the fileset well over 100GB, but very little data on disk outside of the database changes each day.
HostFission - Full Server Monitoring and Management Solutions.
https://hostfission.com/
https://twitter.com/HostFission

I volunteer my time to manage this server, if you would like to support this work I have a patreon here:
https://www.patreon.com/gnif
 

Offline amspire

  • Super Contributor
  • ***
  • Posts: 3633
  • Country: au
Re: POLL: Should I change servers?
« Reply #114 on: March 14, 2018, 11:00:47 pm »
If Gnif is not there, will rebuilding a server be a task that for the new admin person is a hope-it-works-or-else leap in the dark?

A server once bootstrapped can be deployed with puppet in a matter of minutes in a completely automated manner, which is the entire point :).

If in the course of keeping the server running, you have to update anything, change the settings, add modules or whatever, can you do it first in Puppet and then let Puppet make the changes, or do you have to keep the server and Puppet modified in parallel?
Quote

Is MariaDB a bottlekneck at all? If it is handling the load without a problem, then there is probably no need to touch it.

The only changes that will be made here is the replication configuration so we can load balance reads between the servers. SMF was not designed with multiple masters in mind, ie insert ID problems, so writes will have to be directed to a single server, but this will be done with keepalived, not the database.

One of the fastest and most stable relational databases for large data is, believe it or not, SQlite3 - provided you have lots of RAM. There is a project http://bedrockdb.com that makes SQlite3 a replicated and geo-redundant dB. It provides MySQL compatibility if required. Bedrock does provide features like automatic fallover to a slave, and automatic recovery back to the master. If DB issues are a bit of a headache, Bedrock is worth a look.

Changing databases is not an option, we require the feature set of MySQL.

As a matter of interest, how big is the DB and the associated attached files?
On disk the database is currently consuming 10GB, the fileset well over 100GB, but very little data on disk outside of the database changes each day.
Bedrock is MySQL compatible, but I do understand - you have it running on MySQL/MariaDB and so life is easier if you do not change it. It is hard to be 100% compatible.

10GB is not too bad. I was only mentioning Bedrock in case you needed a big speed boost. Bedrock/SQLite3 can hold the whole 10G database in RAM while also writing it to the drives, so it can be exceptionally fast.

I am sure the daily DB changes probably can be synced in seconds, so as long as running Simple Machines off a synced database does not cause stability problems, I do like the plan.
« Last Edit: March 14, 2018, 11:13:00 pm by amspire »
 

Offline gnif

  • Administrator
  • *****
  • Posts: 1043
  • Country: au
Re: POLL: Should I change servers?
« Reply #115 on: March 14, 2018, 11:37:19 pm »
If in the course of keeping the server running, you have to update anything, change the settings, add modules or whatever, can you do it first in Puppet and then let Puppet make the changes, or do you have to keep the server and Puppet modified in parallel?

Puppet applies the changes.

Bedrock is MySQL compatible, but I do understand - you have it running on MySQL/MariaDB and so life is easier if you do not change it. It is hard to be 100% compatible.

10GB is not too bad. I was only mentioning Bedrock in case you needed a big speed boost. Bedrock/SQLite3 can hold the whole 10G database in RAM while also writing it to the drives, so it can be exceptionally fast.

I am sure the daily DB changes probably can be synced in seconds, so as long as running Simple Machines off a synced database does not cause stability problems, I do like the plan.

The DB is not synced daily, every single update is synchronized live, usually in under a millisecond. The delay is in-perceivable.

I think you don't quite get how this will be working. Active/Active means that both servers will be operating in tandem 100% of the time, they will be sharing the load. To make this work both servers need to keep their databases in constant sync, this is absolutely critical for it to operate properly, things such as auto increment IDs would clash otherwise.

As an example of the performance, have a browse of https://forums.realgm.com/boards, this is a similar configuration, but on a database that is 60GB on disk, consisting of nearly 38 million posts across 128,068 members. It absolutely dwarfs the EEV forum, both in data set size and sheer throughput. There is so much traffic on that website that we had to run an internal 10Gbit private LAN for database communications as we were saturating the 2Gbit bonded link the servers had to each other.

The trick to making it perform, keeping the data-set in RAM, just like Bedrock does... MySQL is capable of magic too when configured correctly. We also have enough ram that we can cluster memcached using CouchBase, which is great for performance and reducing database load.

I appreciate the input, but even if Bedrock claimed 100% compatibility it is something that we would have to test extensively over a long period before committing to moving the database across. You also need to note we are not experiencing server/database overload even here at HostGator, most outages are network related, or something screwy with the VM setup that we have (perhaps HG have our storage on a SAN array with inconsistent performance).... All I can say for certain is that we are not on bare metal, and as such we can not confirm any of HG's claims as to how things are configured.
« Last Edit: March 14, 2018, 11:45:23 pm by gnif »
HostFission - Full Server Monitoring and Management Solutions.
https://hostfission.com/
https://twitter.com/HostFission

I volunteer my time to manage this server, if you would like to support this work I have a patreon here:
https://www.patreon.com/gnif
 

Offline zucca

  • Supporter
  • ****
  • Posts: 1692
  • Country: it
  • EE Meid in Itali
Re: POLL: Should I change servers?
« Reply #116 on: March 15, 2018, 12:19:55 am »
Last night I had a dream:

Quote
May be upgrade and allow more MB and freedom in posting/attaching pictures here?  ;)

Everytime I need to attach a picture here my eyes are rolling up in the sky.
Can't know what you don't love. St. Augustine
Can't love what you don't know. Zucca
 

Offline gnif

  • Administrator
  • *****
  • Posts: 1043
  • Country: au
Re: POLL: Should I change servers?
« Reply #117 on: March 15, 2018, 12:25:30 am »
Last night I had a dream:

Quote
May be upgrade and allow more MB and freedom in posting/attaching pictures here?  ;)

Everytime I need to attach a picture here my eyes are rolling up in the sky.

We can't lift that restriction, it's not a server limitation but a problem with how SMF handles image thumbnails.
HostFission - Full Server Monitoring and Management Solutions.
https://hostfission.com/
https://twitter.com/HostFission

I volunteer my time to manage this server, if you would like to support this work I have a patreon here:
https://www.patreon.com/gnif
 

Offline amspire

  • Super Contributor
  • ***
  • Posts: 3633
  • Country: au
Re: POLL: Should I change servers?
« Reply #118 on: March 15, 2018, 01:02:41 am »
Gnif, I didn't know MySQL could keep data in ram. Interesting to know.

When I mentioned daily sync, the thought in my head was that even a whole day's changes would probably take no time to sync. What I didn't realise is that you could keep the dabases synced within a fraction of a second. Very impressive!

It does seem you have a very solid plan, which is really good for Dave.

Just on the use of Unison, does that mean if one server gets the website hacked, then the hacks are duplicated to the other server?
 

Offline gnif

  • Administrator
  • *****
  • Posts: 1043
  • Country: au
Re: POLL: Should I change servers?
« Reply #119 on: March 15, 2018, 01:07:40 am »
Just on the use of Unison, does that mean if one server gets the website hacked, then the hacks are duplicated to the other server?

Yes, clustering servers is not for data backup, just like replicating databases doesn't prevent someone running a bad query that drops the data on both servers in a fraction of a second. We will still be running a daily off site backup.
HostFission - Full Server Monitoring and Management Solutions.
https://hostfission.com/
https://twitter.com/HostFission

I volunteer my time to manage this server, if you would like to support this work I have a patreon here:
https://www.patreon.com/gnif
 

Offline gnif

  • Administrator
  • *****
  • Posts: 1043
  • Country: au
Re: POLL: Should I change servers?
« Reply #120 on: March 15, 2018, 07:42:40 am »
how much more complicated / expensive / etc would it be if dave buys his own hardware to put inside his house to function as a server, assuming there is a low cost way?
and would that possibly be the best way to remove all server issues?

Self-hosting is not only unreliable, but is not an option due to the sheer amount of bandwidth required.

Do the math, leasing servers on a gigabit link with redundant power, fault tolerant connectivity, enterprise grade fire suppression in a HEPA filtered positive pressure environmental controlled environment with 24/7 hands on support and on hand spares at all time for little more then the cost of the monthly power in AU required to run them and guarded physical security/entry/exit to the building

vs.

Self hosting, having to buy 2x the equipment to have spares available, no or very limited power redundancy, on at best a 1Gbit port (if you're very lucky in AU) that costs several hundred dollars a month, times two because you need two for redundancy... etc... etc... etc.

In short, anyone self-hosting something of this scale is just kidding themselves if they think it's better and they are saving money.
« Last Edit: March 15, 2018, 07:44:51 am by gnif »
HostFission - Full Server Monitoring and Management Solutions.
https://hostfission.com/
https://twitter.com/HostFission

I volunteer my time to manage this server, if you would like to support this work I have a patreon here:
https://www.patreon.com/gnif
 

Offline Monkeh

  • Super Contributor
  • ***
  • Posts: 5607
  • Country: gb
Re: POLL: Should I change servers?
« Reply #121 on: March 15, 2018, 07:43:54 am »
But it's just a little website, how hard can it be? ;)

People don't see the scale.
 

Offline gnif

  • Administrator
  • *****
  • Posts: 1043
  • Country: au
Re: POLL: Should I change servers?
« Reply #122 on: March 15, 2018, 07:48:14 am »
I self-host some services, but nothing mission critical, things such as development servers, my personal GitLab server, things that make sense to have locally simply for local bandwidth & convenience reasons. In the past I used to self host much more, but when you go offline due to a small glitch, even something as simple as a rodent/marsupial chewing into some cabling, you quickly realize how frail your connection actually is.
HostFission - Full Server Monitoring and Management Solutions.
https://hostfission.com/
https://twitter.com/HostFission

I volunteer my time to manage this server, if you would like to support this work I have a patreon here:
https://www.patreon.com/gnif
 

Offline Cerebus

  • Super Contributor
  • ***
  • Posts: 3342
  • Country: gb
Re: POLL: Should I change servers?
« Reply #123 on: March 15, 2018, 08:55:37 am »
..., even something as simple as a rodent/marsupial chewing into some cabling, you quickly realize how frail your connection actually is.

Even if you hand over the money to have guaranteed redundancy on links to you, even from separate suppliers, you can find that they share some point of failure further up the line.

I used to be in the ISP business and to avoid exactly that we used to get physical network maps off fibre and other raw data circuit suppliers (under NDA). Firstly, it was an enormous amount of work, secondly, once you do this exercise you find there are a handful of critical physical points in any nations infrastructure. At the time I was doing this there were two critical physical cable routes in London that more than 90% of the UK's Internet traffic passed through. If your redundant paths went under both those manhole covers you were OK, if they went under the same manhole cover, you were stuffed.
Anybody got a syringe I can use to squeeze the magic smoke back into this?
 
The following users thanked this post: gnif

Online beanflying

  • Super Contributor
  • ***
  • Posts: 1189
  • Country: au
  • Toys so very many Toys.
Re: POLL: Should I change servers?
« Reply #124 on: March 15, 2018, 09:08:37 am »
Last night I had a dream:

Quote
May be upgrade and allow more MB and freedom in posting/attaching pictures here?  ;)

Everytime I need to attach a picture here my eyes are rolling up in the sky.

Really bad idea for lots of reasons. Not all the planet has uber broadband. Loading threads with off site images is bad enough now on DSL2.

There are several stable SMF mods that can auto strip images down to size but you can find yourself in a position with a mod (generally built by one person) that doesn't keep compatibility with a patch or major upgrade of the core software.
Coffee, Food, R/C and electronics nerd in no particular order :)
 


Share me

Digg  Facebook  SlashDot  Delicious  Technorati  Twitter  Google  Yahoo
Smf