IMPORTANT: Server Changeover

[EEVblog]

I just got this message from HostGater who host the forum and blog site:

Hello,

Please read this email in its entirety, as it contains very important information regarding your HostGator dedicated server, IP address: 184.173.12.165, and the impending mandatory maintenance and migration that will be occurring within 48 hours.

Your dedicated server(s) are being migrated onto brand new, upgraded hardware. We plan to facilitate this migration both as quickly and easily as possible, but also want to assure you that your total satisfaction with this migration is our primary objective. We will keep you updated via email throughout this entire migration.

Please be aware that there may be small amounts of downtime and that this process does require a change of IP address and nameservers, as such it is vital that you take action in regard to the DNS settings once the actual data migration has been completed.

As mentioned, there are several automatic upgrades involved in this migration:

 -- All servers will now have a RAID1 hard disk configuration. In brief, a RAID1 is two disks containing exact copies of the data (a mirror) which allows for increased read speeds for faster delivery of requested data. Data can be read from both disks independently or in tandem, which also provides redundancy in the event of a disk failure.

 -- All servers also feature an optional Rescue & Hardware Monitoring KVM layer (which is included by default) allowing HostGator to monitor the server for any potential hardware failures and proactively resolve these issues before they result in any downtime and/or possible data loss.

 -- All servers will contain quad-core processors with hyper-threading enabled, which allows the for the processor to effectively share the workload at any given moment between cores resulting in the parallelization of computational processes, or simply put: increased speed.

 -- All servers are pre-configured to allow for the attachment of off-server SAN (Storage Area Network) backup solutions, allowing for increased flexibility and control over your backups.

At this time, there are no changes needed to be made on your part. We'll contact you with another update, within 48 hours, once the migration process has started. When the migration has completed we will forward the traffic from the old server to the new server and we will send you another update which will contain specific details regarding the necessary DNS changes.

So I would expect some downtime on the forum and site over the next two days.

Dave.

[EEVblog]

For those interested, the new server has the following specs:

Server model: Supermicro X9SCD+-F-BH002
Memory: Kingston DDR3 1333MHz
Processor: Intel(R) Xeon(R) CPU E3-1265L V2 @ 2.50GHz
Ethernet controller: Dual Intel 82580 Gigabit (limited to 10MBit port)
Hard drive: Western Digital RE4 Serial ATA (WD1003FBYX-01Y7B1) x2

Dave.

[BravoV]

Does SMF have the ability to put a big banner announcement at upper part of the screen ? Don't forget to announce at eevblog.com front page as well.

[mariush]

10 mbps port in 2013 is stupid and too little for the amount of money they charge.

If you get a lot of visitors on the forum accessing some threads with pictures they'll saturate the port ( 10mbps = 1.2 MB/s)

If you can, choose 100mbps and 2-3 TB bandwidth instead of 10 mbps (which is 3.3 TB spread throughout the month), the forum and website may be more responsive only by not being limited by the 10 mbps bandwidth.

[manicdoc]

10 mbps port in 2013 is stupid and too little for the amount of money they charge.

If you get a lot of visitors on the forum accessing some threads with pictures they'll saturate the port ( 10mbps = 1.2 MB/s)

If you can, choose 100mbps and 2-3 TB bandwidth instead of 10 mbps (which is 3.3 TB spread throughout the month), the forum and website may be more responsive only by not being limited by the 10 mbps bandwidth.

Agreed, but (assuming Dave has the access) better to reduce what you have to send through the NIC first then consider upping the capacity - for instance on the main site jquery is pulled off the server (91k) when that could be served by google for you. Less bandwidth and one less query for the server to worry about, can't get faster than that. With some 'strategic' optimisations you can offload quite a bit.

Lets see how the new super duper server does first, then tweak if needed.

[GeoffS]

Seeing there's an IP address change, you might want to consider reducing the DNS  TTL (time to live) value down to 15 minutes prior to the changes. That way,  the IP address change will propagate more quickly.
It's currently set to 1 hour.

[EEVblog]

Seeing there's an IP address change, you might want to consider reducing the DNS  TTL (time to live) value down to 15 minutes prior to the changes. That way,  the IP address change will propagate more quickly.
It's currently set to 1 hour.

I have no idea were that option even lives...

Dave.

[EEVblog]

10 mbps port in 2013 is stupid and too little for the amount of money they charge.

If you get a lot of visitors on the forum accessing some threads with pictures they'll saturate the port ( 10mbps = 1.2 MB/s)

According the cPanel bandwidth stats, the peak values don't even hit 3Mbps.
I assume it is able to capture the absolute peak value during the 5-10min window the graph shows for each data point.

Dave.

[alm]

It will usually sample the bytes transferred at 5 minute intervals, so you get averages over a 5 minute window.

I have no idea were that option even lives...

Should be in the control panel of whoever is hosting your DNS (domaincontrol.com?). Some don't make it available via the control panel and require you to go through tech support.

[Monkeh]

10 mbps port in 2013 is stupid and too little for the amount of money they charge.

If you get a lot of visitors on the forum accessing some threads with pictures they'll saturate the port ( 10mbps = 1.2 MB/s)

'a lot'? I can saturate that myself. Four times over. I have done, actually. Explains why images are so slow to load.

[mariush]

10 mbps port in 2013 is stupid and too little for the amount of money they charge.

If you get a lot of visitors on the forum accessing some threads with pictures they'll saturate the port ( 10mbps = 1.2 MB/s)

According the cPanel bandwidth stats, the peak values don't even hit 3Mbps.
I assume it is able to capture the absolute peak value during the 5-10min window the graph shows for each data point.

Dave.

That's incorrect.

cPanel simply retrieves those numbers from a bandwidth measuring tool that's independent from cPanel as is generally used for billing people on 95 percentile plans (http://en.wikipedia.org/wiki/Burstable_billing). 

That tool simply does a snapshot of your traffic at the switch port every 5 minutes, and builds a series of instant snapshots for the whole month.. 720 snapshots a day, times 28-31 for a month. On 95% billing, the peak 5% of these are removed and the person pays for the peak that's left... this works great when user is streaming radio, or offering downloads... he doesn't pay for traffic but pays for what  bandwidth he hogs for himself all the time.

So the tool is unable to determine the traffic between two snapshot. 5 seconds before the snapshot someone can access a thread and click on a thumbnail to get a 500 KB picture and download it at 10 mbps making the site slow for everyone else for those 2-3 seconds it takes to download the picture and the bandwidth measurement tool is unable to know that.

[free_electron]

oooh , Red drives....

for the unwashed masses :

western digital has 7 classes of drives

Green : 5400 rpm , power saving , for general purpose infrequent access
blue : 7200rpm , general purpose drive
Black : 7200 RPM double cache , multicore processor : performance drive ideal for boot volume . 5 year warranty
Raptor : 10KRPm : screamer small size but tlightning fast
Red : 7200 rpm with special firmware, optimized for NAs applications.
RE : enterprise drives made for 24/7 operation 5 year warranty
AV : made for 24/7 video streamer applications like settopboxes.

http://www.wdc.com/en/products/internal/desktop/

while you can use the REd drives as a general purpose drive they are tuned for NAS and RAID applications.
http://www.wdc.com/en/products/products.aspx?id=810

so in case of a server where you use RAId : these drives are ideal.

[GeoffS]

Seeing there's an IP address change, you might want to consider reducing the DNS  TTL (time to live) value down to 15 minutes prior to the changes. That way,  the IP address change will propagate more quickly.
It's currently set to 1 hour.

I have no idea were that option even lives...

Dave.

It will depend on which control panel you use, cPanel or WHM. With cPanel there's an entry to modify DNS records, I don't know WHM but assume it has somethig similar.
I doubt that the hosting service manage this for you or they wouldn't have stated:

as such it is vital that you take action in regard to the DNS settings once the actual data migration has been completed.

[EEVblog]

I doubt that the hosting service manage this for you or they wouldn't have stated:

as such it is vital that you take action in regard to the DNS settings once the actual data migration has been completed.

[/quote]

I pay for a fully managed server.
If they break it, they can fix it!

Dave.

[manicdoc]

I doubt that the hosting service manage this for you or they wouldn't have stated:

as such it is vital that you take action in regard to the DNS settings once the actual data migration has been completed.

I pay for a fully managed server.
If they break it, they can fix it!

Dave.
[/quote]

it sounds like they don't manage the eevblog.com zone record - or rather you are expected to make the edits.. Just change what you are comfortable with, the TTL is the longest possible time the value will be cached down the DNS chain, it won't clear down anything under the old value until they refresh any ways (if memory serves), so of limited use during a change; you need to change the TTL down an hour before if you want a faster swap.

[EEVblog]

So the tool is unable to determine the traffic between two snapshot. 5 seconds before the snapshot someone can access a thread and click on a thumbnail to get a 500 KB picture and download it at 10 mbps making the site slow for everyone else for those 2-3 seconds it takes to download the picture and the bandwidth measurement tool is unable to know that.

Ok, in that case, wouldn't the same thinking also extend to a 100Mbps link?
Someone sucks that image file from the whole 100Mbps link.
In which case, what's the difference?, apart from it taking 1/10th the time?
Actually, lets say it's now a 5M image. Isn't it the exact same 2-3 hogging of the server?
And in which case, how does the server even cope with any serious traffic at all??

Dave.

[mariush]

Most of your users won't have 100 mbps at home. I have, but outside my country, I mostly average 4-6 MB/s.  However, they'll have 10-40 mbps download speeds, or they'll have ISPs with  boost schemes (like Comcast PowerBoost) which give higher speeds to files less than a particular file size, usually less than 5-10 MB.

I own a dedicated server with 100mbps unmetered link, which hosts several websites.... for example one website hosts software updates for an open source project, another website is similar to imgur.com but at way lower scale...  so I log the download speeds for these files.

Here's the log for a part of the website with updates for the open source project:

I look at the log in the link above and see that the majority of users download at speeds around 250-400 KB/s, but quite a lot of them manage over 1MB/s.
Sure, there's quite a few with low download speeds but keep in mind the downloads have an .exe extension and quite a few users around the world have ISPs that throttle/traffic shape anything that's not html or pictures or something that's above a specific file size (these downloads average 15-40 MB in size)

Anyway, it's going slightly off topic.

With your 5 MB picture example, a user downloading at full speed would download it in less than 0.2 seconds, leaving plenty of time for the server to upload to others.. if lots of users request files and have high speeds, each connection is slowed down to fit the 100mbps choke point. This is much harder to achieve, compared to a 10mbps link.

The basic idea is that with a 100mbps port, a handful of users won't fill up your server's total bandwidth with their requests - with 100 mbps lots of users can access and retrieve data from your server at the same time, without making it slow for everyone.
100 users accessing the forum will all average 1 mbps, while 100 users on your 10 mbps link will average 0.1 mbps - which one will make the forum feel more responsive, load the pictures faster etc? 
Load on your server will also decrease (edit:said increase originally), but just slightly... the load on your server doesn't increase with 10 mbps link because the operating system has caches, the network card has buffers, etc...  so the forum scripts process the requests, respond with the data and then this data is partially cached by the network card drivers and OS up to a certain point and sent at the slower pace of 10 mbps...  and instead of sending this data in half a second or so as you'd do with 100mbps you're now sending it to the internet in a couple of seconds or more on 10mbps, an interval that's still too small to show up in graphs, which sample every 5 minutes.

The traffic  (amount of MB transferred) will be the same at the end of the month but with 10mbps it's all squeezed and slowed down to the 10mbps speed... it just feels slow for users with high download speeds.

[EEVblog]

Well, the server changeover started at about 8pm last night, and it's 8am now, and still no confirmation email it's done.
I locked the forum down overnight just in case, so no posts would be lost. But bugger that, I expected it would be done by now 

Dave.

[Smokey]

Dude.... What the hell!!

You have no idea how much work you made me do this morning since I couldn't waste time on the forum!!

[SteigsdB]

Woo!

The site's back up. 

Now I can kick that heroin habit I picked up about 8 hours ago.....

Seriously though, thanks for a great resource Dave!

[HackedFridgeMagnet]

It's probably easier to kick that then kicking the electronics habit.

[EEVblog]

I think I'll just leave the forum going. I can't keep it locked down for days until they do this.
If we lose some post, we lose some posts. I presume the very last thing they would do it move the databases? 

Dave.

[EEVblog]

This is what the email said:

We are pleased to inform you that we have started the migration process for your dedicated server 184.173.12.165. At this time we are transferring all data from your current server to brand new hardware. During this migration your websites will remain online and should have very little downtime.  Once the migration has completed, we will send another update that contains new IP addresses and information on any DNS changes that will need to be made. We will also be forwarding all traffic from your old server to the new server. This will allow you to immediately take advantage of the new upgraded hardware.

Dave.

[Chet T16]

The biggest problem lies when both servers are live and you're left with some peoples peoples DNS cache sending them to the old site. Not sure what you should do when you have no control over it.

Seems like a very poor show to me!

[HackedFridgeMagnet]

I think the only real issue with doing the changeover is not losing the data (posts) that were inserted after the database was replicated.
I think most DB guys could handle this in their sleep. They should've already written and tested a script that does this to all the relevant tables.

Hopefully the Hosting service will manage this, though I don't know why the site was down all night, that does seem unnecessary.