IMPORTANT: Server Changeover

[EEVblog]

I just got this message from HostGater who host the forum and blog site:

Hello,

Please read this email in its entirety, as it contains very important information regarding your HostGator dedicated server, IP address: 184.173.12.165, and the impending mandatory maintenance and migration that will be occurring within 48 hours.

Your dedicated server(s) are being migrated onto brand new, upgraded hardware. We plan to facilitate this migration both as quickly and easily as possible, but also want to assure you that your total satisfaction with this migration is our primary objective. We will keep you updated via email throughout this entire migration.

Please be aware that there may be small amounts of downtime and that this process does require a change of IP address and nameservers, as such it is vital that you take action in regard to the DNS settings once the actual data migration has been completed.

As mentioned, there are several automatic upgrades involved in this migration:

 -- All servers will now have a RAID1 hard disk configuration. In brief, a RAID1 is two disks containing exact copies of the data (a mirror) which allows for increased read speeds for faster delivery of requested data. Data can be read from both disks independently or in tandem, which also provides redundancy in the event of a disk failure.

 -- All servers also feature an optional Rescue & Hardware Monitoring KVM layer (which is included by default) allowing HostGator to monitor the server for any potential hardware failures and proactively resolve these issues before they result in any downtime and/or possible data loss.

 -- All servers will contain quad-core processors with hyper-threading enabled, which allows the for the processor to effectively share the workload at any given moment between cores resulting in the parallelization of computational processes, or simply put: increased speed.

 -- All servers are pre-configured to allow for the attachment of off-server SAN (Storage Area Network) backup solutions, allowing for increased flexibility and control over your backups.

At this time, there are no changes needed to be made on your part. We'll contact you with another update, within 48 hours, once the migration process has started. When the migration has completed we will forward the traffic from the old server to the new server and we will send you another update which will contain specific details regarding the necessary DNS changes.

So I would expect some downtime on the forum and site over the next two days.

Dave.

[EEVblog]

For those interested, the new server has the following specs:

Server model: Supermicro X9SCD+-F-BH002
Memory: Kingston DDR3 1333MHz
Processor: Intel(R) Xeon(R) CPU E3-1265L V2 @ 2.50GHz
Ethernet controller: Dual Intel 82580 Gigabit (limited to 10MBit port)
Hard drive: Western Digital RE4 Serial ATA (WD1003FBYX-01Y7B1) x2

Dave.

[BravoV]

Does SMF have the ability to put a big banner announcement at upper part of the screen ? Don't forget to announce at eevblog.com front page as well.

[mariush]

10 mbps port in 2013 is stupid and too little for the amount of money they charge.

If you get a lot of visitors on the forum accessing some threads with pictures they'll saturate the port ( 10mbps = 1.2 MB/s)

If you can, choose 100mbps and 2-3 TB bandwidth instead of 10 mbps (which is 3.3 TB spread throughout the month), the forum and website may be more responsive only by not being limited by the 10 mbps bandwidth.

[manicdoc]

10 mbps port in 2013 is stupid and too little for the amount of money they charge.

If you get a lot of visitors on the forum accessing some threads with pictures they'll saturate the port ( 10mbps = 1.2 MB/s)

If you can, choose 100mbps and 2-3 TB bandwidth instead of 10 mbps (which is 3.3 TB spread throughout the month), the forum and website may be more responsive only by not being limited by the 10 mbps bandwidth.

Agreed, but (assuming Dave has the access) better to reduce what you have to send through the NIC first then consider upping the capacity - for instance on the main site jquery is pulled off the server (91k) when that could be served by google for you. Less bandwidth and one less query for the server to worry about, can't get faster than that. With some 'strategic' optimisations you can offload quite a bit.

Lets see how the new super duper server does first, then tweak if needed.

[GeoffS]

Seeing there's an IP address change, you might want to consider reducing the DNS  TTL (time to live) value down to 15 minutes prior to the changes. That way,  the IP address change will propagate more quickly.
It's currently set to 1 hour.

[EEVblog]

Seeing there's an IP address change, you might want to consider reducing the DNS  TTL (time to live) value down to 15 minutes prior to the changes. That way,  the IP address change will propagate more quickly.
It's currently set to 1 hour.

I have no idea were that option even lives...

Dave.

[EEVblog]

10 mbps port in 2013 is stupid and too little for the amount of money they charge.

If you get a lot of visitors on the forum accessing some threads with pictures they'll saturate the port ( 10mbps = 1.2 MB/s)

According the cPanel bandwidth stats, the peak values don't even hit 3Mbps.
I assume it is able to capture the absolute peak value during the 5-10min window the graph shows for each data point.

Dave.

[alm]

It will usually sample the bytes transferred at 5 minute intervals, so you get averages over a 5 minute window.

I have no idea were that option even lives...

Should be in the control panel of whoever is hosting your DNS (domaincontrol.com?). Some don't make it available via the control panel and require you to go through tech support.

[Monkeh]

10 mbps port in 2013 is stupid and too little for the amount of money they charge.

If you get a lot of visitors on the forum accessing some threads with pictures they'll saturate the port ( 10mbps = 1.2 MB/s)

'a lot'? I can saturate that myself. Four times over. I have done, actually. Explains why images are so slow to load.

[mariush]

10 mbps port in 2013 is stupid and too little for the amount of money they charge.

If you get a lot of visitors on the forum accessing some threads with pictures they'll saturate the port ( 10mbps = 1.2 MB/s)

According the cPanel bandwidth stats, the peak values don't even hit 3Mbps.
I assume it is able to capture the absolute peak value during the 5-10min window the graph shows for each data point.

Dave.

That's incorrect.

cPanel simply retrieves those numbers from a bandwidth measuring tool that's independent from cPanel as is generally used for billing people on 95 percentile plans (http://en.wikipedia.org/wiki/Burstable_billing). 

That tool simply does a snapshot of your traffic at the switch port every 5 minutes, and builds a series of instant snapshots for the whole month.. 720 snapshots a day, times 28-31 for a month. On 95% billing, the peak 5% of these are removed and the person pays for the peak that's left... this works great when user is streaming radio, or offering downloads... he doesn't pay for traffic but pays for what  bandwidth he hogs for himself all the time.

So the tool is unable to determine the traffic between two snapshot. 5 seconds before the snapshot someone can access a thread and click on a thumbnail to get a 500 KB picture and download it at 10 mbps making the site slow for everyone else for those 2-3 seconds it takes to download the picture and the bandwidth measurement tool is unable to know that.

[free_electron]

oooh , Red drives....

for the unwashed masses :

western digital has 7 classes of drives

Green : 5400 rpm , power saving , for general purpose infrequent access
blue : 7200rpm , general purpose drive
Black : 7200 RPM double cache , multicore processor : performance drive ideal for boot volume . 5 year warranty
Raptor : 10KRPm : screamer small size but tlightning fast
Red : 7200 rpm with special firmware, optimized for NAs applications.
RE : enterprise drives made for 24/7 operation 5 year warranty
AV : made for 24/7 video streamer applications like settopboxes.

http://www.wdc.com/en/products/internal/desktop/

while you can use the REd drives as a general purpose drive they are tuned for NAS and RAID applications.
http://www.wdc.com/en/products/products.aspx?id=810

so in case of a server where you use RAId : these drives are ideal.

[GeoffS]

Seeing there's an IP address change, you might want to consider reducing the DNS  TTL (time to live) value down to 15 minutes prior to the changes. That way,  the IP address change will propagate more quickly.
It's currently set to 1 hour.

I have no idea were that option even lives...

Dave.

It will depend on which control panel you use, cPanel or WHM. With cPanel there's an entry to modify DNS records, I don't know WHM but assume it has somethig similar.
I doubt that the hosting service manage this for you or they wouldn't have stated:

as such it is vital that you take action in regard to the DNS settings once the actual data migration has been completed.

[EEVblog]

I doubt that the hosting service manage this for you or they wouldn't have stated:

as such it is vital that you take action in regard to the DNS settings once the actual data migration has been completed.

[/quote]

I pay for a fully managed server.
If they break it, they can fix it!

Dave.

[manicdoc]

I doubt that the hosting service manage this for you or they wouldn't have stated:

as such it is vital that you take action in regard to the DNS settings once the actual data migration has been completed.

I pay for a fully managed server.
If they break it, they can fix it!

Dave.
[/quote]

it sounds like they don't manage the eevblog.com zone record - or rather you are expected to make the edits.. Just change what you are comfortable with, the TTL is the longest possible time the value will be cached down the DNS chain, it won't clear down anything under the old value until they refresh any ways (if memory serves), so of limited use during a change; you need to change the TTL down an hour before if you want a faster swap.

[EEVblog]

So the tool is unable to determine the traffic between two snapshot. 5 seconds before the snapshot someone can access a thread and click on a thumbnail to get a 500 KB picture and download it at 10 mbps making the site slow for everyone else for those 2-3 seconds it takes to download the picture and the bandwidth measurement tool is unable to know that.

Ok, in that case, wouldn't the same thinking also extend to a 100Mbps link?
Someone sucks that image file from the whole 100Mbps link.
In which case, what's the difference?, apart from it taking 1/10th the time?
Actually, lets say it's now a 5M image. Isn't it the exact same 2-3 hogging of the server?
And in which case, how does the server even cope with any serious traffic at all??

Dave.

[mariush]

Most of your users won't have 100 mbps at home. I have, but outside my country, I mostly average 4-6 MB/s.  However, they'll have 10-40 mbps download speeds, or they'll have ISPs with  boost schemes (like Comcast PowerBoost) which give higher speeds to files less than a particular file size, usually less than 5-10 MB.

I own a dedicated server with 100mbps unmetered link, which hosts several websites.... for example one website hosts software updates for an open source project, another website is similar to imgur.com but at way lower scale...  so I log the download speeds for these files.

Here's the log for a part of the website with updates for the open source project:

I look at the log in the link above and see that the majority of users download at speeds around 250-400 KB/s, but quite a lot of them manage over 1MB/s.
Sure, there's quite a few with low download speeds but keep in mind the downloads have an .exe extension and quite a few users around the world have ISPs that throttle/traffic shape anything that's not html or pictures or something that's above a specific file size (these downloads average 15-40 MB in size)

Anyway, it's going slightly off topic.

With your 5 MB picture example, a user downloading at full speed would download it in less than 0.2 seconds, leaving plenty of time for the server to upload to others.. if lots of users request files and have high speeds, each connection is slowed down to fit the 100mbps choke point. This is much harder to achieve, compared to a 10mbps link.

The basic idea is that with a 100mbps port, a handful of users won't fill up your server's total bandwidth with their requests - with 100 mbps lots of users can access and retrieve data from your server at the same time, without making it slow for everyone.
100 users accessing the forum will all average 1 mbps, while 100 users on your 10 mbps link will average 0.1 mbps - which one will make the forum feel more responsive, load the pictures faster etc? 
Load on your server will also decrease (edit:said increase originally), but just slightly... the load on your server doesn't increase with 10 mbps link because the operating system has caches, the network card has buffers, etc...  so the forum scripts process the requests, respond with the data and then this data is partially cached by the network card drivers and OS up to a certain point and sent at the slower pace of 10 mbps...  and instead of sending this data in half a second or so as you'd do with 100mbps you're now sending it to the internet in a couple of seconds or more on 10mbps, an interval that's still too small to show up in graphs, which sample every 5 minutes.

The traffic  (amount of MB transferred) will be the same at the end of the month but with 10mbps it's all squeezed and slowed down to the 10mbps speed... it just feels slow for users with high download speeds.

[EEVblog]

Well, the server changeover started at about 8pm last night, and it's 8am now, and still no confirmation email it's done.
I locked the forum down overnight just in case, so no posts would be lost. But bugger that, I expected it would be done by now 

Dave.

[Smokey]

Dude.... What the hell!!

You have no idea how much work you made me do this morning since I couldn't waste time on the forum!!

[SteigsdB]

Woo!

The site's back up. 

Now I can kick that heroin habit I picked up about 8 hours ago.....

Seriously though, thanks for a great resource Dave!

[HackedFridgeMagnet]

It's probably easier to kick that then kicking the electronics habit.

[EEVblog]

I think I'll just leave the forum going. I can't keep it locked down for days until they do this.
If we lose some post, we lose some posts. I presume the very last thing they would do it move the databases? 

Dave.

[EEVblog]

This is what the email said:

We are pleased to inform you that we have started the migration process for your dedicated server 184.173.12.165. At this time we are transferring all data from your current server to brand new hardware. During this migration your websites will remain online and should have very little downtime.  Once the migration has completed, we will send another update that contains new IP addresses and information on any DNS changes that will need to be made. We will also be forwarding all traffic from your old server to the new server. This will allow you to immediately take advantage of the new upgraded hardware.

Dave.

[Chet T16]

The biggest problem lies when both servers are live and you're left with some peoples peoples DNS cache sending them to the old site. Not sure what you should do when you have no control over it.

Seems like a very poor show to me!

[HackedFridgeMagnet]

I think the only real issue with doing the changeover is not losing the data (posts) that were inserted after the database was replicated.
I think most DB guys could handle this in their sleep. They should've already written and tested a script that does this to all the relevant tables.

Hopefully the Hosting service will manage this, though I don't know why the site was down all night, that does seem unnecessary.

[GeoffS]

They did say they would be forwarding traffic from the old server to the new one so, in theory, there should be no posts going to the old system.

[manicdoc]

I think the only real issue with doing the changeover is not losing the data (posts) that were inserted after the database was replicated.
I think most DB guys could handle this in their sleep. They should've already written and tested a script that does this to all the relevant tables.

Hopefully the Hosting service will manage this, though I don't know why the site was down all night, that does seem unnecessary.

Yes and No - you would need to 'map' the new posts on the old into the new whilst resolving dupe primary keys and chain through foreign references and changes... Not a 10 minute hack.. be easier to do it at the app level and effectively enact the posting API with the new posts off the old server - assuming the forum software has such an API to access.. Remember there might be a whole lot of side effects to take care of (such as text indexes, etc). Certainly not trivial - although if someone has already written a script that does this - then it is possible.

[HackedFridgeMagnet]

you would need to 'map' the new posts on the old into the new whilst resolving dupe primary keys and chain through foreign references and changes... Not a 10 minute hack..

Can't you carry the primary keys across in the replication (backup/restore) ? You can turn the server off for the 5 minutes it takes to run the update script,so there are no duplicate primary keys.

Yes you would have to chain through a few foreign references when doing the inserts but this is why you would script it.
Probably only need to add the new posts and topics, you could disregard the small number of user meta data changes that happen in this time.
Obviously you would test your script first, so you don't get a blowout in down time.

[EEVblog]

Hopefully the Hosting service will manage this, though I don't know why the site was down all night, that does seem unnecessary.

That was my choice, I did it. I got the email about the server change had started, and figured it was safest to just lock the forum down.
It got to midnight and I had to go to bed, so made the choice to leave it locked down overnight just in case.
I thought it would be done by the morning, seems I was wrong.

Dave.

[amspire]

Seeing there's an IP address change, you might want to consider reducing the DNS  TTL (time to live) value down to 15 minutes prior to the changes. That way,  the IP address change will propagate more quickly.
It's currently set to 1 hour.

I have no idea were that option even lives...

Dave.

If you have cpanel, go to Advanced DNS Zone Editor, and edit the A address for eevblog.com - the one that mentions the current IP address.

It will be a line that looks like

Code: [Select]

eevblog.com. 3600  IN  A  184.173.12.165    Edit    Delete
Click edit and change the TTL (Time To Live) from 3600 (1 hour) to a lower number like 600 (10 minutes)

When the server gets transferred, the IP in this record will automatically get updated, and the DNS caches around the world will get the new IP address within 10 minutes.

[EEVblog]

If you have cpanel, go to Advanced DNS Zone Editor, and edit the A address for eevblog.com - the one that mentions the current IP address.

It will be a line that looks like

Code: [Select]

eevblog.com. 3600  IN  A  184.173.12.165    Edit    Delete
Click edit and change the TTL (Time To Live) from 3600 (1 hour) to a lower number like 600 (10 minutes)

When the server gets transferred, the IP in this record will automatically get updated, and the DNS caches around the world will get the new IP address within 10 minutes.

Got it.
Changed to 30min (the lowest option in dropdown box)

Dave.

[manicdoc]

you would need to 'map' the new posts on the old into the new whilst resolving dupe primary keys and chain through foreign references and changes... Not a 10 minute hack..

Can't you carry the primary keys across in the replication (backup/restore) ? You can turn the server off for the 5 minutes it takes to run the update script,so there are no duplicate primary keys.

Yes you would have to chain through a few foreign references when doing the inserts but this is why you would script it.
Probably only need to add the new posts and topics, you could disregard the small number of user meta data changes that happen in this time.
Obviously you would test your script first, so you don't get a blowout in down time.

the problem is if the new system has posts being written which use the same primary keys as the old system - you need to resolve the conflict. Which could get quite hairy if the key is written into some app controlled field (i.e. keyword mapping in articles). hence my suggestion to come in at the app level.

An alternative is to configure the new system to be a MySQL slave to the old until you throw the switch, but you won't be able to write posts on the new system. OR copy across the DB and point the old server database config at the new server - this assumes the forum and other software would deal correctly with being 'multiheaded' and not get its session knickers in a twist 

Basically moving a whole single box system to a new box will usually incur some down time somewhere, you haven't got enough kit to stage it.

[EEVblog]

I asked them what happens with databases:

The way these migrations work has been planned to be as effortless and pain free as possible. First we rsync all of the data from the old server to the new server, once that's done we do it two more times to make sure any data that's been changed while the first rsync was running is updated. then we shut down MySQL on the source box and do a rsync of just the MySQL data to ensure the databases are intact and up to date. MySQL is never restarted on the old box. As soon as that's done we bring up all services on the new server and put iptables NAT rules in place on the old server to forward all traffic from the old IPs to the new ones. This makes the migration nearly transparent. At that time we start prompting the user to make any required DNS changes, which are usually minimal as most customers host DNS on their own servers.

So looks like we won't lose a single post.

Dave.

[Lightages]

Well the SMF not being to connect to database error just occurred again. So it was definitely not a hardware issue.

[gnif]

but you won't be able to write posts on the new system. OR copy across the DB and point the old server database config at the new server - this assumes the forum and other software would deal correctly with being 'multiheaded' and not get its session knickers in a twist 

Basically moving a whole single box system to a new box will usually incur some down time somewhere, you haven't got enough kit to stage it.

This only occurs if you don't know what you are doing. You can write to a MySQL slave, but you must ensure that once you start writing to it, you don't write to the master. This scenario works perfect for an instant switch-over.

1) Slave MySQL database to the old server
2) Configure a reverse proxy to forward HTTP/HTTPs traffic to the new server
3) [optional] use socat or iptables to forward traffic for POP3/IMAP/SMTP, etc. to the new server also
4) De-configure MySQL slave
5) Update DNS, wait for propagation to occur
6) Decommission old server.

Zero down time and zero data loss.

The reverse proxy should be used, as it will then add the X-Forwarded-For header to the connections, which the new server should be configured using mod_rpaf (if Apache is used on the new server, Nginx would be better, but that's another topic) to look for and use as the real IP connecting. Just forwarding the data without this will cause the logs to fill with connections that look like they are coming from the old server and any protection on user account login (such as the forum user login) may trigger lockouts due to all the users seeming to come from the same IP address.

Well the SMF not being to connect to database error just occurred again. So it was definitely not a hardware issue.

This is caused by MySQL running out of connections, just increase the max connections if the server is capable of handling it.

[EEVblog]

The server change over is complete, and it seemed to work even without me changing the DNS IP.
BUt I have just changed the DNS IP to the new one  198.1.64.14
The old one is 184.173.12.165
I presume the site will still work for everyone in that limbo time, working on either IP.

Let me know if there is any problem.

Dave.

[amspire]

The server change over is complete, and it seemed to work even without me changing the DNS IP.
BUt I have just changed the DNS IP to the new one  198.1.64.14

198.1.64.143 you mean. My DNS already points to the new address.

[gnif]

Yeah, they said they would NAT the connections though to the new server, so zero downtime, but you may find that you are now logging everyone's IP address as the old server's IP. And yes, you most certainly needed to change the IP over.

[amspire]

The server change over is complete, and it seemed to work even without me changing the DNS IP.

The DNS has been updated by the host obviously. The A address definitely now point to the new IP.

[gnif]

The server change over is complete, and it seemed to work even without me changing the DNS IP.

The DNS has been updated by the host obviously. The A address definitely now point to the new IP.

BUt I have just changed the DNS IP to the new one  198.1.64.14

No, Dave changed it, the host is obviously doing what they said, using NAT to forward the connections to the new server.

[EEVblog]

The server change over is complete, and it seemed to work even without me changing the DNS IP.

The DNS has been updated by the host obviously. The A address definitely now point to the new IP.

I changed it myself. HostGator do not have access to my DNS. I use a third party company to host all my domain names, that's why they said I have to do it.
I'm sure they would have done it for me if I provided them my login and site details etc.

Dave.

[manicdoc]

but you won't be able to write posts on the new system. OR copy across the DB and point the old server database config at the new server - this assumes the forum and other software would deal correctly with being 'multiheaded' and not get its session knickers in a twist 

Basically moving a whole single box system to a new box will usually incur some down time somewhere, you haven't got enough kit to stage it.

This only occurs if you don't know what you are doing. You can write to a MySQL slave, but you must ensure that once you start writing to it, you don't write to the master. This scenario works perfect for an instant switch-over.

1) Slave MySQL database to the old server
2) Configure a reverse proxy to forward HTTP/HTTPs traffic to the new server
3) [optional] use socat or iptables to forward traffic for POP3/IMAP/SMTP, etc. to the new server also
4) De-configure MySQL slave
5) Update DNS, wait for propagation to occur
6) Decommission old server.

Zero down time and zero data loss.

The reverse proxy should be used, as it will then add the X-Forwarded-For header to the connections, which the new server should be configured using mod_rpaf (if Apache is used on the new server, Nginx would be better, but that's another topic) to look for and use as the real IP connecting. Just forwarding the data without this will cause the logs to fill with connections that look like they are coming from the old server and any protection on user account login (such as the forum user login) may trigger lockouts due to all the users seeming to come from the same IP address.

Agreed, that's a way to do it - but its not zero downtime, you still need to config the Master and reroute traffic on the fly, and get everything spot on. According to what Dave quoted in another email the ops guys seemed to have rsync'ed the table files directly then did traffic routing above the boxes. Probably safer to script wrap and automate. Yes, Nginx would be handy.

[gnif]

but you won't be able to write posts on the new system. OR copy across the DB and point the old server database config at the new server - this assumes the forum and other software would deal correctly with being 'multiheaded' and not get its session knickers in a twist 

Basically moving a whole single box system to a new box will usually incur some down time somewhere, you haven't got enough kit to stage it.

This only occurs if you don't know what you are doing. You can write to a MySQL slave, but you must ensure that once you start writing to it, you don't write to the master. This scenario works perfect for an instant switch-over.

1) Slave MySQL database to the old server
2) Configure a reverse proxy to forward HTTP/HTTPs traffic to the new server
3) [optional] use socat or iptables to forward traffic for POP3/IMAP/SMTP, etc. to the new server also
4) De-configure MySQL slave
5) Update DNS, wait for propagation to occur
6) Decommission old server.

Zero down time and zero data loss.

The reverse proxy should be used, as it will then add the X-Forwarded-For header to the connections, which the new server should be configured using mod_rpaf (if Apache is used on the new server, Nginx would be better, but that's another topic) to look for and use as the real IP connecting. Just forwarding the data without this will cause the logs to fill with connections that look like they are coming from the old server and any protection on user account login (such as the forum user login) may trigger lockouts due to all the users seeming to come from the same IP address.

Agreed, that's a way to do it - but its not zero downtime, you still need to config the Master and reroute traffic on the fly, and get everything spot on. According to what Dave quoted in another email the ops guys seemed to have rsync'ed the table files directly then did traffic routing above the boxes. Probably safer to script wrap and automate. Yes, Nginx would be handy.

This is zero downtime, and configuring the traffic to reroute does not matter how long it takes once the database slave is configured as both servers are kept in sync. A MySQL master/slave can be configured with zero downtime using the percona tools to take a live dump of the database to import into the new server without having to lock the tables. When I said zero downtime with zero loss, I meant it, I do this on a daily basis as I work for a contracted server management company who support the 'Linux pros' when they get stuck.



Edit: sorry, you can be right depending on if the master server has binlogs enabled or not, and the server_id is set, so you may get 1-5 seconds of downtime to enable these features.

Also the Nginx changeover can be seemless also as you make it listen on say port 81 and then insert a port redirect rule into iptables for new connections only, so no need to stop Apache and then start nginx. If this is not desired, there is a <1 second 'service stop apache/http && service start nginx' to run .

[manicdoc]

(snip long quote..)

This is zero downtime, and configuring the traffic to reroute does not matter how long it takes once the database slave is configured as both servers are kept in sync. A MySQL master/slave can be configured with zero downtime using the percona tools to take a live dump of the database to import into the new server without having to lock the tables. When I said zero downtime with zero loss, I meant it, I do this on a daily basis as I work for a contracted server management company who support the 'Linux pros' when they get stuck.



Edit: sorry, you can be right depending on if the master server has binlogs enabled or not, and the server_id is set, so you may get 1-5 seconds of downtime to enable these features.

Also the Nginx changeover can be seemless also as you make it listen on say port 81 and then insert a port redirect rule into iptables for new connections only, so no need to stop Apache and then start nginx. If this is not desired, there is a <1 second 'service stop apache/http && service start nginx' to run .

Yep, that is my (minor) point, you need to have the right set up before hand to get the absolute zero downtime.

BTW what are your thoughts on M/M pairing for a live site - i.e. 2 boxes dual serving with M/M DB set up? I'm researching on this atm, be interested if you have any thoughts. thanks.

[gnif]

(snip long quote..)

This is zero downtime, and configuring the traffic to reroute does not matter how long it takes once the database slave is configured as both servers are kept in sync. A MySQL master/slave can be configured with zero downtime using the percona tools to take a live dump of the database to import into the new server without having to lock the tables. When I said zero downtime with zero loss, I meant it, I do this on a daily basis as I work for a contracted server management company who support the 'Linux pros' when they get stuck.



Edit: sorry, you can be right depending on if the master server has binlogs enabled or not, and the server_id is set, so you may get 1-5 seconds of downtime to enable these features.

Also the Nginx changeover can be seemless also as you make it listen on say port 81 and then insert a port redirect rule into iptables for new connections only, so no need to stop Apache and then start nginx. If this is not desired, there is a <1 second 'service stop apache/http && service start nginx' to run .

Yep, that is my (minor) point, you need to have the right set up before hand to get the absolute zero downtime.

BTW what are your thoughts on M/M pairing for a live site - i.e. 2 boxes dual serving with M/M DB set up? I'm researching on this atm, be interested if you have any thoughts. thanks.

Master-Master set-ups are usually more trouble then they are worth, if you get a sync issue then it is an absolute nightmare to recover from it, it would be better to go with MySQL cluster, which contrary to popular belief, is open source also and freely available. The later versions of the NDB storage engine have resolved most of the issues they used to have with replication across slow links as well as missing features such as key lengths and text search. If you are looking for load balanced configuration and clustering is overkill, you could use a MySQL Master/Slave, and throw MySQL proxy in there to balance read requests to the two servers, this works exceptionally well (note: the warnings about MySQL-Proxy not being ready for production use are rubbish, as MySQL-Proxy is used in the commercial MySQL enterprise set-up which they sell, it is just not advertised).

[EEVblog]

Is it my imagination, or is the new server noticeably quicker to load pages?
BTW, this is the fastest forum I use, and that includes ones on servers in Oz.

Dave.

[gnif]

Coming form AU here it seems quicker for me. Still get that 250ms international latency, but that is not related to your server, just our crummy international link.

[EEVblog]

while you can use the REd drives as a general purpose drive they are tuned for NAS and RAID applications.
http://www.wdc.com/en/products/products.aspx?id=810
so in case of a server where you use RAId : these drives are ideal.

Good to see they picked the right ones then!

Dave.

[manicdoc]

(snip long quote..)

This is zero downtime, and configuring the traffic to reroute does not matter how long it takes once the database slave is configured as both servers are kept in sync. A MySQL master/slave can be configured with zero downtime using the percona tools to take a live dump of the database to import into the new server without having to lock the tables. When I said zero downtime with zero loss, I meant it, I do this on a daily basis as I work for a contracted server management company who support the 'Linux pros' when they get stuck.



Edit: sorry, you can be right depending on if the master server has binlogs enabled or not, and the server_id is set, so you may get 1-5 seconds of downtime to enable these features.

Also the Nginx changeover can be seemless also as you make it listen on say port 81 and then insert a port redirect rule into iptables for new connections only, so no need to stop Apache and then start nginx. If this is not desired, there is a <1 second 'service stop apache/http && service start nginx' to run .

Yep, that is my (minor) point, you need to have the right set up before hand to get the absolute zero downtime.

BTW what are your thoughts on M/M pairing for a live site - i.e. 2 boxes dual serving with M/M DB set up? I'm researching on this atm, be interested if you have any thoughts. thanks.

Master-Master set-ups are usually more trouble then they are worth, if you get a sync issue then it is an absolute nightmare to recover from it, it would be better to go with MySQL cluster, which contrary to popular belief, is open source also and freely available. The later versions of the NDB storage engine have resolved most of the issues they used to have with replication across slow links as well as missing features such as key lengths and text search. If you are looking for load balanced configuration and clustering is overkill, you could use a MySQL Master/Slave, and throw MySQL proxy in there to balance read requests to the two servers, this works exceptionally well (note: the warnings about MySQL-Proxy not being ready for production use are rubbish, as MySQL-Proxy is used in the commercial MySQL enterprise set-up which they sell, it is just not advertised).

Okay, looks like the old M/S set up is the way to go; the boxes would be quite close. I'm used to sharded 'farms' of M/S set ups in my old job. So a familiar stomping ground. thanks.

[EEVblog]

TEST...
I just changed my PHP5 from suphp to fgi
*random pic attachment*

Dave.

[gnif]

TEST...
I just changed my PHP5 from suphp to fgi
*random pic attachment*

Dave.

Looking good . For those that may see this as a security concern, consider that Dave is only hosting the one website on the server, it does not make sense running suPHP on a non-shared server.

[gnif]

Okay, looks like the old M/S set up is the way to go; the boxes would be quite close. I'm used to sharded 'farms' of M/S set ups in my old job. So a familiar stomping ground. thanks.

You're welcome

[EEVblog]

TEST...
I just changed my PHP5 from suphp to fgi
*random pic attachment*

make that fcgi

And is the forum even faster again? 

Dave.

[manicdoc]

TEST...
I just changed my PHP5 from suphp to fgi
*random pic attachment*

Dave.

Looking good . For those that may see this as a security concern, consider that Dave is only hosting the one website on the server, it does not make sense running suPHP on a non-shared server.

Question is - is opcode caching enabled; icing on the cake...

[gnif]

TEST...
I just changed my PHP5 from suphp to fgi
*random pic attachment*

Dave.

Looking good . For those that may see this as a security concern, consider that Dave is only hosting the one website on the server, it does not make sense running suPHP on a non-shared server.

Question is - is opcode caching enabled; icing on the cake...

Possibly not, but this is something that I would not suggest just turning on as it requires a PHP recompile to enable it. Also APC is better then what cPanel provides (eAccelerator, XCache) and is pretty simple to configure and install and can be done without re-building PHP.

Edit: Granted you can do the others without re-building also, but to do it the cPanel way, Dave will need to recompile it.

[EEVblog]

Possibly not, but this is something that I would not suggest just turning on as it requires a PHP recompile to enable it. Also APC is better then what cPanel provides (eAccelerator, XCache) and is pretty simple to configure and install and can be done without re-building PHP.

I believe that Simple Machine forum supports these caches if installed. At the moment, just basic caching is turned on in the forum.

Dave.

[gnif]

Possibly not, but this is something that I would not suggest just turning on as it requires a PHP recompile to enable it. Also APC is better then what cPanel provides (eAccelerator, XCache) and is pretty simple to configure and install and can be done without re-building PHP.

I believe that Simple Machine forum supports these caches if installed. At the moment, just basic caching is turned on in the forum.

Dave.

SMF caching is different, op-code caching is where the compiled interpreted PHP code is stored in RAM or even on disk, and next time the script loads, if the file on disk has not been changed, it uses the pre-compiled one in cache instead of having to re-compile the script again.

[Thor-Arne]

I just had my first database error. 
So the new server didn't help on that problem at all....

[PA0PBZ]

I just had my first database error. 
So the new server didn't help on that problem at all....

Same here...

[EEVblog]

I just had my first database error. 
So the new server didn't help on that problem at all....

I've got SMF set to email me on every database connection error. Nothing today, maybe 6 emails in the last week total.
Once again, the new server is running very low in terms of processing capacity, no where near peak.
It's not an overload capacity, and HostGator upped my max connections (not that I think it matters at all).
Only some people seem to be getting this error. I never get it, and I'm one of the heaviest forum user, and on the other side of the planet. That says it's something more unusual/subtle that is perhaps user condition specific.
If it was truly a random overload issue with the server and/or MySQL et.al, then everyone would see it eventually.
I suspect for SMF to have an option in admin that emails you every time this message appears, it perhaps has somethgin to do with SMF code not being rebust enough in this area? (and it doesn't email me when people say they get see it!)
My mystery continues...

Dave.

[gnif]

I have seen this error a few times myself as well. The reason why the max connections was suspected is exactly because the server is not near peak load, this is an artificial limit imposed, each connection might be doing nothing, but they still count towards the total limit. To track this down some exception handing will need to be added to SMF to log to a file as it seems that SMF logs to the database directly, which if it cant talk to the database, how can it log? Bit of a bad design there.

[EEVblog]

gnif now has the power of god, and is hunting down the culprit 

Dave.

[PA0PBZ]

So come on everybody, make the error happen! 

[EEVblog]

So come on everybody, make the error happen! 

What error? 
 

Dave.

[gnif]

I have added additional logging to SMF to hopefully see the details of the error the next time it occurs. It seems that the max_connections had only been set in the config, but it was set to the default, 150, so I have increased this. The database was also limiting each user to only 50 connections, so this also has been increased.

In short, if any more errors appear, please let me know .

[gnif]

testing uploads

[bingo600]

Testing upload - i have been getting err 500's all day when attaching a .jpg
I'll try a pdf

Hmmm pdf's are ok , but still err-500 on .jpg's  :-(

/Bingo

[gnif]

Testing upload - i have been getting err 500's all day when attaching a .jpg
I'll try a pdf

/Bingo

Can you PM me your IP address? If you do not know how to get it, google 'what is my ip' and google will show it just under the search box in bold text

[bingo600]

Done ...

And the IP is listed in the post it self , where one can report to moderator ...
You can prob only see your own ip ..

/Bingo

[gnif]

Done ...

And the IP is listed in the post it self , where one can report to moderator ...
You can prob only see your own ip ..

/Bingo

Thanks, and yes, Dave gave me the keys to the sever, but I would rather not touch anything I do not need to. This should be fixed now, can you please try it again?

[bingo600]

Still err-500 - pm sent

/Bingo

[gnif]

Thanks, no need for the PM, just did not want you to expose your IP to everyone before.

Can you please try again, it should be fixed now.

[bingo600]

test .jpg

[bingo600]

Heyy ... Nice work 

Thanx

/Bingo

[gnif]

Great! Thanks for testing for me, makes it easier to track these things down.

[gnif]

I just saw the database exceed the old max connection limit, this would indicate that it was indeed the cause of the database error problem and it has now been resolved.

[gnif]

Sigh, the outage that just occurred was due to the way the old server was transferred to the new server, the issue has been fixed, sorry for the any inconvenience that it may have caused.

[bingo600]

You mean this one 

Connection Problems
Sorry, SMF was unable to connect to the database. This may be caused by the server being busy. Please try again later.

Thanx for your effort 

/Bingo

[gnif]

You mean this one 

Connection Problems
Sorry, SMF was unable to connect to the database. This may be caused by the server being busy. Please try again later.

Thanx for your effort 

/Bingo

One of them, there was also a big bolded "500" error, and a a few others. If you see any issues as of now, please let me know.
I may not see them for 8 hours or so though as I am going to bed, it is 4AM here in AU.

[firewalker]

Just gave me the same error. I had to reload 5~6 times.

Alexander.

[gnif]

very odd, let me look into it.

[bingo600]

If you see any issues as of now, please let me know.
I may not see them for 8 hours or so though as I am going to bed, it is 4AM here in AU.

Will post here with message & a time , if i encounter any probs.

Sleep tight down there

/Bingo

[gnif]

Thanks.

My additional logging caught the error which was a time-out on the connection, seems the connection time-out was set to 4 seconds :S. I have increased this, again, if errors are noticed please update this thread.

[manicdoc]

Possibly not, but this is something that I would not suggest just turning on as it requires a PHP recompile to enable it. Also APC is better then what cPanel provides (eAccelerator, XCache) and is pretty simple to configure and install and can be done without re-building PHP.

I believe that Simple Machine forum supports these caches if installed. At the moment, just basic caching is turned on in the forum.

Dave.

SMF caching is different, op-code caching is where the compiled interpreted PHP code is stored in RAM or even on disk, and next time the script loads, if the file on disk has not been changed, it uses the pre-compiled one in cache instead of having to re-compile the script again.

Yep, it can make a big difference to the CPU load. I think Dave was referring to memcache(d) support. Be good to check if that is operational whilst you are in there, given its a dedicated machine it should be safe to enable. SMF should use it.

[croberts]

The home page is not loading if I use www.eevblog.com/forum/index.php (the forum home button uses this url).
The home page will load if I use www.eevblog.com/forum/

[bingo600]

The https://www.eevblog.com/forum/index.php

Loads fine here Firefox 19.0 (Linux)  , just tried your url 2 times

/Bingo

[croberts]

I'm using firefox 19.0 (Windows 7). The url https://www.eevblog.com/forum/index.php worked fine for me until today. If I remove index.php it loads fine.

[gnif]

Can you try the broken address and hit refresh while holding shift? I suspect that you have cached the down site as I can not reproduce this problem.

[manicdoc]

Thanks.

My additional logging caught the error which was a time-out on the connection, seems the connection time-out was set to 4 seconds :S. I have increased this, again, if errors are noticed please update this thread.

I'm assuming thats not the wait_timeout or something is really off with the setup. Also I'm presuming its a local port connect and not socket?

Also given the fact that we seem to require a hell of a lot of connections in proportion to the traffic that the forum s/w isn't that well optimized around db usage. I might download a copy myself and have a peek (btw direct email is keith at aykira dot com).

[croberts]

Hello gnif

I cleared the cache and restarted firefox. Now the url works. Thanks for your help.

[gnif]

Thanks.

My additional logging caught the error which was a time-out on the connection, seems the connection time-out was set to 4 seconds :S. I have increased this, again, if errors are noticed please update this thread.

I'm assuming thats not the wait_timeout or something is really off with the setup. Also I'm presuming its a local port connect and not socket?

Also given the fact that we seem to require a hell of a lot of connections in proportion to the traffic that the forum s/w isn't that well optimized around db usage. I might download a copy myself and have a peek (btw direct email is keith at aykira dot com).

It is a local connect via a unix socket, but the issue was resolved as far as I can tell at the moment. The slow query can not be avoided easily, it is when someone searches the forum, I have already investigated added a fulltext index to the posts to improve this, but it will require some code changes to implement it. The database was exceeding 50 connections across 200+ users, which is not bad at all. On average there is 4-10 connections open to the database, but when a few of these searches occur the numbers jump up a bit. The max connections has been increased and since the last round of fixes I have not seen any more instances in the additional logging I added to SMF.

Hello gnif

I cleared the cache and restarted firefox. Now the url works. Thanks for your help.

You're welcome

[manicdoc]

Thanks.

My additional logging caught the error which was a time-out on the connection, seems the connection time-out was set to 4 seconds :S. I have increased this, again, if errors are noticed please update this thread.

I'm assuming thats not the wait_timeout or something is really off with the setup. Also I'm presuming its a local port connect and not socket?

Also given the fact that we seem to require a hell of a lot of connections in proportion to the traffic that the forum s/w isn't that well optimized around db usage. I might download a copy myself and have a peek (btw direct email is keith at aykira dot com).

It is a local connect via a unix socket, but the issue was resolved as far as I can tell at the moment. The slow query can not be avoided easily, it is when someone searches the forum, I have already investigated added a fulltext index to the posts to improve this, but it will require some code changes to implement it. The database was exceeding 50 connections across 200+ users, which is not bad at all. On average there is 4-10 connections open to the database, but when a few of these searches occur the numbers jump up a bit. The max connections has been increased and since the last round of fixes I have not seen any more instances in the additional logging I added to SMF.

Funny, I thought it was exactly the text search when Dave first had the problem (linear search across all rows if memory serves). See if you can get memcache doing its thing - myself I like to keep requests for static content 'away' from the database and let it focus on the truly dynamic none cacheable queries. How many thread instances of mysqld are running?  Also be careful on the text search coding (i.e. guard against stupid hogging searches).

We'll get the speed stripes on this set up yet 

[EEVblog]

One of them, there was also a big bolded "500" error, and a a few others. If you see any issues as of now, please let me know.
I may not see them for 8 hours or so though as I am going to bed, it is 4AM here in AU.

Huge thanks to gnif who worked some incredible penguin magic into the wee hours of the morning to fix the server that HostGator was supposed to professionally install properly. Turned out they didn't do such a professional job.

AND something else potentially BIG has also been discovered.
I won't effect the server in the short or medium term, but it will effect HostGaotr reputation if they do not explain...   

Dave.

[Monkeh]

One of them, there was also a big bolded "500" error, and a a few others. If you see any issues as of now, please let me know.
I may not see them for 8 hours or so though as I am going to bed, it is 4AM here in AU.

Huge thanks to gnif who worked some incredible penguin magic into the wee hours of the morning to fix the server that HostGator was supposed to professionally install properly. Turned out they didn't do such a professional job.

AND something else potentially BIG has also been discovered.
I won't effect the server in the short or medium term, but it will effect HostGaotr reputation if they do not explain...   

Dave.

They're a large company and they use cPanel. Lack of quality is to be expected.

[gnif]

One of them, there was also a big bolded "500" error, and a a few others. If you see any issues as of now, please let me know.
I may not see them for 8 hours or so though as I am going to bed, it is 4AM here in AU.

Huge thanks to gnif who worked some incredible penguin magic into the wee hours of the morning to fix the server that HostGator was supposed to professionally install properly. Turned out they didn't do such a professional job.

AND something else potentially BIG has also been discovered.
I won't effect the server in the short or medium term, but it will effect HostGaotr reputation if they do not explain...   

Dave.

They're a large company and they use cPanel. Lack of quality is to be expected.

You are more then welcome Dave 

cPanel does not imply bad quality, but that is not so much the issue here. I will not announce what it is and leave it with Dave to follow up and give them a chance to resolve it before they get publicly exposed for it.

As for the migration they performed... well... this is why I hate it when people claim to be a 'Linux Professional', so often they end up pulling this crap.

They installed the new server with CentOS6, then performed a rsync of the entire root filesystem minus /boot to the new server, so the machine was running on a later kernel but the rest of the system was on CentOS5. This caused all sorts of stability problems with MySQL & PHP, which last night when the server started to get loaded down with more requests caused the service to completely crash. Also due to the method of sync, a heap of CentOS6 files were left laying around on the system, specifically the MySQL client libraries for 5.5, so even re-building PHP was a problem as the build kept finding the newer libs and trying to use them for MySQL 5.0. The final fix was to upgrade MySQL to 5.5, remove some HostGator custom cPanel addons that prevented PHP from building, and re-build PHP/Apache via easyapache. Sigh, what a mess.


They have confirmed that they did just do a straight copy between servers, very odd though that there was CentOS6 files hanging about from RPMs, perhaps someone made a mistake some time in the past, or cPanel did something strange. I apologise to HostGator for the quick jump to the incorrect conclusion.

The correct way to do this would have been to install cPanel on the new server, on the nice clean CentOS6 install and then use cPanel's backup and restore feature to copy the site to the new server... rsync JUST the web directory and if not using MySQL master/slace, perform a mysqldump and import on the new server to nothing gets lost during the migration, forward data to the new server and hand the keys to the owner. Due to the way they performed it, the new system is only running 32bit still which is also a reason for limited database performance because it can not use as much RAM as it should be able to use.

Anyway, amongst all those changes I also enabled HTTP compression of text based content, as the new server has ample CPU resources to do it, and prevented the server from advertising it's Apache version, modules and PHP from advertising it's version in the headers, mainly for security.

[manicdoc]

One of them, there was also a big bolded "500" error, and a a few others. If you see any issues as of now, please let me know.
I may not see them for 8 hours or so though as I am going to bed, it is 4AM here in AU.

Huge thanks to gnif who worked some incredible penguin magic into the wee hours of the morning to fix the server that HostGator was supposed to professionally install properly. Turned out they didn't do such a professional job.

AND something else potentially BIG has also been discovered.
I won't effect the server in the short or medium term, but it will effect HostGaotr reputation if they do not explain...   

Dave.

They're a large company and they use cPanel. Lack of quality is to be expected.

You are more then welcome Dave 

cPanel does not imply bad quality, but that is not so much the issue here. I will not announce what it is and leave it with Dave to follow up and give them a chance to resolve it before they get publicly exposed for it.

As for the migration they performed... well... this is why I hate it when people claim to be a 'Linux Professional', so often they end up pulling this crap.

They installed the new server with CentOS6, then performed a rsync of the entire root filesystem minus /boot to the new server, so the machine was running on a later kernel but the rest of the system was on CentOS5. This caused all sorts of stability problems with MySQL & PHP, which last night when the server started to get loaded down with more requests caused the service to completely crash. Also due to the method of sync, a heap of CentOS6 files were left laying around on the system, specifically the MySQL client libraries for 5.5, so even re-building PHP was a problem as the build kept finding the newer libs and trying to use them for MySQL 5.0. The final fix was to upgrade MySQL to 5.5, remove some HostGator custom cPanel addons that prevented PHP from building, and re-build PHP/Apache via easyapache. Sigh, what a mess.

The correct way to do this would have been to install cPanel on the new server, on the nice clean CentOS6 install and then use cPanel's backup and restore feature to copy the site to the new server... rsync JUST the web directory and if not using MySQL master/slace, perform a mysqldump and import on the new server to nothing gets lost during the migration, forward data to the new server and hand the keys to the owner. Due to the way they performed it, the new system is only running 32bit still which is also a reason for limited database performance because it can not use as much RAM as it should be able to use.

Anyway, amongst all those changes I also enabled HTTP compression of text based content, as the new server has ample CPU resources to do it, and prevented the server from advertising it's Apache version, modules and PHP from advertising it's version in the headers, mainly for security.

ugh, thats just plain nasty and so wrong.  Especially given cPanel is there to make this painless; even via WHM you can remote pull a site to a new box...    Obviously dealing with dedicated servers is not their 'strength'. Well done in sorting out the true dogs dinner of an upgrade.

I take it you put mod_security in? Also check the cache settings on .js files - got some strange behaviour with the jquery lib on the WP homepage, kept being pulled down on every load.

[Monkeh]

cPanel does not imply bad quality, but that is not so much the issue here.

cPanel is one gigantic security hole waiting to happen (and doing so fairly regularly). It's also one gigantic limitation on what you can do when it's the only avenue of access you have (which, if present, it usually is).

They installed the new server with CentOS6, then performed a rsync of the entire root filesystem minus /boot to the new server, so the machine was running on a later kernel but the rest of the system was on CentOS5. This caused all sorts of stability problems with MySQL & PHP, which last night when the server started to get loaded down with more requests caused the service to completely crash. Also due to the method of sync, a heap of CentOS6 files were left laying around on the system, specifically the MySQL client libraries for 5.5, so even re-building PHP was a problem as the build kept finding the newer libs and trying to use them for MySQL 5.0. The final fix was to upgrade MySQL to 5.5, remove some HostGator custom cPanel addons that prevented PHP from building, and re-build PHP/Apache via easyapache. Sigh, what a mess.

I don't even have words for that. This is why I don't deal with such companies..

[gnif]

ugh, thats just plain nasty and so wrong.  Especially given cPanel is there to make this painless; even via WHM you can remote pull a site to a new box...    Obviously dealing with dedicated servers is not their 'strength'. Well done in sorting out the true dogs dinner of an upgrade.

I take it you put mod_security in? Also check the cache settings on .js files - got some strange behaviour with the jquery lib on the WP homepage, kept being pulled down on every load.

In the rush to get the site back on-line and discovering broken custom config I just disabled mod_sec and their custom crap for the moment, it is top of my list to fix today . I can not replicate the cache issue, chrome is telling me that all the static files are coming from cache, as expected.

cPanel is one gigantic security hole waiting to happen (and doing so fairly regularly). It's also one gigantic limitation on what you can do when it's the only avenue of access you have (which, if present, it usually is).

Agreed, I do not like cPanel either, standard it is very insecure, but fixable if you know how. The issue is that it is sold as a 'you can be your own admin' type package but they do not educate people that there is way more to administration of a Linux server then just what cPanel provides. cPanel does not even come with a firewall. To be honest I could gripe on about cPanel for weeks. Personally I prefer a Debian based server, no management interface at all, Nginx+php-fpm over Apache, and my monitoring solution to keep an eye on the server and keep it up to date.

[manicdoc]

ugh, thats just plain nasty and so wrong.  Especially given cPanel is there to make this painless; even via WHM you can remote pull a site to a new box...    Obviously dealing with dedicated servers is not their 'strength'. Well done in sorting out the true dogs dinner of an upgrade.

I take it you put mod_security in? Also check the cache settings on .js files - got some strange behaviour with the jquery lib on the WP homepage, kept being pulled down on every load.

In the rush to get the site back on-line and discovering broken custom config I just disabled mod_sec and their custom crap for the moment, it is top of my list to fix today . I can not replicate the cache issue, chrome is telling me that all the static files are coming from cache, as expected.

Okay, could have been some temporary weirdness in amongst all the other weirdness... I'll have a good crawl around when I'm back home this evening and see if there is any more weirdness to be found..

[EEVblog]

cPanel does not imply bad quality, but that is not so much the issue here. I will not announce what it is and leave it with Dave to follow up and give them a chance to resolve it before they get publicly exposed for it.

Ok, the issue is that gnif discovered that my "dedicated" server looked to be in fact a virtual server. By virtue of the drive directory name, the driver, and the drive bios all being virtual.

I asked HostGator about this, and their response was:

Thank you for contacting HostGator. Our new dedicated servers are
virtualized so as to provide a better management back end however this
is not shared between anyone else and you have full access to all
resources on the server. The Rescue & Hardware Monitoring KVM layer
allows us to monitor the server for any potential hardware failures
and proactively resolve these issues before they result in any
downtime and/or possible data loss. If you have any other questions
please let us know. We are more than happy to assist you. Thanks!

Which actually sounds reasonable to my untrained ears.
They even said in the original email about the new server, that I posted at the start of this topic:

-- All servers also feature an optional Rescue & Hardware Monitoring KVM layer (which is included by default) allowing HostGator to monitor the server for any potential hardware failures and proactively resolve these issues before they result in any downtime and/or possible data loss."

I'll leave it up to the penguin initiated to determine if HostGator are actually trying to pull a swifty here with their new dedicated servers, or if they have genuine reasons for adding that virtual layer.
I don't mind running the virtual layer, if it means HostGator can monitor and detect problems etc. But I'll be really pissed if one single byte of another user touches my dedicated hard drive or processor! 

Dave.

[gnif]

If they are being honest here, what they have done is install a single virtual machine on the hardware, so you are not sharing the resources with other users, but how can you be sure of that? and there is a performance loss incurred by this due to the VM layer.

It is more likely that they are doing this so that if hardware fails they can just transfer the virtual machine to a new server and not have to mess about with reconfiguration.

The disadvantage is, if their hardware monitoring is not up to scratch (which I doubt it is) and they do not detect an error, you do not have the ability to detect errors yourself either or have a third party monitor the server's hardware as you are inside the virtual box without direct hardware access.

I would demand access to the physical box for two reasons:

1) It is a security threat, you need to be sure that what they have done is not going to compromise your host. One of the main reasons of having a dedicated server is that you have total control so you can secure the machine and configure it according you your requirements.

2) To verify that they are indeed doing what they say they are, it all sounds a bit dodgy to me.

Edit: This is not a KVM by the way... a KVM is a physical feature of the server that provides a virtual keyboard, video and mouse interface that allows you to use the server as if you are physically sitting on it. That means you can reboot and configure the bios over the internet if you want, ore recover from a critical failure, etc.

Edit 2: Also if there is a hardware failure, say one of your disks, they can decide not to replace it for as long as they want (ie, lazy, cost saving etc) and you will never know as the RAID configuration will hide the problem from you. This means that you will loose your redundancy for who knows how long.

Edit 3: Security is a problem here also because you dont know if they are keeping that virtual layer up to date and patched with the latest updates, your VM may be 100% secure, but could be back-doored via the (and I use this term loosely) "Virtual KVM".

[EEVblog]

Edit 2: Also if there is a hardware failure, say one of your disks, they can decide not to replace it for as long as they want (ie, lazy, cost saving etc) and you will never know as the RAID configuration will hide the problem from you. This means that you will loose your redundancy for who knows how long.

Wouldn't that same thing happen with or without the KVM layer?
If a RAID 1 drive fails in a truly dedicated box, the server would keep running without batting an eyelid. I don't even know if I get alerted when such a thing happens?

Dave.

[gnif]

Edit 2: Also if there is a hardware failure, say one of your disks, they can decide not to replace it for as long as they want (ie, lazy, cost saving etc) and you will never know as the RAID configuration will hide the problem from you. This means that you will loose your redundancy for who knows how long.

Wouldn't that same thing happen with or without the KVM layer?
If a drive fails in a truly dedicated box, the server keep running without batting an eyelid. I don't even know if I get alerted when such a thing happens?

Dave.

With the KVM in there, you would never know if they had been lazy and never replaced the drive. With a dedicated server it can be configured to email you alerts on failures such as this, but because you do not have physical hardware access you would never know you were running a degraded array, and you would not be able to chase them to fix it.

[drewtronics]

I have to disagree almost entirely with gnif. If you want to be super paranoid, run your own gear, in your own place with your own money.

However, running in a VM container has lots of benefits. VMware has a feature called vmotion which is automatic migration technology for failing over a vm from one machine to another automatically when the host pc fails. Its magic, and would prevent downtime at a big hosting farm. I'm sure there are comparable solutions from other vendors/packages.

It can be much easier to check on the health of machines, see if something is going haywire, and respond to customer needs with everything virtualized. I monitor and run a couple of racks worth of equipment at work, among other duties, and virtualization makes it nearly painless. I even use machines on the virtualization pool for engineering and development duties. It all works great. In this day and age, there not a terrible amount of overhead lost, except for some RAM.  The security risk is minimal. It is much, much, more likely someone will find an exploit in this forum software than the hypervisor running the VMs.

I think its a good move hostgator's part.

[gnif]

I have to disagree almost entirely with gnif. If you want to be super paranoid, run your own gear, in your own place with your own money.

However, running in a VM container has lots of benefits. VMware has a feature called vmotion which is automatic migration technology for failing over a vm from one machine to another automatically when the host pc fails. Its magic, and would prevent downtime at a big hosting farm. I'm sure there are comparable solutions from other vendors/packages.

They are not using VMware nor are they auto moving failed hosts to new machines.

It can be much easier to check on the health of machines, see if something is going haywire, and respond to customer needs with everything virtualized. I monitor and run a couple of racks worth of equipment at work, among other duties, and virtualization makes it nearly painless. I even use machines on the virtualization pool for engineering and development duties. It all works great. In this day and age, there not a terrible amount of overhead lost, except for some RAM.  The security risk is minimal. It is much, much, more likely someone will find an exploit in this forum software than the hypervisor running the VMs.

How so? what is the difference between them running their monitoring software on the same host as the client's software? I did not say there was a risk in the VM tech being compromised, I said there was a risk of the DC not keeping the software on the physical host up to date, such as SSH, the Kernel, etc.

Edit:

And how do you know how much they are doing to monitor the server? how heavy is their resource consumption? Are they scheduling checks of things during peak times the client may need the servers resources for other tasks? If you want a VPS you get a VPS, this is a glorified VPS, not a dedicated server.

And do not get me wrong, I like VM environments, I often deploy XEN, the issue at hand here is they were not honest about it, and Dave is paying for a dedicated server, not a VM.

[drewtronics]

How so? what is the difference between them running their monitoring software on the same host as the client's software? I did not say there was a risk in the VM tech being compromised, I said there was a risk of the DC not keeping the software on the physical host up to date, such as SSH, the Kernel, etc.

I've only used VMware, so I can't compare much else, but in that case, the Virtualization package and the monitoring software are the same thing. You don't really separate them. If they're trying to kludge together one of the free software virtualization solutions at a commercial hosting service, they deserve the problems they are almost certainly going to have. Citrix, VMware, w/e Microsoft is calling it these days, they're all pretty manageable.

For instance, just yesterday I was doing something else on a VM host box and noticed one of the cores was pegged at 100% constantly from one of the graphs I saw navigating through the software. I talk to the owner of the VM, and he hasn't used it in months.  I shut it down and archived the VM to a file server. It could have crashed, been compromised, who knows. His responsibility that he can figure out later - but the management was crazy simple.

Again, it doesn't seem like you've done this. You don't run SSH or other services on the hypervisor (aka thing running the VMs). Its a super minimal install with the virtualization software -- that's it.   Usually you also put the management interface on a different VLAN that's under well guarded security. Standard practice.

[EEVblog]

A reply from HostGator, including replying to gnifs concerns:

You have a virtual machine running on a dedicated server. There is one VM on your hardware and it's configured to use the entire server.
These commands were run on the hypervisor host:
>Server model: Supermicro X9SCD+-F-BH002
[ulhglive-root@d9435 ~]# dmidecode --type baseboard | grep -A 1 Super
Manufacturer: Supermicro
Product Name: X9SCD+-F-BH002

>Processor: Intel(R) Xeon(R) CPU E3-1265L V2 @ 2.50GHz
[ulhglive-root@d9435 ~]# grep model\ name /proc/cpuinfo | head -n 1
model name   : Intel(R) Xeon(R) CPU E3-1265L V2 @ 2.50GHz

>Memory: Kingston DDR3 1333MHz
[ulhglive-root@d9435 ~]# dmidecode -t 17 | grep -B 1 King
Speed: 1333 MHz
Manufacturer: Kingston

>Hard drive: Western Digital RE4 Serial ATA (WD1003FBYX-01Y7B1) x2
[ulhglive-root@d9435 ~]# smartctl -a /dev/sda | grep Model
Model Family: Western Digital RE4 Serial ATA
Device Model: WDC WD1003FBYX-01Y7B1
[ulhglive-root@d9435 ~]# smartctl -a /dev/sdb | grep Model
Model Family: Western Digital RE4 Serial ATA
Device Model: WDC WD1003FBYX-01Y7B1

>Is anyone else sharing my hard drive, my processor, or anything to do
with my box?
>Is all that hardware truly dedicated to me?
Absolutely, you're on top of a hypervisor but no one else is using your resources, full stop.
virsh is a command line interface to libvirt, and its output here shows that there are no other instances running on the server:
[ulhglive-root@d9435 ~]# virsh list --all
Id Name State
----------------------------------------------------
1 instance-0000330b running

[ulhglive-root@d9435 ~]#

>Does this mean I can chose to opt-out of having this KVM layer?
Unfortunately our use of the word "optional" was a mistake. Our network infrastructure (among other things) does not allow outside access to the bare metal, we do not offer a non-kvm solution at this time.

I also wanted to address some of the points mentioned by "gnif" on your forum.

>It is more likely that they are doing this so that if hardware fails they can just transfer the virtual machine to a new server and not have to mess about with reconfiguration.
This is actually completely true. Previously our OS reload/hardware failure procedure required hours of downtime while we made backups, provisioned new drives/hardware, restored backups, fixed issues, etc. Not only is this not efficient for us, it means that your sites, email, etc are down for hours. With the VM layer we can simply move your VM to a new node and you'll be back online in under an hour (depending of course on the amount of data you have).

>It is a security threat, you need to be sure that what they have done is not going to compromise your host. One of the main reasons of having a dedicated server is that you have total control so you can secure the machine and configure it according you your requirements.
This is not a concern as the bare metal is completely isolated from the outside world.

>Edit: This is not a KVM by the way... a KVM is a physical feature of the server that provides a virtual keyboard, video and mouse interface that allows you to use the server as if you are physically sitting on it.
KVM in this context stands for Kernel Virtual Machine, not Keyboard Video Mouse.

>Edit 2: Also if there is a hardware failure, say one of your disks, they can decide not to replace it for as long as they want (ie, lazy, cost saving etc) and you will never know as the RAID configuration will hide the problem from you. This means that you will loose your redundancy for who knows how long.
This is simply not true. A 2 disk RAID array with a failed disk will see severe performance issues and would be noticed, plus if we were going to do this why would we make it a RAID in the first place?

Hopefully this addresses these concerns, if not I'd be happy to answer any further questions you may have. We appreciate your understanding and patience.

--
Cheers,

Sean C.
Linux Systems Administration Shift Lead

Hats off to Hostgator for clearing it up, and tying in my Twitter mumble to my ticket and reading and responding to the forum 

Dave.

[gnif]

How so? what is the difference between them running their monitoring software on the same host as the client's software? I did not say there was a risk in the VM tech being compromised, I said there was a risk of the DC not keeping the software on the physical host up to date, such as SSH, the Kernel, etc.

I've only used VMware, so I can't compare much else, but in that case, the Virtualization package and the monitoring software are the same thing. You don't really separate them. If they're trying to kludge together one of the free software virtualization solutions at a commercial hosting service, they deserve the problems they are almost certainly going to have. Citrix, VMware, w/e Microsoft is calling it these days, they're all pretty manageable.

For instance, just yesterday I was doing something else on a VM host box and noticed one of the cores was pegged at 100% constantly from one of the graphs I saw navigating through the software. I talk to the owner of the VM, and he hasn't used it in months.  I shut it down and archived the VM to a file server. It could have crashed, been compromised, who knows. His responsibility that he can figure out later - but the management was crazy simple.

Again, it doesn't seem like you've done this. You don't run SSH or other services on the hypervisor (aka thing running the VMs). Its a super minimal install with the virtualization software -- that's it.   Usually you also put the management interface on a different VLAN that's under well guarded security. Standard practice.

I have done this many times before, I think you should go back and re-read my initial post where I clearly stated that they do this to make it easier to move a server between hosts, which is essentially what you did in the example you stated. My concern is that they did not let Dave know, he should have been fully informed as to what they have done with his server.

[gnif]

A reply from HostGator, including replying to gnifs concerns:

A 2 disk RAID array with a failed disk will see severe performance issues and would be noticed, plus if we were going to do this why would we make it a RAID in the first place?

A clone raid array does not always see huge performance benefits, often the controller compares reads from both disks against each other negating the ability to read from both disks at once. It would only be noticed if they were using RAID5 of RAID1+0. The reason why I raised this was because often I have dealt with DCs that refuse to change a failed disk even after multiple angry email's to the DC. Many of them expect you to notice the issue and then you to chase them on it.

Edit: Just looked at the specs of the board... it does not have hardware raid, so unless they have installed a controller into it, it will be using software raid, which I do like and there is nothing wrong with it, but I can say for certain that RAID1 on a mdadm soft raid device does not see more then a few % performance gain. Assuming that this is what they have done.

[gnif]

And to eat my own words...

They installed the new server with CentOS6, then performed a rsync of the entire root filesystem minus /boot to the new server, so the machine was running on a later kernel but the rest of the system was on CentOS5. This caused all sorts of stability problems with MySQL & PHP, which last night when the server started to get loaded down with more requests caused the service to completely crash. Also due to the method of sync, a heap of CentOS6 files were left laying around on the system, specifically the MySQL client libraries for 5.5, so even re-building PHP was a problem as the build kept finding the newer libs and trying to use them for MySQL 5.0. The final fix was to upgrade MySQL to 5.5, remove some HostGator custom cPanel addons that prevented PHP from building, and re-build PHP/Apache via easyapache. Sigh, what a mess.

They have confirmed that they did just do a straight copy between servers, very odd though that there was CentOS6 files hanging about from RPMs, perhaps someone made a mistake some time in the past, or cPanel did something strange. I apologise to HostGator for the quick jump to the incorrect conclusion.

[EEVblog]

Ok, I very happy with HostGators response and am going to stick with the VM.
The server is currently running a 32bit version of CentOS though, so I have asked them to upgrade this to 64bit.
So we could have another little hiccup in next day or two in the server when that is swapped over, but it should be good for speed and the long term.

Dave.

[manicdoc]

I have to disagree almost entirely with gnif. If you want to be super paranoid, run your own gear, in your own place with your own money.

However, running in a VM container has lots of benefits. VMware has a feature called vmotion which is automatic migration technology for failing over a vm from one machine to another automatically when the host pc fails. Its magic, and would prevent downtime at a big hosting farm. I'm sure there are comparable solutions from other vendors/packages.

They are not using VMware nor are they auto moving failed hosts to new machines.

It can be much easier to check on the health of machines, see if something is going haywire, and respond to customer needs with everything virtualized. I monitor and run a couple of racks worth of equipment at work, among other duties, and virtualization makes it nearly painless. I even use machines on the virtualization pool for engineering and development duties. It all works great. In this day and age, there not a terrible amount of overhead lost, except for some RAM.  The security risk is minimal. It is much, much, more likely someone will find an exploit in this forum software than the hypervisor running the VMs.

How so? what is the difference between them running their monitoring software on the same host as the client's software? I did not say there was a risk in the VM tech being compromised, I said there was a risk of the DC not keeping the software on the physical host up to date, such as SSH, the Kernel, etc.

Edit:

And how do you know how much they are doing to monitor the server? how heavy is their resource consumption? Are they scheduling checks of things during peak times the client may need the servers resources for other tasks? If you want a VPS you get a VPS, this is a glorified VPS, not a dedicated server.

And do not get me wrong, I like VM environments, I often deploy XEN, the issue at hand here is they were not honest about it, and Dave is paying for a dedicated server, not a VM.

To me If (big if) they are genuine about what they are doing, then the overhead should be quite minimal, as modern chipsets have VM specific features to keep the speed up. BUT there is no way of knowing if the box is indeed being shared. Also given their rather 'hacked up' way of doing a migration - I wouldn't be too sure the image is actually worth the effort carting around. KVM is another issue.

[amspire]

cPanel is one gigantic security hole waiting to happen (and doing so fairly regularly). It's also one gigantic limitation on what you can do when it's the only avenue of access you have (which, if present, it usually is).

I have never had a problem finding hosting companies offering SSH access as well as CPanel. Also everybody offers FTP access. I have been with probably 6 different hosting companies and I have always had SSH terminal access.

The cheapest host was $2 a month and it still had SSH access. I currently have a $5 a month single domain site with SSH and a $20 a month reseller package which is great as you are the one who controls the SSH to all the domains you host. You also have full control of the DNS for all the domains, and you can set up a temporary new web and FTP site in seconds (you do not need to have a registered domain to set up a temporary site - I have done it without any domain name plenty of times).

I have never found the need to spend over $5/month for a single domain yet, but then again, my sites have never been high traffic. I have always had great technical support from the hosting companies I have chosen.

If you only have CPanel access, you are probably with the wrong hosting company.

Richard.

[gnif]

cPanel is one gigantic security hole waiting to happen (and doing so fairly regularly). It's also one gigantic limitation on what you can do when it's the only avenue of access you have (which, if present, it usually is).

I have never had a problem finding hosting companies offering SSH access as well as CPanel. Also everybody offers FTP access. I have been with probably 6 different hosting companies and I have always had SSH terminal access.

The cheapest host was $2 a month and it still had SSH access. I currently have a $5 a month single domain site with SSH and a $20 a month reseller package which is great as you are the one who controls the SSH to all the domains you host. You also have full control of the DNS for all the domains, and you can set up a temporary new web and FTP site in seconds (you do not need to have a registered domain to set up a temporary site - I have done it without any domain name plenty of times).

I have never found the need to spend over $5/month for a single domain yet, but then again, my sites have never been high traffic. I have always had great technical support from the hosting companies I have chosen.

If you only have CPanel access, you are probably with the wrong hosting company.

Richard.

Yeah, you are referring to purchasing an account on the box, Dave has full control over his server as the site is pretty resource intensive. The SSH access he has to the server is full root level access, not per account.

[EEVblog]

Yeah, you are referring to purchasing an account on the box, Dave has full control over his server as the site is pretty resource intensive. The SSH access he has to the server is full root level access, not per account.

Yep, I have the ability to create cpanel accounts and sell space if I wanted, just like any shared web host.

Dave.

[BravoV]

Connection Problems
Sorry, SMF was unable to connect to the database. This may be caused by the server being busy. Please try again later.

Just fyi, this happened again, post this straight after I managed to get into this thread.

It happened for almost for 7 minutes, and tried 5 times browser refreshes still no access to the forum.

[gnif]

This time we know why, HostGator was very quick in getting the transfer under way, the server load is real high at the moment, this is unavoidable.

[ve7xen]

Edit: This is not a KVM by the way... a KVM is a physical feature of the server that provides a virtual keyboard, video and mouse interface that allows you to use the server as if you are physically sitting on it. That means you can reboot and configure the bios over the internet if you want, ore recover from a critical failure, etc.

KVM is Kernel-based Virtual Machine, the most modern of Linux's virtualization layers. Most distros have moved from Xen to KVM, and aside from large hosts that have existing infrastructure, most new deployments use it, including OpenStack etc. You're confused.

Edit 3: Security is a problem here also because you dont know if they are keeping that virtual layer up to date and patched with the latest updates, your VM may be 100% secure, but could be back-doored via the (and I use this term loosely) "Virtual KVM".

One would hope that that bare metal is on a separate management network not accessible from the Internet. Of course, what one hopes, and how things actually are are quite likely different things.

As a managed service I don't see where you have any place to complain about how they set up their infrastructure. If you're paying for management, that's their responsibility and you shouldn't need to know or care how it's set up, you're trusting them to do that for you. Not many are any good at it, but that is what you're paying for.

Honestly I don't see what the point of Dave paying for a "managed" server is at this point. gnif is doing most of the work, and all they're charging you for seems to be giving you less flexibility and having a third party with easy root access to your box.

I'd say get an unmanaged dedicated server, ditch fscking cPanel (what the heck is the point of that monstrosity on a site hosting just a couple vhosts?? it's far more of a liability than an asset) and pay one of your competent forum members a token sum to maintain it for you. Should save you a ton of money and get you a better setup to boot.

[manicdoc]

Ok, I very happy with HostGators response and am going to stick with the VM.
The server is currently running a 32bit version of CentOS though, so I have asked them to upgrade this to 64bit.
So we could have another little hiccup in next day or two in the server when that is swapped over, but it should be good for speed and the long term.

Dave.

Pretty please (with ice cream on top) make sure memcached is running and used on the new OS, your forum s/w should make good use of it and it will offload a lot of the 'same as' type queries from the database. Your running dedicated, so no security problem to worry about.  Also look at eAccelerator, which should not require a Php recompile to use (its a module).

BTW If you happen to find yourself by the Bull and Bush between 7 and 9am tomorrow I can say hi.

[gnif]

Ok, I very happy with HostGators response and am going to stick with the VM.
The server is currently running a 32bit version of CentOS though, so I have asked them to upgrade this to 64bit.
So we could have another little hiccup in next day or two in the server when that is swapped over, but it should be good for speed and the long term.

Dave.

Pretty please (with ice cream on top) make sure memcached is running and used on the new OS, your forum s/w should make good use of it and it will offload a lot of the 'same as' type queries from the database. Your running dedicated, so no security problem to worry about.  Also look at eAccelerator, which should not require a Php recompile to use (its a module).

BTW If you happen to find yourself by the Bull and Bush between 7 and 9am tomorrow I can say hi.

I will be doing this amongst other things to improve performance, but if we make too many changes at once then when there are issues from the migration it makes it harder to track them down. We will get it onto the new install, let it run for a while and then look at performance improvements.

As for APC/XCache/eAccelerator... they can all be compiled as modules, but cPanel does not support this, the next time an update to PHP is performed it will break the server as it wont compile the modules for the new version during the upgrade... thankfully there are build hooks that this can be done in however, just pointing out that it is not quite as simple as you would imagine. We need to keep cPanel operating without admin intervention so that Dave can manage things on the server in a pinch, this kind of custom job needs to be done carefully.

PS: cPanel supports eAccelerator compiled into PHP, not as a module, but I would prefer to get APC on here as it is what the PHP devs are officially supporting and working on.

[manicdoc]

Ok, I very happy with HostGators response and am going to stick with the VM.
The server is currently running a 32bit version of CentOS though, so I have asked them to upgrade this to 64bit.
So we could have another little hiccup in next day or two in the server when that is swapped over, but it should be good for speed and the long term.

Dave.

Pretty please (with ice cream on top) make sure memcached is running and used on the new OS, your forum s/w should make good use of it and it will offload a lot of the 'same as' type queries from the database. Your running dedicated, so no security problem to worry about.  Also look at eAccelerator, which should not require a Php recompile to use (its a module).

BTW If you happen to find yourself by the Bull and Bush between 7 and 9am tomorrow I can say hi.

I will be doing this amongst other things to improve performance, but if we make too many changes at once then when there are issues from the migration it makes it harder to track them down. We will get it onto the new install, let it run for a while and then look at performance improvements.

As for APC/XCache/eAccelerator... they can all be compiled as modules, but cPanel does not support this, the next time an update to PHP is performed it will break the server as it wont compile the modules for the new version during the upgrade... thankfully there are build hooks that this can be done in however, just pointing out that it is not quite as simple as you would imagine. We need to keep cPanel operating without admin intervention so that Dave can manage things on the server in a pinch, this kind of custom job needs to be done carefully.

PS: cPanel supports eAccelerator compiled into PHP, not as a module, but I would prefer to get APC on here as it is what the PHP devs are officially supporting and working on.

No worries, sounds like cPanel needs a tad more flexibility around this perhaps. BTW what you think of the whole APS package management? Got a client looking to use it for custom builds, seems solid.

[gnif]

No worries, sounds like cPanel needs a tad more flexibility around this perhaps. BTW what you think of the whole APS package management? Got a client looking to use it for custom builds, seems solid.

APS? Do you mean APT?

[BravoV]

Not sure .. just now I was trying to hammer the forum by opening multiple tabs > 10 concurrently at my browser, it does "feel" very responsive, have you applied those fine-tunning ?

[gnif]

Heh, no, you are just seeing the effects of the earlier changes, no more tuning will be done until we are on the new server install.

[manicdoc]

No worries, sounds like cPanel needs a tad more flexibility around this perhaps. BTW what you think of the whole APS package management? Got a client looking to use it for custom builds, seems solid.

APS? Do you mean APT?

http://apsstandard.org/applications

packaging standard - claims to be management suite independent - although plesk are pushing it.

[EEVblog]

We need to keep cPanel operating without admin intervention so that Dave can manage things on the server in a pinch, this kind of custom job needs to be done carefully.

Yes, I know this breaks the hearts of the penguin elite, but I need to be able to understand, operate, and if necessary fix things myself. That means cPanel.
I don't have the time or patience to learn anything else, I've been using cPanel for a decade, I feel comfortable with it. It was hard enough coming up to speed on WHM with the new dedicated server.

Dave.

[gnif]

No worries, sounds like cPanel needs a tad more flexibility around this perhaps. BTW what you think of the whole APS package management? Got a client looking to use it for custom builds, seems solid.

APS? Do you mean APT?

http://apsstandard.org/applications

packaging standard - claims to be management suite independent - although plesk are pushing it.

Looks like it is just another package management system, nothing that flash. There is no one solution that fits in this industry, as soon as you get beyond the requirements of basic hosting as your organisation grows you quickly find you need to do things that these applications can do, but require a high level of technical understanding and the surrounding infrastructure. That said, I have not looked into APS at all, all I know is that the best package management system I have ever used would have to be APT as it just handles everything gracefully and is fast. If you need to target a packaging scheme I would suggest APT as you will then cover Debian, Ubuntu, Mint, and all the other Debian based systems out there.

[gnif]

We need to keep cPanel operating without admin intervention so that Dave can manage things on the server in a pinch, this kind of custom job needs to be done carefully.

Yes, I know this breaks the hearts of the penguin elite, but I need to be able to understand, operate, and if necessary fix things myself. That means cPanel.
I don't have the time or patience to learn anything else, I've been using cPanel for a decade, I feel comfortable with it. It was hard enough coming up to speed on WHM with the new dedicated server.

Dave.

Heh, doesn't phase me one bit, if it was not for platforms like cPanel the Linux community would suffer due to lack of users. There has to be commercialisation somewhere in it all, and cPanel are pretty quick at responding to and helping their users with technical problems when it does go that way. They have left things open enough to allow modifications and plugins to WHM (especially over the last 12 months), it is not all that bad, just some of it could be done better.

Ps: My number one wishlist item for cPanel is to give the option of running Nginx... but I know that this will break compatibility with so many sites that expect to have .htaccess support, or specific Apache modules, so it does not make much commercial sense to spend the resources on adding it when the amount of users that will use it are rather small in comparison to those that are happy with Apache.

[manicdoc]

No worries, sounds like cPanel needs a tad more flexibility around this perhaps. BTW what you think of the whole APS package management? Got a client looking to use it for custom builds, seems solid.

APS? Do you mean APT?

http://apsstandard.org/applications

packaging standard - claims to be management suite independent - although plesk are pushing it.

Looks like it is just another package management system, nothing that flash. There is no one solution that fits in this industry, as soon as you get beyond the requirements of basic hosting as your organisation grows you quickly find you need to do things that these applications can do, but require a high level of technical understanding and the surrounding infrastructure. That said, I have not looked into APS at all, all I know is that the best package management system I have ever used would have to be APT as it just handles everything gracefully and is fast. If you need to target a packaging scheme I would suggest APT as you will then cover Debian, Ubuntu, Mint, and all the other Debian based systems out there.

Agreed on APT for general package management. Unfortunately they are on windows server atm (working on it). APS looks to 'reach up' into the hosting framework and automates a lot of the set up for you across all the affected hosted services as required. Also covers Ux.

[kioan]

This is not a KVM by the way... a KVM is a physical feature of the server that provides a virtual keyboard, video and mouse interface that allows you to use the server as if you are physically sitting on it.

I'm almost sure that by KVM they mean Kernel-based Virtual Machine and not keyboard/video/mouse switch.

Using a VM on a dedicated host is not necessary a bad thing (with XCP paravirtualization I've never found noticeable performance issues), but the hosting company should inform it's customers about that. When I buy a dedicated server I would expect that full access to the hardware. Some companies offer access to the server hardware via a Remote Access Card so that you can access the bios and boot it remotely from your media.

[gnif]

This is not a KVM by the way... a KVM is a physical feature of the server that provides a virtual keyboard, video and mouse interface that allows you to use the server as if you are physically sitting on it.

I'm almost sure that by KVM they mean Kernel-based Virtual Machine and not keyboard/video/mouse switch.

You are correct if you look at the response that Dave posted.

[EEVblog]

Using a VM on a dedicated host is not necessary a bad thing (with XCP paravirtualization I've never found noticeable performance issues), but the hosting company should inform it's customers about that.

To HostGators credit, they did inform me about that (see my first post), I just didn't know what it really meant at the time.

Dave.

[ve7xen]

if it was not for platforms like cPanel the Linux community would suffer due to lack of users.

ROFLMFAO. 

[EEVblog]

The forum and site is now back up, so I'm assuming the server CentOS upgrade is now complete. 

Dave.

[Monkeh]

cPanel is one gigantic security hole waiting to happen (and doing so fairly regularly). It's also one gigantic limitation on what you can do when it's the only avenue of access you have (which, if present, it usually is).

I have never had a problem finding hosting companies offering SSH access as well as CPanel. Also everybody offers FTP access.

Ahh, FTP, so secure, so modern. Good luck with that.

Any host still running an FTP daemon should be shot.

[Thor-Arne]

Hm, the site went completely down. 

Seems like a post was lost during the down.
I got an email notification that there was new post in a topic, but nothing new there.

[Psi]

Seems to be loading pages faster than it used too.
But that might just be because its 2:36am here.

[EEVblog]

Ok, confirmed. The server is now running CENTOS 6.3 64bit with the KVM layer.
Seems to be working just fine. Server load looks normal.

Yes, sorry, the server went down completely. Guess they couldn't do it without some down time, or that was the best option for them.
I was hoping no posts would be lost, but that could have happened if they didn't do a final database backup after they took it offline.

Dave.

[EEVblog]

Seems to be loading pages faster than it used too.

In theory it could do. The 64bit implementation should be faster.

Dave.

[Monkeh]

Seems to be loading pages faster than it used too.

In theory it could do. The 64bit implementation should be faster.

Dave.

Not really. It should actually be ever so slightly slower until you start needing the address space. It also increases memory usage (not that you care on a proper machine. Embedded cares, though.)

Updating to an OS which is mildly recent, however, should've helped.

[amspire]

Ahh, FTP, so secure, so modern. Good luck with that.

Any host still running an FTP daemon should be shot.

FTP is great. I needed a manual for an '80s bit of gear, and the manufacturer didn't post any manuals that old. But they had an internal company FTP site with an anonomous login. There in one of the employees folders was the manual.

Same thing with a notebook I got 8 years ago. I was able to get manufacturing docs for the notebook from a manufacture's unsecured FTP. Guys like you just want to spoil things. 

[gnif]

Seems to be loading pages faster than it used too.

In theory it could do. The 64bit implementation should be faster.

Dave.

Not really. It should actually be ever so slightly slower until you start needing the address space. It also increases memory usage (not that you care on a proper machine. Embedded cares, though.)

Updating to an OS which is mildly recent, however, should've helped.

Actually he does need the address space, MySQL can be inherently unstable when caching more then 2.4GB of data in RAM on a 32bit system, which we are doing here.

I did not notice the server had been brought back on-line already, and email notifications from the forum seem to be broken, looks like I may have missed Dave as he needs to grant me access to the new install so I can track these issues down.

[gnif]

if it was not for platforms like cPanel the Linux community would suffer due to lack of users.

ROFLMFAO. 

Seems they lost my post in the migration... so much for ensuring database consistency. Anyway, still had the window open, so here it is

Dont see what is so funny, it has allowed/forced many many users to become familiar with Linux in the hosting environment which has pushed them to give it a go in other environments. Bugs get fixed, usability problems are found, etc, simply because more people are using it. I am not saying that it was 'cPamel' that made linux, I am saying that commercial products like cPanel help push more users towards Linux because they either want to cut costs, or they have no choice.

[grumpydoc]

They installed the new server with CentOS6, then performed a rsync of the entire root filesystem minus /boot to the new server,

They did WHAT?

Facepalm++

[Monkeh]

Ahh, FTP, so secure, so modern. Good luck with that.

Any host still running an FTP daemon should be shot.

FTP is great. I needed a manual for an '80s bit of gear, and the manufacturer didn't post any manuals that old. But they had an internal company FTP site with an anonomous login. There in one of the employees folders was the manual.

Same thing with a notebook I got 8 years ago. I was able to get manufacturing docs for the notebook from a manufacture's unsecured FTP. Guys like you just want to spoil things. 

No, I want to secure things. FTP is wide open, unreliable, and just generally shite.

Actually he does need the address space, MySQL can be inherently unstable when caching more then 2.4GB of data in RAM on a 32bit system, which we are doing here.

Never said he didn't need it, just that 64-bit does not necessarily equate to greater performance.

[Thor-Arne]

Hm, now the main site seems to be down.... 

[gnif]

works for me, but like I said, if there are any issues I can't do anything about it until I get access again . I guess it'll be tomorrow some time.

[ve7xen]

Dont see what is so funny, it has allowed/forced many many users to become familiar with Linux in the hosting environment which has pushed them to give it a go in other environments. Bugs get fixed, usability problems are found, etc, simply because more people are using it. I am not saying that it was 'cPamel' that made linux, I am saying that commercial products like cPanel help push more users towards Linux because they either want to cut costs, or they have no choice.

But that's not what you said, you said 'if it was not for platforms like cPanel the Linux community would suffer due to lack of users.', which means there wouldn't be enough users without it, which is hilarious. Linux has been the de-facto standard for shared hosting (and anyone sane doing serious deployments of anything else) long before cPanel was even a glimmer of an idea in some noob PHP3 programmer's brain. cPanel exists because of Linux's popularity in the server space, not vice versa, and without it the Linux community would be just as strong. Maybe we'd have more admins that actually know what they're doing.

[hans]

Server is now quick here too, it was slower earlier today/yesterday, but that seems to be fixed now

[EEVblog]

I did not notice the server had been brought back on-line already, and email notifications from the forum seem to be broken, looks like I may have missed Dave as he needs to grant me access to the new install so I can track these issues down.

Yes, forum email seems broken.
You now have the power of god again. Obviously the migration stripped you of your magic hammer.

Dave.

[EEVblog]

Here is a capture of the live database connection just now.

Dave.

[amspire]

Just had an "Internal Server Error" for about 5 minutes when accessing the forum. It is running again now.

[EEVblog]

Just had an "Internal Server Error" for about 5 minutes when accessing the forum. It is running again now.

Yep, gnif just updated mySQL, fully expected.

Dave.

[EEVblog]

It seems that "persistent connection" and "email on database connection errors" SMF options were both disabled. Not sure why they would have been turned off.
Back on now.

With regards to forum caching, SMF says this:

SMF supports caching through the use of accelerators. The currently supported accelerators include:
APC
eAccelerator
Turck MMCache
Memcached
Zend Platform/Performance Suite (Not Zend Optimizer)
XCache
Caching will work best if you have PHP compiled with one of the above optimizers, or have memcache available. If you do not have any optimizer installed SMF will do file based caching.

SMF performs caching at a variety of levels. The higher the level of caching enabled the more CPU time will be spent retrieving cached information. If caching is available on your machine it is recommended that you try caching at level 1 first.

Note that if you use memcached you need to provide the server details in the setting below. This should be entered as a comma separated list as shown in the example below:
"server1,server2,server3:port,server4"

Note that if no port is specified SMF will use port 11211. SMF will attempt to perform rough/random load balancing across the servers.

SMF has detected that your server has APC installed.

So SMF may possibly be using APC caching at the moment, on "Level 1". There is "Level 2", and "Level 3 (not recommended)" options.
Before the server move, there was no caching detected, so it used file based caching.

Dave.

[gnif]

Dont see what is so funny, it has allowed/forced many many users to become familiar with Linux in the hosting environment which has pushed them to give it a go in other environments. Bugs get fixed, usability problems are found, etc, simply because more people are using it. I am not saying that it was 'cPamel' that made linux, I am saying that commercial products like cPanel help push more users towards Linux because they either want to cut costs, or they have no choice.

But that's not what you said, you said 'if it was not for platforms like cPanel the Linux community would suffer due to lack of users.', which means there wouldn't be enough users without it, which is hilarious. Linux has been the de-facto standard for shared hosting (and anyone sane doing serious deployments of anything else) long before cPanel was even a glimmer of an idea in some noob PHP3 programmer's brain. cPanel exists because of Linux's popularity in the server space, not vice versa, and without it the Linux community would be just as strong. Maybe we'd have more admins that actually know what they're doing.

Ok, sorry, it was a brainfart on my side, I did not intend it to come across that way.

It seems that "persistent connection" and "email on database connection errors" SMF options were both disabled. Not sure why they would have been turned off.
Back on now.

Persistent connections should not be needed any more, but wont hurt to leave them there. As for the caching, we are using APC now and I am currently getting memcached configured which will also improve things .

[gnif]

Ok, everything seems to be operational again, including some odd issues reported by some users. If anyone experiances any issues as of now, please update this thread to let Dave and myself know.

[vk6hdx]

The Wordpress blog on the main site seems to have done the time warp and is showing Posts from August last year at the top of the page.

[gnif]

Looks like a timezone problem, give me a few mins

Edit: It orders fine for me, can you please post the URL you are using?

[EEVblog]

The Wordpress blog on the main site seems to have done the time warp and is showing Posts from August last year at the top of the page.

I was afraid of that.
It's an issue with WP Super Cache.
I re-enabled it thinking it would magically work on the new server. Apparently not.
I'll turn it off again.

Dave.

[gnif]

Dave, the server is still set for an American time zone, should this be changed to Sydney Australia? This may be the reason for the super cache doing this.

[vk6hdx]

Yep all good now gnif seeing the up to date posts now.  I was just visiting http://eevblog.com screenshot attached.

P.s. the forum seems lightning fast now.. 

[manicdoc]

Dont see what is so funny, it has allowed/forced many many users to become familiar with Linux in the hosting environment which has pushed them to give it a go in other environments. Bugs get fixed, usability problems are found, etc, simply because more people are using it. I am not saying that it was 'cPamel' that made linux, I am saying that commercial products like cPanel help push more users towards Linux because they either want to cut costs, or they have no choice.

But that's not what you said, you said 'if it was not for platforms like cPanel the Linux community would suffer due to lack of users.', which means there wouldn't be enough users without it, which is hilarious. Linux has been the de-facto standard for shared hosting (and anyone sane doing serious deployments of anything else) long before cPanel was even a glimmer of an idea in some noob PHP3 programmer's brain. cPanel exists because of Linux's popularity in the server space, not vice versa, and without it the Linux community would be just as strong. Maybe we'd have more admins that actually know what they're doing.

Ok, sorry, it was a brainfart on my side, I did not intend it to come across that way.

It seems that "persistent connection" and "email on database connection errors" SMF options were both disabled. Not sure why they would have been turned off.
Back on now.

Persistent connections should not be needed any more, but wont hurt to leave them there. As for the caching, we are using APC now and I am currently getting memcached configured which will also improve things .

      

Good going - All that will be left to put on will be the speed stripes...

[BravoV]

P.s. the forum seems lightning fast now.. 

Agree, its blazing fast here, props to gnif ! 

Now its time to do QC of his work by hammering the forum ...  ..j/k

[gnif]

Thanks guys, Id love to see how fast it is when you are local(ish) to the server, I can see that the request->response time is exceptional, just our AU lag hides how quick the site is a little.

[EEVblog]

Dave, the server is still set for an American time zone, should this be changed to Sydney Australia? This may be the reason for the super cache doing this.

I set the timezone to Sydney this morning.
No, nothing to do with the Supercache bug IFAIK.
Supercache used to work for me for several years, then it started doing this crazy stuff going back several weeks or months.
IIRC it can be temporarily fixed by clearing the cache, but then it just comes right back again.

Dave.

[gnif]

Dave, the server is still set for an American time zone, should this be changed to Sydney Australia? This may be the reason for the super cache doing this.

I set the timezone to Sydney this morning.
No, nothing to do with the Supercache bug IFAIK.
Supercache used to work for me for several years, then it started doing this crazy stuff going back several weeks or months.
IIRC it can be temporarily fixed by clearing the cache, but then it just comes right back again.

Dave.

Very odd, although I will admit that I have barely used it, I am more into the server side of things then the actual hosted content .

[vk6hdx]

Supercache used to work for me for several years, then it started doing this crazy stuff going back several weeks or months.
IIRC it can be temporarily fixed by clearing the cache, but then it just comes right back again.

Have you tried the HyperCache WP Plugin?  It seems to get a pretty good write up here.

[EEVblog]

Have you tried the HyperCache WP Plugin?  It seems to get a pretty good write up here.

Ah, the bug seems to be in garbage collection somehow.
Might give Hypercache a try.

Dave.

[manicdoc]

Thanks guys, Id love to see how fast it is when you are local(ish) to the server, I can see that the request->response time is exceptional, just our AU lag hides how quick the site is a little.

BTW the local jquery is still on the local homepage, also a lot 1x1 monitoring gif's hanging off the page - just double check the markup has width='1' height='1' in it or the page rendering will hang waiting for those to come through (looks like they are down a long pipe).  also seeing 304 Not Modified responses coming through, so looks like on firefox at least its still pinging the box to see if the file has changed - seems to be doing it with each new forum page one loads (ETag's?) - so if you walk the forum with firebug on you keep seeing the 304 coming in.

[gnif]

BTW the local jquery is still on the local homepage, also a lot 1x1 monitoring gif's hanging off the page - just double check the markup has width='1' height='1' in it

Is that a browser feature? If so I was unaware of this, good to know . As for checking for modified versions of the dynamic pages, this is required so you see if people have modified a post, etc. Not sure how E-Tags would help in this instance, care to elaborate?

The client may then decide to cache the resource, along with its ETag. Later, if the client wants to retrieve the same URL again, it will send its previously saved copy of the ETag along with the request in a "If-None-Match" field.

On this subsequent request, the server may now compare the client's ETag with the ETag for the current version of the resource. If the ETag values match, meaning that the resource has not changed, then the server may send back a very short response with an HTTP 304 Not Modified status. The 304 status tells the client that its cached version is still good and that it should use that.

So this will still cause a 304 response, which you are already getting.

[EEVblog]

Some database stats:
ø per hour: 135,833
ø per minute: 2,264
ø per second: 38

Network traffic since startup: 5 GiB

This MySQL server has been running for 0 days, 7 hours, 18 minutes and 30 seconds. It started up on Mar 06, 2013 at 05:06 PM.

Traffic                    ø per hour
Received   400 MiB   54.7 MiB
Sent   4.6 GiB           650.4 MiB
Total   5 GiB           705.2 MiB

Connections                           ø per hour   %
max. concurrent connections   24   ---   ---
Failed attempts   35           4.79             0.12%
Aborted                   7           0.96             0.02%
Total                           29 k           3,978.47   100.00%

Those error rates are pretty low compared to the previous server.

Aborted clients           7    The number of connections that were aborted because the client died without closing the connection properly.
Aborted connects   35    The number of failed attempts to connect to the MySQL server.

Dave.

[gnif]

Aborted clients           7    The number of connections that were aborted because the client died without closing the connection properly.
Aborted connects   35    The number of failed attempts to connect to the MySQL server.

These occur even when everything is working 100%, they can be caused by clients that started pulling a page and hit the stop button before it was finished, or someone was performing some testing directly on the database on the server (ie, it was possibly me )

[manicdoc]

BTW the local jquery is still on the local homepage, also a lot 1x1 monitoring gif's hanging off the page - just double check the markup has width='1' height='1' in it

Is that a browser feature? If so I was unaware of this, good to know . As for checking for modified versions of the dynamic pages, this is required so you see if people have modified a post, etc. Not sure how E-Tags would help in this instance, care to elaborate?

The client may then decide to cache the resource, along with its ETag. Later, if the client wants to retrieve the same URL again, it will send its previously saved copy of the ETag along with the request in a "If-None-Match" field.

On this subsequent request, the server may now compare the client's ETag with the ETag for the current version of the resource. If the ETag values match, meaning that the resource has not changed, then the server may send back a very short response with an HTTP 304 Not Modified status. The 304 status tells the client that its cached version is still good and that it should use that.

So this will still cause a 304 response, which you are already getting.

I usually run with E-Tag's turned off - on the principal that if you have set it to cache on the browser, then the browser on its own can work out if its gone stale or not. If you want to send out a new version of a library or asset, change the URL - hence why all the libraries/css/etc come with their version baked into the URL... This saves the essentially zero useful info request and the server resources that go with it. I think Etags originally came from windows world if memory serves... Check out my old employers home page (au.yahoo.com) - no Etags anywhere.

FireBug is a plug-in - bees knees for this sort of optimisation.

[gnif]

BTW the local jquery is still on the local homepage, also a lot 1x1 monitoring gif's hanging off the page - just double check the markup has width='1' height='1' in it

Is that a browser feature? If so I was unaware of this, good to know . As for checking for modified versions of the dynamic pages, this is required so you see if people have modified a post, etc. Not sure how E-Tags would help in this instance, care to elaborate?

The client may then decide to cache the resource, along with its ETag. Later, if the client wants to retrieve the same URL again, it will send its previously saved copy of the ETag along with the request in a "If-None-Match" field.

On this subsequent request, the server may now compare the client's ETag with the ETag for the current version of the resource. If the ETag values match, meaning that the resource has not changed, then the server may send back a very short response with an HTTP 304 Not Modified status. The 304 status tells the client that its cached version is still good and that it should use that.

So this will still cause a 304 response, which you are already getting.

I usually run with E-Tag's turned off - on the principal that if you have set it to cache on the browser, then the browser on its own can work out if its gone stale or not. If you want to send out a new version of a library or asset, change the URL - hence why all the libraries/css/etc come with their version baked into the URL... This saves the essentially zero useful info request and the server resources that go with it. I think Etags originally came from windows world if memory serves... Check out my old employers home page (au.yahoo.com) - no Etags anywhere.

FireBug is a plug-in - bees knees for this sort of optimisation.

Well here are the headers the server is sending back:

Code: [Select]

HTTP/1.1 200 OK
Date: Thu, 07 Mar 2013 07:55:11 GMT
Server: Apache
X-Powered-By: PHP/5.4.12
Pragma: no-cache
Cache-Control: private
Expires: Mon, 26 Jul 1997 05:00:00 GMT
Vary: Accept-Encoding
Set-Cookie: PHPSESSID=*REMOVED*; path=/
Last-Modified: Thu, 07 Mar 2013 07:55:11 GMT
Connection: close
Content-Type: text/html; charset=UTF-8

So no ETags, just a expire date in the past, which is what I would expect for dynamic content from the forum. It is showing the PHP version though, forgot to turn that off since the migration to 64bit.

[manicdoc]

BTW the local jquery is still on the local homepage, also a lot 1x1 monitoring gif's hanging off the page - just double check the markup has width='1' height='1' in it

Is that a browser feature? If so I was unaware of this, good to know . As for checking for modified versions of the dynamic pages, this is required so you see if people have modified a post, etc. Not sure how E-Tags would help in this instance, care to elaborate?

The client may then decide to cache the resource, along with its ETag. Later, if the client wants to retrieve the same URL again, it will send its previously saved copy of the ETag along with the request in a "If-None-Match" field.

On this subsequent request, the server may now compare the client's ETag with the ETag for the current version of the resource. If the ETag values match, meaning that the resource has not changed, then the server may send back a very short response with an HTTP 304 Not Modified status. The 304 status tells the client that its cached version is still good and that it should use that.

So this will still cause a 304 response, which you are already getting.

I usually run with E-Tag's turned off - on the principal that if you have set it to cache on the browser, then the browser on its own can work out if its gone stale or not. If you want to send out a new version of a library or asset, change the URL - hence why all the libraries/css/etc come with their version baked into the URL... This saves the essentially zero useful info request and the server resources that go with it. I think Etags originally came from windows world if memory serves... Check out my old employers home page (au.yahoo.com) - no Etags anywhere.

FireBug is a plug-in - bees knees for this sort of optimisation.

Well here are the headers the server is sending back:

Code: [Select]

HTTP/1.1 200 OK
Date: Thu, 07 Mar 2013 07:55:11 GMT
Server: Apache
X-Powered-By: PHP/5.4.12
Pragma: no-cache
Cache-Control: private
Expires: Mon, 26 Jul 1997 05:00:00 GMT
Vary: Accept-Encoding
Set-Cookie: PHPSESSID=*REMOVED*; path=/
Last-Modified: Thu, 07 Mar 2013 07:55:11 GMT
Connection: close
Content-Type: text/html; charset=UTF-8

So no ETags, just a expire date in the past, which is what I would expect for dynamic content from the forum. It is showing the PHP version though, forgot to turn that off since the migration to 64bit.

hmm for the static files _only_ you want it like:

Cache-Control   max-age=86400
Connection   Keep-Alive
Date   Thu, 07 Mar 2013 07:58:58 GMT
Expires   Fri, 08 Mar 2013 07:58:58 GMT
Keep-Alive   timeout=10, max=98
Server   Apache
Vary   Accept-Encoding

that also plays nice with upstream caches. Only a browser refresh will force a fetch.

[gnif]

BTW the local jquery is still on the local homepage, also a lot 1x1 monitoring gif's hanging off the page - just double check the markup has width='1' height='1' in it

Is that a browser feature? If so I was unaware of this, good to know . As for checking for modified versions of the dynamic pages, this is required so you see if people have modified a post, etc. Not sure how E-Tags would help in this instance, care to elaborate?

The client may then decide to cache the resource, along with its ETag. Later, if the client wants to retrieve the same URL again, it will send its previously saved copy of the ETag along with the request in a "If-None-Match" field.

On this subsequent request, the server may now compare the client's ETag with the ETag for the current version of the resource. If the ETag values match, meaning that the resource has not changed, then the server may send back a very short response with an HTTP 304 Not Modified status. The 304 status tells the client that its cached version is still good and that it should use that.

So this will still cause a 304 response, which you are already getting.

I usually run with E-Tag's turned off - on the principal that if you have set it to cache on the browser, then the browser on its own can work out if its gone stale or not. If you want to send out a new version of a library or asset, change the URL - hence why all the libraries/css/etc come with their version baked into the URL... This saves the essentially zero useful info request and the server resources that go with it. I think Etags originally came from windows world if memory serves... Check out my old employers home page (au.yahoo.com) - no Etags anywhere.

FireBug is a plug-in - bees knees for this sort of optimisation.

Well here are the headers the server is sending back:

Code: [Select]

HTTP/1.1 200 OK
Date: Thu, 07 Mar 2013 07:55:11 GMT
Server: Apache
X-Powered-By: PHP/5.4.12
Pragma: no-cache
Cache-Control: private
Expires: Mon, 26 Jul 1997 05:00:00 GMT
Vary: Accept-Encoding
Set-Cookie: PHPSESSID=*REMOVED*; path=/
Last-Modified: Thu, 07 Mar 2013 07:55:11 GMT
Connection: close
Content-Type: text/html; charset=UTF-8

So no ETags, just a expire date in the past, which is what I would expect for dynamic content from the forum. It is showing the PHP version though, forgot to turn that off since the migration to 64bit.

hmm for the static files _only_ you want it like:

Cache-Control   max-age=86400
Connection   Keep-Alive
Date   Thu, 07 Mar 2013 07:58:58 GMT
Expires   Fri, 08 Mar 2013 07:58:58 GMT
Keep-Alive   timeout=10, max=98
Server   Apache
Vary   Accept-Encoding

that also plays nice with upstream caches. Only a browser refresh will force a fetch.

That was for dynamic content, this is what the server is sending for static content, so it could do with the expires header, I will see about correcting this, thanks for pointing it out.

Code: [Select]

HTTP/1.1 200 OK
Date: Thu, 07 Mar 2013 08:03:24 GMT
Server: Apache
Last-Modified: Tue, 06 Sep 2011 12:20:54 GMT
ETag: "50043b-e0-4ac44dbe7d580"
Accept-Ranges: bytes
Content-Length: 224
Keep-Alive: timeout=5, max=100
Connection: Keep-Alive
Content-Type: image/gif

Edit: This has been fixed, all static content has an expiry of 1 day now.

[ve7xen]

BTW the local jquery is still on the local homepage, also a lot 1x1 monitoring gif's hanging off the page - just double check the markup has width='1' height='1' in it

Is that a browser feature? If so I was unaware of this, good to know .

I dunno that I'd call it a feature... but knowing the final size of the element lets the browser compose the final layout of the page and wait for the image to arrive, rather than waiting for the image header before knowing its size, and thus how the page will be laid out. I think browsers these days are pretty clever and will render as much of the page as they can as it comes in, but if those 1x1 gifs are above the fold on a main content div or something, they're going to stall the browser until they load.

[manicdoc]

BTW the local jquery is still on the local homepage, also a lot 1x1 monitoring gif's hanging off the page - just double check the markup has width='1' height='1' in it

Is that a browser feature? If so I was unaware of this, good to know .

I dunno that I'd call it a feature... but knowing the final size of the element lets the browser compose the final layout of the page and wait for the image to arrive, rather than waiting for the image header before knowing its size, and thus how the page will be laid out. I think browsers these days are pretty clever and will render as much of the page as they can as it comes in, but if those 1x1 gifs are above the fold on a main content div or something, they're going to stall the browser until they load.

That's about right - personally I hate the darn things. There was a EC start up in the UK many moons ago that got engineered by a whole boat load of 'con'sultants on a crazed feeding frenzy, all working on their own modules and bits and pieces. I remember looking at the request chain for the home page, literally hundreds of bits of css, js and 1x1 gifs all over the place - like a black hole it literally collapsed under its own inefficient weight... My motto is keep it lean, keep it keen, always.

Plus if Dave ever gets his lovely mug on the telly - all this work will reap massive rewards as his site won't have such a tendency to do a 'click frenzy special'..

[EEVblog]

More graphs from MySQL
The server does seem to peak at using all 8 processors occasionally.
But server load averages are usually under 0.5

Dave.

[gnif]

More graphs from MySQL
The server does seem to peak at using all 8 processors occasionally.
But server load averages are usually under 0.5

Dave.

That is because I increased its limits so that it can use all 8, it might as well take advantage of your new hardware.

[EEVblog]

That is because I increased its limits so that it can use all 8, it might as well take advantage of your new hardware.

 

Dave.

[amspire]

Is the eevblog homepage meant to be offline? I have been getting an Apache Default Website page for the last few hours.

[Thor-Arne]

Is the eevblog homepage meant to be offline? I have been getting an Apache Default Website page for the last few hours.

I had  this problem to when using FireFox, IE worked fine tho.
Needed to clear all cache/history.

[amspire]

Is the eevblog homepage meant to be offline? I have been getting an Apache Default Website page for the last few hours.

I had  this problem to when using FireFox, IE worked fine tho.
Needed to clear all cache/history.

Thanks - that worked. Once the page gets redirected once to the default page cgi script, reloading the page in Firefox has absolutely no effect - it just reloads the cgi script again. It looks like clearing the firefox cache is the only way to get the home page back. Seems an odd way to implement a default page.

[Thor-Arne]

The problem is FireFox, for some reason it uses the cached version no matter what.
I guess there will be another FireFox update soon. 

[amspire]

The problem is FireFox, for some reason it uses the cached version no matter what.
I guess there will be another FireFox update soon. 

The problem is that the default cgi page is a permanent page (http://www.eevblog.com/cgi-sys/defaultwebpage.cgi) , and you cannot stop Firefox from redirecting first before a full refresh of the page - so it just refreshes the default cgi page. So once a redirect is in the cache, it seems you cannot prevent the redirecting in any way other then clearing the cache. It never tries to load from eevblog.com to see if the redirect is still present.

[gnif]

The problem is FireFox, for some reason it uses the cached version no matter what.
I guess there will be another FireFox update soon. 

The problem is that the default cgi page is a permanent page (http://www.eevblog.com/cgi-sys/defaultwebpage.cgi) , and you cannot stop Firefox from redirecting first before a full refresh of the page - so it just refreshes the default cgi page. So once a redirect is in the cache, it seems you cannot prevent the redirecting in any way other then clearing the cache. It never tries to load from eevblog.com to see if the redirect is still present.

Thanks for the URL, I will see if I can add a redirect in to fix the issue, firefox is a PITA with redirects like that.

[Thor-Arne]

The problem is FireFox, for some reason it uses the cached version no matter what.
I guess there will be another FireFox update soon. 

As I said, there was the update. 

[BravoV]

This thread "DIY Metcal 13.56 MHz RF Supply" at the last page 13 is acting up again -> https://www.eevblog.com/forum/projects/diy-metcal-13-56-mhz-rf-supply/180/

It displays blank screen as previous problem.

Edit : Looks like this thread got stickied at Projects, Designs, and Technical Stuff section with last post by EEVBlog, Dave broke it ! 

[baljemmett]

This thread "DIY Metcal 13.56 MHz RF Supply" at the last page 13 is acting up again [...]Looks like this thread got stickied at Projects, Designs, and Technical Stuff section with last post by EEVBlog, Dave broke it ! 

The last post in that thread is:

Title: Re: DIY Metcal 13.56 MHz RF Supply
Post by: EEVblog on March 08, 2013, 02:08:32 AM
PLEASE READ
There is something wrong with this thread in SMF, it is giving technical troubles. No idea why. 
Thread locked unless a fix can be found.

Dave.

... although I'm not sure how many people will have seen it!  The trick is to open one of the earlier pages in the thread and hit the Print option; the print-friendly version that appears will include all the posts, but alas no attachments.

[gnif]

The thread is fixed, it was caused by trying to upload an image that was way too big, SMF was not allocating enough ram to generate the thumbnail for it, and instead of failing gracefully... got a white screen.

I have increased the memory limit for larger files.

[BravoV]

Is there any restriction or setting at SMF to limit the picture resolution when people upload them ? Its quite annoying to watch pics dumped from camera without resizing it 1st, its just plain lazy. 

I think 1024 horizontal resolution should be enough.

[gnif]

I am not sure, Dave will have to have a look into it when he wakes up .

I have unlocked the thread.

[jancumps]

Old server showed posts one hour off here in cet timezone. New server 14 hours off.
The list of unread items seems shorter, ony a half page - used to be 2-3 pages.

Or maybe I need more sleep 

[gnif]

Old server showed posts one hour off here in cet timezone. New server 14 hours off.
The list of unread items seems shorter, ony a half page - used to be 2-3 pages.

Or maybe I need more sleep 

Check what your timezone is set to in your SMF profile. The server was in the incorrect time zone a few days back which Dave corrected, it could account for the difference.

[jancumps]

Just checked. I couldn't find a specific timezone setting, but my country is set to Belgium. I haven't touched my profile lately.

[justanothercanuck]

It's under account settings, look and layout, time offset. 

[jancumps]

That did it. It was set to autodetect. Behaviour seems to have cheged....
It's fixed now.

[Mr Smiley]

Hi,

I'm getting this today at 17:36 uk time, all other messages seem ok

500 Internal Server Error
https://www.eevblog.com/forum/microcontrollers/is-arduino-the-best/?topicseen

Mr Smiley 

[gnif]

Hi,

I'm getting this today at 17:36 uk time, all other messages seem ok

500 Internal Server Error
https://www.eevblog.com/forum/microcontrollers/is-arduino-the-best/?topicseen

Mr Smiley 

Sorry about that, it was me . We are expecting a load spike in the next 24 hours and as such I am preparing the server for it. Just clear your cache and try again.

[Mr Smiley]

Hi,

It's back now, it wasn’t a problem for me, but i thought you might want to still be updated with any quirks, this is the first error I’ve come across on this forum 

[gnif]

No worries, thanks for the heads up .