IMPORTANT: Server Changeover

[GeoffS]

They did say they would be forwarding traffic from the old server to the new one so, in theory, there should be no posts going to the old system.

[manicdoc]

I think the only real issue with doing the changeover is not losing the data (posts) that were inserted after the database was replicated.
I think most DB guys could handle this in their sleep. They should've already written and tested a script that does this to all the relevant tables.

Hopefully the Hosting service will manage this, though I don't know why the site was down all night, that does seem unnecessary.

Yes and No - you would need to 'map' the new posts on the old into the new whilst resolving dupe primary keys and chain through foreign references and changes... Not a 10 minute hack.. be easier to do it at the app level and effectively enact the posting API with the new posts off the old server - assuming the forum software has such an API to access.. Remember there might be a whole lot of side effects to take care of (such as text indexes, etc). Certainly not trivial - although if someone has already written a script that does this - then it is possible.

[HackedFridgeMagnet]

you would need to 'map' the new posts on the old into the new whilst resolving dupe primary keys and chain through foreign references and changes... Not a 10 minute hack..

Can't you carry the primary keys across in the replication (backup/restore) ? You can turn the server off for the 5 minutes it takes to run the update script,so there are no duplicate primary keys.

Yes you would have to chain through a few foreign references when doing the inserts but this is why you would script it.
Probably only need to add the new posts and topics, you could disregard the small number of user meta data changes that happen in this time.
Obviously you would test your script first, so you don't get a blowout in down time.

[EEVblog]

Hopefully the Hosting service will manage this, though I don't know why the site was down all night, that does seem unnecessary.

That was my choice, I did it. I got the email about the server change had started, and figured it was safest to just lock the forum down.
It got to midnight and I had to go to bed, so made the choice to leave it locked down overnight just in case.
I thought it would be done by the morning, seems I was wrong.

Dave.

[amspire]

Seeing there's an IP address change, you might want to consider reducing the DNS  TTL (time to live) value down to 15 minutes prior to the changes. That way,  the IP address change will propagate more quickly.
It's currently set to 1 hour.

I have no idea were that option even lives...

Dave.

If you have cpanel, go to Advanced DNS Zone Editor, and edit the A address for eevblog.com - the one that mentions the current IP address.

It will be a line that looks like

Code: [Select]

eevblog.com. 3600  IN  A  184.173.12.165    Edit    Delete
Click edit and change the TTL (Time To Live) from 3600 (1 hour) to a lower number like 600 (10 minutes)

When the server gets transferred, the IP in this record will automatically get updated, and the DNS caches around the world will get the new IP address within 10 minutes.

[EEVblog]

If you have cpanel, go to Advanced DNS Zone Editor, and edit the A address for eevblog.com - the one that mentions the current IP address.

It will be a line that looks like

Code: [Select]

eevblog.com. 3600  IN  A  184.173.12.165    Edit    Delete
Click edit and change the TTL (Time To Live) from 3600 (1 hour) to a lower number like 600 (10 minutes)

When the server gets transferred, the IP in this record will automatically get updated, and the DNS caches around the world will get the new IP address within 10 minutes.

Got it.
Changed to 30min (the lowest option in dropdown box)

Dave.

[manicdoc]

you would need to 'map' the new posts on the old into the new whilst resolving dupe primary keys and chain through foreign references and changes... Not a 10 minute hack..

Can't you carry the primary keys across in the replication (backup/restore) ? You can turn the server off for the 5 minutes it takes to run the update script,so there are no duplicate primary keys.

Yes you would have to chain through a few foreign references when doing the inserts but this is why you would script it.
Probably only need to add the new posts and topics, you could disregard the small number of user meta data changes that happen in this time.
Obviously you would test your script first, so you don't get a blowout in down time.

the problem is if the new system has posts being written which use the same primary keys as the old system - you need to resolve the conflict. Which could get quite hairy if the key is written into some app controlled field (i.e. keyword mapping in articles). hence my suggestion to come in at the app level.

An alternative is to configure the new system to be a MySQL slave to the old until you throw the switch, but you won't be able to write posts on the new system. OR copy across the DB and point the old server database config at the new server - this assumes the forum and other software would deal correctly with being 'multiheaded' and not get its session knickers in a twist 

Basically moving a whole single box system to a new box will usually incur some down time somewhere, you haven't got enough kit to stage it.

[EEVblog]

I asked them what happens with databases:

The way these migrations work has been planned to be as effortless and pain free as possible. First we rsync all of the data from the old server to the new server, once that's done we do it two more times to make sure any data that's been changed while the first rsync was running is updated. then we shut down MySQL on the source box and do a rsync of just the MySQL data to ensure the databases are intact and up to date. MySQL is never restarted on the old box. As soon as that's done we bring up all services on the new server and put iptables NAT rules in place on the old server to forward all traffic from the old IPs to the new ones. This makes the migration nearly transparent. At that time we start prompting the user to make any required DNS changes, which are usually minimal as most customers host DNS on their own servers.

So looks like we won't lose a single post.

Dave.

[Lightages]

Well the SMF not being to connect to database error just occurred again. So it was definitely not a hardware issue.

[gnif]

but you won't be able to write posts on the new system. OR copy across the DB and point the old server database config at the new server - this assumes the forum and other software would deal correctly with being 'multiheaded' and not get its session knickers in a twist 

Basically moving a whole single box system to a new box will usually incur some down time somewhere, you haven't got enough kit to stage it.

This only occurs if you don't know what you are doing. You can write to a MySQL slave, but you must ensure that once you start writing to it, you don't write to the master. This scenario works perfect for an instant switch-over.

1) Slave MySQL database to the old server
2) Configure a reverse proxy to forward HTTP/HTTPs traffic to the new server
3) [optional] use socat or iptables to forward traffic for POP3/IMAP/SMTP, etc. to the new server also
4) De-configure MySQL slave
5) Update DNS, wait for propagation to occur
6) Decommission old server.

Zero down time and zero data loss.

The reverse proxy should be used, as it will then add the X-Forwarded-For header to the connections, which the new server should be configured using mod_rpaf (if Apache is used on the new server, Nginx would be better, but that's another topic) to look for and use as the real IP connecting. Just forwarding the data without this will cause the logs to fill with connections that look like they are coming from the old server and any protection on user account login (such as the forum user login) may trigger lockouts due to all the users seeming to come from the same IP address.

Well the SMF not being to connect to database error just occurred again. So it was definitely not a hardware issue.

This is caused by MySQL running out of connections, just increase the max connections if the server is capable of handling it.

[EEVblog]

The server change over is complete, and it seemed to work even without me changing the DNS IP.
BUt I have just changed the DNS IP to the new one  198.1.64.14
The old one is 184.173.12.165
I presume the site will still work for everyone in that limbo time, working on either IP.

Let me know if there is any problem.

Dave.

[amspire]

The server change over is complete, and it seemed to work even without me changing the DNS IP.
BUt I have just changed the DNS IP to the new one  198.1.64.14

198.1.64.143 you mean. My DNS already points to the new address.

[gnif]

Yeah, they said they would NAT the connections though to the new server, so zero downtime, but you may find that you are now logging everyone's IP address as the old server's IP. And yes, you most certainly needed to change the IP over.

[amspire]

The server change over is complete, and it seemed to work even without me changing the DNS IP.

The DNS has been updated by the host obviously. The A address definitely now point to the new IP.

[gnif]

The server change over is complete, and it seemed to work even without me changing the DNS IP.

The DNS has been updated by the host obviously. The A address definitely now point to the new IP.

BUt I have just changed the DNS IP to the new one  198.1.64.14

No, Dave changed it, the host is obviously doing what they said, using NAT to forward the connections to the new server.

[EEVblog]

The server change over is complete, and it seemed to work even without me changing the DNS IP.

The DNS has been updated by the host obviously. The A address definitely now point to the new IP.

I changed it myself. HostGator do not have access to my DNS. I use a third party company to host all my domain names, that's why they said I have to do it.
I'm sure they would have done it for me if I provided them my login and site details etc.

Dave.

[manicdoc]

but you won't be able to write posts on the new system. OR copy across the DB and point the old server database config at the new server - this assumes the forum and other software would deal correctly with being 'multiheaded' and not get its session knickers in a twist 

Basically moving a whole single box system to a new box will usually incur some down time somewhere, you haven't got enough kit to stage it.

This only occurs if you don't know what you are doing. You can write to a MySQL slave, but you must ensure that once you start writing to it, you don't write to the master. This scenario works perfect for an instant switch-over.

1) Slave MySQL database to the old server
2) Configure a reverse proxy to forward HTTP/HTTPs traffic to the new server
3) [optional] use socat or iptables to forward traffic for POP3/IMAP/SMTP, etc. to the new server also
4) De-configure MySQL slave
5) Update DNS, wait for propagation to occur
6) Decommission old server.

Zero down time and zero data loss.

The reverse proxy should be used, as it will then add the X-Forwarded-For header to the connections, which the new server should be configured using mod_rpaf (if Apache is used on the new server, Nginx would be better, but that's another topic) to look for and use as the real IP connecting. Just forwarding the data without this will cause the logs to fill with connections that look like they are coming from the old server and any protection on user account login (such as the forum user login) may trigger lockouts due to all the users seeming to come from the same IP address.

Agreed, that's a way to do it - but its not zero downtime, you still need to config the Master and reroute traffic on the fly, and get everything spot on. According to what Dave quoted in another email the ops guys seemed to have rsync'ed the table files directly then did traffic routing above the boxes. Probably safer to script wrap and automate. Yes, Nginx would be handy.

[gnif]

but you won't be able to write posts on the new system. OR copy across the DB and point the old server database config at the new server - this assumes the forum and other software would deal correctly with being 'multiheaded' and not get its session knickers in a twist 

Basically moving a whole single box system to a new box will usually incur some down time somewhere, you haven't got enough kit to stage it.

This only occurs if you don't know what you are doing. You can write to a MySQL slave, but you must ensure that once you start writing to it, you don't write to the master. This scenario works perfect for an instant switch-over.

1) Slave MySQL database to the old server
2) Configure a reverse proxy to forward HTTP/HTTPs traffic to the new server
3) [optional] use socat or iptables to forward traffic for POP3/IMAP/SMTP, etc. to the new server also
4) De-configure MySQL slave
5) Update DNS, wait for propagation to occur
6) Decommission old server.

Zero down time and zero data loss.

The reverse proxy should be used, as it will then add the X-Forwarded-For header to the connections, which the new server should be configured using mod_rpaf (if Apache is used on the new server, Nginx would be better, but that's another topic) to look for and use as the real IP connecting. Just forwarding the data without this will cause the logs to fill with connections that look like they are coming from the old server and any protection on user account login (such as the forum user login) may trigger lockouts due to all the users seeming to come from the same IP address.

Agreed, that's a way to do it - but its not zero downtime, you still need to config the Master and reroute traffic on the fly, and get everything spot on. According to what Dave quoted in another email the ops guys seemed to have rsync'ed the table files directly then did traffic routing above the boxes. Probably safer to script wrap and automate. Yes, Nginx would be handy.

This is zero downtime, and configuring the traffic to reroute does not matter how long it takes once the database slave is configured as both servers are kept in sync. A MySQL master/slave can be configured with zero downtime using the percona tools to take a live dump of the database to import into the new server without having to lock the tables. When I said zero downtime with zero loss, I meant it, I do this on a daily basis as I work for a contracted server management company who support the 'Linux pros' when they get stuck.



Edit: sorry, you can be right depending on if the master server has binlogs enabled or not, and the server_id is set, so you may get 1-5 seconds of downtime to enable these features.

Also the Nginx changeover can be seemless also as you make it listen on say port 81 and then insert a port redirect rule into iptables for new connections only, so no need to stop Apache and then start nginx. If this is not desired, there is a <1 second 'service stop apache/http && service start nginx' to run .

[manicdoc]

(snip long quote..)

This is zero downtime, and configuring the traffic to reroute does not matter how long it takes once the database slave is configured as both servers are kept in sync. A MySQL master/slave can be configured with zero downtime using the percona tools to take a live dump of the database to import into the new server without having to lock the tables. When I said zero downtime with zero loss, I meant it, I do this on a daily basis as I work for a contracted server management company who support the 'Linux pros' when they get stuck.



Edit: sorry, you can be right depending on if the master server has binlogs enabled or not, and the server_id is set, so you may get 1-5 seconds of downtime to enable these features.

Also the Nginx changeover can be seemless also as you make it listen on say port 81 and then insert a port redirect rule into iptables for new connections only, so no need to stop Apache and then start nginx. If this is not desired, there is a <1 second 'service stop apache/http && service start nginx' to run .

Yep, that is my (minor) point, you need to have the right set up before hand to get the absolute zero downtime.

BTW what are your thoughts on M/M pairing for a live site - i.e. 2 boxes dual serving with M/M DB set up? I'm researching on this atm, be interested if you have any thoughts. thanks.

[gnif]

(snip long quote..)

This is zero downtime, and configuring the traffic to reroute does not matter how long it takes once the database slave is configured as both servers are kept in sync. A MySQL master/slave can be configured with zero downtime using the percona tools to take a live dump of the database to import into the new server without having to lock the tables. When I said zero downtime with zero loss, I meant it, I do this on a daily basis as I work for a contracted server management company who support the 'Linux pros' when they get stuck.



Edit: sorry, you can be right depending on if the master server has binlogs enabled or not, and the server_id is set, so you may get 1-5 seconds of downtime to enable these features.

Also the Nginx changeover can be seemless also as you make it listen on say port 81 and then insert a port redirect rule into iptables for new connections only, so no need to stop Apache and then start nginx. If this is not desired, there is a <1 second 'service stop apache/http && service start nginx' to run .

Yep, that is my (minor) point, you need to have the right set up before hand to get the absolute zero downtime.

BTW what are your thoughts on M/M pairing for a live site - i.e. 2 boxes dual serving with M/M DB set up? I'm researching on this atm, be interested if you have any thoughts. thanks.

Master-Master set-ups are usually more trouble then they are worth, if you get a sync issue then it is an absolute nightmare to recover from it, it would be better to go with MySQL cluster, which contrary to popular belief, is open source also and freely available. The later versions of the NDB storage engine have resolved most of the issues they used to have with replication across slow links as well as missing features such as key lengths and text search. If you are looking for load balanced configuration and clustering is overkill, you could use a MySQL Master/Slave, and throw MySQL proxy in there to balance read requests to the two servers, this works exceptionally well (note: the warnings about MySQL-Proxy not being ready for production use are rubbish, as MySQL-Proxy is used in the commercial MySQL enterprise set-up which they sell, it is just not advertised).

[EEVblog]

Is it my imagination, or is the new server noticeably quicker to load pages?
BTW, this is the fastest forum I use, and that includes ones on servers in Oz.

Dave.

[gnif]

Coming form AU here it seems quicker for me. Still get that 250ms international latency, but that is not related to your server, just our crummy international link.

[EEVblog]

while you can use the REd drives as a general purpose drive they are tuned for NAS and RAID applications.
http://www.wdc.com/en/products/products.aspx?id=810
so in case of a server where you use RAId : these drives are ideal.

Good to see they picked the right ones then!

Dave.

[manicdoc]

(snip long quote..)

This is zero downtime, and configuring the traffic to reroute does not matter how long it takes once the database slave is configured as both servers are kept in sync. A MySQL master/slave can be configured with zero downtime using the percona tools to take a live dump of the database to import into the new server without having to lock the tables. When I said zero downtime with zero loss, I meant it, I do this on a daily basis as I work for a contracted server management company who support the 'Linux pros' when they get stuck.



Edit: sorry, you can be right depending on if the master server has binlogs enabled or not, and the server_id is set, so you may get 1-5 seconds of downtime to enable these features.

Also the Nginx changeover can be seemless also as you make it listen on say port 81 and then insert a port redirect rule into iptables for new connections only, so no need to stop Apache and then start nginx. If this is not desired, there is a <1 second 'service stop apache/http && service start nginx' to run .

Yep, that is my (minor) point, you need to have the right set up before hand to get the absolute zero downtime.

BTW what are your thoughts on M/M pairing for a live site - i.e. 2 boxes dual serving with M/M DB set up? I'm researching on this atm, be interested if you have any thoughts. thanks.

Master-Master set-ups are usually more trouble then they are worth, if you get a sync issue then it is an absolute nightmare to recover from it, it would be better to go with MySQL cluster, which contrary to popular belief, is open source also and freely available. The later versions of the NDB storage engine have resolved most of the issues they used to have with replication across slow links as well as missing features such as key lengths and text search. If you are looking for load balanced configuration and clustering is overkill, you could use a MySQL Master/Slave, and throw MySQL proxy in there to balance read requests to the two servers, this works exceptionally well (note: the warnings about MySQL-Proxy not being ready for production use are rubbish, as MySQL-Proxy is used in the commercial MySQL enterprise set-up which they sell, it is just not advertised).

Okay, looks like the old M/S set up is the way to go; the boxes would be quite close. I'm used to sharded 'farms' of M/S set ups in my old job. So a familiar stomping ground. thanks.

[EEVblog]

TEST...
I just changed my PHP5 from suphp to fgi
*random pic attachment*

Dave.