Author Topic: FORUM DISRUPTION: Server Upgrade  (Read 27107 times)

0 Members and 1 Guest are viewing this topic.

Online EEVblog

  • Administrator
  • *****
  • Posts: 31414
  • Country: au
    • EEVblog
FORUM DISRUPTION: Server Upgrade
« on: March 27, 2017, 02:14:58 pm »
 
The following users thanked this post: gnif, dr.diesel, Kean, nugglix, MK14

Offline gnif

  • Administrator
  • *****
  • Posts: 1160
  • Country: au
Re: FORUM DISRUPTION: Server Upgrade
« Reply #1 on: March 28, 2017, 05:09:52 am »
Please also note that after the server comes back online there will be a period of tuning which will cause various errors and timeouts, I will update this thread when this is completed, there is no need to report these and I apologize in advance for the frustration it will cause.
HostFission - Full Server Monitoring and Management Solutions.
https://hostfission.com/
https://twitter.com/HostFission

Note: I am NOT a moderator or arbiter of disputes, my Admin level of access is so that I can perform management of the server on behalf of Dave. Do not contact me over such issues
 
The following users thanked this post: tautech, tpowell1830

Offline gnif

  • Administrator
  • *****
  • Posts: 1160
  • Country: au
Re: FORUM DISRUPTION: Server Upgrade
« Reply #2 on: March 28, 2017, 08:06:34 am »
The server has been configured to take advantage of the additional RAM, it will be monitored closely over the next few days to see if any additional improvements can be made. There should be no more outages unless noted here prior. Performance feedback is very welcome here.
HostFission - Full Server Monitoring and Management Solutions.
https://hostfission.com/
https://twitter.com/HostFission

Note: I am NOT a moderator or arbiter of disputes, my Admin level of access is so that I can perform management of the server on behalf of Dave. Do not contact me over such issues
 
The following users thanked this post: EEVblog, voltsandjolts, nugglix

Offline mrpackethead

  • Super Contributor
  • ***
  • Posts: 2831
  • Country: nz
  • D Size Cell
Re: FORUM DISRUPTION: Server Upgrade
« Reply #3 on: March 28, 2017, 08:20:57 am »
Am getting the 502 bad gateway errors again just a few minutes ago, though the forum is going just fine. 

On a quest to find increasingly complicated ways to blink things
 

Online EEVblog

  • Administrator
  • *****
  • Posts: 31414
  • Country: au
    • EEVblog
Re: FORUM DISRUPTION: Server Upgrade
« Reply #4 on: March 28, 2017, 08:21:20 am »
Was supposed to get 16GB RAM, but got 32GB for some reason  ;D

The MySQL queries has jumped fairly dramatically, seems it now able to process many more requests per second  :-+

 
The following users thanked this post: nugglix

Offline The Soulman

  • Frequent Contributor
  • **
  • Posts: 762
  • Country: nl
  • The sky is the limit!
Re: FORUM DISRUPTION: Server Upgrade
« Reply #5 on: March 28, 2017, 08:25:23 am »
Thanks dudes!  :-+
 

Offline FrankBuss

  • Supporter
  • ****
  • Posts: 2319
  • Country: de
    • Frank Buss
Re: FORUM DISRUPTION: Server Upgrade
« Reply #6 on: March 28, 2017, 08:28:23 am »
It feels a lot faster now when opening topics. Before the upgrade sometimes I had to wait a few seconds, couldn't see this now while reading some topics.
So Long, and Thanks for All the Fish
Electronics, hiking, retro-computing, electronic music etc.: https://www.youtube.com/c/FrankBussProgrammer
 

Online EEVblog

  • Administrator
  • *****
  • Posts: 31414
  • Country: au
    • EEVblog
Re: FORUM DISRUPTION: Server Upgrade
« Reply #7 on: March 28, 2017, 08:30:04 am »
Seems faster for me too.
Opening topics, opening a reply, and posting a reply.
 

Offline nugglix

  • Regular Contributor
  • *
  • Posts: 209
  • Country: de
Re: FORUM DISRUPTION: Server Upgrade
« Reply #8 on: March 28, 2017, 08:34:31 am »
Thread refresh (F5 in open thread page) is now in the sub-second range.
Clearly some sort of success, I'd say.  :-+

Thanks for the time/money spent!

Hmm... question: do I need to read faster now?   ;)

PS: Posting preview also sub-second response.
 

Offline MrBungle

  • Supporter
  • ****
  • Posts: 75
  • Country: au
Re: FORUM DISRUPTION: Server Upgrade
« Reply #9 on: March 28, 2017, 08:34:41 am »
When you said tomorrow, I figured tomorrow.... also known as Wednesday.
All good, minimal disruption  :-+
Great work guys.
 

Offline boffin

  • Supporter
  • ****
  • Posts: 958
  • Country: ca
Re: FORUM DISRUPTION: Server Upgrade
« Reply #10 on: March 28, 2017, 08:42:18 am »
After you've run for a couple of days, I highly recommend mysqltuner.pl ; it'll help show bottlenecks (and over-allocations) in your existing my.cnf

 

Offline gnif

  • Administrator
  • *****
  • Posts: 1160
  • Country: au
Re: FORUM DISRUPTION: Server Upgrade
« Reply #11 on: March 28, 2017, 08:42:43 am »
After you've run for a couple of days, I highly recommend mysqltuner.pl ; it'll help show bottlenecks (and over-allocations) in your existing my.cnf

It is the tool of the trade :)
HostFission - Full Server Monitoring and Management Solutions.
https://hostfission.com/
https://twitter.com/HostFission

Note: I am NOT a moderator or arbiter of disputes, my Admin level of access is so that I can perform management of the server on behalf of Dave. Do not contact me over such issues
 

Online mikeselectricstuff

  • Super Contributor
  • ***
  • Posts: 12135
  • Country: gb
    • Mike's Electric Stuff
Re: FORUM DISRUPTION: Server Upgrade
« Reply #12 on: March 28, 2017, 09:04:57 am »
The slowness I've been seing over the last week or so seems to have gone away.

Youtube channel:Taking wierd stuff apart. Very apart.
Mike's Electric Stuff: High voltage, vintage electronics etc.
Day Job: Mostly LEDs
 

Offline JPortici

  • Super Contributor
  • ***
  • Posts: 2645
  • Country: it
Re: FORUM DISRUPTION: Server Upgrade
« Reply #13 on: March 28, 2017, 09:10:36 am »
Browsing o  tapatalk... super fast :)
 

Offline Muttley Snickers

  • Supporter
  • ****
  • Posts: 2117
  • Country: au
Re: FORUM DISRUPTION: Server Upgrade
« Reply #14 on: March 28, 2017, 09:19:39 am »
In gnif we trust, well done and many thanks.   :-+ :)
 

Offline gnif

  • Administrator
  • *****
  • Posts: 1160
  • Country: au
Re: FORUM DISRUPTION: Server Upgrade
« Reply #15 on: March 28, 2017, 09:30:15 am »
The number of update and insert requests seems quite stable over the past month. But the number of select requests has increased markedly. Why is that? It predates todays RAM upgrade. IIRC you re-jigged the MySQL tables. Is the increase simply because MySQL does more selects as a result of the DB work? Does it reflect new selects being generated as a result of a similar load?
Were you RAM constrained before and did that contribute to the record locking for long periods during maintenance? Which lead to the 502's. Just reading what has been reported I don't quite see how more RAM has addressed the problems reported. Were there performance issue at times outside the DB maintenance intervals?

Sorry gnif I don't mean to bombard you with questions it is just that I am not sure of the right question in MySQL speak.

No worries mate. The pickup in qps is coincides with the MyIASM to InnoDB conversion on those final few tables, we are seeing far fewer lock contentions and as such the total server throughput went up. It went up because prior when a table was locked, it would put the next query into a queue, this is fine provided the queue length doesn't blow out, like it started to. In short, we were limited by locking, which throttled throughput.

Once we switched to row level locking, there is no longer (much) lock contention, and as such all these requests now can run concurrently, so this increases our throughput, people get faster response, etc... This explains the jump in queries per second. I also went through and added some indexes to fix some slow queries, SMF developers don't seem to understand the importance of them, this is not the first time I have had to fix queries.

But then we hit I/O issues as we had also increased our IOPS, so instead of seeing as many 502's due to PHP being too busy, we were seeing more slower responses due to the now much increased IO load, which in turn affects the entire server, not just DB requests. Most people throw a SSD in to resolve this, but it is unwarranted in most cases and at best a SSD is a temporary fix. The usually better solution is to throw a bit more ram at the server and cache the indexes and entire working data set, since RAM is orders of a magnitude faster then a SSD, and once the server has warmed up (data is in RAM), it barely touches the disk except for updates/inserts.

Finally since we have more RAM then we expected, I thew some extra to Memcached.
HostFission - Full Server Monitoring and Management Solutions.
https://hostfission.com/
https://twitter.com/HostFission

Note: I am NOT a moderator or arbiter of disputes, my Admin level of access is so that I can perform management of the server on behalf of Dave. Do not contact me over such issues
 

Offline rrinker

  • Super Contributor
  • ***
  • Posts: 2036
  • Country: us
Re: FORUM DISRUPTION: Server Upgrade
« Reply #16 on: March 28, 2017, 12:38:25 pm »
 I hadn't really noticed it being slower than normal, and just had a bunch of 502's but that was during the time gnif said it was rebuilding the database after the previous changes - that went away.

 But wow - compared to yesterday, it is flying now. Incredibly smooth and responsive clicking on messages and going through pages.  :-+ :-+ :-+  More RAM, more gooder.

 

Offline timgiles

  • Regular Contributor
  • *
  • Posts: 238
  • Country: se
  • Programmer, DB architect
Re: FORUM DISRUPTION: Server Upgrade
« Reply #17 on: March 28, 2017, 01:33:49 pm »
Yes - buttery smooth today  :)
 

Offline grumpydoc

  • Super Contributor
  • ***
  • Posts: 2740
  • Country: gb
Re: FORUM DISRUPTION: Server Upgrade
« Reply #18 on: March 28, 2017, 01:51:28 pm »
Very noticeable improvement!

 :-+ :-+ :-+
 

Online 2N3055

  • Super Contributor
  • ***
  • Posts: 2638
  • Country: hr
Re: FORUM DISRUPTION: Server Upgrade
« Reply #19 on: March 28, 2017, 09:32:18 pm »
Page loading is different: slight delay and than faster loading.. All in all a bit better.. But biggest difference is when replying.. Posting goes much faster....
 

Online dr.diesel

  • Super Contributor
  • ***
  • Posts: 2192
  • Country: us
  • Cramming the magic smoke back in...
Re: FORUM DISRUPTION: Server Upgrade
« Reply #20 on: March 28, 2017, 10:19:11 pm »
Serious improvement from the sticks of Indiana as well, big thanks to gnif/Dave.

Online tautech

  • Super Contributor
  • ***
  • Posts: 18495
  • Country: nz
  • Taupaki Technologies Ltd. NZ Siglent Distributor
    • Taupaki Technologies Ltd.
Re: FORUM DISRUPTION: Server Upgrade
« Reply #21 on: March 28, 2017, 10:26:20 pm »
Big  :-+ here too.  :)
Avid Rabid Hobbyist
 

Offline mrpackethead

  • Super Contributor
  • ***
  • Posts: 2831
  • Country: nz
  • D Size Cell
Re: FORUM DISRUPTION: Server Upgrade
« Reply #22 on: March 28, 2017, 11:24:50 pm »
Are you serving pages on ipv6 yet?
On a quest to find increasingly complicated ways to blink things
 

Offline gnif

  • Administrator
  • *****
  • Posts: 1160
  • Country: au
Re: FORUM DISRUPTION: Server Upgrade
« Reply #23 on: March 29, 2017, 01:57:34 am »
Are you serving pages on ipv6 yet?

Unfortunately not, sadly IPv6 has not been widely deployed by data centers.
HostFission - Full Server Monitoring and Management Solutions.
https://hostfission.com/
https://twitter.com/HostFission

Note: I am NOT a moderator or arbiter of disputes, my Admin level of access is so that I can perform management of the server on behalf of Dave. Do not contact me over such issues
 

Online EEVblog

  • Administrator
  • *****
  • Posts: 31414
  • Country: au
    • EEVblog
Re: FORUM DISRUPTION: Server Upgrade
« Reply #24 on: March 29, 2017, 03:27:11 am »
The forum is seriously slick from my lab too. So both using the new fixed IP trial setting, and without.
That RAM really did the trick.
BTW, I still don't know what it's going to cost me.
The head of marketing at HostGator (who is a fan BTW) said they'll sort it out after I complained about the ludicrous price that would have cost an extra $240/month to go from the 8GB we had to 32GB we have now. PER MONTH!
« Last Edit: March 29, 2017, 03:30:08 am by EEVblog »
 


Share me

Digg  Facebook  SlashDot  Delicious  Technorati  Twitter  Google  Yahoo
Smf