After you've run for a couple of days, I highly recommend mysqltuner.pl ; it'll help show bottlenecks (and over-allocations) in your existing my.cnf
The number of update and insert requests seems quite stable over the past month. But the number of select requests has increased markedly. Why is that? It predates todays RAM upgrade. IIRC you re-jigged the MySQL tables. Is the increase simply because MySQL does more selects as a result of the DB work? Does it reflect new selects being generated as a result of a similar load?
Were you RAM constrained before and did that contribute to the record locking for long periods during maintenance? Which lead to the 502's. Just reading what has been reported I don't quite see how more RAM has addressed the problems reported. Were there performance issue at times outside the DB maintenance intervals?
Sorry gnif I don't mean to bombard you with questions it is just that I am not sure of the right question in MySQL speak.
Are you serving pages on ipv6 yet?
My hosting provider, 1and1, has full dedicated servers with 32 GB RAM, 12 cores x 2.3 GHz and 2 TB RAID 1 harddisk for $200 per month. Just in case you need some arguments for HostGator :)
My hosting provider, 1and1, has full dedicated servers with 32 GB RAM, 12 cores x 2.3 GHz and 2 TB RAID 1 harddisk for $200 per month. Just in case you need some arguments for HostGator :)I am paying $130 for E3-1245, 32GB RAM and 2x 480GB SSD... and that is in Australia! Why are HG so expensive?
My hosting provider, 1and1, has full dedicated servers with 32 GB RAM, 12 cores x 2.3 GHz and 2 TB RAID 1 harddisk for $200 per month. Just in case you need some arguments for HostGator :)I am paying $130 for E3-1245, 32GB RAM and 2x 480GB SSD... and that is in Australia! Why are HG so expensive?
Because, reasons...
If I was to move hosts again then I wouldn't choose a US based one. Simple reason, DMCA.
Have you considered OVH?
Their pricing is pretty competitive. (if you're in Canada, their pricing actually crushes everyone else due to the exchange rate now.)
if you want to customize and evolve over time, colo is really much better.
The head of marketing at HostGator (who is a fan BTW)
The head of marketing at HostGator (who is a fan BTW)
Gentlemen,
That guy is me..... At this point I should probably put on the body armor.... Outing yourself as a HostGator employee is typically seen as a risky move.
Dave is correct, I am a fan. I have been watching for years. I am a Marketing guy who plays with electronics on the weekends (arduino, raspberry pi, ham radio, 3d printing). But I should emphasize, much more Marketing than Engineering. The show and the forum have really helped me out with many personal projects. So a much delayed thank you to all of you.
Anyway, I am willing to field questions if you have them. If you throw anything too technical at me, it may take me a bit to find the answer or make sure it’s something we can actually share publicly.
The head of marketing at HostGator (who is a fan BTW)
Gentlemen,
That guy is me..... At this point I should probably put on the body armor.... Outing yourself as a HostGator employee is typically seen as a risky move.
Dave is correct, I am a fan. I have been watching for years. I am a Marketing guy who plays with electronics on the weekends (arduino, raspberry pi, ham radio, 3d printing). But I should emphasize, much more Marketing than Engineering. The show and the forum have really helped me out with many personal projects. So a much delayed thank you to all of you.
Anyway, I am willing to field questions if you have them. If you throw anything too technical at me, it may take me a bit to find the answer or make sure it’s something we can actually share publicly.
Hi, nice to see you on here :)
I can say personally that I get a ton of use out of the website, videos and forum like yourself, and it is what drew me to offer my services as a way to give back. The questions I would like to ask I do not feel is my right to do so, as they are cost related and I will not be so bold to presume to think that Dave would like to see it discussed openly here.
From a technical POV we are pretty well informed and up to speed on how the HG hosting is setup, and I have spent many many hours tuning the server to run on your systems as optimal as possible. Our latest round of updates, both server side software changes and the memory increase have made an enormous difference. So I will make the same offer in return, if HG are interested in how we have this forum responding so well on this setup, please ask away, but note that some details are trade secrets.
The head of marketing at HostGator (who is a fan BTW)Gentlemen,
That guy is me..... At this point I should probably put on the body armor.... Outing yourself as a HostGator employee is typically seen as a risky move.
Dave is correct, I am a fan. I have been watching for years. I am a Marketing guy who plays with electronics on the weekends (arduino, raspberry pi, ham radio, 3d printing). But I should emphasize, much more Marketing than Engineering. The show and the forum have really helped me out with many personal projects. So a much delayed thank you to all of you.
Anyway, I am willing to field questions if you have them. If you throw anything too technical at me, it may take me a bit to find the answer or make sure it’s something we can actually share publicly.
Another thing... Any thought to putting an SSL on here? We could probably make that happen......
Another thing... Any thought to putting an SSL on here? We could probably make that happen......
That's been discussed many times on here.
I'm sure gnif can fill you in the tech details as he has investigated this extensively I believe.
I would like to, as a lot of people have asked for it, but I believe there are some technical hurdles to overcome.
2) use letsencrypt.org for a free SSL cert, this involves a little bit of setup on the server to renew the cert before it expires as these have a 3 month expiry.
2) use letsencrypt.org for a free SSL cert, this involves a little bit of setup on the server to renew the cert before it expires as these have a 3 month expiry.
Interesting service. It is not much of a problem for this forum, but such a service sounds like the perfect way for the NSA or other 3 letter agencies to get private SSL keys.
The technical hurdles are no longer much of an issue anymore, the concern was always with performance. We have three options hereAnother thing... Any thought to putting an SSL on here? We could probably make that happen......That's been discussed many times on here.
I'm sure gnif can fill you in the tech details as he has investigated this extensively I believe.
I would like to, as a lot of people have asked for it, but I believe there are some technical hurdles to overcome.
1) use CloudFlare for SSL, but that means we must proxy through them.
2) use letsencrypt.org for a free SSL cert, this involves a little bit of setup on the server to renew the cert before it expires as these have a 3 month expiry.
3) purchase a SSL certificate
So if we chose to do that and we disable Cloudflare then the website revokes back to http only?
Which type do we need?
http://support.hostgator.com/articles/ssl-certificates/acquire-ssl/how-do-i-purchase-an-ssl-and-what-type-is-it (http://support.hostgator.com/articles/ssl-certificates/acquire-ssl/how-do-i-purchase-an-ssl-and-what-type-is-it)
LetsEncrypt went through a long period of public security auditing before they offered their service, so it is one of the more secure SSL solutions. It is done properly.2) use letsencrypt.org for a free SSL cert, this involves a little bit of setup on the server to renew the cert before it expires as these have a 3 month expiry.
Interesting service. It is not much of a problem for this forum, but such a service sounds like the perfect way for the NSA or other 3 letter agencies to get private SSL keys.
LetsEncrypt went through a long period of public security auditing before they offered their service, so it is one of the more secure SSL solutions. It is done properly.2) use letsencrypt.org for a free SSL cert, this involves a little bit of setup on the server to renew the cert before it expires as these have a 3 month expiry.
Interesting service. It is not much of a problem for this forum, but such a service sounds like the perfect way for the NSA or other 3 letter agencies to get private SSL keys.
There are many hosting companies that support LetsEncrypt, and they will automatically renew every 3 months - so all you have to do is request a free SSL cert in the CPanel, and you have a certificate that will automatically renew for as long as you want.
Most CPanel hosts now let you enter your own key, so I have 3 LetsEncrypt certs I generate every 3 months (single batch file to update all three), and then I manually add the new keys to tye Cpanels.
For companies, I have always just got a commercial one - RapidSSL is something like A$12 a year if you buy from the right reseller.
Digicert is used by the likes of Facebook, so it is not a bad choice for companies.
For this website since there is no real sensitive information, the cheapest option. You do not need to use HG for this, any certificate authority is fine, personally I have used RapidSSL, and AlphaSSL, but I believe there are better/cheaper options today. To be completely honest though if it was me I would just go for the letsencrypt.org service, it is if anything more secure then a generic SSL cert from other providers as it renews every 3 months, and the setup is quite simple on the server.
For this website since there is no real sensitive information, the cheapest option. You do not need to use HG for this, any certificate authority is fine, personally I have used RapidSSL, and AlphaSSL, but I believe there are better/cheaper options today. To be completely honest though if it was me I would just go for the letsencrypt.org service, it is if anything more secure then a generic SSL cert from other providers as it renews every 3 months, and the setup is quite simple on the server.
I like the idea of a new certificate every few months from a "why not" point of view.
How much work is involved in setting this up?
About 1/2 hour, nothing huge.
About 1/2 hour, nothing huge.
Make it so when you have some free time!
I can add the SSL certificate to Cloudflare I believe if that's required.
About 1/2 hour, nothing huge.
Make it so when you have some free time!
I can add the SSL certificate to Cloudflare I believe if that's required.
Great, I will later today. In the CF configuration you just tell it to use "Strict" mode, then it enforces the SSL cert your server produces is valid.
The 3 month expiry is a great idea. If someone hacks your site and gets the keys to your commercial 2+ year SSL certificate, they can pretend to be you. You can revoke the certificate, but it is up to the browsers to check to see if certificates are revoked or not. As far as I know, Chrome does not actually check for revoked certs. They have their own system that can accept some revoked certificates.For this website since there is no real sensitive information, the cheapest option. You do not need to use HG for this, any certificate authority is fine, personally I have used RapidSSL, and AlphaSSL, but I believe there are better/cheaper options today. To be completely honest though if it was me I would just go for the letsencrypt.org service, it is if anything more secure then a generic SSL cert from other providers as it renews every 3 months, and the setup is quite simple on the server.
I like the idea of a new certificate every few months from a "why not" point of view.
How much work is involved in setting this up?
I'm pretty sure Chrome blocks if the cert is revoked just like other browsers. Just had a customer attempt to replace their SHA-1 certs with SHA-256 and didn't realize that once he requests the existing cert to be re-keyed it revokes the old version of it, and attempting to access the site using this cert with Chrome resulted in a revocation error until it was replaced. Prior to rekeying, the error in Chrome was the one that since it was SHA-1 it wasn't actually secure.Last I heard, Google had decided not to let browsers check if a cert was revoked, but instead Google runs its own in-house tracking of revoked certs, and somehow Chrome uses this Google service. They did have an optional setting to re-enable revoke cert checking in Chrome, but that seems to have disappeared. This solution often works but not all the time. Google's cert database may be much faster, but it is not necessarily comprehensive or up-to-date.
Big jump in the number of forum sessions:Wow. This new traffic from somewhere? Google? I know they like SSLs.....
(https://www.eevblog.com/forum/news/server-upgrade/?action=dlattach;attach=305577;image)
Big jump in the number of forum sessions:There's been some crazy #'s viewing GK's thread:
What is the date of the jump?
What is the cycle of the peaks/troughs in the new session level? Weekly?
What is the old/new session length on average?
Like i've said before I'm no network expert but is there a session reuse or maybe it is called persistent sessions. Has that changed?
Short period of forum outages that lasted only a minute or two @~12.20 NZ time.
522 error.
Gotta say how good the forum's been lately....just brilliant. :-+
Not an error at all for weeks, just perfect.
Thanks Dave for putting your hand in your pocket and gnif for the grafting and tweaks. :-/O
Thought I'd mention it as we all have a bitch quick enough when things aren't going right but we're often a bit slow to commend your efforts after all is well. Been meaning to say something for a few days but you know how it gets sometimes..............Gotta say how good the forum's been lately....just brilliant. :-+
Not an error at all for weeks, just perfect.
Thanks Dave for putting your hand in your pocket and gnif for the grafting and tweaks. :-/O
Thanks mate, I am glad to hear it :)
Gotta say how good the forum's been lately....just brilliant. :-+
Not an error at all for weeks, just perfect.
Thanks Dave for putting your hand in your pocket and gnif for the grafting and tweaks. :-/O
Gotta say how good the forum's been lately....just brilliant. :-+
Not an error at all for weeks, just perfect.
Thanks Dave for putting your hand in your pocket and gnif for the grafting and tweaks. :-/O
I originally did put my hand in my pocket, but then full credit to HostGator who really came to the party and are now offering me the improved server for like half what I was paying before :clap:
And yes, as always a huge thanks to gnif who spends a lot of time tweaking and maintaing the server for free :-+
BTW, I did get a 502 yesterday for like 30 seconds.
Gotta say how good the forum's been lately....just brilliant. :-+
Not an error at all for weeks, just perfect.
Thanks Dave for putting your hand in your pocket and gnif for the grafting and tweaks. :-/O
I originally did put my hand in my pocket, but then full credit to HostGator who really came to the party and are now offering me the improved server for like half what I was paying before :clap:
And yes, as always a huge thanks to gnif who spends a lot of time tweaking and maintaing the server for free :-+
BTW, I did get a 502 yesterday for like 30 seconds.
No problem mate, do you recall roughly what time it occurred at?
Yesterday afternoon some time.
error.log-20170420.gz:2017/04/19 15:07:21 [error] 24952#0: *59443517 FastCGI sent in stderr: "PHP message: PHP Warning: Cannot modify header information - headers already sent by (output started atredacted) inredactedon lineredacted" while reading upstream, client:redacted, server:redacted, request: "GETredactedHTTP/1.1", upstream: "fastcgi://unix:redacted", host: "redacted", referrer: "redacted"
error.log-20170421.gz:2017/04/20 19:32:25 [error] 24953#0: *62660078 FastCGI sent in stderr: "PHP message: PHP Warning: Cannot modify header information - headers already sent by (output started atredacted) inredacted: eval()'d code on lineredacted" while reading upstream, client:redacted, server:redacted, request: "GETredactedHTTP/1.1", upstream: "fastcgi://unix:redacted", host: "redacted", referrer: "redacted"
There is a profile setting in "look and layout" to set a time offset to correct the displayed time to your local time.
Hah, I had not even noticed, I just corrected it! I set it to +11 out of habit without even thinking that the server's timezone is AEST already. Servers I manage generally run on UTC.
Perhaps this one should run on UTC too. Although that might make a mess of DB log/journaling timestamps during the changover. But one isolated server probably doesn't matter in the grand scheme of things.
Does that mean the time is changed for DST? Does the overlapping timestamps when going back an hour not create a problem?
Hah, I had not even noticed, I just corrected it! I set it to +11 out of habit without even thinking that the server's timezone is AEST already. Servers I manage generally run on UTC.
Perhaps this one should run on UTC too. Although that might make a mess of DB log/journaling timestamps during the changover. But one isolated server probably doesn't matter in the grand scheme of things.
Does that mean the time is changed for DST? Does the overlapping timestamps when going back an hour not create a problem?
Timestamps will all actually be stored in Unix time. Everything is actually in UTC (..close enough, anyway) - it just applies an offset.
It does cause everyone not in the same timezone to need to change their timezone four times a year on the forum, however..
I wonder how hard it will be to patch SMF to just do it properly... when I get a little time I will investigate.
I wonder how hard it will be to patch SMF to just do it properly... when I get a little time I will investigate.
That'd be nice.
A little brief searching suggests they've never implemented it because they don't like 'too many options' (okay, there are a lot of timezones, but SMF is presently as dumb as my microwave..).
I wonder how hard it will be to patch SMF to just do it properly... when I get a little time I will investigate.
That'd be nice.
A little brief searching suggests they've never implemented it because they don't like 'too many options' (okay, there are a lot of timezones, but SMF is presently as dumb as my microwave..).
It should be simple, they have already implemented the time offset stuff, clearly they are using the 'date' function everywhere instead of the 'gmdate' function.
Running Win 7 and time linked to some online clock gives zero problems for NZ daylight saving times.
Once you've got your offset from the forum default sorted and set, it works without issue.... does here anyway.
Yeah I get that and used to get the same, that's why I mentioned linking of your PC clock to an online source AND having the correct timezone and country set on your PC too. (didn't mention that before.)Running Win 7 and time linked to some online clock gives zero problems for NZ daylight saving times.
Once you've got your offset from the forum default sorted and set, it works without issue.... does here anyway.
DST start/end dates around the world do not align. I get to change it four times a year. This is boring.
Yeah I get that and used to get the same, that's why I mentioned linking of your PC clock to an online source AND having the correct timezone and country set on your PC too. (didn't mention that before.)Running Win 7 and time linked to some online clock gives zero problems for NZ daylight saving times.
Once you've got your offset from the forum default sorted and set, it works without issue.... does here anyway.
DST start/end dates around the world do not align. I get to change it four times a year. This is boring.
:oYeah I get that and used to get the same, that's why I mentioned linking of your PC clock to an online source AND having the correct timezone and country set on your PC too. (didn't mention that before.)Running Win 7 and time linked to some online clock gives zero problems for NZ daylight saving times.
Once you've got your offset from the forum default sorted and set, it works without issue.... does here anyway.
DST start/end dates around the world do not align. I get to change it four times a year. This is boring.
My clocks are accurate. That does not stop the apparent server time changing.
:o
Gotcha now.
Not seen that, it would piss me off too.
Is it really a SMF issue ? Or something else ? :-//
E: Seems to me the quickest solution is just to go into the admin panel and set the timezone to UTC where it belongs. Pretty sure that was shot down by Dave last time I raised it as he doesn't want to have to set his offset for DST..
E: Seems to me the quickest solution is just to go into the admin panel and set the timezone to UTC where it belongs. Pretty sure that was shot down by Dave last time I raised it as he doesn't want to have to set his offset for DST..
The server's logging, monitoring and other services are all relying on dates that are stored in the local timezone, while it is a quick setting to change, it will affect much more across the server and is too late to change it. If it is done the server's time to a ton of things, including the stats and backup systems will appear to jump back in time 10 hours, which will mess things up in a large way.