Forum Database Upgrade

[EEVblog]

gnif has upgraded the forum database with a new version fo MySQL that will support InnoDB
Please report any issues with performance, old posts etc.

Thanks

[Nusa]

Forum was nearly useless (nonresponsive when trying to view a post) a few hours ago, and I gave up. Seems fine now, however.

[EEVblog]

Forum was nearly useless (nonresponsive when trying to view a post) a few hours ago, and I gave up. Seems fine now, however.

Thanks.
Anyone else notice an improvement?

[TheSteve]

Speed seems fine so far, although I need to retrain my brain as to where the categories are. Moving the EEVblog specific stuff to the top totally makes sense. I feel bad for test equipment(my favorite area) being so close to the bottom though.

[gnif]

The re-sort was not intentional, it seems the ordering has been messed up in the upgrade, which is strange.

Forum was nearly useless (nonresponsive when trying to view a post) a few hours ago, and I gave up. Seems fine now, however.

Sorry about that, the database was busy converting some tables we prior couldn't as the version of MySQL we were on did not support full text searches, which SMF uses quite extensively. There may be more periods like this in the coming few days as I identify other areas that could use improvement, I have already had to add several indexes to tables that SMF should have.

The goal here is to prevent the 502 errors from occurring that people have been reporting.

[BravoV]

The old problem on corrupted attachment still exist, try examples below, look for user's post with photo attachments, and try to view them by clicking them.

What interesting is, at some photo posts, even though they have photo thumbs, but once clicked, it wont pop out the full size photo.

Examples :

-> https://www.eevblog.com/forum/chat/salvaging-old-electronics-parts-and-save-the-world/
-> https://www.eevblog.com/forum/beginners/how-to-better-wire-the-bnc-connector-and-ghz-issue/

[gnif]

I pulled the images in question directly from the server and verified the files on disk are corrupt, these posts are quite old and the issue that caused this may already have been corrected. If you see it on any new posts please be sure to PM me directly, this is the kind of think I want to know about ASAP before too much is corrupted.

[BravoV]

I pulled the images in question directly from the server and verified the files on disk are corrupt, these posts are quite old and the issue that caused this may already have been corrected. If you see it on any new posts please be sure to PM me directly, this is the kind of think I want to know about ASAP before too much is corrupted.

Ok, noted.

Btw, just curious if you could run at background "crawling" for low impact, to check or verify attachment especially picture's files integrity, say like thru their unique header to see how much is the damaged, at least the last date when it was occurred, just an idea.

[JPortici]

did you move eevblog section 'up'?

[Monkeh]

So the forum's been running on MyISAM the whole time?

Boy, does that explain a few things.

[gnif]

So the forum's been running on MyISAM the whole time?

Boy, does that explain a few things.

No it has not, most of the database was on InnoDB but a few of the tables contained text columns that were preventing them from being moved across to InnoDB due to the lack of full text search. The issue was the background maintenance tasks were performing a join with one of these tables, which would then cause a lock contention and stop general forum use. That lock would cause a massive backlog of traffic and a huge spike in CPU load, etc... It has involved a ton of testing to determine where the fault was before throwing "fixes" at the server.

When you have a website that is as busy as this and no dev/test platform, you can't just to haphazardly upgrading packages, until now the remaining two MyISAM tables have not been causing an issue, but as load has gone up on the website it became a problem.

There are a few things that are planned to improve website performance, the server is in need of some TLC when it comes to new/faster technologies, etc, it has just had to wait until I could ensure I had enough time to perform the changes and deal with any new problems that may arise. The up side is we get a faster website, the down side is there will be outages in order to perform this work.

[Monkeh]

So the forum's been running on MyISAM the whole time?

Boy, does that explain a few things.

No it has not, most of the database was on InnoDB but a few of the tables contained text columns that were preventing them from being moved across to InnoDB due to the lack of full text search.

Ah, well, that's why we have you to give the whole story. Still explains things.

The issue was the background maintenance tasks were performing a join with one of these tables, which would then cause a lock contention and stop general forum use. That lock would cause a massive backlog of traffic and a huge spike in CPU load, etc... It has involved a ton of testing to determine where the fault was before throwing "fixes" at the server.

When you have a website that is as busy as this and no dev/test platform, you can't just to haphazardly upgrading packages, until now the remaining two MyISAM tables have not been causing an issue, but as load has gone up on the website it became a problem.

Calm, calm, not attacking you, just observing. You're not the only one in this line of business, and when most of the information on things goes through the Dave filter (no offence, Dave), it's hard to get a good picture.

It's great you've found some time to give it the needed love, the ride's been getting a little bumpy.

[gnif]

Calm, calm, not attacking you, just observing. You're not the only one in this line of business, and when most of the information on things goes through the Dave filter (no offence, Dave), it's hard to get a good picture.

Sorry I did not take your comments as an attack, I was just being explicit in detail for those that are not like you and do not understand why there is outages when this website is worked on.

[EEVblog]

The re-sort was not intentional, it seems the ordering has been messed up in the upgrade, which is strange.

Yeah, no idea why that happened! 

[TheSteve]

It seems to have switched to alphabetic order:

EEVblog
Electronics
General
Products

[DimitriP]

the ride's been getting a little bumpy.

Bumpy? Where ?

Anytime I'm here during the last year and a half  I rarely noticed anything wrong or bumpy (other than the very rare, occasional and not too often 502 bad gateway error)

[gnif]

When you have a website that is as busy as this and no dev/test platform, you can't just to haphazardly upgrading packages,

Perhaps it is time to remedy that situation. Life is so much less stressed if you don't have to reassemble the pieces in a hurry when things go wrong. Much easier to back out to the previous state and try again later.

The issue is cost both financially and time, in an ideal world it would be setup very differently, most of my paying clients have fully redundant setups which makes this much easier, but they come with a pretty hefty price. Many people would be quite amazed at the time involved in setting this up, there are so many options, no one 'best' option, it depends on the requirements and available hardware. For example one of my larger clients (who allows me to be public about my involvement with them ) have one of the largest phpBB3 forums on the internet, the forum hosting alone involves 6 servers, for redundancy and performance.

[Richard Crowley]

Wow, I have never seen a DB related problem, delay or error on this website.  I wish other forums I frequent were as snappy as EEVblog.  GearSlutz in particular has always been painfully slow.  The only errors I see here are related to availability (mostly error messages from Cloudflare.)

[gnif]

Wow, I have never seen a DB related problem, delay or error on this website.  I wish other forums I frequent were as snappy as EEVblog.  GearSlutz in particular has always been painfully slow.  The only errors I see here are related to availability (mostly error messages from Cloudflare.)

Depends on time of day, to be honest this morning at 4:00AM (ADST) was the first time I had ever seen the 502 error personally, which was a lucky break as I was able to inspect the cause while it was occurring.

[Monkeh]

Wow, I have never seen a DB related problem, delay or error on this website.  I wish other forums I frequent were as snappy as EEVblog.  GearSlutz in particular has always been painfully slow.  The only errors I see here are related to availability (mostly error messages from Cloudflare.)

Depends on time of day, to be honest this morning at 4:00AM (ADST) was the first time I had ever seen the 502 error personally, which was a lucky break as I was able to inspect the cause while it was occurring.

The down(under)sides. You miss what must be much of the peak traffic down there in the future.

[Brumby]

The issue is cost both financially and time, in an ideal world it would be setup very differently, most of my paying clients have fully redundant setups which makes this much easier, but they come with a pretty hefty price. Many people would be quite amazed at the time involved in setting this up, there are so many options, no one 'best' option, it depends on the requirements and available hardware.

There are some of us that do understand.

I have no problems with outages - and I appreciate everything that is done behind the scenes.

[gnif]

You would think that since we are in the future we could stop these things before you people in the past see them 

[Brumby]

I do love a dreamer....

[gnif]

It appears the ordering issue is a bug in SMF, it relies on the order of the records being returned from the database being the order they are stored, with the later version of MySQL the records are returned by the order of the first index (the primary key) by default. This bug was already fixed for the boards themselves, but not for the forum categories.

I am hacking in a fix now and will submit a bug report to SMF to get this fixed.

[EEVblog]

The issue is cost both financially and time, in an ideal world it would be setup very differently, most of my paying clients have fully redundant setups which makes this much easier, but they come with a pretty hefty price. Many people would be quite amazed at the time involved in setting this up, there are so many options, no one 'best' option, it depends on the requirements and available hardware. For example one of my larger clients (who allows me to be public about my involvement with them ) have one of the largest phpBB3 forums on the internet, the forum hosting alone involves 6 servers, for redundancy and performance.

 
*hiding my company credit card*

[EEVblog]

I just got this error when going directly into the Test gear section. It only happened once.
I think we have seen this recently.

[gnif]

I just got this error when going directly into the Test gear section. It only happened once.
I think we have seen this recently.

It was me hacking in a order fix, a bug in the query, missing a conditional , ordering is correct now.

[EEVblog]

It was me hacking in a order fix, a bug in the query, missing a conditional , ordering is correct now.

I've always been amazed how well SMF just handled this stuff, like it's bullet proof.

[EEVblog]

Some major sever changes have been done which at the moment can be selectively turned on for individual users.
If you'd like to help out testing this new upgrade please message or email gnif your IP address and it can be enabled.
This will affect the main website and the forum.
It's important to test it before it's enabled for everyone.
Thanks.

[Future soldier]

fitst post of this thread if quoted gives bad gateway error

https://www.eevblog.com/forum/contests/keysight-scope-month-feedback/

idk the warning said not to report it but the problem seems to be of a specific post

[Future soldier]

Some major sever changes have been done which at the moment can be selectively turned on for individual users.
If you'd like to help out testing this new upgrade please message or email gnif your IP address and it can be enabled.
This will affect the main website and the forum.
It's important to test it before it's enabled for everyone.
Thanks.

i am willing to participate

[Richard Crowley]

fitst post of this thread if quoted gives bad gateway error

https://www.eevblog.com/forum/contests/keysight-scope-month-feedback/

idk the warning said not to report it but the problem seems to be of a specific post

It works fine for me (quoting it, etc.).
"Bad Gateway" doesn't sound like any kind of database-related problem to me.

[Future soldier]

fitst post of this thread if quoted gives bad gateway error

https://www.eevblog.com/forum/contests/keysight-scope-month-feedback/

idk the warning said not to report it but the problem seems to be of a specific post

It works fine for me (quoting it, etc.).
"Bad Gateway" doesn't sound like any kind of database-related problem to me.

so its a loclised issue i gues, i know bad gateway doesnt seem to forum issue

[MatthewEveritt]

The gateway error you see isn't specifically database related, but the root cause of the issue is. It's a universal law that error messages can't be too helpful  .

I'm more than happy to test things. The IP I'm posting from covers 99% of my forum use.

[PA0PBZ]

I noticed that the unread replies link takes lots of time to load, by lots I mean up to 30 seconds 

https://www.eevblog.com/forum/unreadreplies/

[T3sl4co1l]

Hmm, a bump in the road, just a moment ago (524 timeout), but it's fine now.  Also had a couple 502s and outages the other day, but obviously, you Aussie bastards were doing the maintenance at local downtime, which is primetime for us... so inconsiderate   

Doesn't seem faster or anything, but my experience might be limited by network speed, and anyway, do whatever it is you need to do with the database to make it less resource intensive / more stable / easier to maintain.

Tim

[ProBang2]

I had only yesterday, for a few hours, the "Bad Gateway 502" error.

Today was something strange, but with the EEVBlog main-site:
Tried to load the main-site. Browser freezes for approx. 90 sec. with a blanc screen, only showing a (translated from german) "transmit data with apis.google.com". After that the site loaded as usual.
What the hell is "apis.google.com"?
Had anyone else this problem, too?

Windows7 64_Bit
Mozilla Firefox 52.0.1 (32Bit)
Adblocker deactivated on www.wwvblog.com

[Richard Crowley]

Yes, the overall performance seems noticably slower. But that is the difference between 500ms and 1500~2000ms

[JPortici]

i noticed that tapatalk is very slow to load too
also, the sections are in alphabetical order!

[Nusa]

Forum is feeling very sluggish at the moment.
My IP is on the testing list if it matters.

[andrewfernie]

i noticed that tapatalk is very slow to load too
also, the sections are in alphabetical order!

Tapatalk has been unusable for me the last few days for EEvblog. Best case is that it is very slow to load, but most of the time I give up and go to a web browser. Same situation with iOS and Android.

Andrew

[gnif]

Reports are no longer required of poor performance, we can clearly see the issue in the server's monitoring metrics. The root cause is that switching the remaining tables to InnoDB has blown out their size and they are unable to be cached in ram, as Dave stated we are getting a RAM upgrade after which there will be some tuning in order to take advantage of the additional RAM which should alleviate the issue.

[JPortici]

i noticed that tapatalk is very slow to load too
also, the sections are in alphabetical order!

Tapatalk has been unusable for me the last few days for EEvblog. Best case is that it is very slow to load, but most of the time I give up and go to a web browser. Same situation with iOS and Android.

Andrew

yesterday my new phone (sigh) arrived, it was a lot faster than the old one to load tapatalk pages.. fast enough to point the finger at the crappy wifi for slowness

[andrewfernie]

i noticed that tapatalk is very slow to load too
also, the sections are in alphabetical order!

Tapatalk has been unusable for me the last few days for EEvblog. Best case is that it is very slow to load, but most of the time I give up and go to a web browser. Same situation with iOS and Android.

Andrew

That RAM upgrade last night seems to have resolved any problems I was having. All good now.

[gnif]

We are aware there seems to be an issue with a database query, I am investigating and will update shortly.

[gnif]

We are aware there seems to be an issue with a database query, I am investigating and will update shortly.

Problem resolved, there was an bug in the database sort conditionals I hacked in last week.