Forum Posting Problem

[EEVblog]

Seems the database got corrupted and wouldn't allow posting. I think it's repaired now.
Testing...

[Skimask]

Looks good...

(P.S.  Pull up your pants!  Or spackle that thing shut!)

[EEVblog]

Just check'n again

[GEuser]

Up late too Dave  , I thought that was just the innernet as I also had slow and disappearing (404's) at that time too .

But just got this after clicking the post button
Connection Problems
Sorry, SMF was unable to connect to the database. This may be caused by the server being busy. Please try again later.

[EEVblog]

Sorry, SMF was unable to connect to the database. This may be caused by the server being busy. Please try again later.

That one, apparently for the last  for the last 3 hours was due to the MySQL server being locked up. A database service reboot fixed it.

[EEVblog]

Hmm, all sorts of problems at the moment it seems. Had to reboot the MySQL server again, and there is a big load on the server. Something isn't right...

[Pack34]

Hmm, all sorts of problems at the moment it seems. Had to reboot the MySQL server again, and there is a big load on the server. Something isn't right...

I remember having a similar issue with my company's web host. Maybe the physical server is starting to go. Any way to request a migration to a different machine?

[EEVblog]

I remember having a similar issue with my company's web host. Maybe the physical server is starting to go. Any way to request a migration to a different machine?

I've been thinking about that. No real evidence that's the case, but it can't hurt I guess, but does seem to have had more downtime/crashes/reboots than I'd expect. I can ask...

[Pack34]

I remember having a similar issue with my company's web host. Maybe the physical server is starting to go. Any way to request a migration to a different machine?

I've been thinking about that. No real evidence that's the case, but it can't hurt I guess, but does seem to have had more downtime/crashes/reboots than I'd expect. I can ask...

Yes, but without knowing what the problem is it is equally likely it can't (won't) help either. If they are willing to migrate you to a different machine as a "goodwill" gesture, which is what it will be in the absence of any evidence of the cause of the problem, then they will become increasingly less willing in the event of a recurrence in future. They'll say no anyway, unless they can see a fault.

I suggest looking at the systems logs to see what messages are present. There may well be an answer there. Something has changed. It could be a server hardware fault. It could also be a capacity problem or a software upgrade has introduced a bug, either on the server itself or with another server it communicates with.

Do you think it could be something dns related? Intermittently pointing towards the website's old location? I don't know if an intermittent DNS problem is possible, just throwing out ideas.

[atferrari]

More than posting, my problem just to access the site. I cannot recall but there was a three-letters acronym involved, something like MPS or similar and an error code, starting with 5, but not sure really.

[GEuser]

I did not have to log on this evening as I could not log off early this morning as that message kept popping up .

The site behaves real snappy here atm , instantaneous practically ..

[T3sl4co1l]

Had this happen a few times, somewhat randomly.  Seems to come and go.

Tim

[Neilm]

I have had 2 or 3 issues getting onto the EEVblog main site and the forum over the last few weeks.

[tautech]

I have had 2 or 3 issues getting onto the EEVblog main site and the forum over the last few weeks.

Me too. Seems reliability is an issue ATM.
Had this again this morning, though it seems all good now.

Connection Problems

Sorry, SMF was unable to connect to the database. This may be caused by the server being busy. Please try again later.

[atferrari]

I have had 2 or 3 issues getting onto the EEVblog main site and the forum over the last few weeks.

Me too. Seems reliability is an issue ATM.
Had this again this morning, though it seems all good now.

Connection Problems

Sorry, SMF was unable to connect to the database. This may be caused by the server being busy. Please try again later.

The same here, several times today.

[retiredcaps]

I also had problems Monday Oct 20 starting at 10AM and ongoing to 10:30AM MST.  I tried later in the afternoon around 2PM MST and was able to get on.

I was able to ping www.eevblog.com, but the forum and main page were down with the same error messages posted by others.

[EEVblog]

Yep, the server is having real problem at the moment, I need to speak with HostGator.
The database issues I'm pretty sure are caused by the server going down, and when it does, there is always a chance a database table can get corrupted.

[gnif]

Hi All,

I am sorry I have not been around to reply to any of this, I have been put out of action as I have been recovering from the whooping cough. I have suggested that Dave have all the RAM in the server replaced, the constant table crashes and now these major crashes point the finger at either RAM or HDD, but since the HDD is a RAID array and as far as I am aware is not reporting any issues, I would not expect this kind of crash (RAM is the easier to replace first too).

[dr.diesel]

Would RAM errors not show up somewhere?

IIRC it's a Linux box, single bit ECC errors would show up in dmesg.

[EEVblog]

Hostgator are saying that it's the MySQL that is going down, and not the whole server. They are working on it...

[gnif]

With regards to RAM, It looks like the server may be overcommited, I have reduced the total ram that MySQL is allowed to use, but this does not explain why it is getting blown way out. It seems to occur when there is heavy load on the server, I am seeing oom errors that is killing MySQL and hundreds of PHP processes running at the same time. I am investigating what PHP is doing during these periods.

[bookaboo]

May be unrelated but the entire site was intermittent for me today from about 11.00am until 2.00PM GMT.

[GEuser]

Doing some internal search's like 10 minutes ago a cupla times it came back > Database error/corruption retry later or something ..

[gnif]

May be unrelated but the entire site was intermittent for me today from about 11.00am until 2.00PM GMT.

It was related, logs show the database had crashed at that point.

Doing some internal search's like 10 minutes ago a cupla times it came back > Database error/corruption retry later or something ..

This may have occured while I was restarting the MySQL service to apply the memory changes.

[T3sl4co1l]

Saw this earlier,

Code: [Select]

Table './eevblog_smp01/smf_members' is marked as crashed and should be repaired
Hope it's not hackers or something

[GEuser]

Saw this earlier,

Code: [Select]

Table './eevblog_smp01/smf_members' is marked as crashed and should be repaired
Hope it's not hackers or something

That was another that popped up too

[gnif]

Saw this earlier,

Code: [Select]

Table './eevblog_smp01/smf_members' is marked as crashed and should be repaired
Hope it's not hackers or something

That was another that popped up too

Thanks for that, I will correct it now, that is a problem from the prior crash.

Edit: This has been done, should not present again.


Edit 2: I noted that even though we setup memcached way back, for some reason it is not in use. I have enabled it in SMF and have already noted quite a substantial reduction in server I/O. The forum also seems to be quite a bit faster for me, but I have not done any testing to properly confirm this.

[EEVblog]

Code: [Select]

Table './eevblog_smp01/smf_members' is marked as crashed and should be repairedHope it's not hackers or something

There are two issues at the moment, that have been persistent over the last two days.

The first and most common is that error message listed above. I believe this is caused because the MySQL database shat itself for some reason, and when that happens there is a chance that any table that was being accessed will corrupt. Hence why you see different error message here. This requires a simple but manual table repair to fix. This seems to be happening every few hours at present.

The second issue is that of a complete MySQL lockup. In this case the MySQL server has to be restarted. This will bring down not only the forum, but the wordpress blog too.

I've disabled Cloudflare until all this gets fixed.

[EEVblog]

Edit 2: I noted that even though we setup memcached way back, for some reason it is not in use. I have enabled it in SMF and have already noted quite a substantial reduction in server I/O.

I haven't fiddled with that. Although I did just update to SMF 2.0.9 yesterday.

[gnif]

I've disabled Cloudflare until all this gets fixed.

Cloudflare has nothing in common with this issue, but with regards to the other issues people have had with it, I do not thing that it is a bad idea to disable it anyway.

[EEVblog]

Cloudflare has nothing in common with this issue, but with regards to the other issues people have had with it, I do not thing that it is a bad idea to disable it anyway.

I've had constant problems with it myself.

[gnif]

Leave it off then, I will keep a close eye on the server and see how things perform.

[alimirjamali]

I would agree with gnif that CloudFlare should have nothing to do with MySql issue. Currently CloudFlare cashing is disabled and only the DNS queries are resolved by them . Let's see if would help at all (which I doubt) . I presume that you check mysql logs (/var/log/mysql/error.log || /var/log/mysqld.log || etc) for errors?

Beside gnif, do you have sudoers (or any other support person) in other TimeZones? Maybe someone in UK or US you trust in person. Someone that could restart the bloody MySql service if needed!

The value of posts on the Forum is substantial. Every time Forum crashes, I panic . Would you confirm that MySql databases and httpd data are backed up regularly (CRON, rsync, etc.)?

With regards to RAM, It looks like the server may be overcommited,

If it is not a huge secret, what is the output of free -m after a fresh restart? I am curious to know how much RAM is installed on the server.

[tautech]

FYI: Had more of this early this morning:
504 - Gateway Timeout
We are sorry for the inconvenience that this error may be causing you. We are aware of the issue and are working to resolve it, please be patient.

There is no need to report this error.

Thank you for your patience,
Dave & gnif

[EEVblog]

FYI: Had more of this early this morning:
504 - Gateway Timeout

Apparently the FTDI thing got picked up on Hacker News and I presume the server got overloaded.

[zapta]

Apparently the FTDI thing got picked up on Hacker News and I presume the server got overloaded.

Or FTDI decided to nuke also these forums...

[tautech]

FYI: Had more of this early this morning:
504 - Gateway Timeout
We are sorry for the inconvenience that this error may be causing you. We are aware of the issue and are working to resolve it, please be patient.

There is no need to report this error.

Thank you for your patience,
Dave & gnif

More again for a short while today.

[nanofrog]

504 - Gateway Timeout
We are sorry for the inconvenience that this error may be causing you. We are aware of the issue and are working to resolve it, please be patient.

There is no need to report this error.

Thank you for your patience,
Dave & gnif

I'm still getting this.

[gnif]

Even though I am not entirely convinced they always DO know about the problem, in spite of this canned response, it is nonetheless clear there is no need to report the problem.

You are sort of correct, that error about us knowing about it was when we had a prolonged window where we expected this to occur (server changes) and the error page was never fixed/updated when we were done.

I have been monitoring the server closely to try to determine where the performance bottlekneck is in the current configuration, we initially assumed these issues were due to the Slashdot traffic, but it is becoming aparrent that there is something more going on.

[tautech]

but it is becoming aparrent that there is something more going on.

Yep for the last few weeks, sometimes for some hours.
Today....infrequent short outages.

[gnif]

but it is becoming aparrent that there is something more going on.

Yep for the last few weeks, sometimes for some hours.
Today....infrequent short outages.

I have just changed the website to store sessions in memcached instead of in the database on disk, this seems to be the bulk of I/O occuring and should make a noticable improvement to performance. If you get timeouts again please don't hesitate to PM me as I will notice that first. The next stage might just be a simple matter of increasing the number of apache processes as the gateway timeout occurs when Nginx can not communicate with the Apache backend (better solution yet would be to remove apache from the equasion, but would prefer not to as this starts to get outside of what cPanel will support).

[T3sl4co1l]

Been getting a "session timeout, please attempt to reply again" message lately.  Yes, I write overly long posts, but still...

Tim

[Thor-Arne]

I sometimes get "session timeout" when marking a board as read, so it's not the long posts. 

[EEVblog]

It went down again a few hours ago.
Even the director of marketing at HostGator emailed me out of the blue and said they noticed it, and asked if it was due to FTDIgate
I guess he's a viewer 

So I think it is simply a spike in traffic every time FTDIgate gets linked somewhere.
gnif has put in place caching that seems to be working well, but I guess there is only so much you can do with a single dedicated server.

[alimirjamali]

So I think it is simply a spike in traffic every time FTDIgate gets linked somewhere.

A proper engineer does not think but measure . Do you monitor RAM usage, Disk IO, CPU load ? There are many utilities in Penguin for both real-time and logged measurement of system metrics (free/top/htop/iftop/iotop/etc. ). Maybe it is time for an upgrade ?

[gnif]

So I think it is simply a spike in traffic every time FTDIgate gets linked somewhere.

A proper engineer does not think but measure . Do you monitor RAM usage, Disk IO, CPU load ? There are many utilities in Penguin for both real-time and logged measurement of system metrics (free/top/htop/iftop/iotop/etc. ). Maybe it is time for an upgrade ?

We do indeed monitor, and nothing directly points to an issue which is why it has been hard to find out what is going on. Since we corrected the issue with the database crashing we are getting some more constant error reporting from the server that is helping to track the issue down. At this point it seems that there is something wrong with PHP as we are now seeing tons of php segfaults in dmesg, I am sill investigating as to the cause.

Edit: The outage that just occured was intentional, we just upgraded php to resolve a known bug with custom error handlers in the version of PHP we were running, I will conitnue to monitor the server and see if this resolves the issue that has been occuring.

[gdewitte]

I'm consistently getting a "504-Gateway timeout" when I try to "Show unread posts…" Anyone else having this problem, or could it be something hosed on my PC? All the other links to various posts seem to work just fine.

[Owen]

I'm consistently getting a "504-Gateway timeout" when I try to "Show unread posts…" Anyone else having this problem, or could it be something hosed on my PC? All the other links to various posts seem to work just fine.

I'm getting this 504-Gateway timeout when i'm trying to enter "www.eevblog.com/forum/testgear/".

[EEVblog]

I'm consistently getting a "504-Gateway timeout" when I try to "Show unread posts…" Anyone else having this problem

Yep, I get that too, but only on my machine at the lab, and only on Firefox. Not other browsers or computers.
https://www.eevblog.com/forum/index.php?action=unread works fine though and does the same thing, but the https://www.eevblog.com/forum/unread/ link does not work.

[alimirjamali]

Yep, I get that too, but only on my machine at the lab, and only on Firefox. Not other browsers or computers.
https://www.eevblog.com/forum/index.php?action=unread works fine though and does the same thing, but the https://www.eevblog.com/forum/unread/ link does not work.

This is a server side issue and has almost nothing to do with your browser (Firefox) or internet connection (@lab).

The reason you were able to get it through index.php is you virtually bypassed browser cashing by an alternative URL and got a fresh copy. The fresh copy might work fine or be another 504-Gateway timeout page.

Hey gnif: Maybe it would be better to add META HTTP-EQUIV="Pragma" CONTENT="no-cache" to head section on the default 504-Gateway Timeout HTML page. Maybe add a meta http-equiv="refresh" content="300" tag too. 

[gnif]

Hey gnif: Maybe it would be better to add META HTTP-EQUIV="Pragma" CONTENT="no-cache" to head section on the default 504-Gateway Timeout HTML page. Maybe add a meta http-equiv="refresh" content="300" tag too. 

You are quite correct, this however has not been a priority to tune as we have been looking for the root cause of the issue. I will adjust this a little later tonight, in the meantime I am preparing to remove apache from the equation and handle requests directly with Nginx, this should both improve performance and assist in determining where any potential bottleknecks are.

[gdewitte]

Quote from: gdewitte on Yesterday at 01:37:38 AM

I'm consistently getting a "504-Gateway timeout" when I try to "Show unread posts…" Anyone else having this problem, or could it be something hosed on my PC? All the other links to various posts seem to work just fine.

My solution so far: cleared browsing history on Google Chrome and everything worked. May be unique to my system and Google Chrome, as all links worked fine when I tried Firefox.

[gnif]

Quote from: gdewitte on Yesterday at 01:37:38 AM

I'm consistently getting a "504-Gateway timeout" when I try to "Show unread posts…" Anyone else having this problem, or could it be something hosed on my PC? All the other links to various posts seem to work just fine.

My solution so far: cleared browsing history on Google Chrome and everything worked. May be unique to my system and Google Chrome, as all links worked fine when I tried Firefox.

I am preparing to remove apache from the equation and handle requests directly with Nginx, this should both improve performance and assist in determining where any potential bottleknecks are.

[alimirjamali]

I am preparing to remove apache from the equation and handle requests directly with Nginx, this should both improve performance and assist in determining where any potential bottleknecks are.

Hey gnif. I see you have enabled CloudFlare cashing again . It is not just DNS anymore .

So you are going to ditch Apache in favor of Nginx . I am happy that I am not in your position to translate all of those nasty .htaccess and Apache VirtualHost config files . You are going to test it on a local copy before applying them?  right? We have some EEVBlog Forumoholics who will be pissed off if Forum is down more than 1 day .

<ConspiracyTheory>
Maybe Dave is DDoSed ? Dave's WordPress Blog and SMF Forum are popular but not that much popular which Apache could not handle. There are less than 2000 Online Users+Guests on Forum. Considering that SMF User online time threshold is 15 minutes (by default) which implies that they do not fetch data simultaneously, Apache should be able to handle them easily.

2000 users / 15 minutes / 60 seconds * Approx. 20 requests per users = Approx. 44 requests per second = piece of cake for Apache.

Even before the #FTDIGate, Dave had made many foes. Maybe if you double check IP of users, you might see 90% from some FTDI server (What Da FTDI).
</ConspiracyTheory>

Good luck.

[gnif]

Hey gnif. I see you have enabled CloudFlare cashing again . It is not just DNS anymore .

This was Dave's decision to do so, it has nothing to do with the gateway issues.

So you are going to ditch Apache in favor of Nginx . I am happy that I am not in your position to translate all of those nasty .htaccess and Apache VirtualHost config files . You are going to test it on a local copy before applying them?  right? We have some EEVBlog Forumoholics who will be pissed of if Forum is down more than 1 day .

Already translated, and live, see the new thread.

<ConspiracyTheory>
Maybe Dave is DDoSed ? Dave's WordPress Blog and SMF Forum are popular but not that much popular which Apache could not handle. There are less than 2000 Online Users+Guests on Forum. Considering that SMF User online time threshold is 15 minutes (by default) which implies that they do not fetch data simultaneously, Apache should be able to handle them easily.

2000 users / 15 minutes / 60 seconds * Approx. 20 requests per users = Approx. 44 requests per second = piece of cake for Apache.

Even before the #FTDIGate, Dave had made many foes. Maybe if you double check IP of users, you might see 90% from some FTDI server (What Da FTDI).
</ConspiracyTheory>

No DDoS and we rate limit per IP anyway, so they would need to have way more servers then you would expect to saturate the server. Since this is a cPanel server it is tuned to host multiple small to medium sized sites for mass reselling, not one large site. It builds Apache in such a way as to suit this purpose and in this instance it is a hinderance. Moving away from Apache to Nginx + PHP-FPM allows us to escape the bounds of the cPanel paradigm .

[T3sl4co1l]

Been getting a "session timeout, please attempt to reply again" message lately.  Yes, I write overly long posts, but still...

Tim

It's back

[gnif]

Been getting a "session timeout, please attempt to reply again" message lately.  Yes, I write overly long posts, but still...

Tim

It's back

I will adjust how memcached is storing sessions and increase the timeout, this should correct this.

[T3sl4co1l]

Thanks!

[EvilGeniusSkis]

could someone have been using something more sophisticated than a simple DDoS, such as SlowLoris or RUDY? 

[gnif]

could someone have been using something more sophisticated than a simple DDoS, such as SlowLoris or RUDY?

Certainly not, logs clearly indicated that Apache was running out of connections well before server load was an issue, Apache is not exposed to the internet as it was behind Nginx which was the reverse proxy. If it was a DoS attack such as SlowLoris or RUDY Nginx would also have stopped responding.

Put simply, Apache is software that should be put down, it was good in it's day, but it is well and truly past it's expiration date. Please do not think that I believe this due to a limited understanding of Apache, I am extremely well versed with it, and I am the maintainer/developer of mod_rpaf which is in use by hundreds if not thousands of large scale deployments around the globe, so I know Apache from a code perspective, not just a deployment perspective.

[alimirjamali]

could someone have been using something more sophisticated than a simple DDoS, such as SlowLoris or RUDY?

FTDI produces many chips for Android smart phones. They could have turned millions of Android phones to zombies and attacked Dave's forum in retaliation to the #FTDIgate
p.s. I know that it is technically impossible; However, conspiracy theories are simply cool .

[gnif]

could someone have been using something more sophisticated than a simple DDoS, such as SlowLoris or RUDY?

FTDI produces many chips for Android smart phones. They could have turned millions of Android phones to zombies and attacked Dave's forum in retaliation to the #FTDIgate
p.s. I know that it is technically impossible; However, conspiracy theories are simply cool .

Haha, you literally made me LOL.

In all this was a good exercise as the server is now running better then ever for pretty much everyone, even those of us who are over the pond and have to already deal with large latency due to the transmission delays.

[EvilGeniusSkis]

this whole thing reminds me of something my multimedia teacher would say

Technology will make a fool out of everyone.

the reason I say this is, to me anyway, it seem that the problem was fixed without anyone figuring out what the problem was exactly. am i right?

also is there a forum features wishlist thread anywhere?

[gnif]

the reason I say this is, to me anyway, it seem that the problem was fixed without anyone figuring out what the problem was exactly. am i right?

I have very clearly stated what the problem was, Apache simply is not capable of serving the sheer number of PHP requests the website has as it is archaic in its design and can not take advantage of modern hardware correctly.

[atferrari]

Hola gnif

I want to stop receiving notifications of new replies to some threads.
I clicked on "UNNOTIFY" at the top of them but I still receive notifications.

One of the threads is this very one.

Gracias for any help.