Author Topic: Forum Posting Problem  (Read 29671 times)

0 Members and 1 Guest are viewing this topic.

Offline EEVblogTopic starter

  • Administrator
  • *****
  • Posts: 38558
  • Country: au
    • EEVblog
Forum Posting Problem
« on: October 18, 2014, 03:18:57 am »
Seems the database got corrupted and wouldn't allow posting. I think it's repaired now.
Testing...
 

Offline Skimask

  • Super Contributor
  • ***
  • Posts: 1433
  • Country: us
Re: Forum Posting Problem
« Reply #1 on: October 18, 2014, 03:27:02 am »
Looks good...

(P.S.  Pull up your pants!  Or spackle that thing shut!)
I didn't take it apart.
I turned it on.

The only stupid question is, well, most of them...

Save a fuse...Blow an electrician.
 

Offline EEVblogTopic starter

  • Administrator
  • *****
  • Posts: 38558
  • Country: au
    • EEVblog
Re: Forum Posting Problem
« Reply #2 on: October 18, 2014, 01:58:58 pm »
Just check'n again
 

Offline GEuser

  • Frequent Contributor
  • **
  • Posts: 502
  • Country: 00
  • Is Leaving
Re: Forum Posting Problem
« Reply #3 on: October 18, 2014, 03:20:46 pm »
Up late too Dave  :-/O , I thought that was just the innernet as I also had slow and disappearing (404's) at that time too .

But just got this after clicking the post button
Connection Problems
Sorry, SMF was unable to connect to the database. This may be caused by the server being busy. Please try again later.
Soon
 

Offline EEVblogTopic starter

  • Administrator
  • *****
  • Posts: 38558
  • Country: au
    • EEVblog
Re: Forum Posting Problem
« Reply #4 on: October 18, 2014, 10:39:29 pm »
Sorry, SMF was unable to connect to the database. This may be caused by the server being busy. Please try again later.

That one, apparently for the last  for the last 3 hours was due to the MySQL server being locked up. A database service reboot fixed it.
 

Offline EEVblogTopic starter

  • Administrator
  • *****
  • Posts: 38558
  • Country: au
    • EEVblog
Re: Forum Posting Problem
« Reply #5 on: October 18, 2014, 11:23:53 pm »
Hmm, all sorts of problems at the moment it seems. Had to reboot the MySQL server again, and there is a big load on the server. Something isn't right...
 

Offline Pack34

  • Frequent Contributor
  • **
  • Posts: 753
Re: Forum Posting Problem
« Reply #6 on: October 18, 2014, 11:29:59 pm »
Hmm, all sorts of problems at the moment it seems. Had to reboot the MySQL server again, and there is a big load on the server. Something isn't right...

I remember having a similar issue with my company's web host. Maybe the physical server is starting to go. Any way to request a migration to a different machine?
 

Offline EEVblogTopic starter

  • Administrator
  • *****
  • Posts: 38558
  • Country: au
    • EEVblog
Re: Forum Posting Problem
« Reply #7 on: October 19, 2014, 12:11:19 am »
I remember having a similar issue with my company's web host. Maybe the physical server is starting to go. Any way to request a migration to a different machine?

I've been thinking about that. No real evidence that's the case, but it can't hurt I guess, but does seem to have had more downtime/crashes/reboots than I'd expect. I can ask...
 

Offline Pack34

  • Frequent Contributor
  • **
  • Posts: 753
Re: Forum Posting Problem
« Reply #8 on: October 19, 2014, 06:17:25 am »
I remember having a similar issue with my company's web host. Maybe the physical server is starting to go. Any way to request a migration to a different machine?

I've been thinking about that. No real evidence that's the case, but it can't hurt I guess, but does seem to have had more downtime/crashes/reboots than I'd expect. I can ask...

Yes, but without knowing what the problem is it is equally likely it can't (won't) help either. If they are willing to migrate you to a different machine as a "goodwill" gesture, which is what it will be in the absence of any evidence of the cause of the problem, then they will become increasingly less willing in the event of a recurrence in future. They'll say no anyway, unless they can see a fault.

I suggest looking at the systems logs to see what messages are present. There may well be an answer there. Something has changed. It could be a server hardware fault. It could also be a capacity problem or a software upgrade has introduced a bug, either on the server itself or with another server it communicates with.

Do you think it could be something dns related? Intermittently pointing towards the website's old location? I don't know if an intermittent DNS problem is possible, just throwing out ideas.
 

Offline atferrari

  • Frequent Contributor
  • **
  • Posts: 314
  • Country: ar
Re: Forum Posting Problem
« Reply #9 on: October 19, 2014, 11:03:47 am »
More than posting, my problem just to access the site. I cannot recall but there was a three-letters acronym involved, something like MPS or similar and an error code, starting with 5, but not sure really.
Agustín Tomás
In theory, there is no difference between theory and practice. In practice, however, there is.
 

Offline GEuser

  • Frequent Contributor
  • **
  • Posts: 502
  • Country: 00
  • Is Leaving
Re: Forum Posting Problem
« Reply #10 on: October 19, 2014, 11:07:58 am »
I did not have to log on this evening as I could not log off early this morning as that message kept popping up .

The site behaves real snappy here atm , instantaneous practically ..
Soon
 

Offline T3sl4co1l

  • Super Contributor
  • ***
  • Posts: 22363
  • Country: us
  • Expert, Analog Electronics, PCB Layout, EMC
    • Seven Transistor Labs
Re: Forum Posting Problem
« Reply #11 on: October 19, 2014, 12:34:47 pm »
Had this happen a few times, somewhat randomly.  Seems to come and go.

Tim
Seven Transistor Labs, LLC
Electronic design, from concept to prototype.
Bringing a project to life?  Send me a message!
 

Offline Neilm

  • Super Contributor
  • ***
  • Posts: 1558
  • Country: gb
Re: Forum Posting Problem
« Reply #12 on: October 19, 2014, 07:16:57 pm »
I have had 2 or 3 issues getting onto the EEVblog main site and the forum over the last few weeks.
Two things are infinite: the universe and human stupidity; and I'm not sure about the the universe. - Albert Einstein
Tesla referral code https://ts.la/neil53539
 

Offline tautech

  • Super Contributor
  • ***
  • Posts: 29355
  • Country: nz
  • Taupaki Technologies Ltd. Siglent Distributor NZ.
    • Taupaki Technologies Ltd.
Re: Forum Posting Problem
« Reply #13 on: October 20, 2014, 09:57:40 pm »
I have had 2 or 3 issues getting onto the EEVblog main site and the forum over the last few weeks.
Me too. Seems reliability is an issue ATM.
Had this again this morning, though it seems all good now.

Connection Problems

Sorry, SMF was unable to connect to the database. This may be caused by the server being busy. Please try again later.
Avid Rabid Hobbyist.
Some stuff seen @ Siglent HQ cannot be shared.
 

Offline atferrari

  • Frequent Contributor
  • **
  • Posts: 314
  • Country: ar
Re: Forum Posting Problem
« Reply #14 on: October 21, 2014, 01:59:21 am »
I have had 2 or 3 issues getting onto the EEVblog main site and the forum over the last few weeks.
Me too. Seems reliability is an issue ATM.
Had this again this morning, though it seems all good now.

Connection Problems

Sorry, SMF was unable to connect to the database. This may be caused by the server being busy. Please try again later.

The same here, several times today.
Agustín Tomás
In theory, there is no difference between theory and practice. In practice, however, there is.
 

Offline retiredcaps

  • Super Contributor
  • ***
  • Posts: 3575
  • Country: ca
Re: Forum Posting Problem
« Reply #15 on: October 21, 2014, 02:02:51 am »
I also had problems Monday Oct 20 starting at 10AM and ongoing to 10:30AM MST.  I tried later in the afternoon around 2PM MST and was able to get on.

I was able to ping www.eevblog.com, but the forum and main page were down with the same error messages posted by others.
 

Offline EEVblogTopic starter

  • Administrator
  • *****
  • Posts: 38558
  • Country: au
    • EEVblog
Re: Forum Posting Problem
« Reply #16 on: October 21, 2014, 02:24:55 am »
Yep, the server is having real problem at the moment, I need to speak with HostGator.
The database issues I'm pretty sure are caused by the server going down, and when it does, there is always a chance a database table can get corrupted.
 

Offline gnif

  • Administrator
  • *****
  • Posts: 1701
  • Country: au
  • Views and opinions are my own
    • AMD
Re: Forum Posting Problem
« Reply #17 on: October 21, 2014, 11:27:21 am »
Hi All,

I am sorry I have not been around to reply to any of this, I have been put out of action as I have been recovering from the whooping cough. I have suggested that Dave have all the RAM in the server replaced, the constant table crashes and now these major crashes point the finger at either RAM or HDD, but since the HDD is a RAID array and as far as I am aware is not reporting any issues, I would not expect this kind of crash (RAM is the easier to replace first too).
SMTS Software Development Engineer @ AMD
 

Offline dr.diesel

  • Super Contributor
  • ***
  • Posts: 2214
  • Country: us
  • Cramming the magic smoke back in...
Re: Forum Posting Problem
« Reply #18 on: October 21, 2014, 12:05:33 pm »
Would RAM errors not show up somewhere?

IIRC it's a Linux box, single bit ECC errors would show up in dmesg.

Offline EEVblogTopic starter

  • Administrator
  • *****
  • Posts: 38558
  • Country: au
    • EEVblog
Re: Forum Posting Problem
« Reply #19 on: October 21, 2014, 12:08:42 pm »
Hostgator are saying that it's the MySQL that is going down, and not the whole server. They are working on it...
 

Offline gnif

  • Administrator
  • *****
  • Posts: 1701
  • Country: au
  • Views and opinions are my own
    • AMD
Re: Forum Posting Problem
« Reply #20 on: October 21, 2014, 01:18:03 pm »
With regards to RAM, It looks like the server may be overcommited, I have reduced the total ram that MySQL is allowed to use, but this does not explain why it is getting blown way out. It seems to occur when there is heavy load on the server, I am seeing oom errors that is killing MySQL and hundreds of PHP processes running at the same time. I am investigating what PHP is doing during these periods.
SMTS Software Development Engineer @ AMD
 

Offline bookaboo

  • Frequent Contributor
  • **
  • Posts: 749
  • Country: ie
Re: Forum Posting Problem
« Reply #21 on: October 21, 2014, 01:21:17 pm »
May be unrelated but the entire site was intermittent for me today from about 11.00am until 2.00PM GMT.
 

Offline GEuser

  • Frequent Contributor
  • **
  • Posts: 502
  • Country: 00
  • Is Leaving
Re: Forum Posting Problem
« Reply #22 on: October 21, 2014, 01:24:11 pm »
Doing some internal search's like 10 minutes ago a cupla times it came back > Database error/corruption retry later or something ..
Soon
 

Offline gnif

  • Administrator
  • *****
  • Posts: 1701
  • Country: au
  • Views and opinions are my own
    • AMD
Re: Forum Posting Problem
« Reply #23 on: October 21, 2014, 01:34:18 pm »
May be unrelated but the entire site was intermittent for me today from about 11.00am until 2.00PM GMT.

It was related, logs show the database had crashed at that point.

Doing some internal search's like 10 minutes ago a cupla times it came back > Database error/corruption retry later or something ..

This may have occured while I was restarting the MySQL service to apply the memory changes.
SMTS Software Development Engineer @ AMD
 

Offline T3sl4co1l

  • Super Contributor
  • ***
  • Posts: 22363
  • Country: us
  • Expert, Analog Electronics, PCB Layout, EMC
    • Seven Transistor Labs
Re: Forum Posting Problem
« Reply #24 on: October 21, 2014, 01:42:16 pm »
Saw this earlier,

Code: [Select]
Table './eevblog_smp01/smf_members' is marked as crashed and should be repaired
Hope it's not hackers or something :(
Seven Transistor Labs, LLC
Electronic design, from concept to prototype.
Bringing a project to life?  Send me a message!
 

Offline GEuser

  • Frequent Contributor
  • **
  • Posts: 502
  • Country: 00
  • Is Leaving
Re: Forum Posting Problem
« Reply #25 on: October 21, 2014, 01:44:21 pm »
Saw this earlier,

Code: [Select]
Table './eevblog_smp01/smf_members' is marked as crashed and should be repaired
Hope it's not hackers or something :(

That was another that popped up too
Soon
 

Offline gnif

  • Administrator
  • *****
  • Posts: 1701
  • Country: au
  • Views and opinions are my own
    • AMD
Re: Forum Posting Problem
« Reply #26 on: October 21, 2014, 01:50:42 pm »
Saw this earlier,

Code: [Select]
Table './eevblog_smp01/smf_members' is marked as crashed and should be repaired
Hope it's not hackers or something :(

That was another that popped up too

Thanks for that, I will correct it now, that is a problem from the prior crash.

Edit: This has been done, should not present again.


Edit 2: I noted that even though we setup memcached way back, for some reason it is not in use. I have enabled it in SMF and have already noted quite a substantial reduction in server I/O. The forum also seems to be quite a bit faster for me, but I have not done any testing to properly confirm this.
« Last Edit: October 21, 2014, 01:55:38 pm by gnif »
SMTS Software Development Engineer @ AMD
 

Offline EEVblogTopic starter

  • Administrator
  • *****
  • Posts: 38558
  • Country: au
    • EEVblog
Re: Forum Posting Problem
« Reply #27 on: October 21, 2014, 01:58:48 pm »
Code: [Select]
Table './eevblog_smp01/smf_members' is marked as crashed and should be repairedHope it's not hackers or something :(

There are two issues at the moment, that have been persistent over the last two days.

The first and most common is that error message listed above. I believe this is caused because the MySQL database shat itself for some reason, and when that happens there is a chance that any table that was being accessed will corrupt. Hence why you see different error message here. This requires a simple but manual table repair to fix. This seems to be happening every few hours at present.

The second issue is that of a complete MySQL lockup. In this case the MySQL server has to be restarted. This will bring down not only the forum, but the wordpress blog too.

I've disabled Cloudflare until all this gets fixed.
 

Offline EEVblogTopic starter

  • Administrator
  • *****
  • Posts: 38558
  • Country: au
    • EEVblog
Re: Forum Posting Problem
« Reply #28 on: October 21, 2014, 02:01:56 pm »
Edit 2: I noted that even though we setup memcached way back, for some reason it is not in use. I have enabled it in SMF and have already noted quite a substantial reduction in server I/O.

I haven't fiddled with that. Although I did just update to SMF 2.0.9 yesterday.
 

Offline gnif

  • Administrator
  • *****
  • Posts: 1701
  • Country: au
  • Views and opinions are my own
    • AMD
Re: Forum Posting Problem
« Reply #29 on: October 21, 2014, 02:10:23 pm »
I've disabled Cloudflare until all this gets fixed.

Cloudflare has nothing in common with this issue, but with regards to the other issues people have had with it, I do not thing that it is a bad idea to disable it anyway.
SMTS Software Development Engineer @ AMD
 

Offline EEVblogTopic starter

  • Administrator
  • *****
  • Posts: 38558
  • Country: au
    • EEVblog
Re: Forum Posting Problem
« Reply #30 on: October 21, 2014, 02:29:19 pm »
Cloudflare has nothing in common with this issue, but with regards to the other issues people have had with it, I do not thing that it is a bad idea to disable it anyway.

I've had constant problems with it myself.
 

Offline gnif

  • Administrator
  • *****
  • Posts: 1701
  • Country: au
  • Views and opinions are my own
    • AMD
Re: Forum Posting Problem
« Reply #31 on: October 21, 2014, 02:45:04 pm »
Leave it off then, I will keep a close eye on the server and see how things perform.
SMTS Software Development Engineer @ AMD
 

Offline alimirjamali

  • Regular Contributor
  • *
  • Posts: 83
  • Country: ir
  • Analog! D2A or A2D?
    • Ali's personal blog (updated once per year!)
Re: Forum Posting Problem
« Reply #32 on: October 21, 2014, 04:25:42 pm »
I would agree with gnif that CloudFlare should have nothing to do with MySql issue. Currently CloudFlare cashing is disabled and only the DNS queries are resolved by them 8). Let's see if would help at all (which I doubt) ???. I presume that you check mysql logs (/var/log/mysql/error.log || /var/log/mysqld.log || etc) for errors?

Beside gnif, do you have sudoers (or any other support person) in other TimeZones? Maybe someone in UK or US you trust in person. Someone that could restart the bloody MySql service if needed!

The value of posts on the Forum is substantial. Every time Forum crashes, I panic :scared:. Would you confirm that MySql databases and httpd data are backed up regularly (CRON, rsync, etc.)?

With regards to RAM, It looks like the server may be overcommited,
If it is not a huge secret, what is the output of free -m after a fresh restart? I am curious to know how much RAM is installed on the server.
 

Offline tautech

  • Super Contributor
  • ***
  • Posts: 29355
  • Country: nz
  • Taupaki Technologies Ltd. Siglent Distributor NZ.
    • Taupaki Technologies Ltd.
Re: Forum Posting Problem
« Reply #33 on: October 25, 2014, 01:35:39 am »
FYI: Had more of this early this morning:
504 - Gateway Timeout
We are sorry for the inconvenience that this error may be causing you. We are aware of the issue and are working to resolve it, please be patient.

There is no need to report this error.

Thank you for your patience,
Dave & gnif
Avid Rabid Hobbyist.
Some stuff seen @ Siglent HQ cannot be shared.
 

Offline EEVblogTopic starter

  • Administrator
  • *****
  • Posts: 38558
  • Country: au
    • EEVblog
Re: Forum Posting Problem
« Reply #34 on: October 25, 2014, 02:00:19 am »
FYI: Had more of this early this morning:
504 - Gateway Timeout

Apparently the FTDI thing got picked up on Hacker News and I presume the server got overloaded.
 

Offline zapta

  • Super Contributor
  • ***
  • Posts: 6289
  • Country: 00
Re: Forum Posting Problem
« Reply #35 on: October 25, 2014, 03:29:41 am »
Apparently the FTDI thing got picked up on Hacker News and I presume the server got overloaded.

Or FTDI decided to nuke also these forums...
 

Offline tautech

  • Super Contributor
  • ***
  • Posts: 29355
  • Country: nz
  • Taupaki Technologies Ltd. Siglent Distributor NZ.
    • Taupaki Technologies Ltd.
Re: Forum Posting Problem
« Reply #36 on: October 26, 2014, 02:41:53 am »
FYI: Had more of this early this morning:
504 - Gateway Timeout
We are sorry for the inconvenience that this error may be causing you. We are aware of the issue and are working to resolve it, please be patient.

There is no need to report this error.

Thank you for your patience,
Dave & gnif
More again for a short while today.
Avid Rabid Hobbyist.
Some stuff seen @ Siglent HQ cannot be shared.
 

Offline nanofrog

  • Super Contributor
  • ***
  • Posts: 5446
  • Country: us
Re: Forum Posting Problem
« Reply #37 on: October 26, 2014, 04:11:20 am »
504 - Gateway Timeout
We are sorry for the inconvenience that this error may be causing you. We are aware of the issue and are working to resolve it, please be patient.

There is no need to report this error.

Thank you for your patience,
Dave & gnif
I'm still getting this.
 

Offline gnif

  • Administrator
  • *****
  • Posts: 1701
  • Country: au
  • Views and opinions are my own
    • AMD
Re: Forum Posting Problem
« Reply #38 on: October 26, 2014, 07:34:35 am »
Even though I am not entirely convinced they always DO know about the problem, in spite of this canned response, it is nonetheless clear there is no need to report the problem.

You are sort of correct, that error about us knowing about it was when we had a prolonged window where we expected this to occur (server changes) and the error page was never fixed/updated when we were done.

I have been monitoring the server closely to try to determine where the performance bottlekneck is in the current configuration, we initially assumed these issues were due to the Slashdot traffic, but it is becoming aparrent that there is something more going on.
SMTS Software Development Engineer @ AMD
 

Offline tautech

  • Super Contributor
  • ***
  • Posts: 29355
  • Country: nz
  • Taupaki Technologies Ltd. Siglent Distributor NZ.
    • Taupaki Technologies Ltd.
Re: Forum Posting Problem
« Reply #39 on: October 26, 2014, 07:40:38 am »
Quote
but it is becoming aparrent that there is something more going on.
Yep for the last few weeks, sometimes for some hours.
Today....infrequent short outages.
Avid Rabid Hobbyist.
Some stuff seen @ Siglent HQ cannot be shared.
 

Offline gnif

  • Administrator
  • *****
  • Posts: 1701
  • Country: au
  • Views and opinions are my own
    • AMD
Re: Forum Posting Problem
« Reply #40 on: October 26, 2014, 07:45:27 am »
Quote
but it is becoming aparrent that there is something more going on.
Yep for the last few weeks, sometimes for some hours.
Today....infrequent short outages.

I have just changed the website to store sessions in memcached instead of in the database on disk, this seems to be the bulk of I/O occuring and should make a noticable improvement to performance. If you get timeouts again please don't hesitate to PM me as I will notice that first. The next stage might just be a simple matter of increasing the number of apache processes as the gateway timeout occurs when Nginx can not communicate with the Apache backend (better solution yet would be to remove apache from the equasion, but would prefer not to as this starts to get outside of what cPanel will support).
SMTS Software Development Engineer @ AMD
 

Offline T3sl4co1l

  • Super Contributor
  • ***
  • Posts: 22363
  • Country: us
  • Expert, Analog Electronics, PCB Layout, EMC
    • Seven Transistor Labs
Re: Forum Posting Problem
« Reply #41 on: October 27, 2014, 01:29:35 pm »
Been getting a "session timeout, please attempt to reply again" message lately.  Yes, I write overly long posts, but still...

Tim
Seven Transistor Labs, LLC
Electronic design, from concept to prototype.
Bringing a project to life?  Send me a message!
 

Offline Thor-Arne

  • Supporter
  • ****
  • Posts: 500
  • Country: no
  • tinker - tinker, little noob.....
Re: Forum Posting Problem
« Reply #42 on: October 27, 2014, 02:23:39 pm »
I sometimes get "session timeout" when marking a board as read, so it's not the long posts.  ;)
 

Offline EEVblogTopic starter

  • Administrator
  • *****
  • Posts: 38558
  • Country: au
    • EEVblog
Re: Forum Posting Problem
« Reply #43 on: October 27, 2014, 09:11:21 pm »
It went down again a few hours ago.
Even the director of marketing at HostGator emailed me out of the blue and said they noticed it, and asked if it was due to FTDIgate
I guess he's a viewer  :-+

So I think it is simply a spike in traffic every time FTDIgate gets linked somewhere.
gnif has put in place caching that seems to be working well, but I guess there is only so much you can do with a single dedicated server.
 

Offline alimirjamali

  • Regular Contributor
  • *
  • Posts: 83
  • Country: ir
  • Analog! D2A or A2D?
    • Ali's personal blog (updated once per year!)
Re: Forum Posting Problem
« Reply #44 on: October 27, 2014, 09:23:32 pm »
So I think it is simply a spike in traffic every time FTDIgate gets linked somewhere.
A proper engineer does not think but measure :P. Do you monitor RAM usage, Disk IO, CPU load 8)? There are many utilities in Penguin for both real-time and logged measurement of system metrics (free/top/htop/iftop/iotop/etc. :blah:). Maybe it is time for an upgrade >:D?
 

Offline gnif

  • Administrator
  • *****
  • Posts: 1701
  • Country: au
  • Views and opinions are my own
    • AMD
Re: Forum Posting Problem
« Reply #45 on: October 28, 2014, 02:49:22 am »
So I think it is simply a spike in traffic every time FTDIgate gets linked somewhere.
A proper engineer does not think but measure :P. Do you monitor RAM usage, Disk IO, CPU load 8)? There are many utilities in Penguin for both real-time and logged measurement of system metrics (free/top/htop/iftop/iotop/etc. :blah:). Maybe it is time for an upgrade >:D?

We do indeed monitor, and nothing directly points to an issue which is why it has been hard to find out what is going on. Since we corrected the issue with the database crashing we are getting some more constant error reporting from the server that is helping to track the issue down. At this point it seems that there is something wrong with PHP as we are now seeing tons of php segfaults in dmesg, I am sill investigating as to the cause.

Edit: The outage that just occured was intentional, we just upgraded php to resolve a known bug with custom error handlers in the version of PHP we were running, I will conitnue to monitor the server and see if this resolves the issue that has been occuring.
« Last Edit: October 28, 2014, 04:00:17 am by gnif »
SMTS Software Development Engineer @ AMD
 

Offline gdewitte

  • Supporter
  • ****
  • Posts: 59
  • Country: us
Re: Forum Posting Problem
« Reply #46 on: October 28, 2014, 02:37:38 pm »
I'm consistently getting a "504-Gateway timeout" when I try to "Show unread posts…" Anyone else having this problem, or could it be something hosed on my PC? All the other links to various posts seem to work just fine.
 

Offline Owen

  • Contributor
  • Posts: 44
Re: Forum Posting Problem
« Reply #47 on: October 28, 2014, 08:31:07 pm »
I'm consistently getting a "504-Gateway timeout" when I try to "Show unread posts…" Anyone else having this problem, or could it be something hosed on my PC? All the other links to various posts seem to work just fine.

I'm getting this 504-Gateway timeout when i'm trying to enter "www.eevblog.com/forum/testgear/".
 

Offline EEVblogTopic starter

  • Administrator
  • *****
  • Posts: 38558
  • Country: au
    • EEVblog
Re: Forum Posting Problem
« Reply #48 on: October 28, 2014, 08:39:03 pm »
I'm consistently getting a "504-Gateway timeout" when I try to "Show unread posts…" Anyone else having this problem

Yep, I get that too, but only on my machine at the lab, and only on Firefox. Not other browsers or computers.
https://www.eevblog.com/forum/index.php?action=unread works fine though and does the same thing, but the https://www.eevblog.com/forum/unread/ link does not work.
 

Offline alimirjamali

  • Regular Contributor
  • *
  • Posts: 83
  • Country: ir
  • Analog! D2A or A2D?
    • Ali's personal blog (updated once per year!)
Re: Forum Posting Problem
« Reply #49 on: October 28, 2014, 09:02:35 pm »
Yep, I get that too, but only on my machine at the lab, and only on Firefox. Not other browsers or computers.
https://www.eevblog.com/forum/index.php?action=unread works fine though and does the same thing, but the https://www.eevblog.com/forum/unread/ link does not work.
This is a server side issue and has almost nothing to do with your browser (Firefox) or internet connection (@lab).

The reason you were able to get it through index.php is you virtually bypassed browser cashing by an alternative URL and got a fresh copy. The fresh copy might work fine or be another 504-Gateway timeout page.

Hey gnif: Maybe it would be better to add META HTTP-EQUIV="Pragma" CONTENT="no-cache" to head section on the default 504-Gateway Timeout HTML page. Maybe add a meta http-equiv="refresh" content="300" tag too.  :-//
 

Offline gnif

  • Administrator
  • *****
  • Posts: 1701
  • Country: au
  • Views and opinions are my own
    • AMD
Re: Forum Posting Problem
« Reply #50 on: October 29, 2014, 12:16:13 pm »
Hey gnif: Maybe it would be better to add META HTTP-EQUIV="Pragma" CONTENT="no-cache" to head section on the default 504-Gateway Timeout HTML page. Maybe add a meta http-equiv="refresh" content="300" tag too.  :-//

You are quite correct, this however has not been a priority to tune as we have been looking for the root cause of the issue. I will adjust this a little later tonight, in the meantime I am preparing to remove apache from the equation and handle requests directly with Nginx, this should both improve performance and assist in determining where any potential bottleknecks are.
SMTS Software Development Engineer @ AMD
 

Offline gdewitte

  • Supporter
  • ****
  • Posts: 59
  • Country: us
Re: Forum Posting Problem
« Reply #51 on: October 29, 2014, 01:31:18 pm »


Quote from: gdewitte on Yesterday at 01:37:38 AM
I'm consistently getting a "504-Gateway timeout" when I try to "Show unread posts…" Anyone else having this problem, or could it be something hosed on my PC? All the other links to various posts seem to work just fine.
My solution so far: cleared browsing history on Google Chrome and everything worked. May be unique to my system and Google Chrome, as all links worked fine when I tried Firefox.


 

Offline gnif

  • Administrator
  • *****
  • Posts: 1701
  • Country: au
  • Views and opinions are my own
    • AMD
Re: Forum Posting Problem
« Reply #52 on: October 29, 2014, 01:33:39 pm »


Quote from: gdewitte on Yesterday at 01:37:38 AM
I'm consistently getting a "504-Gateway timeout" when I try to "Show unread posts…" Anyone else having this problem, or could it be something hosed on my PC? All the other links to various posts seem to work just fine.
My solution so far: cleared browsing history on Google Chrome and everything worked. May be unique to my system and Google Chrome, as all links worked fine when I tried Firefox.


I am preparing to remove apache from the equation and handle requests directly with Nginx, this should both improve performance and assist in determining where any potential bottleknecks are.
« Last Edit: October 29, 2014, 01:36:09 pm by gnif »
SMTS Software Development Engineer @ AMD
 

Offline alimirjamali

  • Regular Contributor
  • *
  • Posts: 83
  • Country: ir
  • Analog! D2A or A2D?
    • Ali's personal blog (updated once per year!)
Re: Forum Posting Problem
« Reply #53 on: October 29, 2014, 05:11:15 pm »
I am preparing to remove apache from the equation and handle requests directly with Nginx, this should both improve performance and assist in determining where any potential bottleknecks are.

Hey gnif. I see you have enabled CloudFlare cashing again 8). It is not just DNS anymore :P.

So you are going to ditch Apache in favor of Nginx :-+. I am happy that I am not in your position to translate all of those nasty .htaccess and Apache VirtualHost config files :phew:. You are going to test it on a local copy before applying them?  ??? right? We have some EEVBlog Forumoholics who will be pissed off if Forum is down more than 1 day :rant:.

<ConspiracyTheory>
Maybe Dave is DDoSed :o? Dave's WordPress Blog and SMF Forum are popular but not that much popular which Apache could not handle. There are less than 2000 Online Users+Guests on Forum. Considering that SMF User online time threshold is 15 minutes (by default) which implies that they do not fetch data simultaneously, Apache should be able to handle them easily.

2000 users / 15 minutes / 60 seconds * Approx. 20 requests per users = Approx. 44 requests per second = piece of cake for Apache.

Even before the #FTDIGate, Dave had made many foes. Maybe if you double check IP of users, you might see 90% from some FTDI server :wtf: (What Da FTDI).
</ConspiracyTheory>

Good luck. ^-^
« Last Edit: October 29, 2014, 06:25:54 pm by alimirjamali »
 

Offline gnif

  • Administrator
  • *****
  • Posts: 1701
  • Country: au
  • Views and opinions are my own
    • AMD
Re: Forum Posting Problem
« Reply #54 on: October 29, 2014, 05:19:41 pm »
Hey gnif. I see you have enabled CloudFlare cashing again 8). It is not just DNS anymore :P.

This was Dave's decision to do so, it has nothing to do with the gateway issues.

So you are going to ditch Apache in favor of Nginx :-+. I am happy that I am not in your position to translate all of those nasty .htaccess and Apache VirtualHost config files :phew:. You are going to test it on a local copy before applying them?  ??? right? We have some EEVBlog Forumoholics who will be pissed of if Forum is down more than 1 day :rant:.

Already translated, and live, see the new thread.

<ConspiracyTheory>
Maybe Dave is DDoSed :o? Dave's WordPress Blog and SMF Forum are popular but not that much popular which Apache could not handle. There are less than 2000 Online Users+Guests on Forum. Considering that SMF User online time threshold is 15 minutes (by default) which implies that they do not fetch data simultaneously, Apache should be able to handle them easily.

2000 users / 15 minutes / 60 seconds * Approx. 20 requests per users = Approx. 44 requests per second = piece of cake for Apache.

Even before the #FTDIGate, Dave had made many foes. Maybe if you double check IP of users, you might see 90% from some FTDI server :wtf: (What Da FTDI).
</ConspiracyTheory>

No DDoS and we rate limit per IP anyway, so they would need to have way more servers then you would expect to saturate the server. Since this is a cPanel server it is tuned to host multiple small to medium sized sites for mass reselling, not one large site. It builds Apache in such a way as to suit this purpose and in this instance it is a hinderance. Moving away from Apache to Nginx + PHP-FPM allows us to escape the bounds of the cPanel paradigm :).
« Last Edit: October 29, 2014, 05:22:02 pm by gnif »
SMTS Software Development Engineer @ AMD
 

Offline T3sl4co1l

  • Super Contributor
  • ***
  • Posts: 22363
  • Country: us
  • Expert, Analog Electronics, PCB Layout, EMC
    • Seven Transistor Labs
Re: Forum Posting Problem
« Reply #55 on: November 06, 2014, 08:48:41 am »
Been getting a "session timeout, please attempt to reply again" message lately.  Yes, I write overly long posts, but still...

Tim

It's back :(
Seven Transistor Labs, LLC
Electronic design, from concept to prototype.
Bringing a project to life?  Send me a message!
 

Offline gnif

  • Administrator
  • *****
  • Posts: 1701
  • Country: au
  • Views and opinions are my own
    • AMD
Re: Forum Posting Problem
« Reply #56 on: November 06, 2014, 04:47:57 pm »
Been getting a "session timeout, please attempt to reply again" message lately.  Yes, I write overly long posts, but still...

Tim

It's back :(

I will adjust how memcached is storing sessions and increase the timeout, this should correct this.
SMTS Software Development Engineer @ AMD
 

Offline T3sl4co1l

  • Super Contributor
  • ***
  • Posts: 22363
  • Country: us
  • Expert, Analog Electronics, PCB Layout, EMC
    • Seven Transistor Labs
Re: Forum Posting Problem
« Reply #57 on: November 07, 2014, 01:39:30 am »
Thanks!
Seven Transistor Labs, LLC
Electronic design, from concept to prototype.
Bringing a project to life?  Send me a message!
 

Offline EvilGeniusSkis

  • Regular Contributor
  • *
  • Posts: 125
  • Country: ca
Re: Forum Posting Problem
« Reply #58 on: November 13, 2014, 01:36:43 am »
could someone have been using something more sophisticated than a simple DDoS, such as SlowLoris or RUDY? 
 

Offline gnif

  • Administrator
  • *****
  • Posts: 1701
  • Country: au
  • Views and opinions are my own
    • AMD
Re: Forum Posting Problem
« Reply #59 on: November 13, 2014, 05:02:47 am »
could someone have been using something more sophisticated than a simple DDoS, such as SlowLoris or RUDY?

Certainly not, logs clearly indicated that Apache was running out of connections well before server load was an issue, Apache is not exposed to the internet as it was behind Nginx which was the reverse proxy. If it was a DoS attack such as SlowLoris or RUDY Nginx would also have stopped responding.

Put simply, Apache is software that should be put down, it was good in it's day, but it is well and truly past it's expiration date. Please do not think that I believe this due to a limited understanding of Apache, I am extremely well versed with it, and I am the maintainer/developer of mod_rpaf which is in use by hundreds if not thousands of large scale deployments around the globe, so I know Apache from a code perspective, not just a deployment perspective.
SMTS Software Development Engineer @ AMD
 

Offline alimirjamali

  • Regular Contributor
  • *
  • Posts: 83
  • Country: ir
  • Analog! D2A or A2D?
    • Ali's personal blog (updated once per year!)
Re: Forum Posting Problem
« Reply #60 on: November 13, 2014, 06:05:08 am »
could someone have been using something more sophisticated than a simple DDoS, such as SlowLoris or RUDY?
FTDI produces many chips for Android smart phones. They could have turned millions of Android phones to zombies and attacked Dave's forum in retaliation to the #FTDIgate >:D

p.s. I know that it is technically impossible; However, conspiracy theories are simply cool ^-^.
 

Offline gnif

  • Administrator
  • *****
  • Posts: 1701
  • Country: au
  • Views and opinions are my own
    • AMD
Re: Forum Posting Problem
« Reply #61 on: November 13, 2014, 06:54:34 am »
could someone have been using something more sophisticated than a simple DDoS, such as SlowLoris or RUDY?
FTDI produces many chips for Android smart phones. They could have turned millions of Android phones to zombies and attacked Dave's forum in retaliation to the #FTDIgate >:D

p.s. I know that it is technically impossible; However, conspiracy theories are simply cool ^-^.

Haha, you literally made me LOL.

In all this was a good exercise as the server is now running better then ever for pretty much everyone, even those of us who are over the pond and have to already deal with large latency due to the transmission delays.
SMTS Software Development Engineer @ AMD
 

Offline EvilGeniusSkis

  • Regular Contributor
  • *
  • Posts: 125
  • Country: ca
Re: Forum Posting Problem
« Reply #62 on: November 14, 2014, 01:39:55 am »
this whole thing reminds me of something my multimedia teacher would say
Quote
Technology will make a fool out of everyone.

the reason I say this is, to me anyway, it seem that the problem was fixed without anyone figuring out what the problem was exactly. am i right?

also is there a forum features wishlist thread anywhere?
« Last Edit: November 14, 2014, 01:43:47 am by EvilGeniusSkis »
 

Offline gnif

  • Administrator
  • *****
  • Posts: 1701
  • Country: au
  • Views and opinions are my own
    • AMD
Re: Forum Posting Problem
« Reply #63 on: November 14, 2014, 11:33:12 am »
the reason I say this is, to me anyway, it seem that the problem was fixed without anyone figuring out what the problem was exactly. am i right?

I have very clearly stated what the problem was, Apache simply is not capable of serving the sheer number of PHP requests the website has as it is archaic in its design and can not take advantage of modern hardware correctly.
SMTS Software Development Engineer @ AMD
 

Offline atferrari

  • Frequent Contributor
  • **
  • Posts: 314
  • Country: ar
Re: Forum Posting Problem
« Reply #64 on: November 15, 2014, 01:23:28 pm »
Hola gnif

I want to stop receiving notifications of new replies to some threads.
I clicked on "UNNOTIFY" at the top of them but I still receive notifications.

One of the threads is this very one.

Gracias for any help.
Agustín Tomás
In theory, there is no difference between theory and practice. In practice, however, there is.
 


Share me

Digg  Facebook  SlashDot  Delicious  Technorati  Twitter  Google  Yahoo
Smf