Bad gateway 502

[Seekonk]

About a week ago I first got this when clicking on, New replies to your posts. Said they were working on it. It still doesn't work.  I don't see any threads mentioning this problem so it must just happen to me. Any solution around this? 

I added the ?c=2  to the end /unreadreplies/?c=2  and that works, but shouldn't it work with just clicking?

[pelule]

Easiest solution is "Ctrl-F5", see at this thread, where it was discussed already:
https://www.eevblog.com/forum/chat/bad-gateway-clicking-show-new-replies/

[Ampera]

Yea, F5 solves it for me when it happens.

No clue why it happens, strange.

[David Hess]

F5 or removing cookies does not always fix it.  I had it happen at about the same time when posting a reply and after that only that discussion thread returned the 502 Bad Gateway error.  Later my reply showed up 3 times in a row.  Later yet it was fixed.

[gnif]

If it occurs again please PM me with the time it occurred and your current IP address, I will have a look at the logs and see what is going on.

[gnif]

Thanks guys, I have identified the fault and we are investigating ways to correct the issue. As always if something like this occurs please feel free to PM me.

[cvanc]

FWIW it was happening earlier today to me.  Crtl-F5 took care of it.

[cvanc]

Consider updating the message that comes up saying you are aware of the problem. Perhaps put a date on it and a means of reporting the issue, or even a Twitter link or something to provide more information.

Very good, but the 502 message should also say "hey, try Crtl-F5".  Lots of people don't know that particular trick...

[lowimpedance]

Consider updating the message that comes up saying you are aware of the problem. Perhaps put a date on it and a means of reporting the issue, or even a Twitter link or something to provide more information.

Very good, but the 502 message should also say "hey, try Crtl-F5".  Lots of people don't know that particular trick...

This and any other tricks too would be good to add to the message fro things to try first.
The Crtl-F5 worked for me today on two different areas of the forum, and I only just found out about this trick today !!.

[StillTrying]

I saw it a couple of times earlier, it was only effecting 1 or 2 threads for me, I might try 1 refresh, but then I just go and visit another thread or group for 2 minutes.

It wouldn't surprise me if everybody constantly hitting F5 actually makes the problem worse!

[tautech]

Thanks guys, I have identified the fault and we are investigating ways to correct the issue. As always if something like this occurs please feel free to PM me.

Had a session of it last week but a tidy up of Chrome fixed it.
Only a few days before it was a pain but somehow came right on it's own before giving trouble again. 
I'll shoot something through quick smart when it happens again if you haven't consigned it to history by then. 

[tkuhmone]

got same error also... Using Mozilla FF browser, I tried to logout from forum during error displayed. That did not work, like any other action on forum. I cleared the history (including cookies etc.) on Mozilla. This forced login out of forum. Just a new login, and forum was ok again...

[gnif]

The issue is caused by a database maintenance operation taking too long which then causes PHP to wait for the database, which in turn causes a backlog of requests running the system out of PHP processes to handle new requests. This is not a history/cookie related issue, and the force refresh is just compounding the issue. All HTTP errors that start with a 5 are server side problems.

[gnif]

The issue is caused by a database maintenance operation taking too long which then causes PHP to wait for the database, which in turn causes a backlog of requests running the system out of PHP processes to handle new requests. This is not a history/cookie related issue, and the force refresh is just compounding the issue. All HTTP errors that start with a 5 are server side problems.

By "force/refresh" do you mean me clearing my browser cookies and cache?  If it is not a history/cookie issue why does that seem to resolve the issue for me? When you say database maintenance operation do you mean backups?

Your force/refresh (Ctrl+F5) or clearing cookies/cache is only working by coincidence, its just the fact that you made another request to the server when it was a little less loaded and as such it was able to return a result. As for DB maintenance, no, not backups, there is more then just backups occurring on the server but I can not divulge details without compromising security.

Response status codes beginning with the digit "5" indicate cases in which the server is aware that it has encountered an error or is otherwise incapable of performing the request. Except when responding to a HEAD request, the server should include an entity containing an explanation of the error situation, and indicate whether it is a temporary or permanent condition. Likewise, user agents should display any included entity to the user. These response codes are applicable to any request method.

The fact that I also see the 502 error in the logs clearly shows that a normal load/refresh is requesting the content from the server, if there was a caching issue here you would just get a cached copy of the working page, not the 502 error message.

[David Hess]

Your force/refresh (Ctrl+F5) or clearing cookies/cache is only working by coincidence, its just the fact that you made another request to the server when it was a little less loaded and as such it was able to return a result. As for DB maintenance, no, not backups, there is more then just backups occurring on the server but I can not divulge details without compromising security.

Last week when this happened, only the thread I was replying to returned the error and it did it for hours.  Access to other discussion threads and parts of the forum worked fine.  Further, there ended up being 3 copies of the same post which magically resolved itself so it is more than just excessive server load.

I cleared cache, cookies, used a different system, and used a different browser with no improvement until it magically corrected itself.  At some point the extra posts were removed.

[gnif]

Your force/refresh (Ctrl+F5) or clearing cookies/cache is only working by coincidence, its just the fact that you made another request to the server when it was a little less loaded and as such it was able to return a result. As for DB maintenance, no, not backups, there is more then just backups occurring on the server but I can not divulge details without compromising security.

Last week when this happened, only the thread I was replying to returned the error and it did it for hours.  Access to other discussion threads and parts of the forum worked fine.  Further, there ended up being 3 copies of the same post which magically resolved itself so it is more than just excessive server load.

I cleared cache, cookies, used a different system, and used a different browser with no improvement until it magically corrected itself.  At some point the extra posts were removed.

It is not more then excessive server load, logs clearly show the problem, the triplicate post would have been caused by refreshing the page and resubmitting the post data... here is how it works:

• You type your post and submit to the server
• Server gets your post and passes it on to PHP
• PHP processes it and attempts to update the database
• The database is write locked (not read, which is why the pages still load) due to maintenance, so PHP waits
• The HTTP server sees PHP taking too long to return, so gives up and returns a 502 error
• You assume the 502 means the server flat out failed, so you refresh, which repeats the above process
• Eventually the database becomes available again and the pending inserts from PHP are executed
• The page finally loads for you and there are three posts

The additional duplicate posts will have been manually removed by a moderator, people report them when they see them, and the mods here clean them up. If a server admin ever tells you something just magically fixed itself be very suspicious about that admin's abilities, there is always a reason.

Most PHP scripts, including the ones that drive this forum follow the flow, input -> process -> output... and the HTTP server waits for the output before returning anything to the client unless forced to flush, this is to avoid TCP fragmentation and results in better performance... Just because you saw an error from the HTTP server, or even the PHP script if it does return something, doesn't mean some processing did not occur that updated things.

If people are getting 502 errors for reading, it means that there are no more PHP processes available as they are all pending on the database, or the server is overloaded with requests, or the database has a fault.

We are seeing a clear load issue during database maintenance that should not be taking more then a few seconds and I am currently investigating as to the cause.

[David Hess]

It is not more then excessive server load, logs clearly show the problem, the triplicate post would have been caused by refreshing the page and resubmitting the post data... here is how it works:

Then why was it *only* happening on the thread I had tried to reply to?  I could not read that thread, could not reply to it although it apparently was accepting the reply but not continuing, and accessing any other part of the forum worked.

[David Hess]

It is not more then excessive server load, logs clearly show the problem, the triplicate post would have been caused by refreshing the page and resubmitting the post data... here is how it works:

Then why was it *only* happening on the thread I had tried to reply to?  I could not read that thread, could not reply to it although it apparently was accepting the reply but not continuing, and accessing any other part of the forum worked.

It's been a while since it last happened to me but that sounds like my experience. One thread was inaccessible but others were OK. Clear the browser and it's back to normal. It's possible I'm not remembering it exactly but it sure sounds familiar.

I cleared the browser cache, reset the cookies, used a different browser, and used a different system with no change in behavior for hours.  The discussion that I replied to was returning the 502 gateway error and other discussion threads were available without problems.

Could specific discussions (1) have been causing excessive server load?

(1) I need to stop calling them discussion threads because they are not threaded.

[gnif]

Then why was it *only* happening on the thread I had tried to reply to?  I could not read that thread, could not reply to it although it apparently was accepting the reply but not continuing, and accessing any other part of the forum worked.

Because the database record was locked, simple... read up on how database row locking works.

I really don't know why people are challenging the answer to the issue, I have explained exactly what the cause is, informed why the forum behaved how it did, stated that this is all logged and clearly visible on the server, and stated exactly what has triggered the fault. I have ample evidence to explain everything here that has been reported, yet it is still being challenged... doesn't make much sense to me.

Simple fact of the matter is, a 502 error comes from the server, not your browser and they are never (I repeat, NEVER) served with headers that tell your browser to cache the error. The logs clearly show a timeout communicating with the PHP instance it had established a connection to, the PHP logs report a loss of connection to the HTTP server when it attempted to write it's reply, and the database showed queries pending completion due to lock contention. The locks reported by the database were caused by the database maintenance tasks not completing in a timely manner (and as such were retaining row locks) due to high load on the server....

The cause of the high load is yet to be identified as it is sporadic and requires further analysis.

In the last instance logged, all the PHP processes were in use waiting on the database (confirmed without a doubt) and the HTTP server had disconnected from them wrongly assuming they had failed/crashed/hung, they were all gradually consumed until there were none left, this is confirmed by 20-30 access log entries with a 499 result code.

*IP-CENSORED* - - [16/Mar/2017:10:00:47 +1100] "GET /forum/stats/?expand=201605 HTTP/1.1" 499 0 "-" "Mozilla/5.0 (compatible; SemrushBot/1.2~bl; +http://www.semrush.co
*IP-CENSORED* - - [16/Mar/2017:10:00:48 +1100] "GET /forum/stats/?expand=201309 HTTP/1.1" 499 0 "-" "Mozilla/5.0 (compatible; SemrushBot/1.2~bl; +http://www.semrush.co
*IP-CENSORED* - - [16/Mar/2017:10:00:48 +1100] "GET /forum/microcontrollers/segger-j-link-edu-or-j-link-clone-which-one-would-you-get/25/ HTTP/1.1" 499 0 "https://www.google.hu/" "Mozilla/5.0 (Windows NT 10.0; WOW64
*IP-CENSORED* - - [16/Mar/2017:10:00:48 +1100] "GET /forum/crowd-funded-projects/asap-connect-the-future-of-usb-cables/?action=dlattach;attach=233543;image;PHPSESSID=*CENSORED* HTTP/1.1" 499 0
*IP-CENSORED* - - [16/Mar/2017:10:00:49 +1100] "GET /forum/metrology/lm399-based-10-v-reference/?action=dlattach;attach=55645;image HTTP/1.1" 499 0 "https://www.eevblog.com/forum/metrology/lm399-based-10-v-referenc
*IP-CENSORED* - - [16/Mar/2017:10:00:49 +1100] "GET /forum/testgear/tektronix-2465b-oscilloscope-teardown/?action=dlattach;attach=299653;image HTTP/1.1" 499 0 "https://www.eevblog.com/forum/testgear/tektronix-2465b-
*IP-CENSORED* - - [16/Mar/2017:10:00:49 +1100] "GET /forum/stats/?expand=201107%3BPHPSESSID%3D*CENSORED* HTTP/1.1" 499 0 "-" "Mozilla/5.0 (compatible; SemrushBot/1.2~bl; +http://www.semrush.com

HTTP 499 in Nginx means that the client closed the connection before the server answered the request.

This is caused by the client (user or bot, whatever) giving up waiting for the page to load, which occurs when there is something preventing PHP returning in a timely manner. This is further proved by the following errors in the log at the same time.

2017/03/16 10:00:50 [error] 23246#0: *701749465 connect() to unix:/usr/local/php-fpm/run/eevblog.sock failed (11: Resource temporarily unavailable) while connecting to upstream...

Upon inspecting the process list at the time of the last incident there were well over 100 queries pending due to lock contention.... This is clearly not a client side cache/refresh issue, this is a server load issue, and forcing a full refresh (Ctrl+F5 or Clearing the Cache) has no effect other then putting more load on the server as you are now not just asking for the HTML for the page you are trying to view (since the content is dynamically generated by PHP here, it is never cached anyway), but making your browser download every image, script, css file even if it has a local cached copy of the static content, turning what would have been one request into tens of requests. Thankfully since this website uses CloudFlare most (not all) of this additional pointless load for static content is handled by the CF CDN.

[Seekonk]

F5 cleared it up for me, thanks for the help.  I'm still amazed that any of this stuff works.

[David Hess]

Because the database record was locked, simple... read up on how database row locking works.

I know what row locking is.

I really don't know why people are challenging the answer to the issue, I have explained exactly what the cause is, informed why the forum behaved how it did, stated that this is all logged and clearly visible on the server, and stated exactly what has triggered the fault. I have ample evidence to explain everything here that has been reported, yet it is still being challenged... doesn't make much sense to me.

I can think of a couple of reasons.  Your initial explanation was not consistent with observed symptoms.  From the user's point of view, the whole situation is very frustrating given how often it happens and in my case how often it has resulted in a long carefully researched post getting lost.  You know something is wrong if users make it a habit of copying the reply form to a text file in case the forum software eats it.

It did not happen this time but one of the error responses which I assume is from Cloudflare results in a redirection preventing backing up the browser to restore the reply form with its text.

[Monkeh]

Sounds like Dave's in for a server upgrade again.

Time for a couple SSDs?

[David Hess]

Sounds like Dave's in for a server upgrade again.

Time for a couple SSDs?

Let us hope it goes better than Drew's server upgrades.

https://beerspill.fark.com/drunkfarksysadmin.jpg

[tautech]

@gnif
Some more info in another thread:
https://www.eevblog.com/forum/chat/502-forum-errors/

Along with the info other incidences I've had could involve any individual thread and in some cases a whole board.

I get that you don't think it's browser based but the more info we can send your way the better.
When we get consistent errors we can generally work around it some...say we want to look at a particular thread in the TE board but trying to open the TE board produces errors....if we know we've posted in it we can look up our posts in our profile and access the thread that way or from New or Last Post links.

[apelly]

I really don't know why people are challenging the answer to the issue

Neither do I. I guess there's always someone thinks they're smarter.

I haven't noticed any issues serious enough to bother mentioning, so it doesn't matter to me, but thanks for taking the time to explain.

FWIW I had assumed this occasional 502 was a cloudflare issue. Even though the message specifically explains there is no response from the server, it is proudly covered in their branding.