Author Topic: Getting a lot of "502 Bad Gateway" errors  (Read 19377 times)

0 Members and 75 Guests are viewing this topic.

Offline EEVblog

  • Administrator
  • *****
  • Posts: 42255
  • Country: au
    • EEVblog
Re: Getting a lot of "502 Bad Gateway" errors
« Reply #225 on: September 27, 2026, 12:26:35 am »
Search engines should still mostly work. Analytic tools can have issues though.

Yes. Ones currently identified by SMF are:
Yandex (111), Google (5), DuckDuckGo (67), Bing (16), Baidu (200)
And this is Under Attack mode.
 
The following users thanked this post: KungFuJosh

Offline EEVblog

  • Administrator
  • *****
  • Posts: 42255
  • Country: au
    • EEVblog
Re: Getting a lot of "502 Bad Gateway" errors
« Reply #226 on: September 27, 2026, 12:28:06 am »
UA mode works, and a lot of sites (forums and retail or whatever) leave it on 24/7. I don't mind occasionally waiting half a second for a verification screen. It's always better than a 502 error.

There is another supposedly better 24/7 mode called Precursor, but my account does not have it, only on the Enterprise edition of Cloudflare
 

Offline EEVblog

  • Administrator
  • *****
  • Posts: 42255
  • Country: au
    • EEVblog
Re: Getting a lot of "502 Bad Gateway" errors
« Reply #227 on: September 27, 2026, 12:31:15 am »
These are the bots in the last 24 hours in Under Attack mode according to Cloudflare.

 

Offline EEVblog

  • Administrator
  • *****
  • Posts: 42255
  • Country: au
    • EEVblog
Re: Getting a lot of "502 Bad Gateway" errors
« Reply #228 on: September 27, 2026, 12:32:00 am »
I just turned Under Attack mode off again. 10:30am Sunday Sydney
 

Offline EEVblog

  • Administrator
  • *****
  • Posts: 42255
  • Country: au
    • EEVblog
Re: Getting a lot of "502 Bad Gateway" errors
« Reply #229 on: September 27, 2026, 12:40:09 am »
I just turned Under Attack mode off again. 10:30am Sunday Sydney

So much for that. 5 minutes later after turning it off I couldn't access the forum.
UA mode back on  >:(
 
The following users thanked this post: Kean

Online KungFuJosh

  • Super Contributor
  • ***
  • Posts: 8291
  • Country: us
  • TEAS is real.
Re: Getting a lot of "502 Bad Gateway" errors
« Reply #230 on: September 27, 2026, 01:33:20 am »
UA mode works, and a lot of sites (forums and retail or whatever) leave it on 24/7. I don't mind occasionally waiting half a second for a verification screen. It's always better than a 502 error.

There is another supposedly better 24/7 mode called Precursor, but my account does not have it, only on the Enterprise edition of Cloudflare

I enabled precursor mode on the forum I run, but I didn't disable UA. I might try it without UA and see how it goes.
"Experience is something you don't get until just after you need it." - Steven Wright
Best Continuity Tester Ever
 

Offline EEVblog

  • Administrator
  • *****
  • Posts: 42255
  • Country: au
    • EEVblog
Re: Getting a lot of "502 Bad Gateway" errors
« Reply #231 on: September 27, 2026, 02:52:57 am »
I enabled precursor mode on the forum I run, but I didn't disable UA. I might try it without UA and see how it goes.

I do not have Precusor mode available, it's only on the Enterprise plan for users in Australia.
 

Online KungFuJosh

  • Super Contributor
  • ***
  • Posts: 8291
  • Country: us
  • TEAS is real.
Re: Getting a lot of "502 Bad Gateway" errors
« Reply #232 on: September 27, 2026, 04:59:48 am »
I enabled precursor mode on the forum I run, but I didn't disable UA. I might try it without UA and see how it goes.

I do not have Precusor mode available, it's only on the Enterprise plan for users in Australia.

I know. I'm not sure why, it might be a regional release thing, or maybe they're going to charge everybody eventually.

Either way, I figure I can test it out and see if it's worth anything or not.
"Experience is something you don't get until just after you need it." - Steven Wright
Best Continuity Tester Ever
 

Offline Siwastaja

  • Super Contributor
  • ***
  • Posts: 11226
  • Country: fi
Re: Getting a lot of "502 Bad Gateway" errors
« Reply #233 on: September 27, 2026, 05:27:45 am »
Quote
so it's one-time, relatively small crawl cost for the site

Should be. Not looking like it is, though  :)

Geez, your reading comprehension seems to be limited within a 10-token context window. Didn't you see "WELL-BEHAVING" in my post a few words before?

Anyhow, I was wrong about UA mode blocking every bot - from Dave's screenshots above, Under Attack mode seems to keep those well-behaving bots allowed, including the APPLE, which demonstrates the problem is the abusive, anonymous bots as we knew all along. Who would have thought.
 

Offline EEVblog

  • Administrator
  • *****
  • Posts: 42255
  • Country: au
    • EEVblog
Re: Getting a lot of "502 Bad Gateway" errors
« Reply #234 on: September 27, 2026, 06:59:08 am »
The Baidu thread continues to be one of the most popular threads. There is no other reason for this than direct government filtering.
It might not be the reason for all this, but at this point I have to try some something so I have moved that thread to the Supporters section, so anyone who's not a user will see just the login page.

https://www.eevblog.com/forum/chat/baidu-is-trawling-the-eevblog-forum/
 

Offline EEVblog

  • Administrator
  • *****
  • Posts: 42255
  • Country: au
    • EEVblog
Re: Getting a lot of "502 Bad Gateway" errors
« Reply #235 on: September 27, 2026, 08:05:42 am »
UA mode now off again.  6pm Sunday Sydney.
 

Offline EEVblog

  • Administrator
  • *****
  • Posts: 42255
  • Country: au
    • EEVblog
Re: Getting a lot of "502 Bad Gateway" errors
« Reply #236 on: September 27, 2026, 08:20:58 am »
I'm have now blocked Baidu
 

Offline PA0PBZ

  • Super Contributor
  • ***
  • Posts: 5504
  • Country: nl
Re: Getting a lot of "502 Bad Gateway" errors
« Reply #237 on: September 27, 2026, 08:28:06 am »
I'm have now blocked Baidu

Not by the "Block Crawler" switch it seems?

Can you remind me why you don't want to leave UA enabled? It seems that most of us see it as the better choise to prevent the 502 errors.
Keyboard error: Press F1 to continue.
 

Offline EEVblog

  • Administrator
  • *****
  • Posts: 42255
  • Country: au
    • EEVblog
Re: Getting a lot of "502 Bad Gateway" errors
« Reply #238 on: September 27, 2026, 08:43:16 am »
I'm have now blocked Baidu
Not by the "Block Crawler" switch it seems?
Can you remind me why you don't want to leave UA enabled? It seems that most of us see it as the better choise to prevent the 502 errors.

That screen shot was before I blocked them.
I am toggling UA mode because I want to experiment with whether or not the bots eventually go away.
I don't want to just turn it on and leave it and never think about it again. It may have impacts on how the forum is indexed by the search engines etc.
And it is going lead to more authentication which is annoying.
I want to get back to a point where UA mode is not required, I'm not just giving up.
 
The following users thanked this post: PA0PBZ, edavid, tooki

Offline PA0PBZ

  • Super Contributor
  • ***
  • Posts: 5504
  • Country: nl
Re: Getting a lot of "502 Bad Gateway" errors
« Reply #239 on: September 27, 2026, 11:08:15 am »
123K guests and the bad gateway is back.
Keyboard error: Press F1 to continue.
 

Offline TUMEMBER

  • Frequent Contributor
  • **
  • Posts: 341
  • Country: pl
Re: Getting a lot of "502 Bad Gateway" errors
« Reply #240 on: September 27, 2026, 11:13:13 am »
On the other hand, I’ve noticed a strange correlation with the "introduction of ads on the forum." Some of the ad providers are "LLM promoters." I would venture to say that they are responsible for generating a significant portion of that AI ghost traffic.

Whenever I see over 1,000 guests on Metrologia, I’m certain the forum is about to crash hard.
W życiu nie ma nic za darmo, są tylko różne formy płatności.
Wciąż płacimy okruchami własnego czasu.
 

Offline EEVblog

  • Administrator
  • *****
  • Posts: 42255
  • Country: au
    • EEVblog
Re: Getting a lot of "502 Bad Gateway" errors
« Reply #241 on: September 27, 2026, 01:12:52 pm »
123K guests and the bad gateway is back.

Yep, turning UA mode back on again, regular as clockwork.
 

Offline PA0PBZ

  • Super Contributor
  • ***
  • Posts: 5504
  • Country: nl
Re: Getting a lot of "502 Bad Gateway" errors
« Reply #242 on: September 27, 2026, 02:41:29 pm »
So it's clearly not the Baidu thread, which I already expected because even when there are lots of guests I've never seen a lot in that thread.
Do you have any idea what brings the server down? You clearly have more tools than us to look at, and since it is something that is recognized by UA mode, any statistics from that side?
Keyboard error: Press F1 to continue.
 

Online KungFuJosh

  • Super Contributor
  • ***
  • Posts: 8291
  • Country: us
  • TEAS is real.
Re: Getting a lot of "502 Bad Gateway" errors
« Reply #243 on: September 27, 2026, 02:49:20 pm »
Dave is doing the right thing experimenting with which bots & how to block them, but the process is tedious AF. He's more patient than I am, that's for sure. ;)
"Experience is something you don't get until just after you need it." - Steven Wright
Best Continuity Tester Ever
 

Offline PA0PBZ

  • Super Contributor
  • ***
  • Posts: 5504
  • Country: nl
Re: Getting a lot of "502 Bad Gateway" errors
« Reply #244 on: September 27, 2026, 03:00:00 pm »
What I just now noticed is that the total crawler requests in the screenshot Dave posted earlier roughly shows the same amount as the number of guests when UA mode is on, so it looks like the troublemaker(s) are not even making it to that statistic.
Keyboard error: Press F1 to continue.
 

Offline Zoli

  • Frequent Contributor
  • **
  • Posts: 799
  • Country: ca
  • Grumpy old men
Re: Getting a lot of "502 Bad Gateway" errors
« Reply #245 on: September 27, 2026, 03:41:20 pm »
UA mode now off again.  6pm Sunday Sydney.
Automating proposal: turning UA on/off as function of guests number(as example: @10k on, @2k off; 2 minute average).
 

Online KungFuJosh

  • Super Contributor
  • ***
  • Posts: 8291
  • Country: us
  • TEAS is real.
Re: Getting a lot of "502 Bad Gateway" errors
« Reply #246 on: September 27, 2026, 08:04:33 pm »
Automation doesn't work like that, and the guest list isn't accurate or reliable data for that purpose.

However, allegedly you can use the Cloudflare API to automatically enable/disable under attack mode from the server based on load data.
"Experience is something you don't get until just after you need it." - Steven Wright
Best Continuity Tester Ever
 

Offline EEVblog

  • Administrator
  • *****
  • Posts: 42255
  • Country: au
    • EEVblog
Re: Getting a lot of "502 Bad Gateway" errors
« Reply #247 on: September 27, 2026, 11:09:24 pm »
What I just now noticed is that the total crawler requests in the screenshot Dave posted earlier roughly shows the same amount as the number of guests when UA mode is on, so it looks like the troublemaker(s) are not even making it to that statistic.

Yep, it's one that don't identify themselves that are problem (often hundreds of thousands of them using unique IP's that look legit). But of course every request adds up, so if the forum is holding up with the volume but it's near the limit because of a log of legit bot traffic, then that can be a problem too as it's easier to tip the server over the edge.
 

Offline EEVblog

  • Administrator
  • *****
  • Posts: 42255
  • Country: au
    • EEVblog
Re: Getting a lot of "502 Bad Gateway" errors
« Reply #248 on: September 27, 2026, 11:20:22 pm »
So it's clearly not the Baidu thread, which I already expected because even when there are lots of guests I've never seen a lot in that thread.


Yes, i was just essentially testing the theory whether or not it's Winnie The Poo doing this deliberately to get that thread taken down. Seems it's not.

Quote
Do you have any idea what brings the server down? You clearly have more tools than us to look at, and since it is something that is recognized by UA mode, any statistics from that side?

Only @gnif can authoritatively answer this, but as I understand it, it's basically a database volume request thing. Every page served on the forum is a dynamic database request, and if you have thousands, or even tens of thousands of requests it can handle that. But if you suddenly get several hundred thousand requests from different IP's that all look like legit request then the server (and it's blocking software) has no option but to try and serve them all, and the volume just become too much. Hence the 502 because the server can't satisfy the request.

Before we turned on Cloudflare we had another similar custom system on the server (CrowdSec) and was able to block the Baidu and other bots successfully. But does human authentication at the server level, but it got to a point this month that it required a DNS level solution like Cloudflare to provide authentication before it even hits the server.
 

Offline exxos

  • Newbie
  • Posts: 6
  • Country: gb
    • exxxosforum
Re: Getting a lot of "502 Bad Gateway" errors
« Reply #249 on: September 28, 2026, 12:48:01 pm »
but as I understand it, it's basically a database volume request thing. Every page served on the forum is a dynamic database request, and if you have thousands, or even tens of thousands of requests it can handle that.

I've been fighting these exact problems like everyone else for the past year on my own forum.

But it turned out for me is it was not actually database hits causing the slowdowns at all, but PHP-FPM itself. I optimised everything to Hell and back on my server. Basically only brought a couple of weeks grace at best. Rinse and repeat over two years.. You end up fighting a battle you simply cannot win :(

Long story short, I did a hard IP block on basically bad traffic areas like the Middle East, where we have never once seen good traffic on the forum.

Secondly, I had to build various PHPBB3 extensions to completely rewire how PHPBB3 handles website traffic. I basically did a copy of every thread on the forum to a nginx cache which is held in RAM. And around 80% of legitimate traffic now hit the cache. It may sound simple but believe me it wasn't , as I was fighting various issues with the forum every step of the way.

Furthermore I set up a double pool for resources where "guests" get a 80% share, and members 20%. Basically what this means its members always get their own resource pool and even if the site looks to go down for guests, the members are completely unaware of that even happening as they are in their own pool. Members are then informed to always stay logged in to make sure they never lose access.

I basically spent an entire year trying to block the traffic and even wrote a roundup if anyone is interested..

https://www.exxosforum.co.uk/articles/traffic-floods-2026.html
https://www.exxosforum.co.uk/articles/server_security_2026.html

TL;DR
Trying to block the traffic, you are fighting a losing battle. Believe me I tried incredibly hard for the best part of two years doing exactly that.

The main problem is mobile devices which rotate the IP addresses every few seconds, one moment you have legitimate traffic from a single IP address of then the next few seconds a flood of crap, and then you get rotated back to a genuine user again. It's a conundrum because you have to block the bad traffic, but then everyone mobile IP becomes collateral damage and cannot access the server because the IP ranges now banned.

Cloudflare.. I'm never going to try that. I wrote probably hundreds of posts over the past few years how my home address gets blocked by them, my server struggles to download updates and such because cloudflare block them. Even external and other related services or websites end up blocked. And the kicker, I even get floods of traffic from cloudflares own network!! So my philosophy there is, if they cannot even protect their own network, how are they going to protect anyone else's ? I'm not looking to start arguments with people with this but for me personally Cloudflare has caused enough problems for me to never consider it.   Plus I would much rather have my defences hosted on my own network so if there is a problem I can solve it myself the same day not depend on the third-party company technical support which is generally non-existent anyway these days. I also got told by several "cyber security experts" to use Cloudflare. They simply could not comprehend they was blocking legitimate services which are critical infrastructure for my server.

So the answer is not to block traffic, but to basically "make it cost nothing" . Unfortunately as I found, there is no prebuilt solution for these things and you have to build your own solution from the ground up. PHPBB3 I had to write many extensions and custom nginx integrations which are all done to solve the problems. Plus you need enough RAM to hold copies of the entire forum which presents its own problems but they can at least be compressed. The solution I put together was far from simple but it's unfortunately what you have to do in 2026.

No doubt this post will make a lot of people angry, as a general rule I never post on other forums anymore because of the hatred and abuse I get over basically everything I say and do. But after continuously reading problems one of my favourite channels, motivated me to list post what works for me.. At least for the moment anyway.. I hope it gives you some food for thought anyway Dave..
« Last Edit: September 28, 2026, 12:52:00 pm by exxos »
 
The following users thanked this post: Whales, tooki


Share me

Digg  Facebook  SlashDot  Delicious  Technorati  Twitter  Google  Yahoo
Smf

 

-->