Getting a lot of "502 Bad Gateway" errors

[HwAoRrDk]

The main problem is mobile devices which rotate the IP addresses every few seconds, one moment you have legitimate traffic from a single IP address of then the next few seconds a flood of crap, and then you get rotated back to a genuine user again.

That sounds like CGNAT - Carrier-Grade Network Address Translation. Tens or hundreds of thousands of users all accessing through a relatively small pool of IP addresses. Mobile networks typically use CGNAT, as do an increasing number of residential and business ISPs.

[exxos]

That sounds like CGNAT - Carrier-Grade Network Address Translation. Tens or hundreds of thousands of users all accessing through a relatively small pool of IP addresses. Mobile networks typically use CGNAT, as do an increasing number of residential and business ISPs.

Yes. Starlink is the worst one. Has such a large IP range all over the place it's crazy.

[Monkeh]

I wrote probably hundreds of posts over the past few years how my home address gets blocked by them, my server struggles to download updates and such because cloudflare block them.

Why would they be blocking either? There's a story there..

[exxos]

Why would they be blocking either? There's a story there..

I have no idea. I used to report bad IPs to abuseipdb up until a few months ago, when CF blocked me some sending reports. CF also blocked me from accessing the stopforumspam services. Plus various other things I've probably forgotten about by now.

Similar as I saw when I signed up here, saw problems with hotmail etc.. Been through that as well. Micro$oft said was no reason for them to block me, I jumped through all the hoops. They couldn't tell me why my IP was getting blocked, but they kept on doing it anyway.. I just gave up in the end and blocked Microsoft related accounts from signing up at all.

Even got banned from google business listing. They said I was misleading customers. I wasn't. I hardly even had anything listed. Jumped through all the hoops, sent utility bills to prove who I was etc, still declined, appealed, they wouldn't reverse it and that was that.

Have issues accessing github from my server half the time. Makes it difficult to get updates from repos because of whatever protections are in place are constantly tripping me up. I just end up downloading stuff from my PC, then uploading it to the server extracting it and installing stuff that way. It's just making more work all the time

Always the same with companies. Once you get on their bad side, good luck. Now imagine if any of those companies like Google that you use their login credentials to other sites , suddenly decide you are on their naughty list and they lock you out from your accounts you signed up with google account.. Or if CF decide my home IP is no longer able to access *any* site behind CF.. Or certain sites "accidentally" go down around election times... I find it all very worrying to say the least.

Plus a few months back CF and canonical issues. IIRC, canonical  update servers completely went down because of floods of traffic coming from CF and CF basically told them the need to take out a subscription to stop the traffic ... I've seen floods of traffic hitting my own server from CF as well. . if CF cannot even protect their own servers, I certainly am not going to trust them to protect mine !

Jumping through all the hoops trying to sort these problems out is just a full-time job I don't want. Of course some people moaned because they couldn't use Hotmail anymore, but those people never wanted to battle it out with Microsoft themselves either. I sunk so much time into it all already i'm not doing it any more.

The whole thing just makes me incredibly frustrated. I'm just a guy in a bedroom, like pretty much anyone else with computer and electronics relating hobbies. And yet most my time is battling companies and problems that I should not be dealing with in the first place

[KungFuJosh]

I also run a phpBB forum, and I also wasted time blocking IPs and other fun rules. Cloudflare worked great for me, though I mostly left it in Under Attack mode. Yesterday I turned on Precursor, and the site has been running fine for 24 hours with UA mode off.

That said, two things:

1. Maybe having Cloudflare would help you if you can whitelist your IP. If they're blocking your IP or services to you, there's probably a reason. I assume you'll ignore this, so...

2. Cloudflare is not the only CDN that can provide protection. If you want to avoid CF, then you should look to their competition.

[exxos]

Cloudflare worked great for me, though I mostly left it in Under attack mode. Yesterday I turned on Precursor, and the site has been running fine for 24 hours with UA mode off.

Cloudflare worked well for you, which is good to hear. I’m still a bit cautious though. Is there any way to be sure no members or guests were lost during that period?

UA = user agents? If CF is relying on those, that does make me a bit uneasy. User agents are trivial to fake, so they’re not something you can really depend on. I’ve seen plenty of traffic claiming to be from Google that, after an rDNS check, clearly isn’t.

Even legitimate Google crawlers sometimes still use HTTP/1 and relatively old user agents. A lot of defences end up trying to block older browsers, which sounds sensible until you realise the crawlers you actually want (Google and others) are still using those older identifiers. Then you end up with an increasingly complicated set of exceptions. It can get messy fast.On the testing side, 24 hours might not be long enough to be conclusive.

I used to block so many IPs that I was running out of server RAM. I’d leave them blocked for a month, unban them, and the floods would just come back. After a few cycles of that over a couple of years, the overhead of constantly managing the bans was costing more than just letting the traffic through.

That said, two things:Maybe having Cloudflare would help you if you can whitelist your IP. If they're blocking your IP or services to you, there's probably a reason. I assume you'll ignore this, so...

I get the suggestion, but “maybe” is the sticking point for me. It might work sometimes and might not. At that point it just becomes another external system I have to keep tweaking. If I’m going to put that kind of effort in, I’d rather do it on my own box where I have full control.

2. Cloudflare is not the only CDN that can provide protection. If you want to avoid CF, then you should look to their competition.

That’s true. I did look briefly at other CDNs a while back, but after enough bad experiences with third-party services I’ve become reluctant to go looking for them unless I really have to. There’s always something that ends up going wrong. Given that they’re already blocking both my home IP and my server IP on some sites (and this has been going on for a while), I’m not confident other providers would necessarily be better. Sorting through them all to find one that works, only to potentially have to start over later when something changes, is a job I’m not keen to take on again.

The Microsoft situation that dragged on for two years before I finally gave up is exactly the kind of thing I’m trying to avoid.What works for one person doesn’t always work for another. If Cloudflare is working well for you and others, that’s genuinely great. I’m happy for you.

[PlainName]

Just now I am getting the CF verification thing but the forum is slow. Perhaps someone has forgotten to put gas in CF today and it is running on empty.

[Whales]

TBH I also hope this forum can eventually move off CF, if only because I don't trust the business model of anti-DDOS companies.  If I were to use what happens with Minecraft servers as a microcosm of the rest of the digital world: DDOS companies are financially incentivised to make sure the DDOS problems are getting worse.

I basically did a copy of every thread on the forum to a nginx cache which is held in RAM. And around 80% of legitimate traffic now hit the cache. It may sound simple but believe me it wasn't , as I was fighting various issues with the forum every step of the way.

I wonder how many GB of pages this would be on the EEVBlog forums   So I assume you only cached the HTML pages and a few accessory files, but not things like attachments?

[EEVblog]

I'm thinking about just going on the basic paid Cloudflare plan, as if I ever need support on the free plan I don't have it.
I don't think it gives me any extra stuff though.

[exxos]

TBH I also hope this forum can eventually move off CF, if only because I don't trust the business model of anti-DDOS companies.  If I were to use what happens with Minecraft servers as a microcosm of the rest of the digital world: DDOS companies are financially incentivised to make sure the DDOS problems are getting worse.

They are getting worse as time goes on. I worry that if "everyone" is behind CF, and they decide to not to the free stuff anymore, or charge £500 a year.. how many sites would they kill off overnight.. No company should have that much control.. 

I wonder how many GB of pages this would be on the EEVBlog forums   So I assume you only cached the HTML pages and a few accessory files, but not things like attachments?

Correct. Man, SMF doesn't half have some verbose over PHPBB3. Looks like images are encoded into the HTML which is bad. I looked at the previous forum page HTML here, came out at 694KB vs one of PHPBB at 181KB.  Thats pretty bad considering it's just a forum with basically text like mine.

Max compression on my page, 28KB, SMF 185KB. With some mathematical guesswork , I get around 40-50GB just to break even on max compression. Normal compression (best speed vs cpu) ends up more like 180GB. Madness. So the cache would have to live on a fast nvme drive. Not as fast as RAM.. but not much choice on a forum of the size here.

It's a double issue, much larger pages taking up more resources, then even adding a cache which can speed things up a lot in RAM, would also suffer being read from disk.  But hits to cache while hitting the disk, is still not hitting the database backend. So should easily cope with x10 traffic right there.

Nginx and PHPBB can serve files right from disk, it's basically as fast as a HTML page from cache or similar speed at least IIRC. Apache I gave up with years ago.

Blocking I still do on my server, I actually block around 25% of the internet. But before anyone moans, I run a English retro computing forum. My Primary traffic is UK then USA, some parts of EU etc. We never had any members from like Africa etc. A lot of bad traffic comes from there and many other countries. So rather than wasting CPU and bandwidth on them I just block them. Likely stops 90% of the bad traffic.  I only have a 4 core 4GB box, so I have to be a bit kind to it   No use paying x10 more just to cope with just feeding bots which give nothing back.

Yeah CF I am beyond tired of clicking their boxes every site I visit these days. Seems worse on this forum for some reason as well. I tried to get A.I. to write a "auto clicker" for me, it basically refused ! I said but i'm sick of clicking those boxes.. A.I. said , still not doing it ! 

[exxos]

I'm thinking about just going on the basic paid Cloudflare plan, as if I ever need support on the free plan I don't have it.
I don't think it gives me any extra stuff though.

If you can figure out where your "user base" is mostly located, then you could probably just IP block you way out of trouble. But you have to do it at a high up network level in the firewall before the webserver even sees the requests.

I block a lot of the middle-east etc as we never saw a member from there in the several years my forum has been up. I am nowhere near as big as EEV forum, but if your aiming for mostly English audiences, then you could block 25% of the internet and not really loose any traffic. That's how bad the bots are now and what we are all up against.

Problem is, AFAIK, is CF routes traffic to your own server via CF own servers. So every request comes from CF. At that point, no blocking tools will ever help as there would be no way to figure out where the traffic is really coming from in the first place.

I've had bad bots from CF's own servers hitting me! Imagine trying to figure out bad vs good CF traffic, it be impossible Plus like I said earlier, CF blocks me from all sorts of services already. If I moved to CF, I would just be trading one set of problems for another set.. just not worth the hassle. Plus are CF blocking actual members already ? It be impossible to tell.

But like I said Dave, you can at least setup a dedicated member pool of resources.  its a quick fix, not ideal, but stops annoying members at least.

[KungFuJosh]

UA = user agents?

No, I meant Under Attack [Mode]. It's been a day and a half now with Precursor instead, and no issues so far.

[bson]

I'm thinking about just going on the basic paid Cloudflare plan, as if I ever need support on the free plan I don't have it.
I don't think it gives me any extra stuff though.

It gives you finer granularity, like for example you could rate limit login attempts from a specific IP address independent of the rest of the site.  If you have ssh access then you can also similarly rate limit port 22, and let CF block it rather than relying on sshd's lockout - which also prevents YOU from logging in when tripped globally.

[EEVblog]

I'm thinking about just going on the basic paid Cloudflare plan, as if I ever need support on the free plan I don't have it.
I don't think it gives me any extra stuff though.

It gives you finer granularity, like for example you could rate limit login attempts from a specific IP address independent of the rest of the site.  If you have ssh access then you can also similarly rate limit port 22, and let CF block it rather than relying on sshd's lockout - which also prevents YOU from logging in when tripped globally.

I'm now on the Cloudflare Pro plan.

Under Attack mode is still active, I'm giving a few days before trying turning it back off again.

[m k]

I had earlier a double CF verification where first one probably failed, a red something blinked in place of a circling thing before new verification started.
No interaction needed, usually that way.

[EEVblog]

I'm now on the Cloudflare Pro plan.

Seems like that was good idea, I now have Precusor mode that everyone raves about.
So I have no enabled Precursor and disabled Under Attack mode, let's see what happens now.

[EEVblog]

And I have two options, current on the default minimise friction.

[tautech]

From ~350 guests on the TE board to 20+K in under an hour ! 

[KungFuJosh]

The guest count doesn't really matter. Only the real server load matters.

ETA: Seems like the site is running very well right now.

[EEVblog]

The guest count doesn't really matter. Only the real server load matters.

Correct.
But it is interesting the note the guest user number climbs by orders of magnitude the moment the moment I disable Under Attack mode. Precursor doesn't seem to have changed that, at least for now. I think it need time to analyse traffic and learn though?

[indeterminate]

One problem with Cloudflare  at the moment is that they are using WebGL fingerprinting  technology to try and catch the bots
many browsers now randomize the date returned from the webgl calls to fool the marketing people and there trackers.

so when the site was running maximum protection my main desktop pc could not access it
The Cloudflare Turnstile fingerprinter was throwing an error on suspect data being returned from the WebGL calls , everything else passed.

[KungFuJosh]

The guest count doesn't really matter. Only the real server load matters.

Correct.
But it is interesting the note the guest user number climbs by orders of magnitude the moment the moment I disable Under Attack mode. Precursor doesn't seem to have changed that, at least for now. I think it need time to analyse traffic and learn though?

Could be a learning thing, but it could also be that one Precursor mode will effect that more than the other. You'll see the data update on your security dashboard either way.

I'd ignore the guest count. My forum shows 936 guest accounts logged in the last 5 minutes, and the live CPU load is 0%.

[golden_labels]

The guest count is what is more formally known as a proxy variable. It matters little by itself, but is a convenient indicator.

But this also obscures the situation to us, mere mortals who can’t see server stats directly. And this makes some the advice I see dispensed and debates I see running, both stand on uncertain foundations. I mentioned solutions other sites use, so the options are not missed. I don’t try to make strong arguments for them, because I’m in no position to compare this site’s situation with what happened to other services. A solution that worked for one site is not necessarily applicable here. I myself can spot potential trouble right away and I trust Gnif has even better insight into this.

[EEVblog]

That sounds like CGNAT - Carrier-Grade Network Address Translation. Tens or hundreds of thousands of users all accessing through a relatively small pool of IP addresses. Mobile networks typically use CGNAT, as do an increasing number of residential and business ISPs.

Just discovered this on the weekend. Sagan was trying to set up a minecraft server and no one could access it. Turns out our home ISP was using CGNAT 

[EEVblog]

The guest count doesn't really matter. Only the real server load matters.

Correct.
But it is interesting the note the guest user number climbs by orders of magnitude the moment the moment I disable Under Attack mode. Precursor doesn't seem to have changed that, at least for now. I think it need time to analyse traffic and learn though?

Could be a learning thing, but it could also be that one Precursor mode will effect that more than the other. You'll see the data update on your security dashboard either way.
I'd ignore the guest count. My forum shows 936 guest accounts logged in the last 5 minutes, and the live CPU load is 0%.

Yesterday I saw the Guest count get to 98k and then start dropping, it's 47k this morning.
That didn't happen before I turned precursor on, it just went into the hundreds of thousands and stayed there until the 502's started.
So I think Precursor is doing something.