Getting a lot of "502 Bad Gateway" errors

[EEVblog]

Cloudflare stats since I turned it on, they are still getting hammered

[EEVblog]

I am on free plan too but (UK) get it.
Looks like a regionally controlled feature.

wow, why would they do that?

[EEVblog]

Damn, this is the fastest and most responsive I think I've ever seen the forum! Page loads are near instant.

[PA0PBZ]

Damn, this is the fastest and most responsive I think I've ever seen the forum! Page loads are near instant.

I agree, I always noticed some icons in the top buttons loading slower than the page itself, but now everything looks instant.
Whatever caused this don't undo it 

[PA0PBZ]

Cloudflare stats since I turned it on, they are still getting hammered

The line is dropping fast, maybe there is a delay in the stats.

[KungFuJosh]

Damn, this is the fastest and most responsive I think I've ever seen the forum! Page loads are near instant.

Cloudflare is awesome. The initially weird behavior was probably cache related.

[EEVblog]

Cloudflare stats since I turned it on, they are still getting hammered

The line is dropping fast, maybe there is a delay in the stats.

No, that's just not a complete sample for the latest time period.

[PA0PBZ]

No, that's just not a complete sample for the latest time period.

Ah, I see. Well, the guest counter is still at 1K so something seems to be blocking it.
I have not seen a figure that low since years.

[EEVblog]

Ah, I see. Well, the guest counter is still at 1K so something seems to be blocking it.
I have not seen a figure that low since years.

Yes, Cloudflare is doing the blocking.
The other one we are using, Crowdsec runs on the server, and given that we were getting attacked by hundreds of thousands of different IP's every hour, that load went onto the server before.
Even though both are essentially doing the same authentication thing, Cloudflare runs at the DNS level and not the server box level, so the traffic now doesn't hit the server.
gnif may have also done something magical in the background in the last 2 hours I'm not aware of, but it's sleep time now so fingers crossed that this hold up until the morning.

[EEVblog]

Wow, this is so pleasant now! I can actually see users. Before it was just pages and pages of guests

[madires]

Let's see how long it stays this way. BTW, I've also seen some malicious traffic from AirTel.

[tom66]

A forum I use got hit in the same way, they ended up putting Anubis on, which helped a lot.

What I don't get is why these bot farms go insane, there seriously cannot be much value in hammering a service so often it goes to 502 unavailable... yeah I get they want content but something like a page once per second would probably be mostly unnoticeable.  Maybe they are hitting a lot of old content though, hard to say for sure unless you know what URLs they're accessing.

[n55_6mt]

Looks like even though Dave was adamant that the forum wouldn't switch to using Cloudflare again, I just saw a Cloudflare verification when I reloaded the forum this AM. And it's back to working... Either way I'm happy to be back. 

[Ranayna]

There are two verifications running from the looks of it.
Cloudflare and Crowdstrike.

What i personally find a bit annoying is that the verification repeats quite often, even though i have not closed my browser (only the EEVBlog tab), let alone restarted my PC or my router.

[madires]

What I don't get is why these bot farms go insane, there seriously cannot be much value in hammering a service so often it goes to 502 unavailable... yeah I get they want content but something like a page once per second would probably be mostly unnoticeable.  Maybe they are hitting a lot of old content though, hard to say for sure unless you know what URLs they're accessing.

If I understand that right, the DDoSing AI bots are mostly interested in the Baidu thread and not in general crawling for training data. Looks more like some kind of feedback loop issue (bots going berserk) or possibly a deliberate attack.

[peter-h]

The CF Under Attack captcha (which I have used a lot) is annoying but extremely effective in blocking attacks and bots and you name it. The server loads drops dramatically, because now you are getting only the real human users. You will probably find that - apart from the storage volume - you can now run EEVBLOG on the smallest available CPU Even with nobody running DOS attacks, some 99% of server load is irrelevant random hits.

You don't need the other captcha.

I'd investigate the cost of the Precursor mode because it is invisible and just as good as Under Attack mode. I can give you (by PM) some Precursor rules to allow in Claude, Google and you may need extra ones if you have other machines which access your server (e.g. a Paypal button). The free CF I have allows 3 Precursor rules but each of them can be a huge list of IP subnets.

[madires]

There are two verifications running from the looks of it.
Cloudflare and Crowdstrike.

What i personally find a bit annoying is that the verification repeats quite often, even though i have not closed my browser (only the EEVBlog tab), let alone restarted my PC or my router.

In this situation it's the most efficient solution to control the flood of AI bots. Otherwise we would get 502s for several days or weeks.

[peter-h]

In this situation it's the most efficient solution to control the flood of AI bots

AI bots are not the problem. I have watched their activity for ages. They scrape your site by hitting the server at a few Hz only. They tend to do a few hundred hits (Claude does 200) then pause and maybe dump cookies / change IP. So this is a low load.

So why do most forums block AI bots? Because they don't want their content ripped off and presented elsewhere. This is kind of fair enough but you are losing the battle because scraping the whole of e.g. EEVBLOG would take only a few days and that one scrape will capture the entire content going back to day 1. And this has probably already been done.

This current problem is remotely controlled infected machines which are mostly residential PCs. This is readily discoverable by looking at the traffic. We did that on our site. The hits were at a few hundred Hz to a few kHz, ~50% from the USA and the rest from the 3rd World and the whole spectrum there including every bit of Africa, S. America, Middle East (yes even Iran and Iraq). Each IP only once. All old defences do not work against this.

The CF Precursor presents a JS script which the browser has to run and - I don't know how this works - these botnets can't do that for some reason.

Clever bots will do timing of various links and will identify slow links which load up your server the most. But you can defend against that. You can't defend against a botnet of say 100k machines, each one hitting just once.

Of course the Under Attack mode also works but people hate it and will eventually abandon your forum. You do NOT want this because www forums are already on the decline. It is fine to have it on say the signup page. BTW I also check all new signups against a list of about 2k reg exps of disposable email addresses; this is a big part of the game to stop people flooding the forum with crap. But so many use gmail now that this is kind of limited. BTW, as a funny aside, guess what % of people using protonmail addresses are malicious? Answers on a postcard

[PlainName]

Of course the Under Attack mode also works but people hate it and will eventually abandon your forum.

It's quick and can do it in the time I change tab from email to the opened link, so not a biggie. Plus the forum is super-quick. I would happily put up with this human check (to which one has to do nothing at all) than adverts any day

Seriously, it's not a big enough of an issue for me to even think of abandoning the place.

[Sorama]

It is very, very annoying that one has to confirm twice every x minutes.

This is not how other sites do it.

Back to the drawing table please !

[themadhippy]

Of course the Under Attack mode also works but people hate it and will eventually abandon your forum.

Yea it takes an extra milli second or 3 to get in,but at least were getting in.I bet more would bugger off if they kept getting 502 errors every other mouse click.

[TimFox]

After a full day of 502 errors, I can cope with the extra sign-in process.

[fourfathom]

I am not troubled by the recurring "not a bot" verification, especially if the alternative is what we've been experiencing the last few days.

[djsb]

Sign in no problem here. Whatever keeps the forum working properly is fine with me. Thanks.

[endless∞]

Wow, this is so pleasant now! I can actually see users. Before it was just pages and pages of guests

Apologies if sucking eggs etc, feel free to ignore/delete, sorry.

Those IPs are Cloudflare, so presumably nginx.conf needs set_real_ip_from / real_ip_header.

In /etc/nginx/conf.d/50-cloudflare.conf or wherever for you:

Code: [Select]

set_real_ip_from 103.21.244.0/22;
set_real_ip_from 103.22.200.0/22;
set_real_ip_from 103.31.4.0/22;
set_real_ip_from 104.16.0.0/12;
set_real_ip_from 108.162.192.0/18;
set_real_ip_from 131.0.72.0/22;
set_real_ip_from 141.101.64.0/18;
set_real_ip_from 162.158.0.0/15;
set_real_ip_from 172.64.0.0/13;
set_real_ip_from 173.245.48.0/20;
set_real_ip_from 188.114.96.0/20;
set_real_ip_from 190.93.240.0/20;
set_real_ip_from 197.234.240.0/22;
set_real_ip_from 198.41.128.0/17;
set_real_ip_from 199.27.128.0/21;
set_real_ip_from 2400:cb00::/32;
set_real_ip_from 2606:4700::/32;
set_real_ip_from 2803:f800::/32;
set_real_ip_from 2405:b500::/32;
set_real_ip_from 2405:8100::/32;
set_real_ip_from 2c0f:f248::/32;
set_real_ip_from 2a06:98c0::/29;
real_ip_header CF-Connecting-IP;

That would also very likely explain the seemingly repetitive slightly odd Crowdsec messages mentioning "traffic coming from your IP range" - because it's always seeing the exact same set of source IPs which are passed through by nginx.

Excessive 404s can also sometimes cause high php-fpm use, so IIRC there's a way to prevent those from ever being sent to PHP. But not necessarily needed when CF is doing its thang.

Check php-fpm settings are suited to whatever usuable memory it has.

The two web servers also still need 80+443 firewall rules, to block non-CF traffic. This is assuming you don't have other things also proxying traffic & don't accidentally block SSH or whatever.

Usually - not always, obv - a good CF custom security rule, applying a managed challenge to almost every path bar a few, works fine. 'Under attack' mode isn't usually necessary. UA mode is just a forceful way of applying challenges, without the flexibility of doing the same via custom security rules.

Scrapers using Bright Data (aka Luminati) residential proxies & bypassing CF were my most recent foes. Sometimes they use request consistent predictable paths & can be tarpitted/similar, but on a high-traffic site I imagine they could be awkward to spot in logs.