Getting a lot of "502 Bad Gateway" errors

[EEVblog]

The only issue is the detection screens potentially “eating” text one is trying to post. But perhaps there’s a solution for this, even while using the detection screens.

For the time being just make sure you copy your post text before you hit Post, especially if it's a lengthy post.

[EEVblog]

I'll turn ofd the Cloudflare Under Attack mode and see if things have settled down. 7:45am Monday morning Sydney time.

[EEVblog]

You can do CF or you can, with considerably more server admin expertise, set up Anubis (assuming you have decent bw say 10 gigabits/sec).
Is EEVBLOG on a virtual server or on a physical machine (a "PC")? The former is expensive for a lot of storage but has very fast bw (permitting local anti- DDOS), while the latter gives unlimited storage for free but is much more easily DDOS'd.

To be very clear again, we do not intend to keep these two authentication firewalls, we only did this because for forum was under attack.
I have just switched off Cloudflare under Attack mode to see what happens.

The server is run on three dedicated bare metal boxes, two are a redundant database system.

[EEVblog]

As a reference, just before I switched off the Cloudflare Under Attack mode.

[tautech]

But this captcha system (one gets two captchas now which is obviously pointless)

I get three of them: first cloudflare, then EEVBlog, then cloudflare again. And, starting today, that last one requires me to tick a box too!

But I'm still here

Yes the check box is new today but barely inconvenient like the ongoing 502's where. 

But, returning to the PC after away from it for a bit required going through the captchas again.... 
Is this really necessary ? 

[EEVblog]

But, returning to the PC after away from it for a bit required going through the captchas again.... 
Is this really necessary ? 

The for the third time, we do not intend to keep two levels of capture. I just switched one off as a test, and gnif is working the back end so that we possibly don't need that that one any more.
The forum was under attack by hundreds of thousands of bots rendering it unusable, we had to do something temporary to stop that.

With the one capture we've had for the last few months I've lost quite a few posts I was writing if I left it too long and timed out, this isn't a new thing.

[TimFox]

I'm amused by the strong reactions to defensive moves against attacks that rendered the forum unusable by normal users.

[Muttley Snickers]

I don't know what others are seeing in regards to the Captcha mentioned but I just get a banner which flashes up on the screen for a couple of seconds then the forum landing page opens up, and much faster than it has ever been. Running Windows 10, Firefox with uBlock Origin installed. Two screen grabs below of the Captcha from yesterday, each at different times.

[golden_labels]

I'm amused by the strong reactions to defensive moves against attacks that rendered the forum unusable by normal users.

I am not.

I wholeheartedly support the sentiment they express, BUT they come at a very wrong time and fail to notice that both Dave and gnif gave a explanation.

Perhaps let us all wait until things settle down a bit.

I also remind of self-hosted solutions like Anubis (deployed by many FOSS projects, reduced bot load enormously), or whatever e.g. Codeberg.org is using.

[EEVblog]

An hour and half later, with Under Attack mode off, no increase in guests!

[EEVblog]

So it looks like for now we are back to our original server-side single layer authentication protection with Crowdsec.

[wilfred]

An hour and half later, with Under Attack mode off, no increase in guests!

Not wanting to add to the traffic on the forum speculating about why the attack was happening, I am now wondering if there was some intent behind the attack that goes beyond speculation. I don't know what details are shared by the companies providing DDOS solutions as to whether it is targeted attacks or the EEVblog just got caught up in a more widespread issue. Now that Under Attack mode is off and the issue has at least temporarily ceased I wonder if you wanted to attack the EEVblog why stop now? Just wondering. I don't know a lot about this sort of attack.

[CatalinaWOW]

This is what has happened now.
There are barely any guests left, i guess there are not *that* many "silent readers" after all.

A move that i would discourage strongly though is blocking the forum for anyone who is not registered.

I agree with not blocking non-registered users.   But there are conceptually easy approaches that would kill bot effectiveness while allowing guests and very low count users access.  Don't know how tough the implementation of these concepts would be.  Examples:  Limit of a handful of accesses per time unit for non-vetted users.  Time unit could be minutes, hours or days.   Or put a response lag of a few seconds in non vetted users responses.  There are other possibilities, that could be a bit annoying for non-vetted real users, but which would dramatically impede the bots. 

Any defense does bring an associated risk.  It forces attackers to develop stronger weapons.

[tautech]

But, returning to the PC after away from it for a bit required going through the captchas again.... 
Is this really necessary ? 

The for the third time, we do not intend to keep two levels of capture. I just switched one off as a test, and gnif is working the back end so that we possibly don't need that that one any more.
The forum was under attack by hundreds of thousands of bots rendering it unusable, we had to do something temporary to stop that.

With the one capture we've had for the last few months I've lost quite a few posts I was writing if I left it too long and timed out, this isn't a new thing.

Thanks Dave......just giving user feedback while you and gnif continue to refine things....no real criticism as it is what it is for now.

[EEVblog]

An hour and half later, with Under Attack mode off, no increase in guests!

Not wanting to add to the traffic on the forum speculating about why the attack was happening, I am now wondering if there was some intent behind the attack that goes beyond speculation. I don't know what details are shared by the companies providing DDOS solutions as to whether it is targeted attacks or the EEVblog just got caught up in a more widespread issue. Now that Under Attack mode is off and the issue has at least temporarily ceased I wonder if you wanted to attack the EEVblog why stop now? Just wondering. I don't know a lot about this sort of attack.

Likely because it's not one person deciding to attack the forum, it's AI bots (and maybe the CCP for the Baidu thread) that went rouge. When the Under Attack mode shut them down they probably marked the site as not worth trying any more.
I still have "Block AI training bots" enabled in Cloudflare (in addition to the multitude of filters gnif has set up server side), I'll disable that right now as a trial and see if the Guests increase again. 10am Monday Sydney time for reference.

[EEVblog]

Thanks Dave......just giving user feedback while you and gnif continue to refine things....no real criticism as it is what it is for now.

The site should now be back to "normal" before this latest attack. There is no longer any extra filtering, and you should not notice any difference to what it was before.

[EEVblog]

I still have "Block AI training bots" enabled in Cloudflare (in addition to the multitude of filters gnif has set up server side), I'll disable that right now as a trial and see if the Guests increase again. 10am Monday Sydney time for reference.

An hour later and no increase in Guest activity. 

[Brumby]

But this captcha system (one gets two captchas now which is obviously pointless) is discouraging everybody.

Everybody? I'm not discouraged.

It's a lot easier to use a site waiting the 1 second it takes for the verifications to pass than it is to...not access the site at all because of a 502 bad gateway error.

Completely agree.

A one-off hassle isn't really a big thing, IMHO.  Focussing on it expends more energy than dealing with it.

The forum is working - and I appreciate the efforts made to keep it working.

[Ranayna]

Thanks Dave......just giving user feedback while you and gnif continue to refine things....no real criticism as it is what it is for now.

The site should now be back to "normal" before this latest attack. There is no longer any extra filtering, and you should not notice any difference to what it was before.

I don't think i ever saw that second "EEVBlog Captcha" before the addition of Crowdstrike. I got that single captcha this morning. So there seems to be some difference to before.
But eh, while it was a bit annoying how often this happened during the weekend, it was still better than the shitty performance and the 502s.

[peter-h]

IME the attack will come back. It either retries periodically, or continues at a low level to test whether the script (the proof of work) is still there.

Ask CF why they offer Precursor free in the UK and not in Australia - if you can get anybody to reply to an email

The server is run on three dedicated bare metal boxes, two are a redundant database system.

Your bandwidth allocation will determine whether you can run Anubis or similar.

[EEVblog]

I don't think i ever saw that second "EEVBlog Captcha" before the addition of Crowdstrike. I got that single captcha this morning. So there seems to be some difference to before.
But eh, while it was a bit annoying how often this happened during the weekend, it was still better than the shitty performance and the 502s.

You only saw it if you have what it seems to be a problematic IP, e.g. VPN or you happen to be in some region or IP range that has been troublesome.
Cloudflare Under Attack mode does it for everyone every time.

[EEVblog]

It's  been 10 hours now and no issues 

[EEVblog]

IME the attack will come back. It either retries periodically, or continues at a low level to test whether the script (the proof of work) is still there.

No doubt.

Ask CF why they offer Precursor free in the UK and not in Australia - if you can get anybody to reply to an email

WTF?

[Ranayna]

I don't think i ever saw that second "EEVBlog Captcha" before the addition of Crowdstrike. I got that single captcha this morning. So there seems to be some difference to before.
But eh, while it was a bit annoying how often this happened during the weekend, it was still better than the shitty performance and the 502s.

You only saw it if you have what it seems to be a problematic IP, e.g. VPN or you happen to be in some region or IP range that has been troublesome.
Cloudflare Under Attack mode does it for everyone every time.

I am at work now. While i would not be surprised if there are more people browsing the EEVBlog here, i can't imagine it is *that* many
And the work IP has not changed for years.
I assume, since that captcha also still has the Cloudflare Logo, that this generally learns from Cloudflare and my "IP reputation" is not solely based on data regarding the EEVBlog.
Stackexchange also uses Cloudflare and there it is a lot more annoying.

I got this captcha a couple of times now, but not nearly as often as the combined captchas yesterday.
Don't get me wrong. I understand the neccessity and i don't really mind that single captcha.
But it *is* changed behaviour.

[PA0PBZ]

But it *is* changed behaviour.

I agree, I have never seen this captcha before but now I see it every time (once an hour?).
I have no problem with it, it only takes a second and requires no action but it is definitely new.