Getting a lot of "502 Bad Gateway" errors

[peter-h]

Yes this is all true.

You need to mod the server log code to show the real IPs, not the CF IPs. We did this on NGINX too.

Also you need to firewall your real server to accept port 80/443 traffic only from CF. This is necessary because your real server IP is known, and even if it wasn't there are websites which log every single IP for which a DNS was ever published. CF have some docs on how to set up this firewall.

You do not need to present the user with two captchas in a row!

Try to get the CF Precursor. It is the best tool by far.

[EEVblog]

It is very, very annoying that one has to confirm twice every x minutes.
This is not how other sites do it.
Back to the drawing table please !

UPDATE:

This was an emergency measure I had to implement to get the forum back operational last night, we are obviously not going to keep this.
gnif was away for a bit and is now back working on tidying it all up.

Technically, this is how it happened. A few months back we implemented CloudSec + some custom options when we first got hit with the AI bots. This is run on our dedicated forum servers, so it's "server side". It worked and we solved the problem at the time.
But this recent attack came from literally hundred of thousands of different IP's through free games and browser app extension on users computers, so essentially hundreds of thousands of people were essentially DDoSing the forum, and load was all going on the server bringing it down.
So last night I was forced to enable an additional DNS side Cloudflare protection without gnif being around to make sure that worked properly. So it worked, and guests are now down to under 1000 from 300,000, hence why the forum is now lightning quick as the load is now off the servers, but we now have two layers of conflicting protection which gnif is currently sorting out.

[tautech]

Just praised your good work then when posting got a captcha and lost my post. 

Seems there's still some work to do......

[EEVblog]

Just praised your good work then when posting got a captcha and lost my post. 
Seems there's still some work to do......

That's been happening to me for months with just the one layer of protection.
I encourage everyone to copy their post text before hitting Post for now just in case.

[EEVblog]

Try to get the CF Precursor. It is the best tool by far.

It is not available in my free plan, and only seemed to be on the massive enterprise plan and not the lower paid plans?

[endless∞]

Question:

Would it help to require all users to solve a CloudFlare captcha (like the simplest "click on the box" one) before first logging on or accessing the site?

I wouldn't mind having to do this at all if it helped reduce the bot attacks.

We are already doing this but only for targetted IP ranges. The issue with doing it globally is it breaks indexing by bots we do like, such as Google.

"not cf.client.bot" in a CF custom security rule to enforce a managed/interactive challenge can exempt Cloudflare-verified bots, which is basically all the nice ones like googlebot, while blocking the unpleasant ones. It has worked for me on a production site. YMMV.

See attached. You can also exclude any other paths/files which shouldn't be presented with a challenge, like static image folders, manifest.json, .png icons etc.

It works well. Again, YMMV.

The scrapers in Vietnam which I encountered several months ago were - in my case at least - all requesting an absurd number of invalid paths, which was exhausting PHP-FPM and so needed a 404 handler. But the CF challenge enforce via security rule mitigated this entirely.

The CF "challenge passage" can usually be extended significantly - a week or month. Otherwise (like while writing this reply!) it can fail to upload an attached image because the CF challenge has expired, which affects the user experience.

[PlainName]

Just praised your good work then when posting got a captcha and lost my post. 
Seems there's still some work to do......

That's been happening to me for months with just the one layer of protection.
I encourage everyone to copy their post text before hitting Post for now just in case.

Or install TextArea Cache, an addon for Firefox (looks like Chrome too) that automatically saves edit boxes. Lose what you're doing and just copy it back from the cache.

[kentfielddude]

I thought you weren't going to use cloud flare?

[EEVblog]

I thought you weren't going to use cloud flare?

I had no choice but to enable it, and it fixed the issue in this instance, so makes sense to keep it now given that we have been continually targeted and I don't expect that to stop.
It also allows me to switch on the Under Attack mode if/when it happens again. The other solution we've have for months is more complicated and only gnif knows how to tweak it when attacks happen.

[peter-h]

What is the objection to CF?

[Ranayna]

What is the objection to CF?

One aspect is the sheer amount of power that Cloudflare has by now.
They may not use it maliciously, sure.
But if cloudflare has a hiccup, that essentially breaks the internet. And that has happend quite a number of times already.

[peter-h]

My site has been behind CF since about 2012 when we started it, with no issues I recall.

I would recommend keeping the DNS control panel elsewhere and then if CF were to blow up, you can change the DNS to point back to our real server. This is what I have but never had to use it. CF is big and they have very good people running it 24/7 so issues get fixed because they make huge $$$ from big corporates. If your DNS is at CF and CF blows up, you won't be able to log in

I know some people don't like CF because it gets used to hide the true location of "criminal" websites, but they will reveal it to law enforcement  agencies. CF is not in the USSR

There have been big issues here in the UK, with Mumsnet getting DDOS'd by some fathers' organisations, because it hosted the fairly predictable womens' discussions about useless fathers and how to shaft your ex husband / deny him child contact etc etc. Mumsnet went behind CF then.

It could be I got Precursor on free CF because they introduced it around July 2026 and let it be free to everyone, for a short time.

The US Govt uses CF heavily, too.

There is also Anubis. ISTM that with most virtual server outfits having a huge bw (many gbps) one could run the proof of work on the server, so long as nothing reaches the application, database, etc. It is the reaching of the application that brings down the site. Only if you run a public facing server on your 10mbps home ADSL is there a bw bottleneck The proof of work is only time-consuming for the client browsers; it doesn't take up much CPU time on your server.

Also worth checking your DB structure. On our site we had issues (won't go into details) where a bot query locked a chunk of the DB and this brought down the site without getting all that much traffic. It turned out we were logging a lot of activity which belonged into the server log and not into the DB. Moving it out made the system withstand DDOS a lot better, without using Precursor. The cost of that change is that admins lose easy visibility of malicious activity by certain human users...

Still getting two captchas; the 2nd one saying

EEVblog Captcha
We have seen a lot of robot like traffic coming from your IP range, please confirm you're not a robot

which cannot be true.

[smithnerd]

What is the objection to CF?

Enable JavaScript and cookies to continue

Incompatible browser extension or network configuration

Verify if your internet or firewall settings have blocked your device from reaching “challenges.cloudflare.com”

[coromonadalix]

if CF  does stabilize  EEVBLOG   no objections  even if there is some caveats

i'm behind a vpn and cloudflare let me pass ...   YMMV ...


at some point Dave Gnif  etc .. had to do something, eevblog was barely usable  ...   

[peter-h]

CF won't block VPNs, TOR or the other stuff which can be concerning to someone running a forum, particularly in conjunction with someone attempting to sign up

[coromonadalix]

noob  here      are the "Guess"  status at fault,  being used by bots  etc ....   can it be controlled  or blocked ?   yeah i know   it seems radical ...

[Ranayna]

This is what has happened now.
There are barely any guests left, i guess there are not *that* many "silent readers" after all.

A move that i would discourage strongly though is blocking the forum for anyone who is not registered.

[peter-h]

Probably silent : registered ratio is 100:1, and that is counting real people interested in electronics. It will certainly be at least 10:1.

But this captcha system (one gets two captchas now which is obviously pointless) is discouraging everybody.

[PlainName]

But this captcha system (one gets two captchas now which is obviously pointless)

I get three of them: first cloudflare, then EEVBlog, then cloudflare again. And, starting today, that last one requires me to tick a box too!

But I'm still here

[Siwastaja]

The forum has never been this fast. The performance has always been very slightly irritating; like, page load time variation in range of 1-5 seconds, not enough to complain/nag, but still something that feels a bit off; average modern-day response time, but not better. Now with the human detection in place, seems like page loads are consistently half a second, which gives an excellent user experience. I guess this also means the reason for those 1-5 sec loads were bots all along.

At least for me the captchas (both layers) automagically clear in less than 5 seconds, which is not any worse than every page load often was in the past, and the captchas don't reappear for every page load, so even if they stay, it's a clear net improvement 

[tooki]

The forum has never been this fast. The performance has always been very slightly irritating; like, page load time variation in range of 1-5 seconds, not enough to complain/nag, but still something that feels a bit off; average modern-day response time, but not better. Now with the human detection in place, seems like page loads are consistently half a second, which gives an excellent user experience. I guess this also means the reason for those 1-5 sec loads were bots all along.

At least for me the captchas (both layers) automagically clear in less than 5 seconds, which is not any worse than every page load often was in the past, and the captchas don't reappear for every page load, so even if they stay, it's a clear net improvement 

I was going to say the exact same thing!!

The performance is so much better now!

The only issue is the detection screens potentially “eating” text one is trying to post. But perhaps there’s a solution for this, even while using the detection screens.

[Nucleonics]

The forum is very fast now, just a couple hundred guests, dead internet indeed...

/tinfoil_hat_on
All searches that i did of similar situations had people answering with "just use cloudflare", seems it is getting very hard not to use their service, pretty much everywhere i go is protected by them. They are the gatekeepers of the global internet.
/tinfoil_hat_off

Seriously though, maybe the fact that not using it made here a target for people to test their bots. Just like when you need to check the internet you just ping 8.8.8.8, if you are a shady programmer and need to test your bot/ai/whatever you just hit eevblog.com.

Thanks for the hard work keeping this forum up, like others said there are not many around anymore!

[peter-h]

The forum has never been this fast. The performance has always been very slightly irritating; like, page load time variation in range of 1-5 seconds, not enough to complain/nag, but still something that feels a bit off; average modern-day response time, but not better. Now with the human detection in place, seems like page loads are consistently half a second, which gives an excellent user experience. I guess this also means the reason for those 1-5 sec loads were bots all along.

That is exactly the modern time. >>90% of server load is random irrelevant traffic. Not even real human readers.

Some can be human. For example on the site I run somebody may have posted a report on the airport on Santorini. So, whenever anybody in the world does a google search for accommodation or weather on Santorini, they hit our site Easily 100k-1M hits a day just like that. Humans! The internet is incredibly inefficient.

We firewalled China and Russian (in CF) and actual targeted hacks (99.9% of which are dictionary attacks on root) fell by 95%. Not worth a lot but every bit helps and it was free. EEVBLOG can't firewall Russia due to the number of Russians here (actually IMHO most are pretending to be in Ukraine ).

You can do CF or you can, with considerably more server admin expertise, set up Anubis (assuming you have decent bw say 10 gigabits/sec).

Is EEVBLOG on a virtual server or on a physical machine (a "PC")? The former is expensive for a lot of storage but has very fast bw (permitting local anti- DDOS), while the latter gives unlimited storage for free but is much more easily DDOS'd.

[Analog Kid]

Question about the current anti-bot setup (two authentications, one from CloudFlare, the other from I forget who):

First of all, I don't mind this barrier. It's a small price to pay to keep the site working, even if it recurs every so often.

But why the two separate captchas (just checkbox clicks)?
Are they in series or parallel?
Does one do something that the other can't do?
Is the protection that they offer additive?

[KungFuJosh]

But this captcha system (one gets two captchas now which is obviously pointless) is discouraging everybody.

Everybody? I'm not discouraged.

It's a lot easier to use a site waiting the 1 second it takes for the verifications to pass than it is to...not access the site at all because of a 502 bad gateway error.

CF is great at what it does, and you should never rely on a private server to block all bots like that. Having to fight the bots uses the server resources including bandwidth, so even if it can block bot traffic, at some point there are no resources left for real users. If that weren't true, this thread wouldn't exist.

Using CF blocks the issue before it gets to the server. CF is also a CDN, so besides protecting from the bot attacks, it helps speed up the site across the planet.