Getting a lot of "502 Bad Gateway" errors

[gnif]

Yes, but we have technical reasons why we need to keep it this way. Lets discuss this via email.

This is a normal issue; you set up a bypass in CF.

The real server should be firewalled to accept only CF traffic, plus obviously any other stuff you need internally.

Yes it's a normal issue, and we have protections in place. This is not something we will discuss publicly as it pertains to server security

Edit: at this time using CF directly I would consider a "trial", the last time we used it, CF failed to identify a massive attack against our infrastructure and left us with no tools to filter it. This was a few years ago, here is hoping that the situation has improved.

Edit 2: This is something that smaller site operators seem to fail to consider... if CF doesn't filter it, how do you then filter an attack coming from "trusted" IPs? You now  have to filter it at the protocol/http layer, firewalls become useless. You're now needing to handle SSL sessions, and then inspect headers to get the originating IP and then make a decision. Your ingress HTTP servers have to work far harder to block known abusive IP ranges. Enabling "Under Attack Mode" is a last resort, and really is just a stopgap workaround, not a fix.

[peter-h]

CF failed to identify a massive attack against our infrastructure and left us with no tools to filter it. This was a few years ago, here is hoping that the situation has improved.

CF is supposed to automatically block huge DDOS attacks (gigabit volume). That was the original reason for CF. But - as we found too - it does not block "small" attacks which are enough to trash your site (and ours). For those you have to use "proof of work" like Precursor or, if your server bw is enough, set up Anubis on the server.

Proof of work serving is low server load but very high client load, which is why it works.

I spent months blocking IP ranges and now have almost 1000 blocked, like e.g. all of Hetzner, Digital Ocean, etc. That is a good idea anyway (some will moan because they are on VPNs terminating on these farms). I also block Russia and China (which I know you don't want to). Historically we got a lot of attacks from some guys in Germany... hence the German server farms. But this last attack, a huge rented residential botnet, 50% US and 50% 3rd World, no chance of blocking that.

[gnif]

CF failed to identify a massive attack against our infrastructure and left us with no tools to filter it. This was a few years ago, here is hoping that the situation has improved.

CF is supposed to automatically block huge DDOS attacks (gigabit volume). That was the original reason for CF. But - as we found too - it does not block "small" attacks which are enough to trash your site (and ours). For those you have to use "proof of work" like Precursor or, if your server bw is enough, set up Anubis on the server.

Proof of work serving is low server load but very high client load, which is why it works.

I spent months blocking IP ranges and now have almost 1000 blocked, like e.g. all of Hetzner, Digital Ocean, etc. That is a good idea anyway (some will moan because they are on VPNs terminating on these farms). I also block Russia and China (which I know you don't want to). Historically we got a lot of attacks from some guys in Germany... hence the German server farms. But this last attack, a huge rented residential botnet, 50% US and 50% 3rd World, no chance of blocking that.

Yes, we have a solution for that already but it was not designed to work with CF. I can adjust it if we need to and put it back in place. Again though the specifics of the implementation are not going to be shared for operational security reasons.

Our block lists for DC owned subnets span hundreds of thousands of IPs, and are automatically generated based on external sources of information daily (Attached non-exhaustive sample). We do not block them, but rather force a proof of work.

[peter-h]

What is "DC owned"?

It sounds like you may not need CF except

- it should still block *huge* DDOS attacks
- it gives you a free https certificate with reliable renewal
- it does cache probably 1/2 your data volume (but probably that is not an issue these days)

I fully understand not talking about some things. I don't either

If CF kick us off Precursor (sounds like we got in under the wire) then I will put Anubis on our server. A friend who knows it says it's easy.

IP range blocking is no more than icing on the cake. It blocks mostly individuals with a grudge (all forums get those). But it is probably good practice. OTOH I pissed off some people on our forum by accidentally blocking their IP. It does tend to block a lot of targeted attacks; a bigger issue for you since your code is open source.

[gnif]

DC = Data center. These are IP ranges that are in use by data centres, cloud providers, etc. Ie, Digital Ocean, OVH, Hertzner, HostPapa, to name a few.

Our ISP already blocks huge DDOS style attacks, we don't need CF for this.
Cert renewal is not an issue, we already have a robust free solution.
Caching doesn't much help us as the pages SMF serve are generally non-cachable (it's from a pre-caching era, dynamic content on every page load)

We had successfully filtered everything we needed to until a few weeks ago when these proxy attacks started. The traffic comes from people that have installed free games/apps on their phones or browser extensions that allow their PC to become part of a proxy bot net. The bots then proxy it via their device, a single request every few days, but there are enough installations out there that it causes a flood of incoming requests that can't be identified from real users. It's not a per-ip flood, there is no pattern to search for, and the requests are coming from regular user's networks (DSL/Cellular, etc).

[EEVblog]

We had successfully filtered everything we needed to until a few weeks ago when these proxy attacks started. The traffic comes from people that have installed free games/apps on their phones or browser extensions that allow their PC to become part of a proxy bot net. The bots then proxy it via their device, a single request every few days, but there are enough installations out there that it causes a flood of incoming requests that can't be identified from real users. It's not a per-ip flood, there is no pattern to search for, and the requests are coming from regular user's networks (DSL/Cellular, etc).

I'm curious to know why Cloudflare fixed it in this case?
Was it a case of "we scared them away for now" with the Under Attack mode and forcing the authentication?

[EEVblog]

This option sounds like fun 

[gnif]

under attack never scared them away, it likely just caused them to move onto different servers in their target list until they noticed we were available again.

What I think has confused matters here is your view of the guest count, because this was turned on before I was ready for it, the servers (including SMF) saw all traffic originating from the CloudFlare proxy IPs, not the actual end user's IP, causing the guests online count to be very low as it logs per IP.

[EEVblog]

under attack never scared them away, it likely just caused them to move onto different servers in their target list until they noticed we were available again.

That's what I figured.

What I think has confused matters here is your view of the guest count, because this was turned on before I was ready for it, the servers (including SMF) saw all traffic originating from the CloudFlare proxy IPs, not the actual end user's IP, causing the guests online count to be very low as it logs per IP.

All I know is that the site went from not working at all to instantly working once I switched cloudflare online, even before I switched Under Attack mode on. I'm not really concerned about the SMF Guest count provided that it doesn't impact the performance significantly or bring down the site like it just did, it's just a curiosity really.

[gnif]

under attack never scared them away, it likely just caused them to move onto different servers in their target list until they noticed we were available again.

That's what I figured.

What I think has confused matters here is your view of the guest count, because this was turned on before I was ready for it, the servers (including SMF) saw all traffic originating from the CloudFlare proxy IPs, not the actual end user's IP, causing the guests online count to be very low as it logs per IP.

All I know is that the site went from not working at all to instantly working once I switched cloudflare online, even before I switched Under Attack mode on. I'm not really concerned about the SMF Guest count provided that it doesn't impact the performance significantly or bring down the site like it just did, it's just a curiosity really.

The guest count can no longer bring the site down, I re-developed this the last time it did and implemented safeguards.

[madires]

I've stumbled across a new problem. Since this morning I got "Database Error" when trying to access the forum. After a bit of testing I found out that it's related to the AAAA DNS records (IPv6) for www.eevblog.com. After forcing IPv4 by adding a local A record (IPv4) I'm able the access the forum again. If possible it would great if Dave or gnif could remove the AAAA records. My guess is that other users with IPv6 have the same issue.

[KungFuJosh]

Because we have changed to using CF entirely I have had to disable/remove the CrowdSec layer we had implemented, these to systems are not compatible.
Unfortunately the result is many bots that I was already properly filtering is now relying on CF to filter, as a result of which I'd expect to see numbers rise again.

According to CrowdSec, they can be used together. You do need to change how some things are done though. From a CrowdSec employee on reddit here: https://www.reddit.com/r/sysadmin/comments/1f33alo/protecting_webapp_using_cloudflare_or_crowdsec/

CrowdSec and Cloudflare can work together to provide you CDN level protection and Server level protection (one caveat here is if you want to block IP's then you cant use a firewall remediation as it will only see Cloudflare as the originating IP but you can use web server remediation)

That's okay anyway, since blocking IPs directly is mostly useless with bots.

Thanks,
Josh

[gnif]

Because we have changed to using CF entirely I have had to disable/remove the CrowdSec layer we had implemented, these to systems are not compatible.
Unfortunately the result is many bots that I was already properly filtering is now relying on CF to filter, as a result of which I'd expect to see numbers rise again.

According to CrowdSec, they can be used together. You do need to change how some things are done though. From a CrowdSec employee on reddit here: https://www.reddit.com/r/sysadmin/comments/1f33alo/protecting_webapp_using_cloudflare_or_crowdsec/

CrowdSec and Cloudflare can work together to provide you CDN level protection and Server level protection (one caveat here is if you want to block IP's then you cant use a firewall remediation as it will only see Cloudflare as the originating IP but you can use web server remediation)

That's okay anyway, since blocking IPs directly is mostly useless with bots.

Thanks,
Josh

Yes, they can be used together but as it stands right now with our current configuration they are not going to and requires additional work.

I've stumbled across a new problem. Since this morning I got "Database Error" when trying to access the forum. After a bit of testing I found out that it's related to the AAAA DNS records (IPv6) for www.eevblog.com. After forcing IPv4 by adding a local A record (IPv4) I'm able the access the forum again. If possible it would great if Dave or gnif could remove the AAAA records. My guess is that other users with IPv6 have the same issue.

Regarding removal of AAAA records, this is not a viable solution, CloudFlare are proxying IPv6 traffic and when we pickup the IP it's clearly being mishandled by SMF. Prior we never saw these because the provider we use doesn't support IPv6. I am investigating a fix.

[gnif]

IPv6 issue has been fixed.

[Monkeh]

the provider we use doesn't support IPv6. I am investigating a fix.

Not in any way attacking your choice of provider, but every time I see a line like that, I don't know if I should respond with or  . It's 2026!

[Zucca]

The friction and sloppiness in the transition from ipv4 to ipv6 is hard to compare to anything else.
Maybe from querty to Dvorak keyboard?

Thanks GNIF for keeping the boat afloat, but you should not have such problem today. I agree with Monek.

[gnif]

It seems I was wrong our provider does now have IPv6 support, however now we are proxied via CF though it makes little sense to request it as it just broadens our attack surface.

[TUMEMBER]

I've stumbled across a new problem. Since this morning I got "Database Error" when trying to access the forum. After a bit of testing I found out that it's related to the AAAA DNS records (IPv6) for www.eevblog.com. After forcing IPv4 by adding a local A record (IPv4) I'm able the access the forum again. If possible it would great if Dave or gnif could remove the AAAA records. My guess is that other users with IPv6 have the same issue.

True. Yesterday was a total nightmare. I did so much clicking—I even dug up the notebook where I’d physically written down my forum password—because that "missing from database" error message was really getting my blood boiling... to put it mildly, without resorting to the Latin word *curva*.

[madires]

Thanks for fixing the IPv6 issue!

It seems I was wrong our provider does now have IPv6 support, however now we are proxied via CF though it makes little sense to request it as it just broadens our attack surface.

Would it? I'm running my servers dual-stack-ed for a long time and I see much less malicious traffic from IPv6 addresses than IPv4. And for targeted attacks it doesn't matter if they happen over v4 or v6. A nice side effect is that ACLs for IPv6 are shorter since you usually deal with short prefixes instead of a bunch of non-contiguous longer prefixes.