Getting a lot of "502 Bad Gateway" errors

[voltsandjolts]

I'm seeing 156K guests just now.
Forum is holding up OK though.

[EEVblog]

I'm seeing 156K guests just now.
Forum is holding up OK though.

The cycle for the last few weeks has been like this:
- Forum work great until it suddenly doesn't (usually 200k-300k "guests" on the public counter, not that's it's accurate for this kinds thing, but it's what we have)
- I enable Under Attack mode in Cloudflare and it instantly works again, the Guest count starts to drop (because it' s rolling average 60 minute counter) and eventually reaches 1k-2k after an hour.
- I turn off Under Attack mode and it takes a day or two to build up to 100k+ again

and maybe a few days after that the cycle eventually repeats itself.

[Whales]

PWM the under attack mode until we get a desirable result   These bots sound asymmetrically overdamped.

[EEVblog]

And BOOM, there is is again, 502 error regular as clockwork. UA mode enabled again.

[CatalinaWOW]

Looks like you are going to have to try longer and longer intervals for the UA mode.  Or perhaps turn it on and off with a random number generator.   This kind of crap is why is hard or impossible to have nice things.  The Vandals of the electronic byways.

[EEVblog]

Back down to 3k users on the forum stats, UA mode now switched off.

[EEVblog]

Back down to 3k users on the forum stats, UA mode now switched off.

Well that lasted mess than minute. After I turned it and posted this I got a crazy slow response and 502 again. UA mode back on 

[Sorama]

Why not keeping UA always on?

What is the downside?

[Ranayna]

There is that extra Cloudflare verification that comes with UA Mode from what i have seen so far.

Though interestingly, i did not get that now. Maybe because i have run through the verification at Stackoverflow a couple of minutes ago?

[EEVblog]

Why not keeping UA always on?
What is the downside?

Excess annoying verification.

[rpt]

Personally I set the 24 hour revalidation period on the forums I manage, so the verification is far less intrusive. And in the same time I just keep the UA mode permanently on.

[coromonadalix]

you can not play cat and mouse indefinitely ?  is there a way to mitigate the "Guess" effects  by changing that access type ?   i know i know  it may be bad, but forcing a forum registration / log in   before anything else ?

[Sorama]

Why not keeping UA always on?
What is the downside?

Excess annoying verification.

Often there is a timeout parameter you can set.
Less annoying than a site going down.
Others sites/forum do that too.

[Ranayna]

i know i know  it may be bad, but forcing a forum registration / log in   before anything else ?

In my opinion, that would be an extraordinarily bad move.
Then please keep UA mode always on before doing that.

That you require a login to even see stuff is my biggest problem i have with Discord. My reddit usage has reduced significantly since they forced a login for old.reddit.com.
I will not join a forum if i cannot take a look at the community before creating an account.

[EEVblog]

you can not play cat and mouse indefinitely ?  is there a way to mitigate the "Guess" effects  by changing that access type ?   i know i know  it may be bad, but forcing a forum registration / log in   before anything else ?

Nope, this forum being open and searchable is one of its biggest benefits.

[Whales]

One of the most hilarious things that I've read during the scraper apocalypse has to be the default settings of Anubis (a popular free PoW captcha).  It only challenges clients that are claiming to be a real web browser:

Anubis decides to present a challenge using this logic:

    User-Agent contains "Mozilla"
    Request path is not in /.well-known, /robots.txt, or /favicon.ico
    Request path is not obviously an RSS feed (ends with .rss, .xml, or .atom)

This should ensure that git clients, RSS readers, and other low-harm clients can get through without issue, but high-risk clients such as browsers and AI scraper bots will get blocked.

For context: almost all web browsers have "Mozilla" in their user agent string for historical compatibility purposes.  This includes webkit (Safari) and chromium

The misbehaving scrapers are desperate to appear normal, whilst the scrapers that politely identify themselves tend to be less of an issue.

[tautech]

Why not keeping UA always on?
What is the downside?

Excess annoying verification.

No problem at all with that if it protects the forum.

The Confirm Form Resubmission OTOH is much more annoying where I always have 2 EEVblog tabs open, the 2nd for Replies to posts which more often than not wants a Confirm.....which in some cases self hides before you can click in the button. 
W10 and Chrome.

[EEVblog]

The Confirm Form Resubmission OTOH is much more annoying where I always have 2 EEVblog tabs open, the 2nd for Replies to posts which more often than not wants a Confirm.....which in some cases self hides before you can click in the button. 
W10 and Chrome.

First I have heard of this.
I have multiple forum thread open all the time, the re-verification just seems to be a timed based thing.

[tautech]

The Confirm Form Resubmission OTOH is much more annoying where I always have 2 EEVblog tabs open, the 2nd for Replies to posts which more often than not wants a Confirm.....which in some cases self hides before you can click in the button. 
W10 and Chrome.

First I have heard of this.
I have multiple forum thread open all the time, the re-verification just seems to be a timed based thing.

I get it a lot and managed to get a screenshot of it, as attached.

[Kean]

I get it a lot and managed to get a screenshot of it, as attached.

That is a browser navigation message, generally appearing when you hit back or refresh on a page which resulted from a submitted form.  I am not as active a user as you, but I don't think I've seen it here in the last month.  And if I've ever seen it, it would be after doing one of those things.  There could be a reason the CF verification is triggering those, but I can't immediately think why.

What I am seeing a lot lately is that clicking on image thumbnails causes them to not load, and sometimes the thumbnail image itself disappears.  This is understandable if the image request is caught by the CF verification check as there is no way for me to authenticate.  I think it only happens on pages which I have had open for some time, either lots to read or coming back after a distraction (i.e. needing to do real work).

[madires]

I get it a lot and managed to get a screenshot of it, as attached.

Same here. Side effect of CF and some timeout.

[KungFuJosh]

UA mode works, and a lot of sites (forums and retail or whatever) leave it on 24/7. I don't mind occasionally waiting half a second for a verification screen. It's always better than a 502 error.

[Siwastaja]

The verification screen is fine. I guess the bigger issue, long-term, is that with "under attack" mode, all bots, including well-behaving, get blocked; correct me if I'm wrong. Being Google-searchable is a big benefit; having allowed AI models to be trained with the great wealth of information has benefited everyone - and the AI providers offer that information through their AIs and own their server cost, so it's one-time, relatively small crawl cost for the site. I'm under the impression that Dave sees this information crawling for search purposes exactly a positive thing. Not everyone agrees, of course, and some sites want to prevent bots completely.

[PlainName]

so it's one-time, relatively small crawl cost for the site

Should be. Not looking like it is, though 

[KungFuJosh]

Search engines should still mostly work. Analytic tools can have issues though.