Server Error Reports

[gnif]

edit: I conducted some tests in another place (to not spam here). Something insane is going on. I can send the characters I did send above, but then I can’t, then I can send them again.

Sounds a bit as if the two EEVBlog hosts do not have identical configuration?

They do, it's not possible for them to not be as they are maintained by puppet.
I am sorry everyone, my health has prevented me from getting back to this yet. I know it's frustrating, hopefully ill be able to investigate this in the coming few days.

[2N3055]

Getting better should be your first priority... 

[squadchannel]

Code: [Select]

Loading [MathJax]/extensions/Safe.js
am I only one who noticed that a small message pops up in the bottom-left corner of the browser when I refresh the page?

[golden_labels]

Sounds a bit as if the two EEVBlog hosts do not have identical configuration?

Such a possibility is why I mentioned the exact IP address and certificate in another report.

But latter tests were done in a single, short session. I was interacting with only one IP address. This implies a single host unless anycast or a front load balancer is at play.

More so, I can now confirm it happens on both addresses.

They do, it's not possible for them to not be as they are maintained by puppet.

My first report was for 192.154.109.3. Just now I tested it on 192.154.111.219. Problems appear on both.

I am sorry everyone, my health has prevented me from getting back to this yet. I know it's frustrating, hopefully ill be able to investigate this in the coming few days.

In me it causes curiosity, not frustration.

Get better, take your time. I believe we can live with avoiding the @ character where it triggers the bug.

[gnif]

Sounds a bit as if the two EEVBlog hosts do not have identical configuration?

Such a possibility is why I mentioned the exact IP address and certificate in another report.

This is because the LetsEncrypt SSL certificates are automatically generated and signed per host on a 15 day cycle. The certs do not get distributed by Puppet, but the config that generates them does. Because of various things that have occurred over the years (fire at the DC/replaced server, etc), they are simply no longer in sync for the refresh period, which doesn't really matter.

But latter tests were done in a single, short session. I was interacting with only one IP address. This implies a single host unless anycast or a front load balancer is at play.

Load balancer, you will ping pong between servers no matter which IP you hit depending on server load.

More so, I can now confirm it happens on both addresses.

As expected, the public have no control over which server handles the request, by design.

Get better, take your time. I believe we can live with avoiding the @ character where it triggers the bug.

Thanks

[golden_labels]

Earlier there was a question of reproducibility. It seems that this exact sequence is a sufficient condition to trigger the bug (but not a ncessary condition):

At-sign Pound Euro (separated by spaces).

Exact codepoints:  U+0040, U+00A3, U+20AC.

It may not be the minimal one, but should be enough for debugging.

[gnif]

FYI: https://www.eevblog.com/forum/chat/scheduled-eevblog-server-maintenance/

[44kgk1lkf6u]

I often get the "you are banned from using this forum" message after the CrowdSec check passes.  Is it necessary to have both?

[gnif]

Crowdsec is absolutely required, the ban however seems to be something different. Please DM me a screenshot and your IP and I will investigate the ban itself.

In future please do not though reply to this thread unless there is actually a server error.

[jc101]

Not the forum, but the main EEVBlog website.

If you go to www.eevblog.com you get an NGINX 502 bad gateway error. Drop the www at the front and all is good, bar a huge list of cookies to untick that wasn't there the last time.

[gnif]

The cookies are ad related, dave has been making some changes.

The 502 I can't reproduce, both http servers are reporting they are healthy.

[Muttley Snickers]

Page loading is extremely slow and also multiple Bad Gateway error messages displayed, as described by other members in the newly created thread linked below. Also, an unusually high number of "guests" on the forum is also noted. 

https://www.eevblog.com/forum/news/getting-a-lot-of-502-bad-gateway-errors/

[EEVblog]

Also, an unusually high number of "guests" on the forum is also noted. 

Yes, they absolutely dominate by orders of magnitude.

[golden_labels]

I don’t know the details of this setup, but perhaps the reverse proxy could distribute traffic based on the Cookie header? Bots are unlikely to send a string “EEVblogForum=” in it, while I believe that all logged in users will. This way guest (and bot) traffic may go to a smaller subset of backend nodes, while everything else gets a relatively smooth ride.

Using string “PHPSESSID=” instead may work too, but I guess agentic LLMs nowadays will simulate that part pretty well. But if the offending guests are not sending it, it’s one more way to send them to tarpit without affecting normal users beyond the first request.

[gnif]

I don’t know the details of this setup, but perhaps the reverse proxy could distribute traffic based on the Cookie header? Bots are unlikely to send a string “EEVblogForum=” in it, while I believe that all logged in users will. This way guest (and bot) traffic may go to a smaller subset of backend nodes, while everything else gets a relatively smooth ride.

Using string “PHPSESSID=” instead may work too, but I guess agentic LLMs nowadays will simulate that part pretty well. But if the offending guests are not sending it, it’s one more way to send them to tarpit without affecting normal users beyond the first request.

We already properly load balance without resorting to pinning via cookies or IP due to proper sync between servers. The problem we we are literally overloaded with requests and I am having to trawl through millions of them to determine ways to detect the sources and force them through a captcha on mass.

[Psi]

Trick the AI into giving away that it's an AI by having a button that is white on white and invisible to everyone else but looks like something the AI will always need to click as part of its web trawling process.

[gnif]

Trick the AI into giving away that it's an AI by having a button that is white on white and invisible to everyone else but looks like something the AI will always need to click as part of its web trawling process.

We already have such measures, they simply do not work anymore

[EEVblog]

The problem we we are literally overloaded with requests and I am having to trawl through millions of them to determine ways to detect the sources and force them through a captcha on mass.

At this point I'm starting to think that this is being done deliberately so that we take down the Baidu thread...

[gnif]

The problem we we are literally overloaded with requests and I am having to trawl through millions of them to determine ways to detect the sources and force them through a captcha on mass.

At this point I'm starting to think that this is being done deliberately so that we take down the Baidu thread...

Perhaps, but while they are hammering it I can use the metrics to identify their networks and blacklist them.

[Psi]

The problem we we are literally overloaded with requests and I am having to trawl through millions of them to determine ways to detect the sources and force them through a captcha on mass.

At this point I'm starting to think that this is being done deliberately so that we take down the Baidu thread...

If it's all users without a login then probably no need to delete anything, just limit some pages/threads to only users who are logged in.

hehe, limit the entire forum to logged in users only except for the cooking page. 

[gnif]

Seems I have won, server load is returning to normal and logs show no 502 errors anymore.


Edit: spoke too soon, still seeing some but not as bad as earlier

[xrunner]

Happening to me this morning. When I first logged on I saw 170k guests now inching up to 200k ... error popping up every 3 or 4 page refreshes.

[coromonadalix]

241244 Guests, 133 Users (1 Hidden)

slow connection(s)   Bad Gateway etc ...

[gnif]

Please stop reporting what I am clearly already working on

[wilfred]

The problem we we are literally overloaded with requests and I am having to trawl through millions of them to determine ways to detect the sources and force them through a captcha on mass.

FYI. It is "en masse". French for all together.

And I just got a 502 error.