Server Error Reports

[EEVblog]

Tempting...
 
https://zadzmo.org/code/nepenthes/

[EEVblog]

Count was just reset.

EDIT: And a few minutes later it's back up to 21,000 guests.

[TimFox]

And now it’s ok again, here.

[gnif]

I am still investigating but at this point it seems the fault of google's user-triggered fetcher (aka, google-proxy).

We were seeing massive amounts of traffic originating from it which looked like general website browsing traffic. Either google have missconfigured their proxies as open proxies and/or someone has found a way to exploit them to hide their origin.

As the google-proxy service does not affect our search rankings or accessibility by general users, I have blocked the subnet that was involved in the traffic.

[gnif]

SMF strikes again, while it's not the direct cause, it's certainly part of the problem.

SMF tracks who's online, which is fine, but it also lumps guest tracking into the same table. This is also fine except for two things.

1) It stores the current status of the user as a PHP serialised string, this includes fields like current forum and thread the user is in. It then uses `LIKE` (aka, substring) searches in this text field to determine these values. This is SLOW, and beyond ridiculous.

I have solved this by adding two columns to the table for these fields and altered SMF to populate them correctly and use them for the lookups.

2) When SMF generates the number of users online vs guest stats on the bottom of the page, it does it the worst way possible. It selects EVERY record from the table, guest or not, and if the record isn't associated with a guest, it increments a counter as it iterates over the data set. When we get totally spammed like today, the table fills up with 100k+ "guest/null" entries. This means that on every page load, 100k+ records are pulled from the database, just to count them. 

I have solved this by limiting the first query to non-guests only and then running a second `select count(*) from...` query to get the guest count.

While this doesn't solve the bot crawling issue, it should hopefully help to keep things responsive during these high load periods.

Here is the before & after implementing the change, CPU load graph:

[EEVblog]

We were seeing massive amounts of traffic originating from it which looked like general website browsing traffic. Either google have missconfigured their proxies as open proxies and/or someone has found a way to exploit them to hide their origin.

My money is on the later. They are evolving.

[tautech]

I noticed that the peak number of guests today set a new record.

Another new one today:
Most Online Ever: 191125 (Today at 08:03:35 am) NZ time

gnif is on the case...

And he did a wonderful job....forum seems back to normal. 

[gnif]

thanks mate, appreciate it.

I am working on a more robust solution to this issue, but it wont be ready for deployment for a while yet.

Seems the SMF fix I put in place is holding up

[TimFox]

Seems to be running slow again this morning in Chicago.

[squadchannel]

me too. from japan

[gnif]

Yup, working on it

[TimFox]

Seems OK now in Chicago.

[gnif]

Yeah, it's not region based. The site keeps coming under massive traffic loads from hundreds of thousands of addresses.

Thankfully I saw the writing on the wall a few months back and have been slowly getting all the data access agreements into place to develop this:
https://github.com/gnif/RackRadar

I have just deployed it, it will require some tuning over the next few weeks but so far it's performing extremely well.
The idea is to identify entire DC owned sub nets and filter them since the forum should really never see much, if any, traffic from these networks.

I had hoped to wait a bit longer before deploying it, but well, things keep breaking

[m k]

Very few users, but over 10k guests.

Are you sure one value is not reversed?

E,
now guests are >22k.
Seems that bots are finding alternative routes.

[gnif]

Very few users, but over 10k guests.

Are you sure one value is not reversed?

E,
now guests are >22k.
Seems that bots are finding alternative routes.

Absolutely sure, there are a lot of genuine guests on this forum, always. Note the numbers though at the moment can't be trusted as I have been purging the table to keep things operational while I tune.

[gnif]

Ok, I did a ton of tuning and altered SMF again to cache the counts for 2 minutes instead of running those heavy queries on every page load.

RackRadar is in place and has been configured so far on the ip orgs that are hitting us.

It's currently configured to find net blocks belonging to the following based on a manual rule set I have defined.

Code: [Select]

AlibabaCloud
DigitalOcean
HostPapa
Amazon
GoogleCloud
Contabo
Hetzner
OVH
LogicWeb
DataCamp
velia_net
CTGServer
IQ-PL
ACE-SG #tencent
TtyPointNet
MicrosoftCloud
SEMRUSH
ZScaler
Webair
HayHost
EBOX
HSTQ
Ril

This is covering a total of 7,141 net blocks, covering a total of 293,805,946 IPv4 addresses.
Over the next few weeks I will continue to monitor, identify and add problem networks to the system.

[EEVblog]

Will it still allow the traditional search engine bots?

[gnif]

Yes, I have been very careful to not impact the bots we like.

[paulca]

Amazon as in AWS?

That'll block a lot more than bots won't it?  Like half the internet?  Anyone that uses it as a private VPN too.

[gnif]

Amazon as in AWS?

That'll block a lot more than bots won't it?  Like half the internet?  Anyone that uses it as a private VPN too.

Yes, yes and no, AWS is only a fraction of the IP space. Unfortunately they are one of the worst offenders, we hate to be blocking these networks but we are left with little choice.
That said, if we have blocked someone's private VPN, feel free to DM me and I will whitelist it.
Note though that I will not whitelist IPs that belong to VPN providers, if they were blocked, it's because we were seeing too much malicious traffic from them.

For those that are interested, you can see here at around 6AM the new system was activated

Across the day, RackRadar has rejected 2,488,102 connections

[kloetpatra]

unfortunately my ip (Hetzner) got blocked as well (connection refused)

[gnif]

As I said, DM me the IP and I will whitelist you. No need to pollute this thread with "me too" posts.

[SteveThackery]

Uh oh, here we go again.

Saturday 13th December, 15:49 GMT. It's starting to slow down again.

[TimFox]

It’s ok here, shortly after that report.

[gnif]

Sorry, that was actually me this time adjusting a few things with the new firewall deployment.

EDIT 2025-12-16: Due to a firewall configuration error the site was slow for for a while, this has been corrected and there is no need to report this.