Server Error Reports

[sorin]

I'm not sure if this is the right place to post this, is not exactly a server error.

For 2 months I have been unable to access eevblog.com from my home internet.
It seems that eevblog.com has blocked (blacklisted) all my ISP IP range.
I'm in Germany, and I'm using 1&1 Versatel / IONOS as a internet provider.
I have also tried to access it using a Proton VPN, but 90% of its VPN servers are also blacklisted from eevblog.com.

This problem happens to me only with eevblog.com other websites work fine!

[gnif]

We have had to blacklist many large networks due to AI bots that have been crawling the website so aggressively and faking their identity as normal users that it's affecting general availability.

Can you please DM me the IP of your home internet and I will look into why this network was blocked, the blocking should only be targeting servers in data centres.

[sorin]

I have sent you the DM.
Maybe you are getting many false positives because if you do a query with a AI, it uses your internet connection to search the web before responding to you.

[gnif]

Your ISPs network was blocked manually, I did this some time ago as NordVPN was routing a large amount of traffic through it that was malicious. Unfortunately your consumer network was lumped in with this IP range, I have now corrected this.

I have sent you the DM.
Maybe you are getting many false positives because if you do a query with a AI, it uses your internet connection to search the web before responding to you.

Actually it does not, the AI service (ChatGPT, etc) itself fetches the resources to process, it would make no sense to fetch them and then have you send them to the AI, doubling the traffic and time to respond.

In any case, the blocking is not an automated process, I manually vet each problematic IP and verify it's owned by a datacenter before researching what IP ranges are owned by the datacenter and then adding them to the block list. This is based on public records from the RIRs.

[gnif]

Please be aware that we are performing maintenance today and errors/outages are to be expected

[gnif]

The major work has been completed.
As usual there will still be some tweaking and tuning to perform, but for now, this is it.

[EEVblog]

FYI, you might see a new captcha thingo, especially if using a VPN.

[MK14]

FYI, you might see a new captcha thingo, especially if using a VPN.

Thanks!
I'm VERY pleased to hear that.

NOT trying to off-topic/politicize this thread, but..  The UK government (unfortunately other countries are tending to do similar things as well), is increasingly implementing legislation, that means that.  Even for law abiding adults, VPN use, is getting more and more necessary (for both privacy, and in some cases, even to be ALLOWED to access the material in the first place, e.g. social media, in some countries, not the UK yet, but rumours think it is coming to the UK, sooner rather than later).  End of off-topic.

Hence, being able to use publicly available VPN's (rather than setting up private ones), is getting more and more important.  Even if the odd captcha is now needed from time to time.

I'm also encouraged to see Cloudflare mentioned, as I was worried, it wasn't going to be used here on EEVblog.

[EEVblog]

I'm also encouraged to see Cloudflare mentioned, as I was worried, it wasn't going to be used here on EEVblog.

It has been used in the past but has given problems. It's not actually being used for the DDOS reasons you might think, we are using another feature of it to implement this new anti-bot thing.

[gnif]

Yes, the intention wasn't to block VPNs but to protect the servers.
We are still not using the CloudFlare proxy service, it has proven in the past to be more trouble then it's worth when the site comes under attack, leaving us with zero recourse if CF doesn't block the malicious data that's saturating our link.

The only aspect of CloudFlare we are using is Turnstile for suspicious IPs, which is like Google's reCAPCHA, but is better from a privacy point of view. I have deployed CrowdSec which does the actual protection.

CrowdSec do have a curated list of AI bots but they are asking $500/m USD for access to it, so instead I have augmented it with my own project RackRadar which provides it with a curated list of IPs that belong to data centres which seems to be doing the trick. Our guest count has returned from the 170-200k peaks we were seeing to the 60-70k average we had before all this started.

[MK14]

Yes, the intention wasn't to block VPNs but to protect the servers.
We are still not using the CloudFlare proxy service, it has proven in the past to be more trouble then it's worth when the site comes under attack, leaving us with zero recourse if CF doesn't block the malicious data that's saturating our link.

The only aspect of CloudFlare we are using is Turnstile for suspicious IPs, which is like Google's reCAPCHA, but is better from a privacy point of view. I have deployed CrowdSec which does the actual protection.

CrowdSec do have a curated list of AI bots but they are asking $500/m USD for access to it, so instead I have augmented it with my own project RackRadar which provides it with a curated list of IPs that belong to data centres which seems to be doing the trick. Our guest count has returned from the 170-200k peaks we were seeing to the 60-70k average we had before all this started.

Thanks for the very detailed, and carefully thought out reply.

As a user here, we are relatively hidden from all the hard work (Thanks indeed!), that is keeping this server (ok two in practice, when I last heard the details), going.

This current massive "Gold Rush" AI situation, seems to be powering a lot of the current issues.

It seemed crazy before your handiwork, where there was almost a 1000 : 1 ratio, between real user page request activities and bots (presumably mostly AI related ones).

Although, as you have said before, some of those 'guests' are actually real humans.  But perhaps more like 10 : 1 or 20 : 1, rather than up to 1000 : 1.

[Analog Kid]

Just got a strange error message:

An Error Has Occurred!
Too few arguments to function HFHooks::{closure}(), 0 passed in /home/forum/public_html/Sources/HFHooks.php on line 63 and exactly 1 expected

Went away when I reloaded the page ...

[gnif]

Just got a strange error message:

An Error Has Occurred!
Too few arguments to function HFHooks::{closure}(), 0 passed in /home/forum/public_html/Sources/HFHooks.php on line 63 and exactly 1 expected

Went away when I reloaded the page ...

Yes, I am making some changes.

[gnif]

Things should be stable now, please report any issues.

I have re-developed parts of the CrowdSec nginx interface to use memcached instead of the nginx cache for captcha results and configured both servers to use the same cache unless there is a failure where it will use a backup cache.

TL;DR; repeat capture prompts shouldn't occur anymore unless you have hit the 1hr timeout.

[isometrik]

Simply reporting on-going forum access issues:

• I am still getting tons of "EEVblog Captcha" requests. Basically one per access (first login of the day, and when accessing pretty much any new discussion topic.
• I don't use a VPN nor AI to access the forum.

[EEVblog]

Same here today. Wasn't like that the other day.
I just lost a post because it Captured me after pressing Post, and it just vanished.

[EEVblog]

I just had to disable my VPN to post that last post. It no amount of Capture verification would allow me to post.

[gnif]

If it was a flat 403 Forbidden, not CrowdSec it's the new anti-bot code, I will need to adjust it a bit by the sounds of it.
Dave can you please DM me the IP you're on when it doesn't work so I can check the logs to see what's going on?

[Analog Kid]

Dang, now I feel left out.
I have not once ever gotten a captcha when using this site. Never.
I guess I must be among the super-trusted around here ...

[EEVblog]

VPN TEST
EDIT: No Capture, so seems like the fix worked.
EDIT2: US VPN

[gnif]

It should be fixed though we still need some minor tweaks to stop posts getting lost if a captcha prompt coincides with a user making a post.

[EEVblog]

It should be fixed though we still need some minor tweaks to stop posts getting lost if a captcha prompt coincides with a user making a post.

Testing another post, one just worked, Aussie VPN.
EDIT: Yep, worked 

[EEVblog]

And nope, failed again, US VPN.
Tried to post but got capture checked every time and lost the post every time.
Had to disable to VPN to post it, and this.

[gnif]

And nope, failed again, US VPN.
Tried to post but got capture checked every time and lost the post every time.
Had to disable to VPN to post it, and this.

When this happens, please use a service like https://whatismyipaddress.com/ so I can dig into the logs. Without the IP I cant investigate.

For anyone that might wonder why this is taking a fair amount tweaking to get this right, I figure it might be worth filling in some details.
There are only a few usual answers to solve the issue we have been having here.

• Use a service like CloudFlare's proxy (not an option here as mentioned prior)
• Use a turnkey WAF solution (very expensive)
• Use a self-hosted WAF solution (cheap, but additional management load)

Obviously we went with #3. The issue is we needed some features that are not available out of the box. For instance, the library used to do the checking in nginx doesn't support a clustered configuration with multiple ingress servers which is why people were getting spammed with captcha prompts.

Another issue we ran into was some VPNs and services like Apple's proxy seem to rotate the IP the user comes from and the original code didn't account for that, it only logged if an IP had passed the captcha. The instant the user's IP changed it would do the human verification again.

This took several days of development time to learn what was going on, and develop and deploy a workable solution (which is here if anyone is interested: https://github.com/gnif/lua-cs-bouncer).
The solutions were two fold:

• Add memcached support to the library
• Make cookie verification token based instead of IP based (for those that care, the cookie is called `eevblog_cs`)

Obviously this code is new and fairly untested so I am still working through the bugs to stabilise it.

[gnif]

And nope, failed again, US VPN.
Tried to post but got capture checked every time and lost the post every time.
Had to disable to VPN to post it, and this.

This should be fixed now, there was a bug in the way the cookie token was being rotated for security. I was rotating it on a negative cache hit, which can occur when a lot of assets on the page try to load at once essentially corrupting the token value and shoving the browser onto one that had not been validated.

I have fixed this by instead encoding the expire time into the cookie token and hmac signing it to prevent tampering. This way the token wont be rotated until it's due to rotate, rather then if there is a cache miss.