FAIL2BAN - is it really problematic?

[sleemanj]

I use fail2ban on all my ec2 and lightsail servers running multiple sites on each. Configured correctly it does a fine job.

Could you suggest some fail2ban rules, which work with rotating IPs?

I doubt my rules would be that useful generally, they are made for my specific circumstances, the way my sites work, the type of users that I have and their typical browsing patterns, but with that said....

My systems all present pages to the browser as .html or trailing-slash directory requests, so I simply detect all requests for .html and trailing slash directories, excluding some known bot UAs , 302 & 301 redirects, and some special locations that I don't want to count. 

Normal people on my sites do not hit full pages in rapid succession, they certainly hit resources of those pages (css, images, js) rapidly so I don't look for those.

Code: [Select]

# Filter: gogo-apache-dos.conf
[Definition]
# Note: Match all requests for .html, / with or without query string
failregex = ^<HOST> -.+(HEAD|GET|POST|PUT|DELETE) [^"]*((\.html(\?[^"]*)?)|/) HTTP
ignoreregex = (REDACTED)|msnbot|Googlebot|bingbot|googlebot|AdsBot-Google-Mobile|facebookexternalhit|(HTTP/1.1" 302)|(HTTP/1.1" 301)

I also have a filter which looks at the fail2ban log to find reoffenders

Code: [Select]

# Filter: gogo-reoffender-fast.conf
[Definition]
failregex = NOTICE\s*\[g-apache-dos-fast\] Unban <HOST>$
ignoreregex = NOTICE \[g-reoffender.*\].*$

In my jail, I have the following configurations, for that

Code: [Select]


# People who request 8 html files in the space of 10 seconds are blocked 5m
[g-apache-dos]
enabled  = true
port     = http,https
filter   = gogo-apache-dos
name     = g-apache-dos
logpath  = /home/*/logs/access-*.log
maxretry = 8
findtime = 10
bantime  = 300
action   = iptables-multiport-log[name="g-apache-dos", port="http,https"]

# People who request 3 html files in the space of 1 second are blocked 30s
[g-apache-dos-fast]
enabled  = true
port     = http,https
filter   = gogo-apache-dos
name     = g-apache-dos-fast
logpath  = /home/*/logs/access-*.log
maxretry = 3
findtime = 1
bantime  = 30
action   = iptables-multiport-log[name="g-apache-dos-fast", port="http,https"]

# People who do a "fast" DOS 3 times in 30 minutes get blocked for an hour
[g-reoffender-fast]
enabled = true
port = all
filter = gogo-reoffender-fast
name = g-reoffender
logpath = /var/log/fail2ban.log
maxretry = 3
findtime = 1800
bantime = 3600
action   = iptables-multiport-log[name="g-reoffender-fast", port="http,https"]



[peter-h]

Normal people on my sites do not hit full pages in rapid succession

I do see lots of cases of high speed, say 10Hz, fetching of multiple pages from real users, which I assume is browser prefetch.

[sleemanj]

Conversely, I never see that on my sites.  When I made my rules I was careful to analyse and examine logs for a period of time to determine appropriate limits.

Every site is different, every user base is different, every tolerance for false positives is different.

In the last 2 days I can only see one possible DOS false positive, the rest is all bots from EC2, digital ocean, googleusercontent (apps running on google), various china based, a couple through tor, and a handful of others.

[Nominal Animal]

I hope that you have password authentication turned off on any server which opens any SSH port to the 'net.

Whenever I know the clients will use a protected key storage, like ssh-agent with passphrase-protected key files, I definitely prefer public key authentication.  Otherwise, it is a balance between the risk of keylogging (and leaking a password), or some process swiping the unprotected key files.
Notably, the browsers I currently use, do not let any JavaScript do either of those.

For allowing trusted friends temporary access to my machine, I set up a new account, and a randomly-generated but easily spoken password (for conveying via telephone or similar).

When I have to use passwords, and for the passphrases protecting my keys, I default to cryptographically secure random number generator generated ones with 96–144 bits, 16 to 24 characters (openssl rand or /dev/random source).  To store these on paper, I use large cryptos (on 5mm gridded A4 paper), with no password in a straight line in it.  Elsewhere, I have "artwork" which describes the shape the various passwords have.  Overkill, because keylogging is how they'll leak if they leak, but I like it.  (Or by someone drugging me and hitting me with a $5 wrench till I reveal the password, per XKCD #538.)

Fail2Ban is a bit too heavy to use on OpenWRT routers, so I've been thinking about writing a lightweight userspace service to maintain a ban list based on port scanning.  The idea is to reduce garbage traffic at the internet-facing edge.  There are two basic approaches: netfilter logging, and monitoring honeypot sockets for incoming connection attempts.  The former is more versatile, but the latter is simpler (if one is only interested in TCP and possibly UDP honeypot sockets).

For SMTP I get another form of "attack" - throwing email at random user names within my domain. For this one fail2ban does not really work - the system doesn't error, it accepts the mail, runs it through spam and virus filters and then ... tries to return it to the sender, completely gumming up email.

Yup.  Unless it is an authenticated user, bounces (return to sender) should be disabled.  Filtering on RCPT TO makes a lot of sense.  I don't think fail2ban can help in this.

I don't currently run my own email server; I use a hosted one, and lots of spam filters on my own mail program.

[hans]

Normal people on my sites do not hit full pages in rapid succession

I do see lots of cases of high speed, say 10Hz, fetching of multiple pages from real users, which I assume is browser prefetch.

Never heard about it.

You can prefetch DNS queries or linked files though.. like:

Code: [Select]

<link rel="dns-prefetch" href="https://www.mozilla.org" />

Code: [Select]

<link rel="prefetch" href="main.js" />

If you then have some JS at end of page or something that injects code, the browser has already fetched these endpoints/domains.

I don't think browsers will go out and request GET on new HTML bodies out of the blue. In theory REST model recommends GET requsts to be read-only, but in reality they not always are.
 You could look at the Sec headers to see how a browser has ended up a certain point. E.g. Sec-Fetch-Mode:

Code: [Select]

Sec-Fetch-Mode: cors
Sec-Fetch-Mode: navigate
Sec-Fetch-Mode: no-cors
Sec-Fetch-Mode: same-origin
Sec-Fetch-Mode: websocket
You would mostly expect 'navigate' from directly browsing HTML pages, same-origin for resources loaded on that page, and cors/no-cors/websocket for AJAX or websocket requests from JS.

Very basic Python scraping bots don't send these headers. Heck they don't even send the headers in the right order. Browsers like Chrome, Firefox, etc. have fairly static HTTP and TLS libraries, so services like Cloudflare use these hints for fingerprinting against bots.

[peter-h]

FWIW I see constant activity by Sentibot, at around 15 secs between hits. This is not a significant load. 95.216.173.102. I googled it. What a strange machine...

A colleague has also generated extra indexes which have speeded up access quite a lot, which will also help.

[Algoma]

You mentioned the site Being behind Cloudflare, They'll have their own external anonymous appearing bots monitoring, checking and caching your page on a regular interval to verify service status. Block Cloudflare's bots and your visitors will get Cloudflare status errors.

https://developers.cloudflare.com/bots/concepts/bot/

You had the Free plan: There is a basic mode to fight bots.
https://developers.cloudflare.com/bots/get-started/free/

Perhaps a temporary upgrade to Cloudflare Pro to figure out what is really happening using their tools, the back to free once you've got an idea what it really is.

[peter-h]

Yes.

We are using the hidden google recaptcha on the signup page, and AFAICT this is working 100% to block bot signups. Unfortunately manufacturing humans is a well developed activity I also check against a list of about 2k disposable email domains, which helps, but with gmail... There is also a Cloudflare browser integrity check on the signup page, so somehow together these work.

FWIW, since most signups are malicious, a few years ago I got a Ruby programmer to code a graphical captcha for the signup, which presented some numbers to solve, as a graphic image. Then a friend (who works for a penetration test company) hacked it in minutes, using standard unix OCR tools So I never deployed this, especially as I manually approve all signups anyway.

I tested Cloudflare's Under Attack mode yesterday and it puts up a big visible captcha (just a tick box, not the utterly mad "spot all the buses" one) and after that it works fine, and re-asks after some time (hours?).

I already have Bot Fight Mode set - been like that for ages. I have now blocked AI bots.

[Algoma]

China has been so heavy into AI development, with a few useful models released already (Deepseek, QWEN etc..) but they're so aggressively targeting and seeking to scrape and agglomerate any and all technical knowledge sources they can find. While I can see the value, having worked with such systems myself. but they really need to calm their bots down from being so aggressive with the collection of public data from small sites, constantly scraping entire forums just to pickup a tiny fragments of new material to feed their bots.

I can only imagine what traffic statistics changes have occurred with EEVBLOG Forum since the release of ChatGPT, when this AI push all started.

[peter-h]

For sure forums have filled up with people (humans) posting bland questions to generate LLM food. You see it heavily on reddit, facebook, probably stackexchange etc. On the one I admin, those posters don't get in to start with, plus I IP-block china, russia...

[madires]

In my experience blocking specific countries mostly helps just against script kiddies, spammers and other rookies, while also causing collateral damage. Doesn't work for more sophisticated groups using proxy services like the Mirai successors. Specifically from China and Russia I see a lot of ssh dictionary attacks, occasional spam attempts, and website scanning for known vulnerabilities. The prefixes I block completely are usually the most active bulletproof hosters and a few aggressive security companies (scanning).

[DiTBho]

Fail2Ban is a bit too heavy to use on OpenWRT routers, so I've been thinking about writing a lightweight userspace service to maintain a ban list based on port scanning.  The idea is to reduce garbage traffic at the internet-facing edge.  There are two basic approaches: netfilter logging, and monitoring honeypot sockets for incoming connection attempts.  The former is more versatile, but the latter is simpler (if one is only interested in TCP and possibly UDP honeypot sockets).

Depends on the router it's running on, but here's what I did: Userspace collects bad-ips in a list, sends them to IP tables.
Better idea: Userspace collects a list of bad-ips that somehow get past the firewall, sends them to the "firewall", dedicated machine.
and anything that knocks on the default SSH port is blocked!

[DiTBho]

S.N.O.R.T.? Anyone?

[Nominal Animal]

Fail2Ban is a bit too heavy to use on OpenWRT routers, so I've been thinking about writing a lightweight userspace service to maintain a ban list based on port scanning.  The idea is to reduce garbage traffic at the internet-facing edge.  There are two basic approaches: netfilter logging, and monitoring honeypot sockets for incoming connection attempts.  The former is more versatile, but the latter is simpler (if one is only interested in TCP and possibly UDP honeypot sockets).

Depends on the router it's running on, but here's what I did: Userspace collects bad-ips in a list, sends them to IP tables.

Yep, that's the point.  The trick is how do you collect the bad IP addresses in the first place.  Fail2Ban looks at log files.  Netfilter logging reads any addresses logged from the firewall rules.  Userspace honeypot listens to incoming connections, and immediately closes any connections.

Of these, I think I like the netfilter logging best, because then the userspace component can use an external command (shell script) to update the set of blocked IP addresses, allowing for maximum flexibility - including when the actual firewall is a separate machine.  It's not like that command will be executed very often, so the small overhead does not matter.
 
For recording the state of suspected IP addresses, I'll use a compact in-memory storage.  I don't mind that the blocked addresses are lost at reboot: it's not like this is foolproof in any way, just to reduce unwanted inbound traffic.

S.N.O.R.T.? Anyone?

I don't use Snort, simply because I do not trust Cisco at all.  In this, I am definitely paranoid; don't be like me.

The source code is at GitHub for everyone to check out and use, that's not the issue even for me.  The issue for me is the Snort rules –– the very point of using Snort in the first place! –– and how Cisco has last-word control over these.  I just cannot bring myself to trust Cisco on this.  The fact that even the Community Ruleset comes with the Cisco Ruleset License/Agreement (that however in 2.3 states that are not bound by that Agreement) gives me the heebie-jeebies.  I do not want to have any legally binding contract/agreement with Cisco.

Snort applies regular expressions over raw packet contents.  Rules are source-target address-port and content regular expression rules, that when matching, causes an alert.  Only packets matching the source-target address-port part are checked for contents.  It is very reasonable and useful approach for suspicious packet inspection; it's just that what rules will you use –– effectively, subscribe to, because you do want to keep them up to date –– and trust?  If you do not mind trusting Cisco, Snort is a very good addition to use.

[grumpydoc]

Bots and scrapers and how to defend against them being discussed on LWWN here https://lwn.net/Articles/1008897

[peter-h]

That URL is behind a paywall.

[kjpye]

It was posted on Feb 14, and will be generally available after a week. so try again on Friday or Saturday.