Author Topic: Baidu is trawling the EEVblog forum (Read 1261046 times)

gnif · « **Reply #100 on:** July 22, 2026, 06:05:47 pm »

Quote from: PA0PBZ on July 22, 2026, 10:30:30 am

By the way, has the bot filter been compromised? I see 40K guests on average, right after gnif implemented the filter it was a lot less.

Nothing has been "compromised", the word your looking for here is "defeated".
It's doubtful but note we can't catch every single bot, if we get to aggressive we start causing issues for general users.

RoGeorge · « **Reply #101 on:** July 22, 2026, 06:24:48 pm »

If the bots are keep flocking on this particular topic only (and not on the rest of the forum), maybe just delete this topic entirely, and thus get rid of the bots (I guess they won't still make thousands of requests if they receive URL not found error).

fzabkar · « **Reply #102 on:** July 22, 2026, 07:17:20 pm »

Quote from: eti on March 24, 2023, 01:05:22 am

remember, we have been here many many hundreds of years before todays’ America came to be - we discovered you in 1492!

European imperialists applied the concept of "terra nullius" to any lands that weren't already inhabited by Europeans. They raped, murdered and pillaged everywhere they went. Australia, in particular, recognises the consequences of such practices in the form of native title legislation. All the British had to do was to buy the land from the aboriginal inhabitants, and it would probably have cost them a pittance. Are native Americans claiming title to Manhattan? No, because it was bought for a price that they determined, even though it may seem to be a bad deal by modern standards. Americans bought Alaska from the Russians and Louisiana from the French. Europeans, on the other hand, preferred to steal everything.

thm_w · « **Reply #103 on:** July 22, 2026, 10:11:54 pm »

Quote from: fzabkar on July 22, 2026, 07:17:20 pm

eti is a banned troll, and this forum is not about politics, don't bother responding.

Whales · « **Reply #104 on:** July 22, 2026, 11:04:46 pm »

Quote from: RoGeorge on July 22, 2026, 06:24:48 pm

If the bots are keep flocking on this particular topic only (and not on the rest of the forum), maybe just delete this topic entirely

Could that be a goal?

SiliconWizard · « **Reply #105 on:** July 22, 2026, 11:14:36 pm »

Quote from: RoGeorge on July 22, 2026, 06:24:48 pm

If the bots are keep flocking on this particular topic only (and not on the rest of the forum), maybe just delete this topic entirely, and thus get rid of the bots (I guess they won't still make thousands of requests if they receive URL not found error).

Not sure about that. If it really "attracts" bots, it's possibly a good idea to keep the thread so that admins can add more filters for those bots.
Looks like a nice bait.

EEVblog · « **Reply #106 on:** July 23, 2026, 02:13:03 am »

Quote from: Psi on July 22, 2026, 12:56:01 pm

It's probably something really stupid, like the AI finds a thread called "Baidu is trawling the EEVblog forum" just from crawling pages, then it spawns some analytical AI tasks which go looking into what "EEVBLOG forum" is, and one of those finds the same thread again. Recursive thinking AI model analysis until depth limit/time limit exceeded.

I assumed that's what it was.

hans · « **Reply #107 on:** July 23, 2026, 07:22:10 am »

Quote from: gnif on July 22, 2026, 06:05:47 pm

Quote from: PA0PBZ on July 22, 2026, 10:30:30 am
By the way, has the bot filter been compromised? I see 40K guests on average, right after gnif implemented the filter it was a lot less.

Nothing has been "compromised", the word your looking for here is "defeated".
It's doubtful but note we can't catch every single bot, if we get to aggressive we start causing issues for general users.

Is my scraping framework also a bot?

Code: [Select]

GET [url]https://www.eevblog.com/forum/[/url]
STATUS: 200 Ok
Response:
<!DOCTYPE html>
<html lang="en">
  <head>
    <title>EEVblog Captcha</title>

Just kidding. Not interested in scraping this forum, I have quite an elaborate scraping framework set up for other non-AI projects (like sifting through mind numbing news sites).
But certainly these defenses catch my crawler.

Tech stack is 1000 IP pool + C# + wreq and a whole lot of massaging & heuristics.
Knowing the depths of foolery going on in that world, good job setting these defenses up

Normally speaking, just TLS spoofing catches out a lot of websites, although ASNs are getting increasingly more sticky nowadays.

I've no idea why a bot would keep monitoring this thread though. Indeed maybe its a few keywords that it pushes certain URLs into high priority queues.
In my own projects, I limit crawling to only once half an hour with everything being incremental. Index crawlers get cut off when it sees no new links, article crawlers are set on a bounded refresh timer (cut out after a few days). Such crawlers are very efficient and certainly not just following every link blindly as most AI/organic crawlers do afaik, so one certainly wouldn't see this insane amount of traffic. I think this is where a "good bot" (if such a thing exists) falls over to a "bad bot", even if it is not breaking past logins or other forms as such.

Still 1150+ guests reading this thread

Thomas Ballantine · « **Reply #108 on:** July 23, 2026, 10:09:56 am »

Has adding 402: payment required support been considered for the scrapers? It's slowly but surely picking up adoption, although there's a lot of cryptocurrency nonsense around it too. I can see it being useful for firmware developers or similar putting together a RAG to help with programming - something I can see myself doing (Is there a respectful way to pull a few dozen pages at present?). It's conceivably worth a few cents a page.

PlainName · « **Reply #109 on:** July 23, 2026, 01:30:17 pm »

Why would a scraper pay any more attention to a 402 that they do to robots.txt file?

hans · « **Reply #110 on:** July 23, 2026, 04:16:11 pm »

How is a 402 different from 403? Or 429? Or 418? /s
One would need to deny serving content as is usual with the 400-range error codes. But it just challenges bad actors to find a different strategy to scrape.. Nevertheless "pay us for access" is perhaps a bit more clear than "403 Forbidden, not going to tell you why".. And 418 is a fairly obvious "we see you" kind of error.

The difficulty is always in detection. Bots aren't honest with user agents or any other behaviour, like TCP/IP fingerprinting, TLS fingerprinting, HTTP headers, javascript environment, etc. They are also getting more cleverer with IPs these days.
Its pretty easy to find which IP is part of which ASN. Some ASNs reside in datacenters, while others are part of residential ISPs. Its relatively safe to block datacenter ASNs, although it may affect some businesses and other users (e.g. VPNs)
However, in residential ASNs, bot traffic is often scattered. It could be a home developer running their DIY python script with curl-cffi or some other self-hosted service.

Or it could be part of a botnet like Bright Data, which pays developers to add their middleware libraries in an application. This library starts a proxy server in the background which will tunnel requests for BrightData customers. So when applications are free, no ads.. then you can pretty much bet this is how they make money from you. LG and Samsung are cracking down on these libraries being installed with many apps on SmartTV's: https://securityboulevard.com/2026/07/lg-to-ban-residential-proxies-from-smart-tv-apps/

Its almost the perfect crime: SmartTVs are almost never unplugged, and these libraries keep running in the background once installed. Moreover its a high quality residential IP that may only be used for some Netflix, Youtube and online shopping, as thats what the average person does. In contrast, a tech savvy person may know they can resell their internet connection for money (even if its against their ISP's TOS). There are also mobile proxy farms and a few other ways people get residential IPs.
BrightData recently introduced a KYC policy instead of just blindly accepting everyone to make use of their network. I presume because their "library install base" is shrinking rapidly.

And then these AI companies are big enough fish to probably have their own way of sourcing network infrastructure for the data hunger/stealing (just like Google, Bing, Yahoo, etc. can afford to run a search engine).

EEVblog · « **Reply #111 on:** July 24, 2026, 12:40:24 am »

Quote from: hans on July 23, 2026, 07:22:10 am

Still 1150+ guests reading this thread

And today!

PlainName · « **Reply #112 on:** July 24, 2026, 10:48:01 am »

Can't you embed paying adverts that look like content? Could buy gnif half a glass of mineral water.

EEVblog · « **Reply #113 on:** September 10, 2026, 03:22:26 am »

humgar · « **Reply #114 on:** September 10, 2026, 03:46:09 am »

Would it be possible that, when a specific thread is getting hammered (counted within a rolling time window), set a timeout period where all requests will land on a static page with the link to its archive.org backup that the user has to manually click on (do not auto redirect).

Not sure if Wayback Machine will yell at the website who does that though. Maybe their response times are slow enough to make the bots give up.

humgar · « **Reply #115 on:** September 10, 2026, 03:48:44 am »

Quote from: EEVblog on September 10, 2026, 03:22:26 am

Andy Chee · « **Reply #116 on:** September 10, 2026, 05:46:21 am »

Quote from: EEVblog on September 10, 2026, 03:22:26 am

With the advent of generative AI, I would think you could do better than a stock image!

humgar · « **Reply #117 on:** September 10, 2026, 05:52:19 am »

Quote from: Andy Chee on September 10, 2026, 05:46:21 am

With the advent of generative AI, I would think you could do better than a stock image!

Obummer! :rofl:

tszaboo · « **Reply #118 on:** September 10, 2026, 10:45:04 am »

Quote from: EEVblog on July 24, 2026, 12:40:24 am

Quote from: hans on July 23, 2026, 07:22:10 am
Still 1150+ guests reading this thread

And today!

I would consider blacklisting the entire region, until they stop doing it.

PA0PBZ · « **Reply #119 on:** September 10, 2026, 10:59:06 am »

493 guests in this topic, 337298 guests in total. I don't think this topic is the problem

elliottveares · « **Reply #120 on:** September 10, 2026, 11:12:17 am »

Can't something like CloudFlare or a similar 3rd party service protect the forum from DDOS attacks and other malicious activity, although I expect that costs money and may not be financially viable?

elliottveares · « **Reply #121 on:** September 10, 2026, 11:13:58 am »

Quote from: tszaboo on September 10, 2026, 10:45:04 am

Quote from: EEVblog on July 24, 2026, 12:40:24 am
Quote from: hans on July 23, 2026, 07:22:10 am
Still 1150+ guests reading this thread

And today!
I would consider blacklisting the entire region, until they stop doing it.

Blocking the region may be effective, but may also not be if they end up using something like Tor or another proxy service to get around the GeoIP block.

madires · « **Reply #122 on:** September 10, 2026, 12:21:20 pm »

Quote from: tszaboo on September 10, 2026, 10:45:04 am

I would consider blacklisting the entire region, until they stop doing it.

Could work only temporarily as many AI crawlers moved to using proxy services, mostly involuntary residential proxies (installed a malicious app).

Quote from: elliottveares on September 10, 2026, 11:12:17 am

Can't something like CloudFlare or a similar 3rd party service protect the forum from DDOS attacks and other malicious activity, although I expect that costs money and may not be financially viable?

CloudFlare added filtering of AI bots/agents to their services. And they offer (possibly still beta) something called pay-per-crawl.

ftg · « **Reply #123 on:** September 10, 2026, 02:59:37 pm »

I guess the recent OpenAI KiCAD PCB work did not help.
And now all other vendors also want that sweet sweet eevblog forum wisdom in their training sets.

SiliconWizard · « **Reply #124 on:** September 10, 2026, 04:06:40 pm »

As this particular thread is what appears to have been targeted in priority, it looks more like it may be from political reasons than AI training, but certainly, AI training is another matter that causes issues as well.


EEVblog® Main Site	EEVblog® on Youtube	EEVblog® on Twitter	EEVblog® on Facebook	EEVblog® on Odysee

Author Topic: Baidu is trawling the EEVblog forum (Read 1261046 times)

Share me