EEVblog > EEVblog Specific

RSS feed is broken by rate limiter

(1/1)

ataradov:
Last night RSS feed was broken by rate limiter. I have it set to 15 minutes poll time, does not sound like a lot, yet I get "Until you rate limit your crawler, you can get lost!" response. First of all, seems pretty rude. But then - what is the maximum rate then?

I understand that this was probably introduced to combat AI crawlers or something like that, but for RSS feed, this limit seems unreasonable.

ataradov:
Well, I switched it to hourly and it did not help. So, it looks like may be a ban is by IP?

Also, this probably belongs in "News/Suggestions/Help".

ataradov:
Yeah, I think it is IP ban. I switched to 4 h update, but still no response. I tried to download a lot from my home PC, and there does not seem to be a limit. The request here would come from Amazon IP range (self hosted RSS reader), so I guess it is IP ban.

This is not great. RSS is the way I keep track of the posts.

ataradov:
Just in case anyone experiences the same issue (unlikely, I know), passing the feed though feedburner solves it, since now it is a google IP that does the request.

gnif:

--- Quote from: ataradov on March 16, 2024, 06:14:31 pm ---Last night RSS feed was broken by rate limiter. I have it set to 15 minutes poll time, does not sound like a lot, yet I get "Until you rate limit your crawler, you can get lost!" response. First of all, seems pretty rude. But then - what is the maximum rate then?

I understand that this was probably introduced to combat AI crawlers or something like that, but for RSS feed, this limit seems unreasonable.

--- End quote ---

We were seeing 20k hits per IP every hour, coming from AmazonBot. As Amazon refuse to rate limit their crawlers we had to block them.

See: https://developer.amazon.com/amazonbot

Seems we need to be more specific about the IPs we block here if people are using a feed that comes from Amazon also. I will look into this as this was not the intention here.

Navigation

[0] Message Index