Maybe only allow logged in users in the near future?
Then the AI, can create user(s) (as many as needed, perhaps hundreds of new ones), just to do the web-crawling, so that wouldn't necessarily work.
I thought there were already rate limits (per time period, such as hourly), per IP address, to control/limit web-crawling and similar from loading/slowing this website by too much.
Edit: Typos