Archive .org (waybackmachine) is crawling several sites on my Apache server and crashing them due to very high burst traffic.
I don't want to block their crawler, but I want to rate limit them. I have contacted their support and their answer was "No, we don't offer a crawl rate - there's nothing we can do about it".
How can I speed rate limit their IPs?
207.241.230.103
207.241.232.92
207.241.230.131
207.241.232.90
207.241.232.89
And it looks as though I am not the only one;
https://www.abuseipdb.com/check/207.241.232.90
https://www.abuseipdb.com/check/207.241.230.131
https://www.abuseipdb.com/check/207.241.230.103