This repository was archived by the owner on Jun 17, 2025. It is now read-only.
This repository was archived by the owner on Jun 17, 2025. It is now read-only.
[Feature] Adjustable requests limit per website #6
Description
I have ran into major IP address blocking because of the number of requests.
Recently, it happened with the fixed IP in my job and we received a question from our ISP.
Not like most people fall into this but may harm crawling even in dynamic IP addresses environments.
Maybe we should have an option to limit the number of requests per time to a specific website / domain / IP address.