Gather statistics on HTTP rate limit errors

[janbuchar]

This is necessary for apify/crawlee-python#60 to bring benefit to running crawlee on Apify

Rate limit errors should have status code 429 (please recheck)

[vdusek]

@Mantisus

[B4nan]

This is handled on client level, not SDK. Client is making requests, and saving stats on the rate limits (yes, its about the 429 status codes). Crawlee then reads this from the client.

https://github.com/apify/apify-client-js/blob/master/src/statistics.ts#L18
https://github.com/apify/crawlee/blob/master/packages/core/src/autoscaling/snapshotter.ts#L383

The only thing that SDK is doing here is switching the storage client to the apify client on the platform, which I know you have a bit differently in the python versions.

[janbuchar]

In Python SDK, we wrap the client instead of using it directly as a StorageManager. Also, the Python API client does not seem to collect those statistics (please correct me if I'm wrong).

[B4nan]

Also, the Python API client does not seem to collect those statistics (please correct me if I'm wrong).

Maybe, but that doesn't mean it shouldn't be implemented there, right? SDK is not making requests, the client is.

[janbuchar]

Also, the Python API client does not seem to collect those statistics (please correct me if I'm wrong).

Maybe, but that doesn't mean it shouldn't be implemented there, right? SDK is not making requests, the client is.

Eh, the design philosophy of the javascript counterpart seems to suggest that, even though I'm not terribly happy about it. Having the client handle errors, retries and so on makes it kinda accumulate responsibilites that one (me at least) might not expect there.

Anyways, regardless of whether we implement rate limit statistics tracking here or in the client, we'll need to make modifications to the ApifyStorageClient which resides in the SDK codebase.

[B4nan]

Having the client handle errors, retries and so on makes it kinda accumulate responsibilites that one (me at least) might not expect there.

But it's designed that way already, we cant just change everything now. Python client already handles retries, right? So it feels natural to track this there. I also don't like the idea of having things implemented differently here and in the JS version, it will make things harder to reason about, to understand, to support, to transfer knowledge and whatnot.