improv: shuffle indexablePages

[tomasz-stefaniak]

What did I change?

Shuffle indexablePages to prevent using API quota on checking the same pages over and over again

Why did I change it?

I want my quota to be used on checking new pages instead on rechecking the same pages

[goenning]

that shouldn't be necessary as there's a 14 days cache, so if a page is was checked a few minutes ago and the next page failed due to API Quota, the next run should start from the latest failed page. Is that not happening?

[tomasz-stefaniak]

It's not, at least not in my case. It might be that the cache isn't working as expected then. I haven't reviewed the code very thoroughly but it looks like the cache is used to get the indexing status of all pages. I think any page with a non-indexed status gets an indexing request, whether it's present in the cache or not.

I'm not the only one running into this issue: https://x.com/charlestehio/status/1795505617341419532

[antoinekm]

it's linked to another issue that needs to be investigated, I don't think this is a good idea to shuffle the array, it doesn't fix the initial problem.

[kolyasya]

I have the same problem that the script goes through the same array of pages over and over again.

[goenning]

I think this PR fixed this, no? #62

[kolyasya]

I think this PR fixed this, no? #62

I'll double check. Meanwhile I was using this fork which shuffles pages.