Cache the repodata file

[the-maldridge]

The repodata file should be cached and the prober should issue HEAD requests to determine if it needs to retrieve the file again. This should dramatically cut down on bandwidth usage by the exporter.

[WendelHime]

Hi @the-maldridge , I'd like to implement this feature but I have some questions about the context/implementation of this project.

This project run as a distributed system or single instance? With this question I hope to understand if you mean to implement a cache using redis, rabbitmq or just store in a map or other in-memory cache lib.

Do you have a target example? And there's a target dev instance or docker image for integration test purposes?

There's any restriction to work with other third party libs? Like, can I use gorilla/mux for HEAD requests?

There's a contribution guideline to follow?

[the-maldridge]

At present the system runs as a single shard, but it would be nice to maintain the capability to run in a distributed fashion. Consider groupcache for this purpose. The targets are just mirrors. You can pull any configured target from the production prometheus instance and try with that. Why on earth would you pull in all of gorilla/mux for something that's in the stdlib? That would not be accepted.

PRs should pass gofmt, go vet, and should generally speaking lint clean with staticcheck.

[WendelHime]

At present the system runs as a single shard, but it would be nice to maintain the capability to run in a distributed fashion. Consider groupcache for this purpose. The targets are just mirrors. You can pull any configured target from the production prometheus instance and try with that.
PRs should pass gofmt, go vet, and should generally speaking lint clean with staticcheck.

Ok, thanks!

Why on earth would you pull in all of gorilla/mux for something that's in the stdlib? That would not be accepted.

Basically to split the functionalities using different HTTP methods for the same route, using different functions for each one. Do you mean to use the same function (doProbe) to do the HEAD job?

[Duncaen]

The idea is to cache the downloaded repository indexes and do a HEAD with if-changes-since or whatever to not download each index everytime /probe is called and the repoindex did not change, caching the resulting metrics is unreliable.

Meaning, the fetch function should HEAD the url first based on previous responses.

[WendelHime]

aaaaaaaaah, got it. Thank you! I'll work on it on weekend