Integration tests with Grafana scraper

[LesnyRumcajs]

Summary

There were some issues in the past with the Prometheus metrics Forest produces. Apparently, there is no validation of whether they are correct or not. The crate we're using for Prometheus metrics has an open issue for doing some of them prometheus/client_rust#52.

For example:

Error scraping target: invalid metric type "receipt>_6_size_bytes gauge"

resolved via #6065

or #4594

the Info metric is not compliant with the default Prometheus/Grafana stack and so we should not use it (unless proven it works fine).

Those broke existing monitoring setups in the past. Unfortunately, even a small issue in the Prometheus report results in a complete halt to metrics ingestion. Given this, we should have integration tests that assert the metrics are readable.

Completion Criteria

• Introduce integration tests that would assert that scraper works correctly. Can we do it without a full-fledged observability stack? If not, https://github.com/ChainSafe/forest/tree/main/monitored-stack could be used.

Additional Links & Resources

[hanabi1224]

Hey @LesnyRumcajs would you please update the description?

[LesnyRumcajs]

We could use https://pkg.go.dev/github.com/grafana/prometheus@v2.5.0+incompatible/scrape for testing.