feat(insights): Add fixed limits to concurrent report and source processing when uploading #179
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Add fixed limits to concurrent report and source processing when uploading.
Presently, the theoretical infinite number of goroutines can be spawned when the client is processing and uploading reports. Where
r
is the number of reports across all targeted sources ands
is the number of targeted sources, there is a maximum limit ofr+s
goroutines that may be spawned, with r routines potentially attempting to send out HTTP requests.There is a 150 report limit per source on how many reports are allowed to be on the disk. Typically, a collection call will remove old reports if there are too many already on the disk, limiting the impact of the go routine spawns. But it is possible for abuse to result in a considerable number of sources and reports that the uploader will treat as valid. Should the HTTP requests block or be stuck in retry loops, this can be problematic.
This PR also includes a test to ensure that the limits are being respected.
There is also a small fix to ensure that the root directory isn't being processed when traversing a directory to search for reports. This change doesn't cause any behavioral changes other than preventing a bit of unnecessary log spam.
UDENG-7470