[tests] update benchmark tests, config #65
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What it does
The benchmarks were showing slow results for
AsyncFileHandler, much slower compared to what they were until v0.4.0. The reason is that a couple of bugs were recently fixed in that handler, that resulted in the following configurable property, used for the benchmark tests, to not have the desired effect:org.eclipse.tracecompass.traceeventlogger.AsyncFileHandler.maxSize = 1000After the bugs were fixed,
AsyncFileHandlerstarted to correctly use "maxSize" as the size of its record buffer. However, a buffer of 1000 is not enough for the logging load generated by the benchmark tests, making the new async handler look bad.Setting "bufferSize" to value 100k makes it big enough to "absorb" the logging load of the benchmark tests well, and brings back the previous level of performance for the async handler, such that the benchmark results are comparable to those from until v0.4.0.
To detect such a problem in the future, the benchmark tests now compute the relative performance of the async file handler vs the old sync handler and fails the test if the former is not performing above a certain threshold. The threshold may need to be lowered a bit if e.g. it's not met once in a while during CI (there is variation in the results)
In the benchmark results printed on STDOUT, the relative performance of the async file handlers now displayed as an extra column in the benchmark result tables.
Also, the benchmark test results are now printed in two versions: one un-formatted in "csv" format that's suitable to import e.g. in LibreOffice Calc, and another that's formatted for human readability.
How to test
org.eclipse.tracecompass.traceeventlogger.AsyncFileHandler.maxSize, used in the benchmark tests, to1000. This should decrease the performance fornewAsyncmaking it close tooldSyncand cause the benchmark tests to fail.Follow-ups
N/A
Review checklist