Differences between JtlReporter and JMeter HTML Report (Scenario Trends vs. Statistics Table)

[MoritzSalein]

I compare the summary results between JMeter HTML Report and JtlReporter and I found following differences between Statistics table and scenario trends:

1. Response Time P90
2. Network Bytes

The same differences are on the Overview panel of the test run.

Which report is incorrect?

.

[ludeknovy]

Hi @MoritzSalein!
Regarding the differences in P90 - I think JtlReporter calculates it only for successful requests. In both examples you provided, I can see there was a small number of failed requests and that could make the difference.

Regarding the differences in - this looks correct in the report detail. But please note they use different units.
If you convert ~1883KB/sec (JMeter HTML report) to Mbps, it makes ~15. https://www.convertunits.com/from/KB/s/to/Mbps
But it definitely looks off in the scenario chart.

[MoritzSalein]

Hi @ludeknovy,

I understand your explanation to the P90 value. But I ran the same scenario again later without errors and I got a similiar difference. Look at my new screenshot.

The small difference of calculating the network bytes in the report details is acceptable to me. Of course, I had a little confusion of the units.

[ludeknovy]

1. I checked the code and for the calculation I use PSQL percentile_cont function - continuous percentile: returns a value corresponding to the specified fraction in the ordering, interpolating between adjacent input items if needed. see: https://www.postgresql.org/docs/14/functions-aggregate.html

2. If I recall correctly, the unit transformation is done in UI, so I could make a setting where you could set-up the units per scenario. If that would help.

[MoritzSalein]

to 1. ok, I don't know how the JMeter HTML Report works, but it must be different. I try to ask the JMeter project how they calculate it.
to 2. yes, it will be a nice feature to set up the units in the interface.

[MoritzSalein]

@ludeknovy Perhaps the difference in the scenario trend graph could be the same issue as reported in #98

[MoritzSalein]

Thank you for your support.

[ludeknovy]

I did a quick check and it works fine for me. Could you please try to upgrade to a newer version?

[MoritzSalein]

@ludeknovy
With the latest version the dashboard shows the right value for network.
But now in the overview isn't the value of network data, there ist the value of avg. latency. Do you know why?

[ludeknovy]

I believe that's because of the settings - navigate to project settings and there you can select which metrics get displayed.

[MoritzSalein]

Thank you, you're right. I have a different project setting in my new installation. I looked in the test run settings and found nothing to set it. Therefore I asked you.
5 stars for you support.