[Perf] Linux/x64: 4 Regressions on 12/30/2022 6:15:13 AM

[performanceautofiler]

Run Information

Architecture	x64
OS	ubuntu 18.04
Baseline	96b849ecfc8c535eafd01e0ae82c8a54488aacfb
Compare	c71ad8cfdd0065395b01b3af6c06a0214f04c249
Diff	Diff

Regressions in System.Memory.ReadOnlySpan

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
IndexOfString - Duration of single invocation	437.59 ns	514.54 ns	1.18	0.12	False
IndexOfString - Duration of single invocation	289.04 ns	348.50 ns	1.21	0.11	False
IndexOfString - Duration of single invocation	320.20 ns	445.41 ns	1.39	0.13	False
IndexOfString - Duration of single invocation	231.69 ns	279.19 ns	1.21	0.17	False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Memory.ReadOnlySpan*'

Payloads

Baseline
Compare

Histogram

System.Memory.ReadOnlySpan.IndexOfString(input: "foobardzsdzs", value: "rddzs", comparisonType: InvariantCulture)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 514.5388159226967 > 452.18107559005864.
IsChangePoint: Marked as a change because one of 10/11/2022 6:23:40 AM, 10/22/2022 11:54:51 PM, 12/30/2022 6:06:02 AM, 1/3/2023 6:29:42 AM falls between 12/23/2022 9:40:21 PM and 1/3/2023 6:29:42 AM.
IsRegressionStdDev: Marked as regression because -20.355749884721657 (T) = (0 -514.7970980871519) / Math.Sqrt((60.70143357557084 / (33)) + (132.0208005668129 / (10))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (10) - 2, .025) and -0.1811320467917317 = (435.8505888359202 - 514.7970980871519) / 435.8505888359202 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Memory.ReadOnlySpan.IndexOfString(input: "AAAAA5AAAA", value: "5", comparisonType: InvariantCulture)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 348.5012465477906 > 304.0939800708053.
IsChangePoint: Marked as a change because one of 10/11/2022 6:23:40 AM, 12/18/2022 12:43:31 PM, 12/30/2022 6:06:02 AM, 1/3/2023 6:29:42 AM falls between 12/23/2022 9:40:21 PM and 1/3/2023 6:29:42 AM.
IsRegressionStdDev: Marked as regression because -17.23444724225909 (T) = (0 -334.9141477296196) / Math.Sqrt((29.76659841787175 / (33)) + (50.46295192405744 / (10))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (10) - 2, .025) and -0.14351702320542387 = (292.880771281229 - 334.9141477296196) / 292.880771281229 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Memory.ReadOnlySpan.IndexOfString(input: "string1", value: "string2", comparisonType: InvariantCulture)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 445.4095278610983 > 338.8064296772343.
IsChangePoint: Marked as a change because one of 10/22/2022 11:54:51 PM, 12/30/2022 6:06:02 AM, 1/3/2023 6:29:42 AM falls between 12/23/2022 9:40:21 PM and 1/3/2023 6:29:42 AM.
IsRegressionStdDev: Marked as regression because -27.353568489859335 (T) = (0 -432.9844174022045) / Math.Sqrt((39.320636739028686 / (33)) + (140.72479757709223 / (10))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (10) - 2, .025) and -0.327699618565273 = (326.1162474913507 - 432.9844174022045) / 326.1162474913507 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Memory.ReadOnlySpan.IndexOfString(input: "ABCDE", value: "c", comparisonType: InvariantCultureIgnoreCase)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 279.1868331193049 > 246.48819031441124.
IsChangePoint: Marked as a change because one of 10/17/2022 12:47:34 PM, 12/30/2022 6:06:02 AM, 1/3/2023 6:29:42 AM falls between 12/23/2022 9:40:21 PM and 1/3/2023 6:29:42 AM.
IsRegressionStdDev: Marked as regression because -16.802923377085133 (T) = (0 -282.12044769546003) / Math.Sqrt((41.222897278141836 / (33)) + (51.495536715894346 / (10))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (10) - 2, .025) and -0.17738445278311885 = (239.61625026437162 - 282.12044769546003) / 239.61625026437162 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

[kotlarmilos]

@MihaZupan it could be related to dotnet/runtime@5b8ebea

[MihaZupan]

Yep, this is dotnet/runtime#79787.

This looks like the same scenario as dotnet/runtime#78980 (comment) - the new approach uses Vector128 vectorization.
Does the mono interpreter still report true for things like Sse2.IsSupported?

[kotlarmilos]

We can look back when intrinsics are implemented then.

Does the mono interpreter still report true for things like Sse2.IsSupported?

To my knowledge no intrinsics are implemented yet in mono interpreter. @BrzVlad please confirm.