[Perf] Linux/arm64: 25 Improvements on 3/21/2023 8:47:40 PM #14662
Description
Run Information
| Name | Value |
|---|---|
| Architecture | arm64 |
| OS | ubuntu 20.04 |
| Queue | AmpereUbuntu |
| Baseline | 7b651743dc951384f6c3d8455c88518022c7c3b9 |
| Compare | b0b7aaefb88aa8d01b3d64fb40ac2f73a9d98c3e |
| Diff | Diff |
| Configs | CompilationMode:tiered, RunKind:micro |
Improvements in System.Globalization.Tests.StringSearch
| Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
|---|---|---|---|---|---|---|---|---|---|---|
| IsSuffix_SecondHalf - Duration of single invocation | 229.70 ns | 192.81 ns | 0.84 | 0.00 | True | |||||
| IsPrefix_FirstHalf - Duration of single invocation | 229.67 ns | 191.92 ns | 0.84 | 0.00 | True | |||||
| IsSuffix_SecondHalf - Duration of single invocation | 274.93 ns | 246.29 ns | 0.90 | 0.01 | False | |||||
| IsSuffix_SecondHalf - Duration of single invocation | 230.07 ns | 192.62 ns | 0.84 | 0.01 | True | |||||
| IsPrefix_FirstHalf - Duration of single invocation | 276.77 ns | 238.78 ns | 0.86 | 0.00 | True | |||||
| IsPrefix_FirstHalf - Duration of single invocation | 229.91 ns | 191.92 ns | 0.83 | 0.00 | True | |||||
| IsPrefix_FirstHalf - Duration of single invocation | 229.21 ns | 191.65 ns | 0.84 | 0.01 | True | |||||
| IsSuffix_SecondHalf - Duration of single invocation | 230.46 ns | 193.55 ns | 0.84 | 0.01 | True | |||||
| IsPrefix_FirstHalf - Duration of single invocation | 274.93 ns | 237.12 ns | 0.86 | 0.00 | True | |||||
| IsSuffix_SecondHalf - Duration of single invocation | 276.19 ns | 238.74 ns | 0.86 | 0.01 | False |
Repro
General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md
Payloads
git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Globalization.Tests.StringSearch*'Details
Payloads
Histogram
System.Globalization.Tests.StringSearch.IsSuffix_SecondHalf(Options: (en-US, IgnoreNonSpace, False))
Description of detection logic
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 192.81369071205185 < 218.44515021873295.
IsChangePoint: Marked as a change because one of 3/21/2023 2:53:21 PM, 3/30/2023 4:24:24 AM falls between 3/21/2023 2:53:21 PM and 3/30/2023 4:24:24 AM.
IsImprovementStdDev: Marked as improvement because 545.6855198423127 (T) = (0 -192.57657457832755) / Math.Sqrt((0.7563867012948795 / (299)) + (0.0651137768622719 / (30))) is greater than 1.9672451058622664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (30) - 2, .975) and 0.16266543089762286 = (229.98760792208745 - 192.57657457832755) / 229.98760792208745 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.
System.Globalization.Tests.StringSearch.IsPrefix_FirstHalf(Options: (en-US, None, False))
Description of detection logic
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 191.92378090047893 < 218.11696776992636.
IsChangePoint: Marked as a change because one of 3/21/2023 2:53:21 PM, 3/30/2023 4:24:24 AM falls between 3/21/2023 2:53:21 PM and 3/30/2023 4:24:24 AM.
IsImprovementStdDev: Marked as improvement because 568.4646905806234 (T) = (0 -191.87120794962115) / Math.Sqrt((0.2791637418715514 / (299)) + (0.10265224098076794 / (30))) is greater than 1.9672451058622664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (30) - 2, .975) and 0.16354909070893295 = (229.38729077626488 - 191.87120794962115) / 229.38729077626488 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.
System.Globalization.Tests.StringSearch.IsSuffix_SecondHalf(Options: (en-US, IgnoreCase, False))
Description of detection logic
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 246.2922994355137 < 260.6976016659852.
IsChangePoint: Marked as a change because one of 2/16/2023 5:39:51 AM, 3/21/2023 2:53:21 PM, 3/30/2023 4:24:24 AM falls between 3/21/2023 2:53:21 PM and 3/30/2023 4:24:24 AM.
IsImprovementStdDev: Marked as improvement because 13.228342927400526 (T) = (0 -237.5433970502851) / Math.Sqrt((320.4700363666071 / (299)) + (2.7974325529336577 / (30))) is greater than 1.9672451058622664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (30) - 2, .975) and 0.0567002789828229 = (251.8217611621233 - 237.5433970502851) / 251.8217611621233 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
System.Globalization.Tests.StringSearch.IsSuffix_SecondHalf(Options: (, None, False))
Description of detection logic
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 192.6216784121548 < 218.4083079904805.
IsChangePoint: Marked as a change because one of 3/21/2023 2:53:21 PM, 3/30/2023 4:24:24 AM falls between 3/21/2023 2:53:21 PM and 3/30/2023 4:24:24 AM.
IsImprovementStdDev: Marked as improvement because 470.22748395796464 (T) = (0 -192.54508505789107) / Math.Sqrt((1.7603584224818531 / (299)) + (0.015225681906550017 / (30))) is greater than 1.9672451058622664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (30) - 2, .975) and 0.16338802774393427 = (230.14861302864298 - 192.54508505789107) / 230.14861302864298 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.
System.Globalization.Tests.StringSearch.IsPrefix_FirstHalf(Options: (, IgnoreCase, False))
Description of detection logic
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 238.78266938799052 < 262.9352135950355.
IsChangePoint: Marked as a change because one of 3/21/2023 2:53:21 PM, 3/30/2023 4:24:24 AM falls between 3/21/2023 2:53:21 PM and 3/30/2023 4:24:24 AM.
IsImprovementStdDev: Marked as improvement because 297.3646308569498 (T) = (0 -238.68250391970102) / Math.Sqrt((2.4743429346364962 / (299)) + (0.22129167329001864 / (30))) is greater than 1.9672451058622664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (30) - 2, .975) and 0.13484761028690467 = (275.884927046036 - 238.68250391970102) / 275.884927046036 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.
System.Globalization.Tests.StringSearch.IsPrefix_FirstHalf(Options: (, None, False))
Description of detection logic
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 191.91652489495738 < 217.8065408450468.
IsChangePoint: Marked as a change because one of 3/21/2023 2:53:21 PM, 3/30/2023 4:24:24 AM falls between 3/21/2023 2:53:21 PM and 3/30/2023 4:24:24 AM.
IsImprovementStdDev: Marked as improvement because 627.4625113743365 (T) = (0 -191.9050721126097) / Math.Sqrt((0.9257061784682077 / (299)) + (0.015482570874402192 / (30))) is greater than 1.9672451058622664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (30) - 2, .975) and 0.16423476475258222 = (229.61600222076552 - 191.9050721126097) / 229.61600222076552 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.
System.Globalization.Tests.StringSearch.IsPrefix_FirstHalf(Options: (en-US, IgnoreNonSpace, False))
Description of detection logic
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 191.64618253645565 < 217.6996352895458.
IsChangePoint: Marked as a change because one of 3/21/2023 2:53:21 PM, 3/30/2023 4:24:24 AM falls between 3/21/2023 2:53:21 PM and 3/30/2023 4:24:24 AM.
IsImprovementStdDev: Marked as improvement because 412.57258403840837 (T) = (0 -191.9184895217218) / Math.Sqrt((1.0940115024244947 / (299)) + (0.13389515874716235 / (29))) is greater than 1.9672675222597202 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (29) - 2, .975) and 0.1635760175636326 = (229.45120363800947 - 191.9184895217218) / 229.45120363800947 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.
System.Globalization.Tests.StringSearch.IsSuffix_SecondHalf(Options: (en-US, None, False))
Description of detection logic
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 193.54517734201718 < 218.77345785075607.
IsChangePoint: Marked as a change because one of 3/21/2023 2:53:21 PM, 3/30/2023 4:24:24 AM falls between 3/21/2023 2:53:21 PM and 3/30/2023 4:24:24 AM.
IsImprovementStdDev: Marked as improvement because 565.556871824167 (T) = (0 -193.47264300203778) / Math.Sqrt((0.9169105005717608 / (299)) + (0.034512198845195324 / (30))) is greater than 1.9672451058622664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (30) - 2, .975) and 0.15954179777768474 = (230.1990063163915 - 193.47264300203778) / 230.1990063163915 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.
System.Globalization.Tests.StringSearch.IsPrefix_FirstHalf(Options: (en-US, IgnoreCase, False))
Description of detection logic
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 237.12043098357736 < 261.18960951781975.
IsChangePoint: Marked as a change because one of 3/21/2023 2:53:21 PM, 3/30/2023 4:24:24 AM falls between 3/21/2023 2:53:21 PM and 3/30/2023 4:24:24 AM.
IsImprovementStdDev: Marked as improvement because 541.5608523705594 (T) = (0 -237.0004037396872) / Math.Sqrt((0.4795050823225292 / (299)) + (0.10034386971351177 / (30))) is greater than 1.9672451058622664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (30) - 2, .975) and 0.13848354494539705 = (275.09678120386695 - 237.0004037396872) / 275.09678120386695 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.
System.Globalization.Tests.StringSearch.IsSuffix_SecondHalf(Options: (, IgnoreCase, False))
Description of detection logic
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 238.74466858129162 < 262.47311048987217.
IsChangePoint: Marked as a change because one of 2/16/2023 5:39:51 AM, 3/21/2023 2:53:21 PM, 3/30/2023 4:24:24 AM falls between 3/21/2023 2:53:21 PM and 3/30/2023 4:24:24 AM.
IsImprovementStdDev: Marked as improvement because 13.33546597060727 (T) = (0 -238.4762362210983) / Math.Sqrt((330.44602818067 / (299)) + (0.08006917130253885 / (30))) is greater than 1.9672451058622664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (30) - 2, .975) and 0.05558582225053031 = (252.51234240191627 - 238.4762362210983) / 252.51234240191627 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
Run Information
| Name | Value |
|---|---|
| Architecture | arm64 |
| OS | ubuntu 20.04 |
| Queue | AmpereUbuntu |
| Baseline | 7b651743dc951384f6c3d8455c88518022c7c3b9 |
| Compare | b0b7aaefb88aa8d01b3d64fb40ac2f73a9d98c3e |
| Diff | Diff |
| Configs | CompilationMode:tiered, RunKind:micro |
Improvements in System.Memory.Span<Int32>
| Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
|---|---|---|---|---|---|---|---|---|---|---|
| LastIndexOfAnyValues - Duration of single invocation | 12.08 ns | 8.79 ns | 0.73 | 0.04 | True | |||||
| IndexOfAnyTwoValues - Duration of single invocation | 10.46 ns | 8.21 ns | 0.78 | 0.05 | False | |||||
| IndexOfAnyThreeValues - Duration of single invocation | 267.33 ns | 163.12 ns | 0.61 | 0.00 | True | |||||
| IndexOfAnyThreeValues - Duration of single invocation | 15.95 ns | 11.49 ns | 0.72 | 0.05 | True | |||||
| LastIndexOfAnyValues - Duration of single invocation | 179.59 ns | 120.78 ns | 0.67 | 0.00 | True | |||||
| IndexOfAnyTwoValues - Duration of single invocation | 179.12 ns | 120.97 ns | 0.68 | 0.00 | True |
_1.png)
_2.png)
_3.png)
_4.png)
_5.png)
_6.png)
Repro
General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md
Payloads
git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Memory.Span<Int32>*'Details
Payloads
Histogram
System.Memory.Span<Int32>.LastIndexOfAnyValues(Size: 33)
Description of detection logic
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 8.78636178986266 < 11.57268638127106.
IsChangePoint: Marked as a change because one of 3/21/2023 2:53:21 PM, 3/30/2023 4:24:24 AM falls between 3/21/2023 2:53:21 PM and 3/30/2023 4:24:24 AM.
IsImprovementStdDev: Marked as improvement because 112.41259151502476 (T) = (0 -8.803457963460177) / Math.Sqrt((0.23768292111288922 / (299)) + (0.007104779812967692 / (30))) is greater than 1.9672451058622664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (30) - 2, .975) and 0.29085873666640544 = (12.414251459682513 - 8.803457963460177) / 12.414251459682513 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.
System.Memory.Span<Int32>.IndexOfAnyTwoValues(Size: 33)
Description of detection logic
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 8.210843943013025 < 10.00433278079095.
IsChangePoint: Marked as a change because one of 2/7/2023 2:48:42 AM, 2/14/2023 11:21:41 AM, 3/21/2023 2:53:21 PM, 3/30/2023 4:24:24 AM falls between 3/21/2023 2:53:21 PM and 3/30/2023 4:24:24 AM.
IsImprovementStdDev: Marked as improvement because 65.70804325343836 (T) = (0 -8.16148609421462) / Math.Sqrt((0.44083909457598897 / (299)) + (0.00786868087678003 / (30))) is greater than 1.9672451058622664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (30) - 2, .975) and 0.2512233202431908 = (10.899759988339035 - 8.16148609421462) / 10.899759988339035 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
System.Memory.Span<Int32>.IndexOfAnyThreeValues(Size: 512)
Description of detection logic
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 163.1161721616684 < 253.95299318704323.
IsChangePoint: Marked as a change because one of 3/21/2023 2:53:21 PM, 3/30/2023 4:24:24 AM falls between 3/21/2023 2:53:21 PM and 3/30/2023 4:24:24 AM.
IsImprovementStdDev: Marked as improvement because 1673.9923282456505 (T) = (0 -163.2182683644545) / Math.Sqrt((0.9798475145348821 / (299)) + (0.01468975745105529 / (30))) is greater than 1.9672451058622664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (30) - 2, .975) and 0.3862994362968535 = (265.9575011298261 - 163.2182683644545) / 265.9575011298261 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.
System.Memory.Span<Int32>.IndexOfAnyThreeValues(Size: 33)
Description of detection logic
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 11.494285972944377 < 15.306549485841611.
IsChangePoint: Marked as a change because one of 2/7/2023 2:48:42 AM, 2/14/2023 11:21:41 AM, 3/21/2023 2:53:21 PM, 3/30/2023 4:24:24 AM falls between 3/21/2023 2:53:21 PM and 3/30/2023 4:24:24 AM.
IsImprovementStdDev: Marked as improvement because 98.12669483446935 (T) = (0 -11.471350862874052) / Math.Sqrt((0.5597990004002362 / (299)) + (0.014412774050035219 / (30))) is greater than 1.9672451058622664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (30) - 2, .975) and 0.2932405937353629 = (16.230913605384334 - 11.471350862874052) / 16.230913605384334 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.
System.Memory.Span<Int32>.LastIndexOfAnyValues(Size: 512)
Description of detection logic
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 120.78254620359971 < 170.6074134713576.
IsChangePoint: Marked as a change because one of 3/21/2023 2:53:21 PM, 3/30/2023 4:24:24 AM falls between 3/21/2023 2:53:21 PM and 3/30/2023 4:24:24 AM.
IsImprovementStdDev: Marked as improvement because 536.2843328189279 (T) = (0 -120.92032900943722) / Math.Sqrt((0.3213726561811616 / (299)) + (0.32980201305870704 / (30))) is greater than 1.9672451058622664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (30) - 2, .975) and 0.3276006629347655 = (179.83409908940143 - 120.92032900943722) / 179.83409908940143 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.
System.Memory.Span<Int32>.IndexOfAnyTwoValues(Size: 512)
Description of detection logic
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 120.96569491478282 < 170.30912924127352.
IsChangePoint: Marked as a change because one of 3/21/2023 2:53:21 PM, 3/30/2023 4:24:24 AM falls between 3/21/2023 2:53:21 PM and 3/30/2023 4:24:24 AM.
IsImprovementStdDev: Marked as improvement because 1255.7900783687162 (T) = (0 -121.07499330457813) / Math.Sqrt((0.41674485663668953 / (299)) + (0.022789606487870363 / (30))) is greater than 1.9672451058622664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (30) - 2, .975) and 0.3249243900939782 = (179.35027058885026 - 121.07499330457813) / 179.35027058885026 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
Run Information
| Name | Value |
|---|---|
| Architecture | arm64 |
| OS | ubuntu 20.04 |
| Queue | AmpereUbuntu |
| Baseline | 7b651743dc951384f6c3d8455c88518022c7c3b9 |
| Compare | b0b7aaefb88aa8d01b3d64fb40ac2f73a9d98c3e |
| Diff | Diff |
| Configs | CompilationMode:tiered, RunKind:micro |
Improvements in System.Tests.Perf_Uri
| Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
|---|---|---|---|---|---|---|---|---|---|---|
| Ctor - Duration of single invocation | 265.23 ns | 242.12 ns | 0.91 | 0.01 | True | |||||
| Ctor - Duration of single invocation | 278.08 ns | 257.53 ns | 0.93 | 0.00 | True | |||||
| CtorIdnHostPathAndQuery - Duration of single invocation | 855.98 ns | 720.98 ns | 0.84 | 0.00 | True | |||||
| CtorIdnHostPathAndQuery - Duration of single invocation | 665.03 ns | 615.22 ns | 0.93 | 0.00 | True | |||||
| CtorIdnHostPathAndQuery - Duration of single invocation | 821.63 ns | 744.91 ns | 0.91 | 0.00 | True | |||||
| CtorIdnHostPathAndQuery - Duration of single invocation | 687.43 ns | 627.57 ns | 0.91 | 0.00 | True | |||||
| CtorIdnHostPathAndQuery - Duration of single invocation | 909.67 ns | 845.86 ns | 0.93 | 0.00 | True | |||||
| CtorIdnHostPathAndQuery - Duration of single invocation | 812.34 ns | 751.87 ns | 0.93 | 0.00 | True |
Repro
General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md
Payloads
git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Uri*'Details
Payloads
Histogram
System.Tests.Perf_Uri.Ctor(input: "http://xn--hst-sna.with.xn--nicode-2ya")
Description of detection logic
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 242.12440886168895 < 252.0803940894109.
IsChangePoint: Marked as a change because one of 3/21/2023 2:53:21 PM, 3/30/2023 4:24:24 AM falls between 3/21/2023 2:53:21 PM and 3/30/2023 4:24:24 AM.
IsImprovementStdDev: Marked as improvement because 46.10377996533062 (T) = (0 -243.5858753443923) / Math.Sqrt((2.7124195696177527 / (254)) + (6.445313374121526 / (30))) is greater than 1.968411900500806 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (254) + (30) - 2, .975) and 0.08247065201686693 = (265.4802005841345 - 243.5858753443923) / 265.4802005841345 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.
System.Tests.Perf_Uri.Ctor(input: "https://a.much.longer.domain.name")
Description of detection logic
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 257.5340425862827 < 263.52996510478494.
IsChangePoint: Marked as a change because one of 3/21/2023 2:53:21 PM, 3/30/2023 4:24:24 AM falls between 3/21/2023 2:53:21 PM and 3/30/2023 4:24:24 AM.
IsImprovementStdDev: Marked as improvement because 59.384382026174364 (T) = (0 -258.93243423813715) / Math.Sqrt((2.986918410861068 / (254)) + (3.0327703031557145 / (30))) is greater than 1.968411900500806 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (254) + (30) - 2, .975) and 0.07153302911229367 = (278.88168600178864 - 258.93243423813715) / 278.88168600178864 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.
System.Tests.Perf_Uri.CtorIdnHostPathAndQuery(input: "http://host/path?key1=value1&key2=value2&key3=value3&key4=value4")
Description of detection logic
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 720.9804208067866 < 810.3848745946977.
IsChangePoint: Marked as a change because one of 3/21/2023 2:53:21 PM, 3/30/2023 4:24:24 AM falls between 3/21/2023 2:53:21 PM and 3/30/2023 4:24:24 AM.
IsImprovementStdDev: Marked as improvement because 184.59078775856588 (T) = (0 -719.9347695183171) / Math.Sqrt((37.24847817959369 / (252)) + (10.274843574228935 / (29))) is greater than 1.9685031265482282 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (252) + (29) - 2, .975) and 0.15375078587430085 = (850.7361159113352 - 719.9347695183171) / 850.7361159113352 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.
System.Tests.Perf_Uri.CtorIdnHostPathAndQuery(input: "http://dot.net/path/with?key=value#fragment")
Description of detection logic
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 615.2157157836649 < 632.1891896276089.
IsChangePoint: Marked as a change because one of 3/21/2023 2:53:21 PM, 3/30/2023 4:24:24 AM falls between 3/21/2023 2:53:21 PM and 3/30/2023 4:24:24 AM.
IsImprovementStdDev: Marked as improvement because 54.61803957862095 (T) = (0 -614.4313735323866) / Math.Sqrt((54.452124547550895 / (254)) + (21.66165529531104 / (30))) is greater than 1.968411900500806 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (254) + (30) - 2, .975) and 0.07920694422294854 = (667.2849775282806 - 614.4313735323866) / 667.2849775282806 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.
System.Tests.Perf_Uri.CtorIdnHostPathAndQuery(input: "https://a.much.longer.domain.name/path/with?key=value#fragment")
Description of detection logic
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 744.9063326997223 < 777.2412788925575.
IsChangePoint: Marked as a change because one of 3/21/2023 2:53:21 PM, 3/30/2023 4:24:24 AM falls between 3/21/2023 2:53:21 PM and 3/30/2023 4:24:24 AM.
IsImprovementStdDev: Marked as improvement because 67.51335125677451 (T) = (0 -748.3812138314881) / Math.Sqrt((24.122787623644385 / (253)) + (23.142160141228253 / (30))) is greater than 1.968442091833948 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (253) + (30) - 2, .975) and 0.07748007816678121 = (811.2358292971228 - 748.3812138314881) / 811.2358292971228 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.
System.Tests.Perf_Uri.CtorIdnHostPathAndQuery(input: "https://contoso.com/path/with?key=value#fragment")
Description of detection logic
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 627.5748261880323 < 646.2801174837334.
IsChangePoint: Marked as a change because one of 3/21/2023 2:53:21 PM, 3/30/2023 4:24:24 AM falls between 3/21/2023 2:53:21 PM and 3/30/2023 4:24:24 AM.
IsImprovementStdDev: Marked as improvement because 70.78984580959916 (T) = (0 -627.8131167974782) / Math.Sqrt((29.384186167440692 / (254)) + (11.677620555340505 / (30))) is greater than 1.968411900500806 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (254) + (30) - 2, .975) and 0.07418001716723438 = (678.1157551563482 - 627.8131167974782) / 678.1157551563482 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.
System.Tests.Perf_Uri.CtorIdnHostPathAndQuery(input: "http://host/path with escapable values?key=va lue")
Description of detection logic
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 845.8619401672396 < 856.7114252614246.
IsChangePoint: Marked as a change because one of 3/21/2023 2:53:21 PM, 3/30/2023 4:24:24 AM falls between 3/21/2023 2:53:21 PM and 3/30/2023 4:24:24 AM.
IsImprovementStdDev: Marked as improvement because 37.47693200228131 (T) = (0 -851.9673381302042) / Math.Sqrt((62.982702954966285 / (253)) + (45.738482849398196 / (30))) is greater than 1.968442091833948 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (253) + (30) - 2, .975) and 0.055340041956553586 = (901.8772637455444 - 851.9673381302042) / 901.8772637455444 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.
System.Tests.Perf_Uri.CtorIdnHostPathAndQuery(input: "http://xn--hst-sna.with.xn--nicode-2ya/path/with?key=value#fragment")
Description of detection logic
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 751.8666052072352 < 775.70237687341.
IsChangePoint: Marked as a change because one of 3/21/2023 2:53:21 PM, 3/30/2023 4:24:24 AM falls between 3/21/2023 2:53:21 PM and 3/30/2023 4:24:24 AM.
IsImprovementStdDev: Marked as improvement because 107.41285502950451 (T) = (0 -746.5466790236235) / Math.Sqrt((34.79410451001779 / (254)) + (6.685641699665012 / (30))) is greater than 1.968411900500806 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (254) + (30) - 2, .975) and 0.07945119942325643 = (810.980013830767 - 746.5466790236235) / 810.980013830767 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
Run Information
| Name | Value |
|---|---|
| Architecture | arm64 |
| OS | ubuntu 20.04 |
| Queue | AmpereUbuntu |
| Baseline | 7b651743dc951384f6c3d8455c88518022c7c3b9 |
| Compare | b0b7aaefb88aa8d01b3d64fb40ac2f73a9d98c3e |
| Diff | Diff |
| Configs | CompilationMode:tiered, RunKind:micro |
Improvements in MicroBenchmarks.Serializers.Json_ToStream<IndexViewModel>
| Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
|---|---|---|---|---|---|---|---|---|---|---|
| DataContractJsonSerializer_ - Duration of single invocation | 117.74 μs | 109.95 μs | 0.93 | 0.00 | True |
_1.png)
Repro
General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md
Payloads
git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'MicroBenchmarks.Serializers.Json_ToStream<IndexViewModel>*'Details
Payloads
Histogram
MicroBenchmarks.Serializers.Json_ToStream<IndexViewModel>.DataContractJsonSerializer_
Description of detection logic
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 109.9531853901332 < 110.86607683285365.
IsChangePoint: Marked as a change because one of 3/21/2023 2:53:21 PM, 3/30/2023 4:24:24 AM falls between 3/21/2023 2:53:21 PM and 3/30/2023 4:24:24 AM.
IsImprovementStdDev: Marked as improvement because 45.70948876218136 (T) = (0 -109269.91691012472) / Math.Sqrt((1002178.0874455189 / (299)) + (829152.417007072 / (30))) is greater than 1.9672451058622664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (30) - 2, .975) and 0.06858969028890695 = (117316.62809703955 - 109269.91691012472) / 117316.62809703955 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository