[Perf] Windows/arm64: 27 Improvements on 3/21/2023 8:47:40 PM #14617
Description
Run Information
| Name | Value |
|---|---|
| Architecture | arm64 |
| OS | Windows 10.0.19041 |
| Queue | SurfaceWindows |
| Baseline | 7b651743dc951384f6c3d8455c88518022c7c3b9 |
| Compare | b0b7aaefb88aa8d01b3d64fb40ac2f73a9d98c3e |
| Diff | Diff |
| Configs | CompilationMode:tiered, RunKind:micro |
Improvements in System.Tests.Perf_Uri
| Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
|---|---|---|---|---|---|---|---|---|---|---|
| Ctor - Duration of single invocation | 258.18 ns | 240.51 ns | 0.93 | 0.02 | True | |||||
| CtorIdnHostPathAndQuery - Duration of single invocation | 798.78 ns | 677.92 ns | 0.85 | 0.00 | True | |||||
| CtorIdnHostPathAndQuery - Duration of single invocation | 631.01 ns | 574.25 ns | 0.91 | 0.00 | True | |||||
| Ctor - Duration of single invocation | 247.84 ns | 224.76 ns | 0.91 | 0.00 | True | |||||
| CtorIdnHostPathAndQuery - Duration of single invocation | 762.63 ns | 699.90 ns | 0.92 | 0.00 | True | |||||
| CtorIdnHostPathAndQuery - Duration of single invocation | 765.42 ns | 700.65 ns | 0.92 | 0.00 | True | |||||
| CtorIdnHostPathAndQuery - Duration of single invocation | 649.96 ns | 600.91 ns | 0.92 | 0.00 | True | |||||
| CtorIdnHostPathAndQuery - Duration of single invocation | 690.65 ns | 652.45 ns | 0.94 | 0.00 | False | |||||
| CtorIdnHostPathAndQuery - Duration of single invocation | 638.49 ns | 587.79 ns | 0.92 | 0.00 | True |
Repro
General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md
Payloads
git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Uri*'Details
Payloads
Histogram
System.Tests.Perf_Uri.Ctor(input: "https://a.much.longer.domain.name")
Description of detection logic
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 240.50853923082585 < 244.08549249312802.
IsChangePoint: Marked as a change because one of 3/21/2023 2:53:21 PM, 3/30/2023 4:24:24 AM falls between 3/21/2023 2:53:21 PM and 3/30/2023 4:24:24 AM.
IsImprovementStdDev: Marked as improvement because 64.70640623654543 (T) = (0 -237.83889468409157) / Math.Sqrt((8.030450079824709 / (255)) + (1.6289620042817063 / (30))) is greater than 1.9683819234322317 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (255) + (30) - 2, .975) and 0.07380517969504341 = (256.7914325042126 - 237.83889468409157) / 256.7914325042126 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.
System.Tests.Perf_Uri.CtorIdnHostPathAndQuery(input: "http://host/path?key1=value1&key2=value2&key3=value3&key4=value4")
Description of detection logic
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 677.9204683449541 < 759.6830361038201.
IsChangePoint: Marked as a change because one of 3/21/2023 2:53:21 PM, 3/30/2023 4:24:24 AM falls between 3/21/2023 2:53:21 PM and 3/30/2023 4:24:24 AM.
IsImprovementStdDev: Marked as improvement because 145.63590732876085 (T) = (0 -671.4284264315155) / Math.Sqrt((35.823319599275194 / (255)) + (18.926869085200813 / (30))) is greater than 1.9683819234322317 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (255) + (30) - 2, .975) and 0.16001919382798294 = (799.3378199811101 - 671.4284264315155) / 799.3378199811101 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.
System.Tests.Perf_Uri.CtorIdnHostPathAndQuery(input: "http://dot.net/path/with?key=value#fragment")
Description of detection logic
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 574.2462789240419 < 598.6295786964341.
IsChangePoint: Marked as a change because one of 3/21/2023 2:53:21 PM, 3/30/2023 4:24:24 AM falls between 3/21/2023 2:53:21 PM and 3/30/2023 4:24:24 AM.
IsImprovementStdDev: Marked as improvement because 62.210339380011966 (T) = (0 -573.180724601083) / Math.Sqrt((32.38130157258675 / (255)) + (12.348146420613622 / (30))) is greater than 1.9683819234322317 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (255) + (30) - 2, .975) and 0.07377619093071786 = (618.8360944608462 - 573.180724601083) / 618.8360944608462 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.
System.Tests.Perf_Uri.Ctor(input: "http://xn--hst-sna.with.xn--nicode-2ya")
Description of detection logic
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 224.7593019252183 < 234.78695868719748.
IsChangePoint: Marked as a change because one of 3/21/2023 2:53:21 PM, 3/30/2023 4:24:24 AM falls between 3/21/2023 2:53:21 PM and 3/30/2023 4:24:24 AM.
IsImprovementStdDev: Marked as improvement because 100.98910124927073 (T) = (0 -226.60144510050114) / Math.Sqrt((2.00717942484255 / (254)) + (1.155911266855265 / (30))) is greater than 1.968411900500806 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (254) + (30) - 2, .975) and 0.08761928334688286 = (248.36281714911993 - 226.60144510050114) / 248.36281714911993 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.
System.Tests.Perf_Uri.CtorIdnHostPathAndQuery(input: "http://xn--hst-sna.with.xn--nicode-2ya/path/with?key=value#fragment")
Description of detection logic
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 699.8950520183904 < 725.8260909594475.
IsChangePoint: Marked as a change because one of 3/21/2023 2:53:21 PM, 3/30/2023 4:24:24 AM falls between 3/21/2023 2:53:21 PM and 3/30/2023 4:24:24 AM.
IsImprovementStdDev: Marked as improvement because 112.47329171888711 (T) = (0 -699.7421993058902) / Math.Sqrt((23.34000512989029 / (255)) + (5.834066672271794 / (30))) is greater than 1.9683819234322317 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (255) + (30) - 2, .975) and 0.07915521194605457 = (759.8915782372843 - 699.7421993058902) / 759.8915782372843 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.
System.Tests.Perf_Uri.CtorIdnHostPathAndQuery(input: "https://a.much.longer.domain.name/path/with?key=value#fragment")
Description of detection logic
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 700.650233974322 < 723.7760303633949.
IsChangePoint: Marked as a change because one of 3/21/2023 2:53:21 PM, 3/30/2023 4:24:24 AM falls between 3/21/2023 2:53:21 PM and 3/30/2023 4:24:24 AM.
IsImprovementStdDev: Marked as improvement because 62.2044132383208 (T) = (0 -700.970486245264) / Math.Sqrt((27.6052441932513 / (254)) + (22.17443721958879 / (30))) is greater than 1.968411900500806 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (254) + (30) - 2, .975) and 0.07553790831489987 = (758.2468686926277 - 700.970486245264) / 758.2468686926277 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.
System.Tests.Perf_Uri.CtorIdnHostPathAndQuery(input: "https://CONTOSO.com/path/with?key=value#fragment")
Description of detection logic
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 600.9130673763533 < 617.4315921374338.
IsChangePoint: Marked as a change because one of 3/21/2023 2:53:21 PM, 3/30/2023 4:24:24 AM falls between 3/21/2023 2:53:21 PM and 3/30/2023 4:24:24 AM.
IsImprovementStdDev: Marked as improvement because 75.58029103032328 (T) = (0 -603.4975541060354) / Math.Sqrt((19.090412145309976 / (255)) + (8.970214890440067 / (30))) is greater than 1.9683819234322317 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (255) + (30) - 2, .975) and 0.07112944742540767 = (649.711149129762 - 603.4975541060354) / 649.711149129762 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.
System.Tests.Perf_Uri.CtorIdnHostPathAndQuery(input: "http://host/path%20with%20escapable%20values?key=va%20lue")
Description of detection logic
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 652.4538680607029 < 658.081706367136.
IsChangePoint: Marked as a change because one of 3/21/2023 2:53:21 PM, 3/30/2023 4:24:24 AM falls between 3/21/2023 2:53:21 PM and 3/30/2023 4:24:24 AM.
IsImprovementStdDev: Marked as improvement because 28.079850939704283 (T) = (0 -655.8114839113458) / Math.Sqrt((97.24922345096488 / (255)) + (60.21972189626328 / (30))) is greater than 1.9683819234322317 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (255) + (30) - 2, .975) and 0.06206794670637218 = (699.2100137834167 - 655.8114839113458) / 699.2100137834167 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
System.Tests.Perf_Uri.CtorIdnHostPathAndQuery(input: "https://contoso.com/path/with?key=value#fragment")
Description of detection logic
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 587.7915662264116 < 608.4209279255197.
IsChangePoint: Marked as a change because one of 3/21/2023 2:53:21 PM, 3/30/2023 4:24:24 AM falls between 3/21/2023 2:53:21 PM and 3/30/2023 4:24:24 AM.
IsImprovementStdDev: Marked as improvement because 56.538467725246676 (T) = (0 -590.8163562206531) / Math.Sqrt((22.746124097959154 / (255)) + (16.716362990648236 / (30))) is greater than 1.9683819234322317 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (255) + (30) - 2, .975) and 0.07144228387867582 = (636.2731642450299 - 590.8163562206531) / 636.2731642450299 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
Run Information
| Name | Value |
|---|---|
| Architecture | arm64 |
| OS | Windows 10.0.19041 |
| Queue | SurfaceWindows |
| Baseline | 7b651743dc951384f6c3d8455c88518022c7c3b9 |
| Compare | b0b7aaefb88aa8d01b3d64fb40ac2f73a9d98c3e |
| Diff | Diff |
| Configs | CompilationMode:tiered, RunKind:micro |
Improvements in System.Globalization.Tests.StringSearch
| Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
|---|---|---|---|---|---|---|---|---|---|---|
| IsSuffix_SecondHalf - Duration of single invocation | 218.88 ns | 183.41 ns | 0.84 | 0.00 | True | |||||
| IsPrefix_FirstHalf - Duration of single invocation | 262.28 ns | 226.38 ns | 0.86 | 0.00 | True | |||||
| IsSuffix_SecondHalf - Duration of single invocation | 262.25 ns | 226.60 ns | 0.86 | 0.01 | False | |||||
| IsSuffix_SecondHalf - Duration of single invocation | 261.54 ns | 226.60 ns | 0.87 | 0.01 | False | |||||
| IsPrefix_FirstHalf - Duration of single invocation | 217.90 ns | 182.85 ns | 0.84 | 0.00 | True | |||||
| IsSuffix_SecondHalf - Duration of single invocation | 218.10 ns | 183.84 ns | 0.84 | 0.00 | True | |||||
| IsSuffix_SecondHalf - Duration of single invocation | 219.31 ns | 183.41 ns | 0.84 | 0.00 | True | |||||
| IsPrefix_FirstHalf - Duration of single invocation | 218.14 ns | 182.73 ns | 0.84 | 0.00 | True | |||||
| IsPrefix_FirstHalf - Duration of single invocation | 262.27 ns | 226.26 ns | 0.86 | 0.00 | True | |||||
| IsPrefix_FirstHalf - Duration of single invocation | 217.79 ns | 182.79 ns | 0.84 | 0.00 | True |
Repro
General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md
Payloads
git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Globalization.Tests.StringSearch*'Details
Payloads
Histogram
System.Globalization.Tests.StringSearch.IsSuffix_SecondHalf(Options: (en-US, IgnoreNonSpace, False))
Description of detection logic
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 183.40820577609793 < 207.96709452491842.
IsChangePoint: Marked as a change because one of 3/21/2023 2:53:21 PM, 3/30/2023 4:24:24 AM falls between 3/21/2023 2:53:21 PM and 3/30/2023 4:24:24 AM.
IsImprovementStdDev: Marked as improvement because 685.3859283441378 (T) = (0 -183.43930736973533) / Math.Sqrt((0.17496582243788603 / (299)) + (0.06251312070083868 / (30))) is greater than 1.9672451058622664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (30) - 2, .975) and 0.16179418557836836 = (218.8475720564046 - 183.43930736973533) / 218.8475720564046 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.
System.Globalization.Tests.StringSearch.IsPrefix_FirstHalf(Options: (en-US, IgnoreCase, False))
Description of detection logic
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 226.38138117887962 < 249.1747981387898.
IsChangePoint: Marked as a change because one of 3/21/2023 2:53:21 PM, 3/30/2023 4:24:24 AM falls between 3/21/2023 2:53:21 PM and 3/30/2023 4:24:24 AM.
IsImprovementStdDev: Marked as improvement because 654.6039833341777 (T) = (0 -226.29798577392094) / Math.Sqrt((0.5518750164744353 / (299)) + (0.03509187605680161 / (30))) is greater than 1.9672451058622664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (30) - 2, .975) and 0.13707224945574753 = (262.24441806535225 - 226.29798577392094) / 262.24441806535225 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.
System.Globalization.Tests.StringSearch.IsSuffix_SecondHalf(Options: (, IgnoreCase, False))
Description of detection logic
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 226.6046716889187 < 248.95122837327793.
IsChangePoint: Marked as a change because one of 2/16/2023 5:39:51 AM, 3/21/2023 2:53:21 PM, 3/30/2023 4:24:24 AM falls between 3/21/2023 2:53:21 PM and 3/30/2023 4:24:24 AM.
IsImprovementStdDev: Marked as improvement because 13.871032762015394 (T) = (0 -226.4553810967146) / Math.Sqrt((304.5826576568102 / (299)) + (0.10310931374103334 / (30))) is greater than 1.9672451058622664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (30) - 2, .975) and 0.058314991712981545 = (240.47890653866364 - 226.4553810967146) / 240.47890653866364 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
System.Globalization.Tests.StringSearch.IsSuffix_SecondHalf(Options: (en-US, IgnoreCase, False))
Description of detection logic
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 226.6042865763583 < 248.97477429001722.
IsChangePoint: Marked as a change because one of 2/16/2023 5:39:51 AM, 3/21/2023 2:53:21 PM, 3/30/2023 4:24:24 AM falls between 3/21/2023 2:53:21 PM and 3/30/2023 4:24:24 AM.
IsImprovementStdDev: Marked as improvement because 13.960064122106608 (T) = (0 -226.565660632422) / Math.Sqrt((294.01632090137923 / (299)) + (0.09001345411480524 / (30))) is greater than 1.9672451058622664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (30) - 2, .975) and 0.05766475616589495 = (240.42999783239364 - 226.565660632422) / 240.42999783239364 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
System.Globalization.Tests.StringSearch.IsPrefix_FirstHalf(Options: (, None, False))
Description of detection logic
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 182.85150432225836 < 207.04343822065675.
IsChangePoint: Marked as a change because one of 3/21/2023 2:53:21 PM, 3/30/2023 4:24:24 AM falls between 3/21/2023 2:53:21 PM and 3/30/2023 4:24:24 AM.
IsImprovementStdDev: Marked as improvement because 691.5618180091029 (T) = (0 -182.92367952502588) / Math.Sqrt((0.09888739336666007 / (299)) + (0.06740709816743504 / (30))) is greater than 1.9672451058622664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (30) - 2, .975) and 0.16103343270646325 = (218.03452802073903 - 182.92367952502588) / 218.03452802073903 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.
System.Globalization.Tests.StringSearch.IsSuffix_SecondHalf(Options: (, None, False))
Description of detection logic
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 183.8373618051473 < 207.90551879333478.
IsChangePoint: Marked as a change because one of 3/21/2023 2:53:21 PM, 3/30/2023 4:24:24 AM falls between 3/21/2023 2:53:21 PM and 3/30/2023 4:24:24 AM.
IsImprovementStdDev: Marked as improvement because 488.2850745332985 (T) = (0 -183.50248589885632) / Math.Sqrt((0.09876409010043846 / (299)) + (0.1464244135890235 / (30))) is greater than 1.9672451058622664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (30) - 2, .975) and 0.16113486883585013 = (218.75088030443882 - 183.50248589885632) / 218.75088030443882 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.
System.Globalization.Tests.StringSearch.IsSuffix_SecondHalf(Options: (en-US, None, False))
Description of detection logic
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 183.41177722894443 < 207.83586560657938.
IsChangePoint: Marked as a change because one of 3/21/2023 2:53:21 PM, 3/30/2023 4:24:24 AM falls between 3/21/2023 2:53:21 PM and 3/30/2023 4:24:24 AM.
IsImprovementStdDev: Marked as improvement because 371.99593868784376 (T) = (0 -183.4728988328513) / Math.Sqrt((0.11315303837246318 / (299)) + (0.2598287525134681 / (30))) is greater than 1.9672451058622664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (30) - 2, .975) and 0.1616143233015122 = (218.84068863790293 - 183.4728988328513) / 218.84068863790293 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.
System.Globalization.Tests.StringSearch.IsPrefix_FirstHalf(Options: (en-US, None, False))
Description of detection logic
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 182.73323814146175 < 207.12503058301292.
IsChangePoint: Marked as a change because one of 3/21/2023 2:53:21 PM, 3/30/2023 4:24:24 AM falls between 3/21/2023 2:53:21 PM and 3/30/2023 4:24:24 AM.
IsImprovementStdDev: Marked as improvement because 712.7858987199694 (T) = (0 -182.87047021713218) / Math.Sqrt((0.10248147774298999 / (299)) + (0.06246877783672412 / (30))) is greater than 1.9672451058622664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (30) - 2, .975) and 0.16103454509692489 = (217.97139458889762 - 182.87047021713218) / 217.97139458889762 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.
System.Globalization.Tests.StringSearch.IsPrefix_FirstHalf(Options: (, IgnoreCase, False))
Description of detection logic
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 226.26120790630415 < 249.09823564007817.
IsChangePoint: Marked as a change because one of 3/21/2023 2:53:21 PM, 3/30/2023 4:24:24 AM falls between 3/21/2023 2:53:21 PM and 3/30/2023 4:24:24 AM.
IsImprovementStdDev: Marked as improvement because 345.0466228225767 (T) = (0 -226.25882919459485) / Math.Sqrt((0.6737127855286732 / (299)) + (0.2610495028591767 / (30))) is greater than 1.9672451058622664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (30) - 2, .975) and 0.13764541204854583 = (262.3733118091 - 226.25882919459485) / 262.3733118091 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.
System.Globalization.Tests.StringSearch.IsPrefix_FirstHalf(Options: (en-US, IgnoreNonSpace, False))
Description of detection logic
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 182.7910896005366 < 206.94312358093177.
IsChangePoint: Marked as a change because one of 3/21/2023 2:53:21 PM, 3/30/2023 4:24:24 AM falls between 3/21/2023 2:53:21 PM and 3/30/2023 4:24:24 AM.
IsImprovementStdDev: Marked as improvement because 819.6416703410594 (T) = (0 -182.86079600104733) / Math.Sqrt((0.17921823477893847 / (299)) + (0.03747120897218512 / (30))) is greater than 1.9672451058622664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (30) - 2, .975) and 0.16157352257901067 = (218.0999776671289 - 182.86079600104733) / 218.0999776671289 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
Run Information
| Name | Value |
|---|---|
| Architecture | arm64 |
| OS | Windows 10.0.19041 |
| Queue | SurfaceWindows |
| Baseline | 7b651743dc951384f6c3d8455c88518022c7c3b9 |
| Compare | b0b7aaefb88aa8d01b3d64fb40ac2f73a9d98c3e |
| Diff | Diff |
| Configs | CompilationMode:tiered, RunKind:micro |
Improvements in System.Memory.Span<Int32>
| Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
|---|---|---|---|---|---|---|---|---|---|---|
| LastIndexOfAnyValues - Duration of single invocation | 170.46 ns | 115.64 ns | 0.68 | 0.00 | True | |||||
| IndexOfAnyTwoValues - Duration of single invocation | 170.96 ns | 115.79 ns | 0.68 | 0.00 | True | |||||
| IndexOfAnyTwoValues - Duration of single invocation | 10.31 ns | 7.80 ns | 0.76 | 0.11 | False | |||||
| IndexOfAnyThreeValues - Duration of single invocation | 252.44 ns | 155.09 ns | 0.61 | 0.00 | True | |||||
| LastIndexOfAnyValues - Duration of single invocation | 10.56 ns | 8.42 ns | 0.80 | 0.05 | True | |||||
| IndexOfAnyThreeValues - Duration of single invocation | 15.05 ns | 10.76 ns | 0.72 | 0.06 | True |
_1.png)
_2.png)
_3.png)
_4.png)
_5.png)
Repro
General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md
Payloads
git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Memory.Span<Int32>*'Details
Payloads
Histogram
System.Memory.Span<Int32>.LastIndexOfAnyValues(Size: 512)
Description of detection logic
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 115.63608250726368 < 161.9552422875369.
IsChangePoint: Marked as a change because one of 3/21/2023 2:53:21 PM, 3/30/2023 4:24:24 AM falls between 3/21/2023 2:53:21 PM and 3/30/2023 4:24:24 AM.
IsImprovementStdDev: Marked as improvement because 2782.2711209598797 (T) = (0 -115.57874036155272) / Math.Sqrt((0.07432439355261873 / (299)) + (0.004232407886095802 / (30))) is greater than 1.9672451058622664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (30) - 2, .975) and 0.3221188776823483 = (170.50001328609517 - 115.57874036155272) / 170.50001328609517 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.
System.Memory.Span<Int32>.IndexOfAnyTwoValues(Size: 512)
Description of detection logic
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 115.78580510310414 < 162.4523571711186.
IsChangePoint: Marked as a change because one of 3/21/2023 2:53:21 PM, 3/30/2023 4:24:24 AM falls between 3/21/2023 2:53:21 PM and 3/30/2023 4:24:24 AM.
IsImprovementStdDev: Marked as improvement because 869.0574162433705 (T) = (0 -115.47942749557961) / Math.Sqrt((0.16997234879716197 / (299)) + (0.10415589619581367 / (30))) is greater than 1.9672451058622664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (30) - 2, .975) and 0.32357353708011805 = (170.71985474533003 - 115.47942749557961) / 170.71985474533003 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.
System.Memory.Span<Int32>.IndexOfAnyTwoValues(Size: 33)
Description of detection logic
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 7.798979812258853 < 9.748030303693636.
IsChangePoint: Marked as a change because one of 3/21/2023 2:53:21 PM, 3/30/2023 4:24:24 AM falls between 3/21/2023 2:53:21 PM and 3/30/2023 4:24:24 AM.
IsImprovementStdDev: Marked as improvement because 71.61197320230931 (T) = (0 -7.732351464596709) / Math.Sqrt((0.21713412601231957 / (299)) + (0.008632098541934193 / (30))) is greater than 1.9672451058622664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (30) - 2, .975) and 0.22774158590021995 = (10.012647739954112 - 7.732351464596709) / 10.012647739954112 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
System.Memory.Span<Int32>.IndexOfAnyThreeValues(Size: 512)
Description of detection logic
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 155.09280280968017 < 239.83927885769816.
IsChangePoint: Marked as a change because one of 3/21/2023 2:53:21 PM, 3/30/2023 4:24:24 AM falls between 3/21/2023 2:53:21 PM and 3/30/2023 4:24:24 AM.
IsImprovementStdDev: Marked as improvement because 1134.9370147893094 (T) = (0 -155.14885461169558) / Math.Sqrt((0.7894163820124774 / (299)) + (0.14290962307027827 / (30))) is greater than 1.9672451058622664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (30) - 2, .975) and 0.38629082516761654 = (252.80517380902756 - 155.14885461169558) / 252.80517380902756 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.
System.Memory.Span<Int32>.LastIndexOfAnyValues(Size: 33)
Description of detection logic
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 8.420647213655684 < 10.210020648064043.
IsChangePoint: Marked as a change because one of 3/21/2023 2:53:21 PM, 3/30/2023 4:24:24 AM falls between 3/21/2023 2:53:21 PM and 3/30/2023 4:24:24 AM.
IsImprovementStdDev: Marked as improvement because 21.76984691347717 (T) = (0 -8.267416308570208) / Math.Sqrt((0.1436897242549493 / (299)) + (0.3856852190705465 / (30))) is greater than 1.9672451058622664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (30) - 2, .975) and 0.2331851753103947 = (10.781502968355795 - 8.267416308570208) / 10.781502968355795 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.
System.Memory.Span<Int32>.IndexOfAnyThreeValues(Size: 33)
Description of detection logic
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 10.761169249323395 < 14.103299673280787.
IsChangePoint: Marked as a change because one of 3/21/2023 2:53:21 PM, 3/30/2023 4:24:24 AM falls between 3/21/2023 2:53:21 PM and 3/30/2023 4:24:24 AM.
IsImprovementStdDev: Marked as improvement because 112.43698296309876 (T) = (0 -10.930870379302311) / Math.Sqrt((0.2106755828149204 / (299)) + (0.01941771071439999 / (30))) is greater than 1.9672451058622664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (30) - 2, .975) and 0.27441516843200925 = (15.064910267874083 - 10.930870379302311) / 15.064910267874083 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository