Roslyn source generator crash on mono/linux/arm64 #81123
Assignees
Labels
area-VM-meta-mono
blocking-clean-ci
Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms'
Known Build Error
Use this to report build issues in the .NET Helix tab
source-generator
Indicates an issue with a source generator feature
Milestone
Comments
|
I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label. |
radical
added
blocking-clean-ci
|
Tagging subscribers to this area: @dotnet/area-system-text-regularexpressions Issue Details
|
|
I also hit this in an unrelated PR: #81070 The shared failure message seems to be: It's affecting at least 3 Generator Roslyn tests:
Microsoft.Extensions.Logging.Generators.Roslyn3.11.TestsCallstackSystem.Text.Json.SourceGeneration.Roslyn4.4.TestsCallstackMicrosoft.Extensions.Logging.Generators.Roslyn4.0.TestsCallstack |
carlossanlop
added
the
Known Build Error
steveharter
added
the
source-generator
stephentoub
changed the title
System.Text.RegularExpressions.Tests native crash on mono/linux/arm64
|
This issue has heavy impact in many PRs. Looking for an area owner. |
This was referenced Feb 1, 2023
Merged
steveisok
removed
blocking-clean-ci
This seems to happen in different suites, so disabling may turn into a whack-a-mole game. For example, I spotted one happening in System.Security.Cryptography tests. |
|
going to try to decode the crashed ip address in-process in the crash handler. #83219 |
|
I can reproduce a crash on osx with the Microsoft.Extensions.Logging.Abstractions/tests/Microsoft.Extensions.Logging.Generators.Tests/Microsoft.Extensions.Logging.Generators.Roslyn4.0.Tests.csproj |
|
Some of the failures appear to be races with delegate invocation. This fails when ran in a loop on on an apple m1. |
|
So the problem seems to be related to the invoke trampolines which do: These trampolines are stored into delegate->invoke_impl by mono_delegate_trampoline (). Another thread makes an indirect call through delegate->invoke_impl and observes delegate->method_ptr being NULL. delegate->method_ptr is also set by mono_delegate_trampoline (), but even with a memory barrier between the stores, its not guaranteed that another thread will not observed the previous NULL value. |
vargaz
added a commit
to vargaz/runtime
that referenced
this issue
Mar 20, 2023
vargaz
added a commit
to vargaz/runtime
that referenced
this issue
Mar 20, 2023
vargaz
added a commit
that referenced
this issue
Mar 20, 2023
vargaz
added a commit
that referenced
this issue
Mar 21, 2023
|
We think The numbers on March 21 2023 before the change were:
|
|
Thanks @lambdageek and @vargaz. Once we confirm that fixes the issue, please consider backporting it to 7.0. We are seeing the failure happen there too. [Pending verifying benchmark results as @lambdageek told me offline]. |
|
#83688 is probably needed too. |
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as outdated.
This comment was marked as outdated.
|
Most of the recent hits are either on the 7.0 branch, or are unrelated false positives (the query catches too many other native crashes) There is one hit from #84004 from System.Security.Cryptography.Tests.WorkItemExecution that seems real: |
|
It still seems to crash, but the crashes are less frequent. |
|
The XML test lately failing are showing something like: with reported native stack: and managed stack: These are fixed by #88892 |
ghost
locked as resolved and limited conversation to collaborators
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
System.Text.RegularExpressions.Testsnative crash onBuild Libraries Test Run release mono linux arm64 Debugon unrelated #81066 .Build, and log:
{ "ErrorMessage": "futex-internal.h: No such file or directory.", "BuildRetry": false }Report
Summary
Known issue validation
Build: 🔎⚠️ Validation could not be done without an Azure DevOps build URL on the issue. Please add it to the "Build: 🔎" line.
Result validation:
Validation performed at: 6/28/2023 10:04:05 PM UTC
The text was updated successfully, but these errors were encountered: