Crash inside libmsquic.so/libcrypto..so in System.Net.Http.Functional.Tests

[jkotas]

Build Information

Build: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=1203539
Build error leg or test failing: System.Net.Http.Functional.Tests.WorkItemExecution
Pull request: #121634

Error Message

Fill the error message using step by step known issues guidance.

{
  "ErrorMessage": "",
  "ErrorPattern": "",
  "BuildRetry": false,
  "ExcludeConsoleLog": false
}

Report

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
0	0	0

[jkotas]

From runfo get-helix-payload -j aba1f875-f698-420d-b4aa-03e134688e5a -w System.Net.Http.Functional.Tests -o c:\helix_payload - crashing at:

(lldb) bt
* thread #1, name = 'dotnet', stop reason = signal SIGSEGV: invalid address
  * frame #0: 0x00007c28410a7acd ld-musl-x86_64.so.1`get_meta(p=<unavailable>) at meta.h:141:2
    frame #1: 0x00007c28410a7b9f ld-musl-x86_64.so.1`__libc_free(p=<unavailable>) at free.c:105:19
    frame #2: 0x00007c28410a7658 ld-musl-x86_64.so.1`free(p=<unavailable>) at free.c:5:2 [artificial]
    frame #3: 0x00007c28410a2f8d ld-musl-x86_64.so.1`closedir(dir=0x00007c283e9e90b0) at closedir.c:9:2
    frame #4: 0x00007c283ace4885 libcrypto.so.3`OPENSSL_DIR_end + 28
    frame #5: 0x00007c283ae6ef79 libcrypto.so.3`___lldb_unnamed_symbol11660 + 32
    frame #6: 0x00007c283adfdd68 libcrypto.so.3`___lldb_unnamed_symbol9022 + 45
    frame #7: 0x00007c283adfea18 libcrypto.so.3`OSSL_STORE_close + 14
    frame #8: 0x00007c283ae0b711 libcrypto.so.3`___lldb_unnamed_symbol9127 + 326
    frame #9: 0x00007c283ae0b9d8 libcrypto.so.3`___lldb_unnamed_symbol9132 + 100
    frame #10: 0x00007c283ae22862 libcrypto.so.3`___lldb_unnamed_symbol9326 + 273
    frame #11: 0x00007c283ae2304b libcrypto.so.3`X509_STORE_CTX_get1_issuer + 111
    frame #12: 0x00007c283ae2669f libcrypto.so.3`___lldb_unnamed_symbol9354 + 945
    frame #13: 0x00007c283ae27ab1 libcrypto.so.3`___lldb_unnamed_symbol9360 + 36
    frame #14: 0x00007c283ae286ec libcrypto.so.3`___lldb_unnamed_symbol9361 + 449
    frame #15: 0x00007c283b32d181 libmsquic.so.2`___lldb_unnamed_symbol2126 + 1297
    frame #16: 0x00007c283b3372ed libmsquic.so.2`___lldb_unnamed_symbol2241 + 450
    frame #17: 0x00007c283b360ec3 libmsquic.so.2`___lldb_unnamed_symbol3116 + 61
    frame #18: 0x00007c283b35db0b libmsquic.so.2`___lldb_unnamed_symbol3078 + 1726
    frame #19: 0x00007c283b33e138 libmsquic.so.2`___lldb_unnamed_symbol2484 + 211
    frame #20: 0x00007c283b3304e4 libmsquic.so.2`___lldb_unnamed_symbol2137 + 212
    frame #21: 0x00007c283b2e90e3 libmsquic.so.2`___lldb_unnamed_symbol1445 + 115
    frame #22: 0x00007c283b2e9663 libmsquic.so.2`___lldb_unnamed_symbol1447 + 83
    frame #23: 0x00007c283b2deb0b libmsquic.so.2`___lldb_unnamed_symbol1407 + 1739
    frame #24: 0x00007c283b2e2156 libmsquic.so.2`___lldb_unnamed_symbol1409 + 294
    frame #25: 0x00007c283b2e2ac6 libmsquic.so.2`___lldb_unnamed_symbol1414 + 502
    frame #26: 0x00007c283b2e3289 libmsquic.so.2`___lldb_unnamed_symbol1415 + 153
    frame #27: 0x00007c283b2e67aa libmsquic.so.2`___lldb_unnamed_symbol1427 + 698
    frame #28: 0x00007c283b2c7766 libmsquic.so.2`___lldb_unnamed_symbol1278 + 150
    frame #29: 0x00007c283b2c7e89 libmsquic.so.2`___lldb_unnamed_symbol1280 + 153
    frame #30: 0x00007c283b3215d3 libmsquic.so.2`___lldb_unnamed_symbol1902 + 115
    frame #31: 0x00007c283b3217e3 libmsquic.so.2`___lldb_unnamed_symbol1904 + 131
    frame #32: 0x00007c28410c89d2 ld-musl-x86_64.so.1`start(p=0x00007c23d667ee60) at pthread_create.c:207:2
    frame #33: 0x00007c28410ca314 ld-musl-x86_64.so.1`__clone at clone.s:22

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

[MihuBot]

I'm a bot. Here is a possible related and/or duplicate issue (I may be wrong):

• [QUIC] [Linux] Segmentation fault in System.Net.Http.Functional.Tests #103703

[MichalStrehovsky]

Looks like this crash is Alpine too. We've been collecting these is in #121575.

[MichalStrehovsky]

Looks like this crash is Alpine too. We've been collecting these is in #121575.

Oh, right, different test. Possibly same root cause.

[ManickaP]

I haven't yet looked into the dump. But first of all, libmsquic 2.4.5 is more than a year old, so we should upgrade to 2.4.16+. That's on us, we're the ones maintaining libmsquic aport package.
Secondly, there wasn't any change in libmsquic, but there was a change in OpenSSL on 30.9. https://git.alpinelinux.org/aports/commit/?h=3.21-stable&id=945b6087f50008062ae5dc0515f08f861c7e3240 which is likely the root cause.
@liveans unless you want to look into this, I'll take this.