After upgrading to Eclipse 2024-09 on Windows Server 2016, it seems there's a crash loading swt-win32-4966r5.dll

[kgibm]

Describe the bug

Custom RCP of Eclipse Memory Analyzer Tool using Eclipse 2024-09 and running it with Java 21 (OpenJDK 21 with Eclipse OpenJ9 through IBM Semeru Runtimes) on Windows Server 2016 x64 causes a crash seemingly in swt-win32-4966r5.dll:

18:36:41.656 0x30900          j9scar.113       > JVM_LoadLibrary(name=E:\kgrigor\.eclipse\MAT_IBMJava32GB_20240923c\org.eclipse.osgi\401\0\.cp\swt-win32-4966r5.dll)
18:36:41.656 0x30900         omrport.475        > omrsl_open_shared_library name=E:\kgrigor\.eclipse\MAT_IBMJava32GB_20240923c\org.eclipse.osgi\401\0\.cp\swt-win32-4966r5.dll, flags=2
18:36:41.656 0x30900         omrport.476         - omrsl_open_shared_library using mangledName E:\kgrigor\.eclipse\MAT_IBMJava32GB_20240923c\org.eclipse.osgi\401\0\.cp\swt-win32-4966r5.dll
Unhandled exception
Type=Segmentation error vmState=0x00040000
Windows_ExceptionCode=c0000139 J9Generic_Signal=00000004 ExceptionAddress=00007FFFB1085A60 ContextFlags=0010000f

Running windbg on the resulting core file shows the JVM calling LoadLibraryExW and a crash in ntdll (then there's a subsequent hang in Eclipse OpenJ9 catching the segfault and trying to create a core dump which is being investigated separately; it's likely due to insufficient permissions on these tightly controlled support servers we use but I'm able to create a core dump of the hung process starting Task Manager as Administrator and requesting a core dump). Relevant stack is between frames 0x1a - 0x0f:

0:000> kn
 # Child-SP          RetAddr           Call Site
00 000000a1`d9337518 00007fff`ae1e6d1f ntdll!NtWaitForSingleObject+0x14
01 000000a1`d9337520 00007fff`672b9096 KERNELBASE!WaitForSingleObjectEx+0x8f
02 000000a1`d93375c0 00007fff`67255e33 j9prt29!omrdump_create+0x306 [c:\workspace\openjdk-build\workspace\build\src\omr\port\win32\omrosdump.c @ 185] 
03 000000a1`d9337660 00007fff`67255925 j9dmp29!doSystemDump+0xa3 [c:\workspace\openjdk-build\workspace\build\src\openj9\runtime\rasdump\dmpagent.c @ 763] 
04 000000a1`d93376c0 00007fff`672bb6f6 j9dmp29!protectedDumpFunction+0x15 [c:\workspace\openjdk-build\workspace\build\src\openj9\runtime\rasdump\dmpagent.c @ 2897] 
05 000000a1`d93376f0 00007fff`672bce30 j9prt29!runInTryExcept+0x16 [c:\workspace\openjdk-build\workspace\build\src\omr\port\win64amd\omrsignal.c @ 220] 
06 000000a1`d9337730 00007fff`672538cc j9prt29!omrsig_protect+0x210 [c:\workspace\openjdk-build\workspace\build\src\omr\port\win64amd\omrsignal.c @ 285] 
07 (Inline Function) --------`-------- j9dmp29!runDumpFunction+0x6d [c:\workspace\openjdk-build\workspace\build\src\openj9\runtime\rasdump\dmpagent.c @ 2875] 
08 000000a1`d9337910 00007fff`6726cec2 j9dmp29!runDumpAgent+0x33c [c:\workspace\openjdk-build\workspace\build\src\openj9\runtime\rasdump\dmpagent.c @ 2794] 
09 000000a1`d9337df0 00007fff`673df50f j9dmp29!triggerDumpAgents+0x532 [c:\workspace\openjdk-build\workspace\build\src\openj9\runtime\rasdump\trigger.c @ 1041] 
0a 000000a1`d9338270 00007fff`672bb6f6 j9vm29!generateDiagnosticFiles+0x1ef [c:\workspace\openjdk-build\workspace\build\src\openj9\runtime\vm\gphandle.c @ 1166] 
0b 000000a1`d9338730 00007fff`672bce30 j9prt29!runInTryExcept+0x16 [c:\workspace\openjdk-build\workspace\build\src\omr\port\win64amd\omrsignal.c @ 220] 
0c 000000a1`d9338770 00007fff`673df712 j9prt29!omrsig_protect+0x210 [c:\workspace\openjdk-build\workspace\build\src\omr\port\win64amd\omrsignal.c @ 285] 
0d 000000a1`d9338950 00007fff`672badb4 j9vm29!vmSignalHandler+0x1d2 [c:\workspace\openjdk-build\workspace\build\src\openj9\runtime\vm\gphandle.c @ 843] 
0e 000000a1`d9339840 00007fff`b1046dc0 j9prt29!mainVectoredExceptionHandler+0x154 [c:\workspace\openjdk-build\workspace\build\src\omr\port\win64amd\omrsignal.c @ 1207] 
0f 000000a1`d9339a00 00007fff`b10146cb ntdll!RtlInitializeCriticalSection+0x190
10 000000a1`d9339aa0 00007fff`b1085a9f ntdll!RtlImageNtHeaderEx+0x17b
11 000000a1`d933a1a0 00007fff`b10a7a33 ntdll!RtlRaiseStatus+0x5f
12 000000a1`d933a740 00007fff`b1099b12 ntdll!memset+0x1b0f3
13 000000a1`d933a930 00007fff`b0fede78 ntdll!memset+0xd1d2
14 000000a1`d933aab0 00007fff`b0fed3c4 ntdll!RtlCheckTokenCapability+0x3e8
15 000000a1`d933ab10 00007fff`b0ff8b0e ntdll!RtlInitializeSid+0x134
16 000000a1`d933ab50 00007fff`b0ff513d ntdll!RtlUnicodeToMultiByteN+0x43e
17 000000a1`d933abd0 00007fff`b0ff1c5c ntdll!RtlFormatCurrentUserKeyPath+0x1fd
18 000000a1`d933ad70 00007fff`ae1ef28f ntdll!LdrLoadDll+0x8c
19 000000a1`d933ae70 00007fff`672be91f KERNELBASE!LoadLibraryExW+0x16f
1a 000000a1`d933aee0 00007fff`67543265 j9prt29!omrsl_open_shared_library+0x26f [c:\workspace\openjdk-build\workspace\build\src\omr\port\win32\omrsl.c @ 181] 
1b 000000a1`d933c5d0 00007fff`670b3761 jvm_7fff67530000!JVM_LoadLibrary+0x145 [c:\workspace\openjdk-build\workspace\build\src\openj9\runtime\j9vm\jvm.c @ 3990] 
1c 000000a1`d933c870 00007fff`674b0b91 java!Java_jdk_internal_loader_NativeLibraries_load+0x89 [s\src\java.base\share\native\libjava\NativeLibraries.c @ 139] 
1d 000000a1`d933ca00 00007fff`674b09e3 j9vm29!ffi_call_win64+0x41 [c:\workspace\openjdk-build\workspace\build\src\build\windows-x86_64-server-release\vm\runtime\libffi\win64_intel.asm @ 1105] 
1e 000000a1`d933ca40 00007fff`674b07f2 j9vm29!ffi_call_int+0x133 [c:\workspace\openjdk-build\workspace\build\src\openj9\runtime\libffi\x86\ffiw64.c @ 181] 
1f 000000a1`d933ca90 00007fff`67329b2b j9vm29!ffi_call+0x12 [c:\workspace\openjdk-build\workspace\build\src\openj9\runtime\libffi\x86\ffiw64.c @ 190] 
20 000000a1`d933cad0 00000000`00146528 j9vm29!VM_BytecodeInterpreterCompressed::run+0x18a3b
21 000000a1`d933cad8 00000000`000008a0 0x146528
22 000000a1`d933cae0 0000020f`5744ce79 0x8a0
23 000000a1`d933cae8 000000a1`d933cfa0 0x0000020f`5744ce79
24 000000a1`d933caf0 000000a1`d933cbb0 0x000000a1`d933cfa0
25 000000a1`d933caf8 00000000`00000000 0x000000a1`d933cbb0

To Reproduce

Steps to reproduce the behavior:

1. Eclipse MAT Tycho build with:

    	<repository>
    		<id>2024-09</id>
    		<url>https://download.eclipse.org/releases/2024-09/</url>
    		<layout>p2</layout>
    	</repository>

   And .target file:

   <?xml version="1.0" encoding="UTF-8" standalone="no"?>
   <?pde version="3.5"?>
   <target name="eclipse base">
     <locations>
       <location includeAllPlatforms="false" includeMode="planner" type="InstallableUnit">
         <unit id="org.eclipse.rcp.feature.group" version="0.0.0"/>
         <unit id="org.eclipse.equinox.p2.user.ui.feature.group" version="0.0.0"/>
         <unit id="org.eclipse.equinox.executable.feature.group" version="0.0.0"/>
         <repository location="https://download.eclipse.org/releases/2024-09/"/>
       </location>
     </locations>
   </target>

Expected behavior

No crash

Screenshots

N/A

Environment:

1. Select the platform(s) on which the behavior is seen:

• • All OS
• • Windows
• • Linux
• • macOS

3. Additional OS info (e.g. OS version, Linux Desktop, etc)

Windows Server 2016. I think this is equivalent to Windows 10, so Eclipse 2024-09 is Eclipse 4.33 so I think Windows 10 should be supported: https://eclipse.dev/eclipse/development/plans/eclipse_project_plan_4_33.xml#target_environments

4. JRE/JDK version

openjdk version "21.0.4" 2024-07-16 LTS
IBM Semeru Runtime Open Edition 21.0.4.1 (build 21.0.4+7-LTS)
Eclipse OpenJ9 VM 21.0.4.1 (build openj9-0.46.1, JRE 21 Windows Server 2016 amd64-64-Bit Compressed References 20240716_260 (JIT enabled, AOT enabled)
OpenJ9   - 4760d5d320
OMR      - 840a9adba
JCL      - db3fffb417c based on jdk-21.0.4+7)

Version since

Works on the same machines with Eclipse 4.31

Workaround (or) Additional context

None

[kgibm]

I should have also noted that the same exact binaries work on other Windows x64 machines (we publish the build to git and then pull the tag to machines), so the DLL swt-win32-4966r5.dll is not corrupt or broken in any obvious way. These particular machines are in an especially secure environment, so it's possible something is interfering with the DLL loading; however, Eclipse 2024-03 SWT DLLs load in these environments work fine. I will ask our support team for these servers if they have any kernel-level DLL security introspection products.

[kgibm]

The problem does not occur in these same systems when running with Windows Server 2022. We will build out some new machines as a workaround; however, if any additional diagnostics can be gathered on the 2016 machines with the crash, just let me know. Thanks

[iloveeclipse]

Probably something similar to #1252, too old Windows or something Windows Server specific. Have you tried to run on Oracle/Temurin JDK build? May be the error would be more precise ?

[hazendaz]

It's happening for me, too, on Windows Server 10. Build is 1607. How can we manually revert? I have to build shared eclipse bundles due to security on loading plugins which takes forever adding 100 or so plugins. There was no warning nor documentation stating this would entirely break. Landing page still states it's supported by eclipse. I think the write up here is as good as it gets. I'm using oracles jdk 21.0.4.

[iloveeclipse]

@hazendaz assuming Oracle JVM crashes in your case, could you please attach JVM crash dump and error printed on the shell?

[Phillipus]

Works on the same machines with Eclipse 4.31

What about 4.32?

[Phillipus]

Found these reports:

dbeaver/dbeaver#35722
dbeaver/dbeaver#35694

Operating System
Windows Server 2016

[Phillipus]

Assuming something changed to native Windows code between 4.32 and 4.33 that would cause this crash we have:

7e60295
e2d0fd3
3e44b4b

So, the added Windows API calls are:

GetThreadDpiAwarenessContext
SetThreadDpiAwarenessContext
MonitorFromRect
OpenThemeDataForDpi
GetAncestor

And the documentation states that Windows Server 2016 is supported:

https://learn.microsoft.com/en-us/windows/win32/api/winuser/nf-winuser-getthreaddpiawarenesscontext
https://learn.microsoft.com/en-us/windows/win32/api/winuser/nf-winuser-setthreaddpiawarenesscontext
https://learn.microsoft.com/en-us/windows/win32/api/winuser/nf-winuser-monitorfromrect
https://learn.microsoft.com/en-us/windows/win32/api/uxtheme/nf-uxtheme-openthemedatafordpi
https://learn.microsoft.com/en-us/windows/win32/api/winuser/nf-winuser-getancestor

Would be good to have a more refined stack trace.

[Phillipus]

Assuming that one of these native calls is causing a problem on Windows Server 2016 the message is going to be "The specified procedure could not be found". In order to find out which procedure could not be found somebody needs to check different SWT commits on Windows Server 2016 as I did here.

[Phillipus]

Assuming that one of these native calls is causing a problem on Windows Server 2016 the message is going to be "The specified procedure could not be found". In order to find out which procedure could not be found somebody needs to check different SWT commits on Windows Server 2016 as I did here.

Being a glutton for punishment and doubtful that anyone else would do this, I loaded up Windows Server 2016 in a VM and traced it to the problem commit:

e2d0fd3

It's OK before that but gives a "The specified procedure could not be found" after it (actually it's ba4442b where the binaries are committed)

So it's either OpenThemeDataForDpi or MonitorFromRect.

[kgibm]

Aren't both of those supported in Windows Server 2016?

• https://learn.microsoft.com/en-us/windows/win32/api/uxtheme/nf-uxtheme-openthemedatafordpi

  Minimum supported server Windows Server 2016 [desktop apps only]

• https://learn.microsoft.com/en-us/windows/win32/api/winuser/nf-winuser-monitorfromrect

  Minimum supported server Windows 2000 Server [desktop apps only]

[Phillipus]

Aren't both of those supported in Windows Server 2016?

As I already said here, they are supposed to be supported. But maybe they're not?