parallel gradle build failure

[shuffle2]

Describe the bug
I have added org.gradle.parallel=true to gradle.properties. The cpu has 64 cores.
Often, executing something like gradle assembleAll -x ip -x zipJavadocs fails with an error like:

The :Project:jar task failed.
Unable to delete file 'C:\src\ghidra\Ghidra\Framework\Project\build\libs\Project.jar'

where the falting .jar is nondeterministic (but always within Ghidra\Framework\Project\build\libs).

After looking at timeline of gradle build scan, it seemed like the jar target was always overlapping with a buildHelpFiles task.

After adding -x buildHelp to the gradle command line, the issue goes away.
It seems there is some unsafe behavior going on with the buildHelp task or dependencies.

To Reproduce
On machine with plenty of cores, run gradle --parallel assembleAll -x ip -x zipJavadocs.

Environment (please complete the following information):

• OS: Microsoft Windows [Version 10.0.22631.3737]
• Java Version: jdk-21.0.3+9
• Ghidra Version: 76977bd
• Ghidra Origin: locally built

-Gradle version: 8.8

[dragonmacher]

I scanned the helpProject.gradle looking for anything obvious that could cause this. So far all I'm seeing is the use of caching in the project. Specifically,

// This method contains logic for calculating help inputs based on the classpath of the project
// The work is cached, as the inputs may be requested multiple times during a build
ext.helpInputsCache = null
def getHelpInputs(Collection<File> fullClasspath) {
	
	if (ext.helpInputsCache != null) {
		return ext.helpInputsCache
	}
	
	def results = new HashSet<File>()
	Collection<File> modules = getMyModules(fullClasspath)
	modules.each { m -> 
		getHelpInputsFromModule(m.getAbsolutePath(), results)
	}
	
	// the classpath above does not include my module's contents, so add that manually
	def modulePath = file('.').getAbsolutePath()
	getHelpInputsFromModule(modulePath, results)
	
	ext.helpInputsCache = results.findAll(File::exists)	
	return ext.helpInputsCache
}

If the parallel grade mode is sharing the data structures in the ext space, then it is possible that multiple processes are attempting to build and use that cache. I'm not familiar enough with gradle to say for sure though.

[astrelsky]

#4543 is unfortunately relevant

I have only encountered this a few times. Every time I have, simply deleting the file in question and trying again fixes it. Seems you knew this already.

[shuffle2]

I'm again trying to build ghidra and the parallel build situation seems worse than before. Maybe this is because gradle 9.x is more strict about thread safety checks...
The problem is explained here: https://docs.gradle.org/9.2.0/userguide/viewing_debugging_dependencies.html#sub:resolving-unsafe-configuration-resolution-errors

here's how some others dealt with it: gradle/gradle#33938

my current env:
windows 10.0.26100.6584
jdk-21.0.9+10
gradle-9.2.0
Ghidra version: 14ead1a
Ghidra Origin: locally built

[shuffle2]

After applying #8622 , the errors are back to their original state (causing this issue, relating to buildHelp). Maybe that user could help here with gradle wizardry :D (cc @jobermayr )

[shuffle2]

Another difference I've noticed since the last time I was building ghidra, is that now, using gradlew.bat , I need to increase -Xmx to -Xmx1024m. I guess this should be determined dynamically instead of hardcoded to current value of 64m?

[jobermayr]

Does #8622 together with 3612 fix it?

[shuffle2]

Does #8622 together with 3612 fix it?

I think it does! (it built successfully twice in a row)

initially I was hitting a different issue, which turned out to be because of an unrelated problem on the system (DeviceAssociationService was causing some memleak and soaking up majority of my system's memory...). I've since edited this comment.