Find an alternative to 'registering' a custom AbstractArtifactRepository

[Aaron1011]

Currently, ForgeGradle (through Artifactural) 'registers' a custom AbstractArtifactRepository with Gradle. This a huge hack - Gradle does not support registering custom artifact repositories, which requires us to use a combination of reflection and internal Gradle apis.

This has the effect of tying all users of ForgeGradle to a specific minor (or even point) release of Gradle. Upgrading Gradle becomes very difficult, as there's no guarantee that ForgeGradle will work at all.

This problem is only going to get worse over time. Since Artifactural is using internal Gradle APIs, Gradle is free to change those APIs at any time. We've already dealt with this once - a change to a constructor signature required a horrible build-time bytecode hack in order to a produce a ForgeGradle jar compatible with multiple versions of Gradle. As new versions of Gradle are released, ForgeGradle will need to choose between dropping support for old minor (or even patch!) releases, or adding even more version-specific hacks to hold things together.

We should investigate officially supported alternatives to the current custom artifact repository hack. If acceptable solutions exist, I will try to write up a Gradle feature request describing the changes that would be necessary.

[LexManos]

I would LOVE to move away from ugly hacks, but at the time this was written Gradle had nothing.
Gradle 5.5 introduced Artifact Transformers, but 4.10+ have caching issues that I don't want to deal with {It deletes random files from the cache in the name of 'automatic cleaning'}. As well as the transformers not being fully functional/fleshed out last time I looked at them.

As it sits, I'm jusst really burnt out on trying to get Gradle to do what SHOULD be a simple concept of 'let me make code that generates a dependency', but it comes with the baggage of the other crap in Gradle's dep management.

[DenWav]

I really have no idea why Gradle doesn't allow registering a task as a configuration dependency. It seems like that fits perfectly inside Gradle's model, but for no reason at all Gradle doesn't allow it. It feels so natural in Gradle I almost frequently find myself trying to use a task output as a configuration dependency just to be sadly reminded you can't do this.

I don't really know for sure but I would imagine that would be exactly what FG3 needs - register everything as tasks and depend on said tasks, no setupDecW required and everything would naturally work. I wonder if there are any PRs for Gradle open for this kind of thing, or if Gradle would be open to accepting a PR adding the feature.

[tomdcc]

Since this is being discussed, just popping in to mention that this line in Artifactural also currently causes Gradle build scans to fall over, because under normal circumstances a flat dir repo never has an empty directory list.

[wolfs]

@LexManos I would really love to help bring this project to a place where artifact transforms can be used instead of the hack described above. What would be necessary to do so? I don't quite understand what the custom artifact repository is doing. Artifact transforms are working well enough that they are used widely in the Gradle plugin for Android.

[LexManos]

I haven't had time to dig to deeply into the transformer system, and the caching system of newer versions. FG is a means to a end, and I've been focusing on that end project since we have a system that is working.

But, A basic over view of what we do:

MinecraftRepo: Serves net.minecraft:[client|server]:version[:slim|data|extra|mappings]
This is the simplest repo, it's just downloading a few files, and applying static transformations.

• Pom: Downloads the manifest json, to download the version json, and converts its dependency list to POM format.
• raw: Downloads version manifest json, to download the version json, uses a url from that to download and serve the raw obfuscated jar files.
• Mappings: Downloads MCPConfig zip, extracts mappings.txt, and serves it.
• Data: Converts raw to JUST the non-classs files.
• Slim: Uses Mappings to convert raw to just contain classes listed in the mappings.
• Extra: Uses Mappings to convert raw to just contain classes NOT listed in mappings.

MCPRepo: Serves: net.minecraft:[client|server|joined|mappings_(channel)|:version:[srg|extra]
This one is slightly more complicated, as we run a executable jar to produce some of the artifacts.

• pom: Same as MinecraftRepo's pom, except supports joined, and extra dependencies from MCPConfig
• raw: Uses MCPConfig to build a chain of processors and stops after the 'merge' step.
• srg: Uses MCPConfig to build a chain of processors and stops after the 'rename' step.

DeobfuscatingRepo: Serves: (group):(artifact):(version)_mapped_(mapping_version)[:sources]

• pom: Copies the upstream artifact's pom, updating self references to mapped version.
• raw: Downloads MCP Mappings, and runs a jar file to remap the upstream jar to those names.
• source: Downloads MCP Mappings, does basic text-replace to produce a named sources jar.

MinecraftUserRepo: Serves a single target artifact: (group):(artifact):(version)[_mapped_(mapping_version)][_at_(ATHash)][:sources|*]
This is the most complicated repo as it is the end modder facing one, it combines all the other repos together, producing an artifact set that the end modders can compile against. The AT Hash part is just a dirty thing to bust the gradle cache. Would love to see that go.

• pom: Downloads the chain of userdev configs, and vanilla pom, adding libraries as needed from config
• sources:
  • Downloads the chain of userdev configs resulting in a MCPConfig processor chain
  • Injects classes from root MCPConfig creating a package-info.java for every package from the template
  • Injects nessasary source files from userdev config chain.
  • Applies source patches from userdev config chain.
  • If mapping_version specified:
    • Download mappings
    • Do text replacement to sources jar to get version in mapped names
• raw:
  • If Sources exists, runs java compile on them and serves the compiled jar.
  • If sources does not exist
    • Downloads the vanilla jar in SRG name from MCPRepo.
    • Downloads the chain of userdev configs
    • Applies binary patches and injects new classes from them
    • Runs MCInjector executable jar, which cleans up some bytecode stupidity from the Mojang Obfusicator
    • Compiles and injects classes from root MCPConfig creating a package-info.java for every package from the template.
    • If the user has specified custom Access transformers, gather them up and run our executable AccessTransformer jar against the current jar.
    • If mapping_version supplied:
      • Download Mappings
      • Generate Srg->Mapped config from MCPConfig, and downloaded mappings
      • Run executable jar file to rename jar using that mapping

We also have other tasks, for downloading assets, creating run configs, etc.. but that's not dirty hacks.

If you want more specifics on certain steps, we can talk but that's a overview and anyone who wants to try and make it work with transformers feel free to go ahead.

[melix]

We may need more than artifact transforms to do what you need and probably more than a single discussion to get to a working solution. However, from what I read, I think a good amount of the problems you're facing are due to the fact you're trying to warp the POM model for your needs, and indeed, it's complicated.

I don't feel like custom repositories is the way to go here. In fact, I think we have a better solution, which is Gradle Module Metadata. This module metadata is here to describe complex software components, in particular components which are available in different forms (that we call variants). In short, we have the following:

• a component is a set of variants
• a variant consists of a set of files and dependencies

Each variant is associated with a unique set of attributes and capabilities, meaning that under the same GAV coordinates, you can provide different "versions" (variants) of the same thing. That is to say, if I understand properly, that you could have a "MinecraftComponent" that has several variants: mappings, data, ...

We started documenting how to implement variant-aware plugins and this is for example already used by the Kotlin folks for their Kotlin multi-platform plugin.

If you're not already, we're happy to talk to you in the dependency management channel of our community Slack.

[fuzzyweapon]

I haven't worked in gradle in a year, but I was working on their latest because I was using kotlin and I'm pretty sure all of the goals of this as outline are much easier in the newer versions of gradle. I mean, obviously that would need to be verified for each of these, but some of these stages aren't really helpful. Like, why are you guys generating POMs and stuff like this. Why are you working the Maven model where you don't have to? It's a lot easier in terms of stability, etc, to lean into the way the Gradle model is...and that might be a lot easier to work toward today if some of the blockers to a new version were addressed.

[LexManos]

Feel free to try and provide a proof of concept. Until then we are sticking with what works. I am tired of bashing my head against the wall to make gradle do what we need to do. If people care then try it.

[fuzzyweapon]

Yeah, I totally understand that frustration. 1.5 years of headbashing under my belt. I'll think about it when I've got a little more time on my plate. Right now I'm pretty busy.