Unable to generate IntelliJ Idea xml file

[zainab-ali]

The bug

The mill mill.scalalib.GenIdea.idea task fails:

1 targets failed
mill.scalalib.GenIdea.idea java.nio.file.NoSuchFileException: /home/zainab/.cache/coursier/v1/https/repo1.maven.org/maven2/io/netty/netty-transport-native-epoll/4.1.46.Final/netty-transport-native-epoll-4.1.46.Final-linux-x86_64.pom
    sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
    sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
    sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
    sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214)
    java.nio.file.Files.newByteChannel(Files.java:361)
    java.nio.file.Files.newByteChannel(Files.java:407)
    java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:384)
    java.nio.file.Files.newInputStream(Files.java:152)
    os.Path.getInputStream(Path.scala:474)
    os.read$bytes$.apply(ReadWriteOps.scala:258)
    os.read$.apply(ReadWriteOps.scala:218)
    os.read$.apply(ReadWriteOps.scala:216)
    mill.scalalib.GenIdeaImpl.sbtLibraryNameFromPom$1(GenIdeaImpl.scala:300)
    mill.scalalib.GenIdeaImpl.libraryNames$1(GenIdeaImpl.scala:313)
    mill.scalalib.GenIdeaImpl.$anonfun$xmlFileLayout$86(GenIdeaImpl.scala:347)
    scala.collection.StrictOptimizedIterableOps.flatMap(StrictOptimizedIterableOps.scala:117)
    scala.collection.StrictOptimizedIterableOps.flatMap$(StrictOptimizedIterableOps.scala:104)
    scala.collection.immutable.HashSet.flatMap(HashSet.scala:34)
    mill.scalalib.GenIdeaImpl.xmlFileLayout(GenIdeaImpl.scala:347)
    mill.scalalib.GenIdeaImpl.run(GenIdeaImpl.scala:36)
    mill.scalalib.GenIdea$.$anonfun$idea$2(GenIdea.scala:14)
    scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)

To Reproduce

Run mill mill.scalalib.GenIdea.idea in the root directory.

Version details

• OS: Linux
• mill version: 0.8.0

Additional information

This is likely due to the GenIdea.idea task attempting to inspect netty's POM file:

In the mill GenIdea code, we see:

    // Hack so that Intellij does not complain about unresolved magic
    // imports in build.sc when in fact they are resolved
    def sbtLibraryNameFromPom(pomPath : os.Path) : String = {
      val pom = xmlParseDom(os.read(pomPath)).flatMap(Pom.project).right.get

The library netty-transport-native-epoll doesn't have a POM on maven.

I'm as yet unsure why this is (are POM files necessary for libraries in maven?).

Quite possibly, the GenIdea.idea task should skip the POM if it doesn't exist. If so, we should work to fix this in the mill codebase.

[zainab-ali]

This is a bug in the way that jar classifiers are handled when searching for POMs in the mill GenIdeaImpl task.

@kxa14, you mentioned that you weren't sure where to start investigating. I kept a log of the steps taken as I looked into this. I wouldn't usually upload them in a GitHub comment (I doubled back on myself and found a few red herrings, so they aren't too concise), but I think you'll find them useful. They may give you some pointers on how to troubleshoot these sorts of issues.

1. Checkout and run the master version of mill

   According to the mill README, we can run mill with

   ./mill -i dev.run ~/scala/lsug-website mill.scalalib.GenIdea/idea

   This fails with the same error

2. Add some logging

   Looking in the out directory, where mill writes its outputs, there's a out/mill/scalalib/GenIdea/idea/log file. This contains messages outputted by the GenIdeaImpl ctx.log function. Incidentally, these log lines helped me locate the GenIdeaImpl class without knowing much about the mill codebase.

   We can add a log line to sbtLibraryNameFromPom:

     def sbtLibraryNameFromPom(pomPath : os.Path) : String = {
       ctx.map(_.log.info(s"Parsing POM $pomPath ..."))
       val pom = xmlParseDom(os.read(pomPath)).flatMap(Pom.project).right.get

   Running the GenIdea task again gives the log message:

   Parsing POM ~/.cache/coursier/v1/https/repo1.maven.org/maven2/io/netty/netty-transport-native-epoll/4.1.46.Final/netty-transport-native-epoll-4.1.46.Final-linux-x86_64.pom ...
   
   

   This is the last thing printed before the task fails.

3. Check the path ~/.cache/coursier...netty-transport-native-epoll-4.1.46.Final-linux-x86_64.pom

   The pom file doesn't exist.

4. Why are we searching for netty-transport-native-epoll?

   Following the code, we search for the pom for libraries in buildDepsPaths. This seems like a collection of paths to all of our build dependencies. We expect netty-transport-native-epoll to be a dependency in here. Let's add some log messages to check:

    val allBuildLibraries : Set[ResolvedLibrary] =
      resolvedLibraries(buildLibraryPaths ++ buildDepsPaths).toSet
   
    buildDepsPaths.map(p => ctx.map(_.log.info(s"buildDepPath $p ...")))

   This outputs the log message:

   buildDepPath ~/.cache/coursier/v1/https/repo1.maven.org/maven2/io/netty/netty-transport-native-epoll/4.1.46.Final/netty-transport-native-epoll-4.1.46.Final-linux-x86_64.jar ...
   

   So netty-transport-native-epoll is indeed a dependency.

5. What is netty-transport-native-epoll anyway?

   Epoll is an event notification syscall. The netty-transport-native-epoll jar contains JNI bindings for it. Loosely speaking, it's a wrapper around the non-Java shared library files for the syscall. It's probably pulled in by http4s. According to the Netty docs on native transports, it should be treated as an ordinary jar on Maven Central.

6. Should it have a POM?

   According to the Maven Central publishing instructions, all releases should have a POM. The Netty build is no exception.

7. Hold on a minute. Does it have a POM?

   Looking into the ~/.cache/coursier.../4.1.46.Final directory, there actually is one:

   netty-transport-native-epoll-4.1.46.Final-linux-x86_64.jar
   netty-transport-native-epoll-4.1.46.Final.pom
   netty-transport-native-epoll-4.1.46.Final-sources.jar
   

   It's missing a linux-x86_64 suffix, so is different to the file we're searching for.

8. What's that linux-x86_64 suffix for?

   Maven jars can have classifiers. These are arbitrary suffixes that let a release contain artifacts with different names. Looking back at the Netty build, it has a linux-x86_64 classifier. And looking at the Netty docs on native transports again, we see how the classifier is specified when adding the dependency for sbt:

   "io.netty" % "netty-transport-native-epoll" % "${project.version}" classifier "linux-x86_64"
   

   Mill should strip this classifier when looking for a POM.

9. Why doesn't mill strip the classifier?

   The logic in GenIdeaImpl looks for a POM extension with the same filename as the jar:

   val withoutExt = path.last.dropRight(path.ext.length + 1)
   val pom = path / os.up / s"$withoutExt.pom"

   It doesn't cater for classifiers. This is a bug in the mill code.

10. How do we fix this?

    One solution would be to look for any file with a POM extension in the same directory as the jar. Releases should (I think) only have a single POM associated with them, so we couldn't accidentally pick the wrong POM.

    Another would be to make the POM field optional. I'm not sure what the repercussions on the intellij idea file would be.