Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Gradle plugin's forbiddenApisMain throws ClassNotFoundException despite compileJava succeeding #109

Closed
hakanai opened this issue Aug 17, 2016 · 9 comments
Assignees

Comments

@hakanai
Copy link

hakanai commented Aug 17, 2016

I have been trying to crack down on incorrect dependencies in our projects and have a tool in the making which automates removing random dependencies or replacing "compile" with "testCompile" to see if the build still works. If it still works, it removes the line.

Overnight (the tool takes forever because our build is slow) it removed a line which somehow broke running forbiddenApisMain but I'm not quite sure why.

The one particular module can still compile:

$ gradle forbiddenApisMain
...omitting...
:engine-impl:compileJava
BUILD SUCCESSFUL

But when I run the checks:

$ gradle forbiddenApisMain
...
:engine-impl:forbiddenApisMain FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':engine-impl:forbiddenApisMain'.
> de.thetaphi.forbiddenapis.ForbiddenApiException: Check for forbidden API calls failed: java.lang.ClassNotFoundException: org.picocontainer.Startable

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.

BUILD FAILED

Indeed, the line it had just removed from the build was:

compile libraries.picocontainer

So seemingly I have ended up with a module where the classpath used for compiling is adequate for compiling but not adequate for forbidden-apis to perform its checks. I'm still trying to figure out how this can be possible.

We do turn off transitive dependencies for all our compile dependencies, as part of trying to ensure that random rubbish isn't pulled into our compile classpath, to stop people accidentally importing the wrong classes and to encourage proper encapsulation of dependencies. So my initial hunch is that perhaps forbidden-apis is running with the compile classpath but should be running with the runtime classpath, or a modified version of the compile classpath with transitive dependencies turned on.

The complete stack trace is as follows:

org.gradle.api.tasks.TaskExecutionException: Execution failed for task ':engine-impl:forbiddenApisMain'.
        at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeActions(ExecuteActionsTaskExecuter.java:69)
        at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.execute(ExecuteActionsTaskExecuter.java:46)
        at org.gradle.api.internal.tasks.execution.PostExecutionAnalysisTaskExecuter.execute(PostExecutionAnalysisTaskExecuter.java:35)
        at org.gradle.api.internal.tasks.execution.SkipUpToDateTaskExecuter.execute(SkipUpToDateTaskExecuter.java:66)
        at org.gradle.api.internal.tasks.execution.ValidatingTaskExecuter.execute(ValidatingTaskExecuter.java:58)
        at org.gradle.api.internal.tasks.execution.SkipEmptySourceFilesTaskExecuter.execute(SkipEmptySourceFilesTaskExecuter.java:52)
        at org.gradle.api.internal.tasks.execution.SkipTaskWithNoActionsExecuter.execute(SkipTaskWithNoActionsExecuter.java:52)
        at org.gradle.api.internal.tasks.execution.SkipOnlyIfTaskExecuter.execute(SkipOnlyIfTaskExecuter.java:53)
        at org.gradle.api.internal.tasks.execution.ExecuteAtMostOnceTaskExecuter.execute(ExecuteAtMostOnceTaskExecuter.java:43)
        at org.gradle.execution.taskgraph.DefaultTaskGraphExecuter$EventFiringTaskWorker.execute(DefaultTaskGraphExecuter.java:203)
        at org.gradle.execution.taskgraph.DefaultTaskGraphExecuter$EventFiringTaskWorker.execute(DefaultTaskGraphExecuter.java:185)
        at org.gradle.execution.taskgraph.AbstractTaskPlanExecutor$TaskExecutorWorker.processTask(AbstractTaskPlanExecutor.java:66)
        at org.gradle.execution.taskgraph.AbstractTaskPlanExecutor$TaskExecutorWorker.run(AbstractTaskPlanExecutor.java:50)
        at org.gradle.execution.taskgraph.DefaultTaskPlanExecutor.process(DefaultTaskPlanExecutor.java:25)
        at org.gradle.execution.taskgraph.DefaultTaskGraphExecuter.execute(DefaultTaskGraphExecuter.java:110)
        at org.gradle.execution.SelectedTaskExecutionAction.execute(SelectedTaskExecutionAction.java:37)
        at org.gradle.execution.DefaultBuildExecuter.execute(DefaultBuildExecuter.java:37)
        at org.gradle.execution.DefaultBuildExecuter.access$000(DefaultBuildExecuter.java:23)
        at org.gradle.execution.DefaultBuildExecuter$1.proceed(DefaultBuildExecuter.java:43)
        at org.gradle.execution.DryRunBuildExecutionAction.execute(DryRunBuildExecutionAction.java:32)
        at org.gradle.execution.DefaultBuildExecuter.execute(DefaultBuildExecuter.java:37)
        at org.gradle.execution.DefaultBuildExecuter.execute(DefaultBuildExecuter.java:30)
        at org.gradle.initialization.DefaultGradleLauncher$4.run(DefaultGradleLauncher.java:153)
        at org.gradle.internal.Factories$1.create(Factories.java:22)
        at org.gradle.internal.progress.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:91)
        at org.gradle.internal.progress.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:53)
        at org.gradle.initialization.DefaultGradleLauncher.doBuildStages(DefaultGradleLauncher.java:150)
        at org.gradle.initialization.DefaultGradleLauncher.access$200(DefaultGradleLauncher.java:32)
        at org.gradle.initialization.DefaultGradleLauncher$1.create(DefaultGradleLauncher.java:98)
        at org.gradle.initialization.DefaultGradleLauncher$1.create(DefaultGradleLauncher.java:92)
        at org.gradle.internal.progress.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:91)
        at org.gradle.internal.progress.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:63)
        at org.gradle.initialization.DefaultGradleLauncher.doBuild(DefaultGradleLauncher.java:92)
        at org.gradle.initialization.DefaultGradleLauncher.run(DefaultGradleLauncher.java:83)
        at org.gradle.launcher.exec.InProcessBuildActionExecuter$DefaultBuildController.run(InProcessBuildActionExecuter.java:99)
        at org.gradle.tooling.internal.provider.ExecuteBuildActionRunner.run(ExecuteBuildActionRunner.java:28)
        at org.gradle.launcher.exec.ChainingBuildActionRunner.run(ChainingBuildActionRunner.java:35)
        at org.gradle.launcher.exec.InProcessBuildActionExecuter.execute(InProcessBuildActionExecuter.java:48)
        at org.gradle.launcher.exec.InProcessBuildActionExecuter.execute(InProcessBuildActionExecuter.java:30)
        at org.gradle.launcher.exec.ContinuousBuildActionExecuter.execute(ContinuousBuildActionExecuter.java:81)
        at org.gradle.launcher.exec.ContinuousBuildActionExecuter.execute(ContinuousBuildActionExecuter.java:46)
        at org.gradle.launcher.exec.DaemonUsageSuggestingBuildActionExecuter.execute(DaemonUsageSuggestingBuildActionExecuter.java:51)
        at org.gradle.launcher.exec.DaemonUsageSuggestingBuildActionExecuter.execute(DaemonUsageSuggestingBuildActionExecuter.java:28)
        at org.gradle.launcher.cli.RunBuildAction.run(RunBuildAction.java:43)
        at org.gradle.internal.Actions$RunnableActionAdapter.execute(Actions.java:173)
        at org.gradle.launcher.cli.CommandLineActionFactory$ParseAndBuildAction.execute(CommandLineActionFactory.java:239)
        at org.gradle.launcher.cli.CommandLineActionFactory$ParseAndBuildAction.execute(CommandLineActionFactory.java:212)
        at org.gradle.launcher.cli.JavaRuntimeValidationAction.execute(JavaRuntimeValidationAction.java:35)
        at org.gradle.launcher.cli.JavaRuntimeValidationAction.execute(JavaRuntimeValidationAction.java:24)
        at org.gradle.launcher.cli.ExceptionReportingAction.execute(ExceptionReportingAction.java:33)
        at org.gradle.launcher.cli.ExceptionReportingAction.execute(ExceptionReportingAction.java:22)
        at org.gradle.launcher.cli.CommandLineActionFactory$WithLogging.execute(CommandLineActionFactory.java:205)
        at org.gradle.launcher.cli.CommandLineActionFactory$WithLogging.execute(CommandLineActionFactory.java:169)
        at org.gradle.launcher.Main.doAction(Main.java:33)
        at org.gradle.launcher.bootstrap.EntryPoint.run(EntryPoint.java:45)
        at org.gradle.launcher.bootstrap.ProcessBootstrap.runNoExit(ProcessBootstrap.java:55)
        at org.gradle.launcher.bootstrap.ProcessBootstrap.run(ProcessBootstrap.java:36)
        at org.gradle.launcher.GradleMain.main(GradleMain.java:23)
Caused by: org.gradle.internal.UncheckedException: de.thetaphi.forbiddenapis.ForbiddenApiException: Check for forbidden API calls failed: java.lang.ClassNotFoundException: org.picocontainer.Startable
        at org.gradle.internal.UncheckedException.throwAsUncheckedException(UncheckedException.java:45)
        at org.gradle.internal.reflect.JavaMethod.invoke(JavaMethod.java:78)
        at org.gradle.api.internal.project.taskfactory.AnnotationProcessingTaskFactory$StandardTaskAction.doExecute(AnnotationProcessingTaskFactory.java:228)
        at org.gradle.api.internal.project.taskfactory.AnnotationProcessingTaskFactory$StandardTaskAction.execute(AnnotationProcessingTaskFactory.java:221)
        at org.gradle.api.internal.project.taskfactory.AnnotationProcessingTaskFactory$StandardTaskAction.execute(AnnotationProcessingTaskFactory.java:210)
        at org.gradle.api.internal.AbstractTask$TaskActionWrapper.execute(AbstractTask.java:621)
        at org.gradle.api.internal.AbstractTask$TaskActionWrapper.execute(AbstractTask.java:604)
        at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeAction(ExecuteActionsTaskExecuter.java:80)
        at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeActions(ExecuteActionsTaskExecuter.java:61)
        ... 57 more
Caused by: de.thetaphi.forbiddenapis.ForbiddenApiException: Check for forbidden API calls failed: java.lang.ClassNotFoundException: org.picocontainer.Startable
        at de.thetaphi.forbiddenapis.Checker.run(Checker.java:624)
        at de.thetaphi.forbiddenapis.gradle.CheckForbiddenApis.checkForbidden(CheckForbiddenApis.java:585)
        at org.gradle.internal.reflect.JavaMethod.invoke(JavaMethod.java:75)
        ... 64 more
Caused by: java.lang.ClassNotFoundException: org.picocontainer.Startable
        at de.thetaphi.forbiddenapis.Checker.getClassFromClassLoader(Checker.java:314)
        at de.thetaphi.forbiddenapis.Checker.lookupRelatedClass(Checker.java:326)
        at de.thetaphi.forbiddenapis.ClassScanner.checkClassDefinition(ClassScanner.java:152)
        at de.thetaphi.forbiddenapis.ClassScanner.checkType(ClassScanner.java:172)
        at de.thetaphi.forbiddenapis.ClassScanner.checkType(ClassScanner.java:183)
        at de.thetaphi.forbiddenapis.ClassScanner.checkDescriptor(ClassScanner.java:210)
        at de.thetaphi.forbiddenapis.ClassScanner$2.<init>(ClassScanner.java:324)
        at de.thetaphi.forbiddenapis.ClassScanner.visitMethod(ClassScanner.java:316)
        at de.thetaphi.forbiddenapis.asm.ClassReader.b(Unknown Source)
        at de.thetaphi.forbiddenapis.asm.ClassReader.accept(Unknown Source)
        at de.thetaphi.forbiddenapis.asm.ClassReader.accept(Unknown Source)
        at de.thetaphi.forbiddenapis.Checker.checkClass(Checker.java:602)
        at de.thetaphi.forbiddenapis.Checker.run(Checker.java:619)
        ... 66 more
@uschindler
Copy link
Member

uschindler commented Aug 18, 2016

Hi.

theoretically, the compile classpath should be fine for forbidden-apis checks. It can just happen under some circumstances that - luckily - javac does not need to look into superclasses of types found in code. But those types should be available on compile classpath, too (e.g., a newer version of javac may possibly also look into them).

In contrast, forbiddenapis has to look into types more thorougly, because if you forbid a very generic superclass, code that uses any subclass of it should also be forbidden. Typical example: java.io.FileReader is forbidden, because it does not allow to pass charset. If some class in our code or from a library in your classpath extends java.io.FileReader, it must be detected, so every code calling this subclass must be identified to use the FileReader class. Javac does not have the need to do this unless you call a virtual method, not overridden in the subclass (currently, but may need to in the future).

On the other hand, the runtime classpath is also not 100% correct. A workaround if you trigger such issues is to redefine the "classpath" property of forbiddenapis to point to the runtime. By default it is initialized with the compile classpath, but that's easy to override!:

forbiddenApisMain {
  classpath = project.sourceSets.main.runtimeClasspath;
}

FYI: This issue is similar to #43

@uschindler uschindler self-assigned this Aug 18, 2016
@uschindler
Copy link
Member

uschindler commented Aug 18, 2016

In the above stack trace it looks like the following is happening:

  • Some method signature in your code uses a bunch of types from some external library that itsself depends on the missing dependency.
  • Forbiddenapis scans the method descriptor (checkDescriptor delegating to checkType using a method type build from descriptor), which iterates over all types (again checkType) found inside (parameter types, return types).
  • On each type it calls checkClassDefinition, which check if its is forbidden. This check involves loading all superclasses up to Object. And this breaks, only in forbidden, because javac does not necessarily do this while compiling.

@hakanai
Copy link
Author

hakanai commented Aug 18, 2016

I guess PicoContainer's Startable is a very common interface for random things to implement and it's very likely that some component somewhere has implemented it, but we never call start on whatever it is, so we never hit the issue.

So I guess I really have two options:

  1. Consider that it should be on the compile classpath and change my tool to run "check" instead of "compileJava" to verify that the compile classpath is correct.
  2. Consider that it shouldn't be on the compile classpath and change our config to use the runtime classpath.

I bet each has its pros and cons, and I bet that leaving it off would result in devs complaining about compile failing when they just ran the compile from IDEA and it worked. (The task of getting forbidden-apis to run alongside the IDEA compiler is difficult.) :D

@uschindler
Copy link
Member

Another alternative to investigate is to add the picocontainer element to forbiddenapis classpath only:

forbiddenApisMain {
  classpath += libraries.picocontainer // not sure how to do this correctly, just as idea
}

@hakanai
Copy link
Author

hakanai commented Aug 18, 2016

True enough. I could also potentially do something like this:

configurations {
  compileForForbiddenApis {
    extendsFrom compile
    transitive = true
  }
}
forbiddenApisMain {
  classpath = configurations.compileForForbiddenApis
}

But I do fear what a future javac might do, so maybe I'm better off just adding the dependency and changing the way I check whether the build is still OK.

@uschindler
Copy link
Member

+1 looks like a cool workaround.

Anyways, another quick solution is: forbiddenApisMain.failOnMissingClasses = false
This would print a warning only; complaining about missing class. In that case the checks are not so thorough, because it would stop iterating to superclasses and interfaces once a missing one is found.

@uschindler
Copy link
Member

Can I close this issue?

@hakanai
Copy link
Author

hakanai commented Aug 28, 2016

I ended up changing my tool to run forbiddenApisMain to check for missing classes, because I can't know whether a future javac might load more classes than the current one.

@hakanai hakanai closed this as completed Aug 28, 2016
@hakanai
Copy link
Author

hakanai commented Jun 26, 2018

Got bitten by one of these again, and starting to think that it would be really helpful if it printed out the chain of references resulting in trying to load the class.

At the moment, I have a vague "couldn't find this class", but there are no direct references to it from the project being checked, so I have to somehow figure out where the indirect references might be, which I don't have any tools to help with.

Created #142 to track this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Development

No branches or pull requests

2 participants