Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Cleanup Error on Scan #9779

Open
Martin-Idel opened this issue Jan 20, 2025 · 4 comments
Open

Cleanup Error on Scan #9779

Martin-Idel opened this issue Jan 20, 2025 · 4 comments
Labels
needs info An issue where further information is required scanner About the scanner tool

Comments

@Martin-Idel
Copy link

Describe the bug

Scanning Excalidraw (https://github.com/excalidraw/excalidraw) results in an error and no ort output.

To Reproduce

Steps to reproduce the behavior:

  1. Check out repository for excalidraw at following link 'https://github.com/excalidraw/excalidraw/releases/tag/v0.17.3'
  2. Run Analyzer from dockerfile built from ORT master (no issues), via:
docker run -v ~/gitRepos/ort_output/:/home/ort/.ort/config/ -v ~/gitRepos/excalidraw/:/download -v ~/gitRepos/ort_output/ort_excalidraw/analyze/:/analyze ort:latest analyze -i /download -o/analyze

Config is just

ort:
  analyzer:
    allowDynamicVersions: true
  1. Finally do run the scanner:
docker run --network=host -e ORT_CONFIG_DIR=/config -v ~/gitRepos/ort_output/ort_excalidraw/:/config/ -v ~/gitRepos/ort_output/ort_excalidraw/analyze/:/analyze -v ~/gitRepos/ort_output/ort_excalidraw/scan/:/scan ort:latest --info scan -i /analyze/analyzer-result.yml -o/scan
  1. See error

I also tried adding options:
-e "JAVA_OPTS=-Xms512M -Xmx10g -XX:MaxMetaspaceSize=6g -Dkotlin.daemon.jvm.options='-Xmx16g'"
--memory="32g"
and various combinations of sizes, that seemed to not help.

I also tried using a postgres database which significantly reduced scan time, however the error remains the same.

Expected behavior

No error.

Console / log output

09:19:42.855 [main] INFO org.ossreviewtoolkit.utils.common.ProcessCapture - Running 'git checkout 0136e8a05ef864666a6e95b0de8b679b0bc93338' in '/tmp/ort-DefaultWorkingTreeCache18354474298420537781'...
Exception in thread "main" java.nio.file.NoSuchFileException: /tmp/ort-DefaultWorkingTreeCache18354474298420537781/.git/gc.log.lock
at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)
at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
at java.base/sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55)
at java.base/sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:171)
at java.base/sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99)
at java.base/java.nio.file.Files.readAttributes(Files.java:1853)
at java.base/java.nio.file.FileTreeWalker.getAttributes(FileTreeWalker.java:220)
at java.base/java.nio.file.FileTreeWalker.visit(FileTreeWalker.java:277)
at java.base/java.nio.file.FileTreeWalker.next(FileTreeWalker.java:374)
at java.base/java.nio.file.Files.walkFileTree(Files.java:2820)
at kotlin.io.path.PathsKt__PathUtilsKt.visitFileTree(PathUtils.kt:1147)
at kotlin.io.path.PathsKt__PathUtilsKt.visitFileTree(PathUtils.kt:1201)
at kotlin.io.path.PathsKt__PathUtilsKt.visitFileTree$default(PathUtils.kt:1193)
at kotlin.io.path.PathsKt__PathRecursiveFunctionsKt.copyToRecursively(PathRecursiveFunctions.kt:234)
at kotlin.io.path.PathsKt__PathRecursiveFunctionsKt.copyToRecursively(PathRecursiveFunctions.kt:81)
at kotlin.io.path.PathsKt__PathRecursiveFunctionsKt.copyToRecursively$default(PathRecursiveFunctions.kt:72)
at org.ossreviewtoolkit.scanner.provenance.DefaultProvenanceDownloader.downloadFromVcs$lambda$1(ProvenanceDownloader.kt:117)
at org.ossreviewtoolkit.downloader.DefaultWorkingTreeCache.use(WorkingTreeCache.kt:65)
at org.ossreviewtoolkit.scanner.provenance.DefaultProvenanceDownloader.downloadFromVcs(ProvenanceDownloader.kt:104)
at org.ossreviewtoolkit.scanner.provenance.DefaultProvenanceDownloader.access$downloadFromVcs(ProvenanceDownloader.kt:80)
at org.ossreviewtoolkit.scanner.provenance.DefaultProvenanceDownloader$download$1.invokeSuspend(ProvenanceDownloader.kt:96)
at org.ossreviewtoolkit.scanner.provenance.DefaultProvenanceDownloader$download$1.invoke(ProvenanceDownloader.kt)
at org.ossreviewtoolkit.scanner.provenance.DefaultProvenanceDownloader$download$1.invoke(ProvenanceDownloader.kt)
at org.ossreviewtoolkit.utils.ort.UtilsKt$runBlocking$1.invokeSuspend(Utils.kt:231)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:100)
at kotlinx.coroutines.EventLoopImplBase.processNextEvent(EventLoop.common.kt:263)
at kotlinx.coroutines.BlockingCoroutine.joinBlocking(Builders.kt:95)
at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking(Builders.kt:69)
at kotlinx.coroutines.BuildersKt.runBlocking(Unknown Source)
at org.ossreviewtoolkit.utils.ort.UtilsKt.runBlocking(Utils.kt:231)
at org.ossreviewtoolkit.utils.ort.UtilsKt.runBlocking$default(Utils.kt:229)
at org.ossreviewtoolkit.scanner.provenance.DefaultProvenanceDownloader.download(ProvenanceDownloader.kt:96)
at org.ossreviewtoolkit.scanner.provenance.ProvenanceDownloader$DefaultImpls.downloadRecursively(ProvenanceDownloader.kt:64)
at org.ossreviewtoolkit.scanner.provenance.DefaultProvenanceDownloader.downloadRecursively(ProvenanceDownloader.kt:80)
at org.ossreviewtoolkit.scanner.Scanner.createMissingArchives(Scanner.kt:708)
at org.ossreviewtoolkit.scanner.Scanner.scan(Scanner.kt:179)
at org.ossreviewtoolkit.scanner.Scanner$scan$3.invokeSuspend(Scanner.kt)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:100)
at kotlinx.coroutines.EventLoopImplBase.processNextEvent(EventLoop.common.kt:263)
at kotlinx.coroutines.BlockingCoroutine.joinBlocking(Builders.kt:95)
at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking(Builders.kt:69)
at kotlinx.coroutines.BuildersKt.runBlocking(Unknown Source)
at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking$default(Builders.kt:47)
at kotlinx.coroutines.BuildersKt.runBlocking$default(Unknown Source)
at org.ossreviewtoolkit.plugins.commands.scanner.ScannerCommand.runScanners(ScannerCommand.kt:242)
at org.ossreviewtoolkit.plugins.commands.scanner.ScannerCommand.run(ScannerCommand.kt:148)
at com.github.ajalt.clikt.core.CoreCliktCommandKt.parse(CoreCliktCommand.kt:107)
at com.github.ajalt.clikt.core.CoreCliktCommandKt.main(CoreCliktCommand.kt:78)
at com.github.ajalt.clikt.core.CoreCliktCommandKt.main(CoreCliktCommand.kt:90)
at org.ossreviewtoolkit.cli.OrtMainKt.main(OrtMain.kt:88)

Environment

Output of the ort requirements -l commands command:

09:42:44.616 [main] INFO  org.ossreviewtoolkit.model.config.OrtConfiguration - Using ORT configuration file '/config/config.yml'.
Hoplite is configured to infer which sealed type to choose by inspecting the config values at runtime. This behaviour is now deprecated in favour of explicitly specifying the type through a discriminator field. In 3.0 this new behavior will become the default. To enable this behavior now (and disable this warning), invoke withExplicitSealedTypes() on the ConfigLoaderBuilder.
09:42:44.936 [main] INFO  org.ossreviewtoolkit.utils.common.EnvironmentVariableFilter - EnvironmentVariableFilter initialized with denySubstrings = [key, pass, pwd, token, user] and allowNames = [CARGO_HTTP_USER_AGENT, COMPOSER_ALLOW_SUPERUSER, CONAN_LOGIN_ENCRYPTION_KEY, CONAN_LOGIN_USERNAME, CONAN_PASSWORD, CONAN_USERNAME, CONAN_USER_HOME, CONAN_USER_HOME_SHORT, DOTNET_CLI_CONTEXT_ANSI_PASS_THRU, GIT_ASKPASS, GIT_HTTP_USER_AGENT, GRADLE_USER_HOME, HACKAGE_USERNAME, HACKAGE_PASSWORD, HACKAGE_KEY, PWD, USER, USERPROFILE].
 ______________________________                                                
/        \_______   \__    ___/ The OSS Review Toolkit, version DOCKER-SNAPSHOT
|    |   | |       _/ |    |    built with JDK 21.0.5+11-LTS, running under Jav
|    |   | |    |   \ |    |    Executing 'requirements' as 'ort' on Linux     
\________/ |____|___/ |____|    with 16 CPUs and a maximum of 10240 MiB of memo
                                                                               
Environment variables:                                                        
ORT_CONFIG_DIR = /config                                                      
ORT_DATA_DIR = /home/ort/.ort                                                 
HOME = /home/ort                                                              
JAVA_HOME = /opt/java/openjdk                                                 
ANDROID_HOME = /opt/android-sdk                                               
                                                                              
Looking for ORT configuration in the following file:
        /config/config.yml

09:42:45.330 [main] INFO  org.reflections.Reflections - Reflections took 270 ms to scan 94 urls, producing 155 keys and 1369 values
Scanners:
        - Askalono: Requires 'askalono' in no specific version. Tool not found.
        - BoyterLc: Requires 'lc' in no specific version. Tool not found.
        - Licensee: Requires 'licensee' in no specific version. Tool not found.
09:42:45.599 [main] INFO  org.ossreviewtoolkit.utils.common.ProcessCapture - Running 'scancode --version' in '/home/ort'...
        * ScanCode: Requires 'scancode' in version >=30.0.0. Found version 32.3.1.

PackageManagers:
09:42:46.760 [main] INFO  org.ossreviewtoolkit.utils.common.ProcessCapture - Running 'bazel --version' in '/home/ort'...
        * BazelCommand: Requires 'bazel' in version >=7.0.0. Found version 7.0.1.
09:42:50.771 [main] INFO  org.ossreviewtoolkit.utils.common.ProcessCapture - Running 'bower --version' in '/home/ort'...
        * BowerCommand: Requires 'bower' in version >=1.8.8. Found version 1.8.14.
09:42:51.138 [main] INFO  org.ossreviewtoolkit.utils.common.ProcessCapture - Running 'buildozer --version' in '/home/ort'...
        + BuildozerCommand: Requires 'buildozer' in no specific version. Found version redacted.
09:42:51.159 [main] INFO  org.ossreviewtoolkit.utils.common.ProcessCapture - Running 'cargo --version' in '/home/ort'...
        * CargoCommand: Requires 'cargo' in no specific version. Found version 1.84.0.
09:42:51.243 [main] INFO  org.ossreviewtoolkit.utils.common.ProcessCapture - Running 'pod --version --allow-root' in '/home/ort'...
        * CocoaPodsCommand: Requires 'pod' in version >=1.11.0. Found version 1.16.2.
09:42:51.915 [main] INFO  org.ossreviewtoolkit.utils.common.ProcessCapture - Running 'composer --no-ansi --version' in '/home/ort'...
        * ComposerCommand: Requires 'composer' in version >=1.5.0. Found version 2.8.4.
09:42:52.000 [main] INFO  org.ossreviewtoolkit.utils.common.ProcessCapture - Running 'conan --version' in '/home/ort'...
        * ConanCommand: Requires 'conan' in version >=1.44.0 and <2.0.0. Found version 1.64.1.
09:42:52.517 [main] INFO  org.ossreviewtoolkit.utils.common.ProcessCapture - Running 'go version' in '/home/ort'...
        * GoCommand: Requires 'go' in version >=1.21.1. Found version 1.23.0.
09:42:52.553 [main] INFO  org.ossreviewtoolkit.utils.common.ProcessCapture - Running 'npm --version' in '/home/ort'...
        * NpmCommand: Requires 'npm' in version >=6.0.0 and <11.0.0. Found version 10.9.2.
09:42:52.671 [main] INFO  org.ossreviewtoolkit.utils.common.ProcessCapture - Running 'nuget-inspector --version' in '/home/ort'...
        + NuGetInspector: Requires 'nuget-inspector' in no specific version. Could not determine the version.
09:42:53.047 [main] INFO  org.ossreviewtoolkit.utils.common.ProcessCapture - Running 'pipenv --version' in '/home/ort'...
        * PipenvCommand: Requires 'pipenv' in version >=2018.10.9. Found version 2023.12.1.
09:42:53.679 [main] INFO  org.ossreviewtoolkit.utils.common.ProcessCapture - Running 'pnpm --version' in '/home/ort'...
        * PnpmCommand: Requires 'pnpm' in version >=5.0.0 and <10.0.0. Found version 9.9.0.
09:42:54.072 [main] INFO  org.ossreviewtoolkit.utils.common.ProcessCapture - Running 'poetry --version' in '/home/ort'...
        * PoetryCommand: Requires 'poetry' in no specific version. Found version 1.8.3.
09:42:54.505 [main] INFO  org.ossreviewtoolkit.utils.common.ProcessCapture - Running 'dart --version' in '/home/ort'...
        * Pub: Requires 'dart' in version >=2.10.0. Found version 2.18.4.
09:42:54.532 [main] INFO  org.ossreviewtoolkit.utils.common.ProcessCapture - Running 'python-inspector --version' in '/home/ort'...
        * PythonInspector: Requires 'python-inspector' in version >=0.9.2. Found version 0.10.0.
09:42:54.837 [main] INFO  org.ossreviewtoolkit.utils.common.ProcessCapture - Running 'sbt --version' in '/home/ort'...
        * SbtCommand: Requires 'sbt' in no specific version. Found version copying runtime jar...
sbt version in this project: 1.10.0
sbt script version: 1.10.0.
09:43:13.913 [main] INFO  org.ossreviewtoolkit.utils.common.ProcessCapture - Running 'stack --version' in '/home/ort'...
        * StackCommand: Requires 'stack' in version >=2.1.1. Found version 3.3.1.
09:43:13.953 [main] INFO  org.ossreviewtoolkit.utils.common.ProcessCapture - Running 'swift --version' in '/home/ort'...
        * SwiftCommand: Requires 'swift' in no specific version. Found version 5.10.1.
09:43:14.375 [main] INFO  org.ossreviewtoolkit.utils.common.ProcessCapture - Running 'yarn --version' in '/home/ort'...
        * YarnCommand: Requires 'yarn' in version >=1.3.0 and <1.23.0. Found version 1.22.22.

VersionControlSystems:
09:43:14.581 [main] INFO  org.ossreviewtoolkit.utils.common.ProcessCapture - Running 'git --version' in '/home/ort'...
        * Git: Requires 'git' in version >=2.29.0. Found version 2.34.1.
09:43:14.595 [main] INFO  org.ossreviewtoolkit.utils.common.ProcessCapture - Running 'repo --version' in '/home/ort'...
        * GitRepo: Requires 'repo' in no specific version. Found version 2.50 (launcher).
09:43:14.672 [main] INFO  org.ossreviewtoolkit.utils.common.ProcessCapture - Running 'hg --version' in '/home/ort'...
        * Mercurial: Requires 'hg' in no specific version. Found version 6.9.1.

Prefix legend:
        - The tool was not found in the PATH environment.
        + The tool was found in the PATH environment, but not in the required version.
        * The tool was found in the PATH environment in the required version.

ScanCode license texts found in '/opt/scancode-license-data'.

Not all tools requirements were satisfied:
        ! Some tools were not found in their required versions.

Additional context

Running with --debug it seems that this happens during some cleanup.
@Martin-Idel Martin-Idel added bug to triage Issues that need triaging labels Jan 20, 2025
@sschuberth
Copy link
Member

Good to read from you again @Martin-Idel 😉

Running 'git checkout 0136e8a05ef864666a6e95b0de8b679b0bc93338' in '/tmp/ort-DefaultWorkingTreeCache18354474298420537781'...
Exception in thread "main" java.nio.file.NoSuchFileException: /tmp/ort-DefaultWorkingTreeCache18354474298420537781/.git/gc.log.lock

I assume this could be a permission problem, i.e. the file actually is there but not accessible for whether reason. If so, the problem should appear with any (smaller) project in your specific setup. Is that something you could confirm?

@sschuberth sschuberth added scanner About the scanner tool needs info An issue where further information is required and removed to triage Issues that need triaging labels Jan 20, 2025
@Martin-Idel
Copy link
Author

Thanks @sschuberth !

Sorry for the delay, but it took me a while to run a couple of tests.
I tried it with the mime-types repository (exact same commands) and all is well, so it seems to be a more specific issue.

I have some issues with permissions, but it would be a bit weird in THIS case, because the file is in a temporary folder, why should access be messed up there? I don't have any mount point on that directory and there is no restriction on the docker container to use some space. I also checked df -h and my main partition has 140 GB available so that should also be a non-issue (hopefully :D ).

@sschuberth
Copy link
Member

Can you attach the (compressed) ORT analyzer result that you used as the input to the ort scan command to this issue?

@Martin-Idel
Copy link
Author

Sorry, I could have thought of that. Here it is:

analyzer-result.zip

@sschuberth sschuberth removed the bug label Feb 10, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
needs info An issue where further information is required scanner About the scanner tool
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@sschuberth @Martin-Idel