Skip to content

Versions of CUDA, cuDNN, and TensorFlow all need to be compatible #22

@uschmidt83

Description

@uschmidt83

Hi,

the plugin currently uses an old TensorFlow (TF) version (1.6.0), which is incompatible with the installation requirements of more recent TF versions. Specifically, TF currently requires cuDNN >= 7.2 to be installed, which is too new for TF 1.6.0 (see error below).

Furthermore, installation of cuDNN is currently not mentioned in the documentation.

Best,
Uwe

$ ./ImageJ-linux64 --java-home /sw/apps/jdk/current
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option PermSize=128m; support was removed in 8.0
Java HotSpot(TM) 64-Bit Server VM warning: Using incremental CMS is deprecated and will likely be removed in a future release
[INFO] imagej-tensorflow version: 1.0.1
[INFO] tensorflow version: 1.6.0
[INFO] The current library path is: LD_LIBRARY_PATH=/sw/apps/cuda/9.0.176/lib64:/sw/apps/cuda/9.0.176/lib:/home/uschmidt/sw/local/lib:/home/uschmidt/tmp/new_fiji/Fiji.app/lib/linux64:/home/uschmidt/tmp/new_fiji/Fiji.app/mm/linux64
[INFO] loading model net_tubulin from source http://csbdeep.bioimagecomputing.com/model-tubulin.zip
[INFO] TensorFlow model cache: /home/uschmidt/tmp/new_fiji/Fiji.app/models
2018-11-19 16:19:18.450296: I tensorflow/core/platform/cpu_feature_guard.cc:140] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX
2018-11-19 16:19:18.451773: I tensorflow/cc/saved_model/loader.cc:240] Loading SavedModel with tags: { serve }; from: /home/uschmidt/tmp/new_fiji/Fiji.app/models/net_tubulin
2018-11-19 16:19:18.631059: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:898] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2018-11-19 16:19:18.631955: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1212] Found device 0 with properties:
name: GeForce GTX TITAN X major: 5 minor: 2 memoryClockRate(GHz): 1.076
pciBusID: 0000:05:00.0
totalMemory: 11.93GiB freeMemory: 11.71GiB
2018-11-19 16:19:18.632026: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1312] Adding visible gpu devices: 0
2018-11-19 16:22:32.125101: I tensorflow/core/common_runtime/gpu/gpu_device.cc:993] Creating TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 11339 MB memory) -> physical GPU (device: 0, name: GeForce GTX TITAN X, pci bus id: 0000:05:00.0, compute capability: 5.2)
2018-11-19 16:22:32.283239: I tensorflow/cc/saved_model/loader.cc:159] Restoring SavedModel bundle.
2018-11-19 16:22:32.329594: I tensorflow/cc/saved_model/loader.cc:194] Running LegacyInitOp on SavedModel bundle.
2018-11-19 16:22:32.330448: I tensorflow/cc/saved_model/loader.cc:289] SavedModel load for tags { serve }; Status: success. Took 193878674 microseconds.
[INFO] Shape of input tensor: [-1, -1, -1, 1]
[INFO] Shape of output tensor: [-1, -1, -1, 2]
datasetAxes:[X, Y]
nodeAxes:[Time, Y, X, Channel]
mapping:[2, 1, 0, 3]
--------------
[INFO] Normalize ..
[INFO] Dataset type: 32-bit signed float, converting to FloatType.
[INFO] Dataset dimensions: [720, 576]
[INFO] INPUT NODE:
datasetAxes:[X, Y]
nodeAxes:[Time, Y, X, Channel]
mapping:[2, 1, 0, 3]
--------------
[INFO] OUTPUT NODE:
datasetAxes:[X, Y]
nodeAxes:[Time, Y, X, Channel]
mapping:[2, 1, 0, 3]
--------------
[INFO] Dividing image into 1 tile(s)..
[INFO] Size of single image tile: [720, 576]
[INFO] Final image tiling: [1, 1]
[INFO] Network input size: [720, 576]
[INFO] Processing tile 1..
2018-11-19 16:22:33.661466: E tensorflow/stream_executor/cuda/cuda_dnn.cc:378] Loaded runtime CuDNN library: 7104 (compatibility version 7100) but source was compiled with 7004 (compatibility version 7000).  If using a binary install, upgrade your CuDNN library to match.  If building from sources, make sure the library loaded at runtime matches a compatible version specified during compile configuration.
2018-11-19 16:22:33.662665: F tensorflow/core/kernels/conv_ops.cc:717] Check failed: stream->parent()->GetConvolveAlgorithms( conv_parameters.ShouldIncludeWinogradNonfusedAlgo<T>(), &algorithms)
[1]    28406 abort (core dumped)  ./ImageJ-linux64 --java-home /sw/apps/jdk/current

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions