Tensorflow crashes

[SHITIANYU-hue]

I found that if I installed the macad-agent and run the basic_agent in macad-gym(https://github.com/praveen-palanisamy/macad-gym/blob/master/examples/basic_agent.py), there will be some issues with importing tensorflow modules. The detailed log is:

Traceback (most recent call last):
  File "/home/tianyushi/code/macad-gym/examples/basic_agent.py", line 5, in <module>
    env = gym.make("HomoNcomIndePOIntrxMASS3CTWN3-v0")
  File "/home/tianyushi/miniconda3/envs/macad-gym/lib/python3.6/site-packages/gym/envs/registration.py", line 183, in make
    return registry.make(id, **kwargs)
  File "/home/tianyushi/miniconda3/envs/macad-gym/lib/python3.6/site-packages/gym/envs/registration.py", line 125, in make
    env = spec.make(**kwargs)
  File "/home/tianyushi/miniconda3/envs/macad-gym/lib/python3.6/site-packages/gym/envs/registration.py", line 88, in make
    cls = load(self._entry_point)
  File "/home/tianyushi/miniconda3/envs/macad-gym/lib/python3.6/site-packages/gym/envs/registration.py", line 17, in load
    mod = importlib.import_module(mod_name)
  File "/home/tianyushi/miniconda3/envs/macad-gym/lib/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/tianyushi/code/macad-gym/src/macad_gym/envs/__init__.py", line 1, in <module>
    from macad_gym.carla.multi_env import MultiCarlaEnv
  File "/home/tianyushi/code/macad-gym/src/macad_gym/carla/multi_env.py", line 220, in <module>
    **from ray.rllib.env import MultiAgentEnv**
  File "/home/tianyushi/miniconda3/envs/macad-gym/lib/python3.6/site-packages/ray/rllib/__init__.py", line 11, in <module>
    from ray.rllib.evaluation.policy_graph import PolicyGraph
  File "/home/tianyushi/miniconda3/envs/macad-gym/lib/python3.6/site-packages/ray/rllib/evaluation/__init__.py", line 2, in <module>
    from ray.rllib.evaluation.policy_evaluator import PolicyEvaluator
  File "/home/tianyushi/miniconda3/envs/macad-gym/lib/python3.6/site-packages/ray/rllib/evaluation/policy_evaluator.py", line 18, in <module>
    from ray.rllib.evaluation.sampler import AsyncSampler, SyncSampler
  File "/home/tianyushi/miniconda3/envs/macad-gym/lib/python3.6/site-packages/ray/rllib/evaluation/sampler.py", line 14, in <module>
    from ray.rllib.evaluation.tf_policy_graph import TFPolicyGraph
  File "/home/tianyushi/miniconda3/envs/macad-gym/lib/python3.6/site-packages/ray/rllib/evaluation/tf_policy_graph.py", line 12, in <module>
    from ray.rllib.models.lstm import chop_into_sequences
  File "/home/tianyushi/miniconda3/envs/macad-gym/lib/python3.6/site-packages/ray/rllib/models/__init__.py", line 1, in <module>
    from ray.rllib.models.catalog import ModelCatalog, MODEL_DEFAULTS
  File "/home/tianyushi/miniconda3/envs/macad-gym/lib/python3.6/site-packages/ray/rllib/models/catalog.py", line 17, in <module>
    from ray.rllib.models.fcnet import FullyConnectedNetwork
  File "/home/tianyushi/miniconda3/envs/macad-gym/lib/python3.6/site-packages/ray/rllib/models/fcnet.py", line 6, in <module>
    import tensorflow.contrib.slim as slim
  File "/home/tianyushi/.local/lib/python3.6/site-packages/tensorflow/contrib/__init__.py", line 28, in <module>
    from tensorflow.contrib import cudnn_rnn
  File "/home/tianyushi/.local/lib/python3.6/site-packages/tensorflow/contrib/cudnn_rnn/__init__.py", line 33, in <module>
    from tensorflow.contrib.cudnn_rnn.python.ops.cudnn_rnn_ops import CudnnCompatibleGRUCell
  File "/home/tianyushi/.local/lib/python3.6/site-packages/tensorflow/contrib/cudnn_rnn/python/ops/cudnn_rnn_ops.py", line 21, in <module>
    from tensorflow.contrib.rnn.python.ops import lstm_ops
  File "/home/tianyushi/.local/lib/python3.6/site-packages/tensorflow/contrib/rnn/__init__.py", line 83, in <module>
    from tensorflow.contrib.rnn.python.ops.gru_ops import *
  File "/home/tianyushi/.local/lib/python3.6/site-packages/tensorflow/contrib/rnn/python/ops/gru_ops.py", line 33, in <module>
    resource_loader.get_path_to_datafile("_gru_ops.so"))
  File "/home/tianyushi/.local/lib/python3.6/site-packages/tensorflow/contrib/util/loader.py", line 55, in load_op_library
    ret = load_library.load_op_library(path)
  File "/home/tianyushi/.local/lib/python3.6/site-packages/tensorflow/python/framework/load_library.py", line 56, in load_op_library
    lib_handle = py_tf.TF_LoadLibrary(library_filename, status)
  File "/home/tianyushi/.local/lib/python3.6/site-packages/tensorflow/python/framework/errors_impl.py", line 473, in __exit__
    c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.InvalidArgumentError: Could not parse default value '4000' from Attr("upper_frequency_limit: float = 4000") for Op Mfcc
Could not parse default value '20' from Attr("lower_frequency_limit: float = 20") for Op Mfcc

I found a discussion related to this problem: tensorflow/tensorflow#13963

[SHITIANYU-hue]

The main problem is due to these lines in the multi_env.py

[praveen-palanisamy]

Hi @SHITIANYU-hue ,

Could you please paste your full log? There is no crash information in the log you posted.

Also, there is no tensorflow import statement in https://github.com/praveen-palanisamy/macad-gym/blob/master/examples/basic_agent.py. I wonder which line is causing the issue you linked to when you run.

[SHITIANYU-hue]

Hello, yes, there is no TensorFlow in basic_agent, but it used the import macad_gym and env = gym.make("HomoNcomIndePOIntrxMASS3CTWN3-v0"). Therefore, the multi_env will be used https://github.com/praveen-palanisamy/macad-gym/blob/master/src/macad_gym/carla/multi_env.py. And the multi_env will use the ray. So it will occur this problem

[AizazSharif]

I am facing a similar issue. Tensorflow gives the same error if I want to merge macad-agents or ray with macad-gym.

[praveen-palanisamy]

Can you post your crash log and your python environment with package version (pip freeze > pyenv.txt)?

As long as you are using the python package versions listed in the conda-env.yml file there should be no crash due to Ray or TensorFlow.

[AizazSharif]

The existing conda-env.yml contains tensorflow-gpu only, and when we install tensorflow (whether of same version or different) the code crashes once env = gym.make is called. Please find the attached files.

output.txt

pyenv.txt