Description
Hello Theano team,
I am trying to load Theano/Keras model, I am getting crash on gpuarray. It happens also even when I use GPU for training. Same code works OK for CPU.
odroid@odroid:~/Desktop/share/thMNIST$ sudo THEANO_FLAGS=device=opencl0:1 python3 th__mnist.py
Using Theano backend.
Mapped name None to device opencl0:1: Mali-T628
Traceback (most recent call last):
File "th__mnist.py", line 50, in
model.fit(X_train, y_train, validation_data=(X_test, y_test), epochs=10, batch_size=200)
File "/usr/local/lib/python3.6/dist-packages/keras/engine/training.py", line 1010, in fit
self._make_train_function()
File "/usr/local/lib/python3.6/dist-packages/keras/engine/training.py", line 519, in _make_train_function
**self._function_kwargs)
File "/usr/local/lib/python3.6/dist-packages/keras/backend/theano_backend.py", line 1397, in function
return Function(inputs, outputs, updates=updates, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/keras/backend/theano_backend.py", line 1383, in init
**kwargs)
File "/usr/local/lib/python3.6/dist-packages/theano/compile/function.py", line 317, in function
output_keys=output_keys)
File "/usr/local/lib/python3.6/dist-packages/theano/compile/pfunc.py", line 486, in pfunc
output_keys=output_keys)
File "/usr/local/lib/python3.6/dist-packages/theano/compile/function_module.py", line 1841, in orig_function
fn = m.create(defaults)
File "/usr/local/lib/python3.6/dist-packages/theano/compile/function_module.py", line 1715, in create
input_storage=input_storage_lists, storage_map=storage_map)
File "/usr/local/lib/python3.6/dist-packages/theano/gof/link.py", line 699, in make_thunk
storage_map=storage_map)[:3]
File "/usr/local/lib/python3.6/dist-packages/theano/gof/vm.py", line 1091, in make_all
impl=impl))
File "/usr/local/lib/python3.6/dist-packages/theano/gof/op.py", line 952, in make_thunk
compute_map=compute_map, impl='c')
File "/usr/local/lib/python3.6/dist-packages/theano/gpuarray/elemwise.py", line 2519, in prepare_node
self.get_kernel_cache(node)
File "/usr/local/lib/python3.6/dist-packages/theano/gpuarray/elemwise.py", line 2533, in get_kernel_cache
redux))
File "/usr/local/lib/python3.6/dist-packages/theano/gpuarray/elemwise.py", line 2728, in generate_kernel
init_nd=node.inputs[0].ndim)
File "/usr/local/lib/python3.6/dist-packages/pygpu-0.7.6+5.g8786e0f-py3.6-linux-armv7l.egg/pygpu/reduction.py", line 194, in init
self.init_local_size = min(context.lmemsize //
File "pygpu/gpuarray.pyx", line 1109, in pygpu.gpuarray.GpuContext.lmemsize.get
File "pygpu/gpuarray.pyx", line 487, in pygpu.gpuarray.ctx_property
pygpu.gpuarray.GpuArrayException: ('The following error happened while compiling the node', GpuReduce{add}(GpuElemwise{Composite{Cast{float32}(NEQ(i0, i1))}}[].0), '\n', b'clGetDeviceInfo(id, CL_DEVICE_LOCAL_MEM_SIZE, sizeof(sz), &sz, NULL): Invalid value')
Could you please help me on this?
Thanks.