Skip to content

gpuarray crashes with ARM Mali GPU #581

Open
@Kishwar

Description

@Kishwar

Hello Theano team,

I am trying to load Theano/Keras model, I am getting crash on gpuarray. It happens also even when I use GPU for training. Same code works OK for CPU.

odroid@odroid:~/Desktop/share/thMNIST$ sudo THEANO_FLAGS=device=opencl0:1 python3 th__mnist.py
Using Theano backend.
Mapped name None to device opencl0:1: Mali-T628

Traceback (most recent call last):
File "th__mnist.py", line 50, in
model.fit(X_train, y_train, validation_data=(X_test, y_test), epochs=10, batch_size=200)
File "/usr/local/lib/python3.6/dist-packages/keras/engine/training.py", line 1010, in fit
self._make_train_function()
File "/usr/local/lib/python3.6/dist-packages/keras/engine/training.py", line 519, in _make_train_function
**self._function_kwargs)
File "/usr/local/lib/python3.6/dist-packages/keras/backend/theano_backend.py", line 1397, in function
return Function(inputs, outputs, updates=updates, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/keras/backend/theano_backend.py", line 1383, in init
**kwargs)
File "/usr/local/lib/python3.6/dist-packages/theano/compile/function.py", line 317, in function
output_keys=output_keys)
File "/usr/local/lib/python3.6/dist-packages/theano/compile/pfunc.py", line 486, in pfunc
output_keys=output_keys)
File "/usr/local/lib/python3.6/dist-packages/theano/compile/function_module.py", line 1841, in orig_function
fn = m.create(defaults)
File "/usr/local/lib/python3.6/dist-packages/theano/compile/function_module.py", line 1715, in create
input_storage=input_storage_lists, storage_map=storage_map)
File "/usr/local/lib/python3.6/dist-packages/theano/gof/link.py", line 699, in make_thunk
storage_map=storage_map)[:3]
File "/usr/local/lib/python3.6/dist-packages/theano/gof/vm.py", line 1091, in make_all
impl=impl))
File "/usr/local/lib/python3.6/dist-packages/theano/gof/op.py", line 952, in make_thunk
compute_map=compute_map, impl='c')
File "/usr/local/lib/python3.6/dist-packages/theano/gpuarray/elemwise.py", line 2519, in prepare_node
self.get_kernel_cache(node)
File "/usr/local/lib/python3.6/dist-packages/theano/gpuarray/elemwise.py", line 2533, in get_kernel_cache
redux))
File "/usr/local/lib/python3.6/dist-packages/theano/gpuarray/elemwise.py", line 2728, in generate_kernel
init_nd=node.inputs[0].ndim)
File "/usr/local/lib/python3.6/dist-packages/pygpu-0.7.6+5.g8786e0f-py3.6-linux-armv7l.egg/pygpu/reduction.py", line 194, in init
self.init_local_size = min(context.lmemsize //
File "pygpu/gpuarray.pyx", line 1109, in pygpu.gpuarray.GpuContext.lmemsize.get
File "pygpu/gpuarray.pyx", line 487, in pygpu.gpuarray.ctx_property
pygpu.gpuarray.GpuArrayException: ('The following error happened while compiling the node', GpuReduce{add}(GpuElemwise{Composite{Cast{float32}(NEQ(i0, i1))}}[].0), '\n', b'clGetDeviceInfo(id, CL_DEVICE_LOCAL_MEM_SIZE, sizeof(sz), &sz, NULL): Invalid value')

Could you please help me on this?

Thanks.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions