GPU Out of memory

[gaushi]

Hi CJ and Andy,
I ran into out of memory issue with GPU while running bootstraps.
Here is a portion of the log:
2021-11-26 16:42:28.228503: W tensorflow/core/framework/op_kernel.cc:1745] OP_REQUIRES failed at matmul_op_impl.h:681 : RESOURCE_EXHAUSTED: OOM when allocating tensor with shape[387840,256] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc

Could you please suggest a solution? When I run with different seeds (and no bootstraps) Locator works just fine.

Thanks,
Best,
Gautam

[andrewkern]

Hi @gaushi -- huh that's strange. it seems to me like a potential memory leak in either our code or tensorflow itself. do you have a minimal working example that you could share so that we can debug this?

[andrewkern]

also it would be helpful to know the versions of everything you have installed-- pip list should do it