[PyCUDA] Memory error when running program more than once

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

[PyCUDA] Memory error when running program more than once

ErisDyschordia
I have a program that I am running written in PyCuda, that works correctly the first time I run it. Because I am using a lot of fast little gpuarray functions, on a TON of different arrays, every time I finish with one I manually remove it from the memory using array.gpudata.free() so that I immediately have enough room to do the next array. I do this every time. So every single array is manually freed. The program will complete successfully, but if I try to run it a second time, the moment I try to move anything over to the GPU it says CudaMalloc fail: out of memory.

Any idea what could be causing this? the only way I have found to "fix" it, is to restart the computer between runs.
Reply | Threaded
Open this post in threaded view
|

Re: Memory error when running program more than once

Andreas Kloeckner
On Thu, 24 May 2012 09:57:38 -0700 (PDT), ErisDyschordia <[hidden email]> wrote:

> I have a program that I am running written in PyCuda, that works correctly
> the first time I run it. Because I am using a lot of fast little gpuarray
> functions, on a TON of different arrays, every time I finish with one I
> manually remove it from the memory using array.gpudata.free() so that I
> immediately have enough room to do the next array. I do this every time. So
> every single array is manually freed. The program will complete
> successfully, but if I try to run it a second time, the moment I try to move
> anything over to the GPU it says CudaMalloc fail: out of memory.
>
> Any idea what could be causing this? the only way I have found to "fix" it,
> is to restart the computer between runs.
Once the first run finishes, can you still find the program using 'ps
aux' (I'm assuming you're on Linux?)? If not, this is very likely a
CUDA bug.

Further, "CudaMalloc fail" indicates that something besides PyCUDA is
complaining. PyCUDA does not use cudaMalloc. (It uses cuMalloc--welcome
to the wonderful world of CUDA.)

Also, if, when you said 'program', you actually meant two things that
happen within the same (OS) process, and if you're using memory pools,
then what you're reporting has every right to happen. A memory pool,
unless told otherwise, *will* hold on to all the device memory that was
once allocated through it.

http://documen.tician.de/pycuda/util.html#pycuda.tools.DeviceMemoryPool.free_held

HTH,
Andreas

_______________________________________________
PyCUDA mailing list
[hidden email]
http://lists.tiker.net/listinfo/pycuda

attachment0 (195 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Memory error when running program more than once

ErisDyschordia
Adding the free held memory command to the end of my program solved the problem, I clearly don't have a great grasp on how memory is stored. Thanks!