[PyCUDA] mpi4py and GPUArray in Python 3

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

[PyCUDA] mpi4py and GPUArray in Python 3

Yiyin Zhou
Hi,

I ran into issues when running the following gist written by Lev.
Basically the code works fine under Python 2.7 but I got the following error in Python 3.6 (both with openmpi 3.0.0 and mpi4py 3.0.0).

Traceback (most recent call last):
  File "mpi4py_pycuda_demo.py", line 64, in <module>
    comm.Recv([bufint(x_gpu), dtype_to_mpi(x_gpu.dtype)], source=0)
  File "mpi4py/MPI/Comm.pyx", line 283, in mpi4py.MPI.Comm.Recv
  File "mpi4py/MPI/msgbuffer.pxi", line 402, in mpi4py.MPI.message_p2p_recv
  File "mpi4py/MPI/msgbuffer.pxi", line 388, in mpi4py.MPI._p_msg_p2p.for_recv
  File "mpi4py/MPI/msgbuffer.pxi", line 155, in mpi4py.MPI.message_simple
  File "mpi4py/MPI/msgbuffer.pxi", line 94, in mpi4py.MPI.message_basic
  File "mpi4py/MPI/asbuffer.pxi", line 308, in mpi4py.MPI.getbuffer
  File "mpi4py/MPI/asbuffer.pxi", line 135, in mpi4py.MPI.PyMPI_GetBuffer
BufferError: memoryview: underlying buffer is not writable

I am using 2017.1.1, so presumably the as_buffer method should already return a writable buffer from a fix two years ago. Any suggestion is appreciated!

Thanks,
Yiyin



_______________________________________________
PyCUDA mailing list
[hidden email]
https://lists.tiker.net/listinfo/pycuda
Reply | Threaded
Open this post in threaded view
|

Re: mpi4py and GPUArray in Python 3

Yiyin Zhou
It seems that I was looking at a wrong line and referred to the as_buffer method for the pointer_holder_base class that was fixed.
The same fix is needed for the device_allocation class as well, I believe:

Yiyin

On Thu, Jun 7, 2018 at 4:53 PM, Yiyin Zhou <[hidden email]> wrote:
Hi,

I ran into issues when running the following gist written by Lev.
Basically the code works fine under Python 2.7 but I got the following error in Python 3.6 (both with openmpi 3.0.0 and mpi4py 3.0.0).

Traceback (most recent call last):
  File "mpi4py_pycuda_demo.py", line 64, in <module>
    comm.Recv([bufint(x_gpu), dtype_to_mpi(x_gpu.dtype)], source=0)
  File "mpi4py/MPI/Comm.pyx", line 283, in mpi4py.MPI.Comm.Recv
  File "mpi4py/MPI/msgbuffer.pxi", line 402, in mpi4py.MPI.message_p2p_recv
  File "mpi4py/MPI/msgbuffer.pxi", line 388, in mpi4py.MPI._p_msg_p2p.for_recv
  File "mpi4py/MPI/msgbuffer.pxi", line 155, in mpi4py.MPI.message_simple
  File "mpi4py/MPI/msgbuffer.pxi", line 94, in mpi4py.MPI.message_basic
  File "mpi4py/MPI/asbuffer.pxi", line 308, in mpi4py.MPI.getbuffer
  File "mpi4py/MPI/asbuffer.pxi", line 135, in mpi4py.MPI.PyMPI_GetBuffer
BufferError: memoryview: underlying buffer is not writable

I am using 2017.1.1, so presumably the as_buffer method should already return a writable buffer from a fix two years ago. Any suggestion is appreciated!

Thanks,
Yiyin




_______________________________________________
PyCUDA mailing list
[hidden email]
https://lists.tiker.net/listinfo/pycuda
Reply | Threaded
Open this post in threaded view
|

Re: mpi4py and GPUArray in Python 3

Freddie Witherden
In reply to this post by Yiyin Zhou
On 07/06/18 21:53, Yiyin Zhou wrote:

> I ran into issues when running the following gist written by Lev.
> https://gist.github.com/lebedov/8514d3456a94a6c73e6d#file-mpi4py_pycuda_demo-py
> Basically the code works fine under Python 2.7 but I got the following
> error in Python 3.6 (both with openmpi 3.0.0 and mpi4py 3.0.0).
>
> Traceback (most recent call last):
>   File "mpi4py_pycuda_demo.py", line 64, in <module>
>     comm.Recv([bufint(x_gpu), dtype_to_mpi(x_gpu.dtype)], source=0)
>   File "mpi4py/MPI/Comm.pyx", line 283, in mpi4py.MPI.Comm.Recv
>   File "mpi4py/MPI/msgbuffer.pxi", line 402, in mpi4py.MPI.message_p2p_recv
>   File "mpi4py/MPI/msgbuffer.pxi", line 388, in
> mpi4py.MPI._p_msg_p2p.for_recv
>   File "mpi4py/MPI/msgbuffer.pxi", line 155, in mpi4py.MPI.message_simple
>   File "mpi4py/MPI/msgbuffer.pxi", line 94, in mpi4py.MPI.message_basic
>   File "mpi4py/MPI/asbuffer.pxi", line 308, in mpi4py.MPI.getbuffer
>   File "mpi4py/MPI/asbuffer.pxi", line 135, in mpi4py.MPI.PyMPI_GetBuffer
> BufferError: memoryview: underlying buffer is not writable
>
> I am using 2017.1.1, so presumably the as_buffer method should already
> return a writable buffer from a fix two years ago. Any suggestion is
> appreciated!
The fix was incomplete.  If I recall there are two places where it need
be applied, and it somehow only ended up being fixed in one of them.

As a workaround:

from ctypes import c_int, c_ssize_t, c_void_p, pythonapi, py_object

_make_pybuf = pythonapi.PyMemoryView_FromMemory
_make_pybuf.argtypes = [c_void_p, c_ssize_t, c_int]
_make_pybuf.restype = py_object

buf = _make_pybuf(my_alloc, nbytes, 0x200)

Regards, Freddie.

>
> Thanks,
> Yiyin
>
>
>
>
> _______________________________________________
> PyCUDA mailing list
> [hidden email]
> https://lists.tiker.net/listinfo/pycuda
>


_______________________________________________
PyCUDA mailing list
[hidden email]
https://lists.tiker.net/listinfo/pycuda

signature.asc (849 bytes) Download Attachment