[PyCUDA] Create gpuarrays on different GPUs

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

[PyCUDA] Create gpuarrays on different GPUs

Zhangsheng Lai
Hi,

I'm trying to create different GPU arrays on different GPUs.

```
import pycuda
import pycuda.driver as cuda
from pycuda.compiler import SourceModule
import pycuda.curandom as curandom

d = 2 ** 15

cuda.init()
dev1 = cuda.Device(1)
ctx1 = dev1.make_context()

curng1 = curandom.XORWOWRandomNumberGenerator()

x1 = curng1.gen_normal((d,d), dtype = np.float32) # so x1 is stored in GPU 1 memory

ctx1.pop() # clearing ctx of GPU1

dev2 = cuda.Device(1)
ctx2 = dev2.make_context()

curng2 = curandom.XORWOWRandomNumberGenerator()

x2 = curng2.gen_normal((d,d), dtype = np.float32) # so x2 is stored in GPU 2

```

with the setup above, I tried to check by poping ctx2 and pushing ctx1, can I access x1 and not x2 and vice versa, popping ctx1 and pushing ctx2, I can access x2 and not x1. However, I realise that I can access x1 and x2 in both contexts. 

Thus I'm wondering if my assumptions of x1 stored in GPU1 and x2 stored in GPU2 are correct, or if it is actually the UVA and peer access that allows me to access both x1 and x2 even if only one of the two ctx is active.

Thanks,
Zhangsheng

_______________________________________________
PyCUDA mailing list
[hidden email]
https://lists.tiker.net/listinfo/pycuda
Reply | Threaded
Open this post in threaded view
|

Re: Create gpuarrays on different GPUs

Andreas Kloeckner
Zhangsheng Lai <[hidden email]> writes:
> with the setup above, I tried to check by poping ctx2 and pushing ctx1, can
> I access x1 and not x2 and vice versa, popping ctx1 and pushing ctx2, I can
> access x2 and not x1. However, I realise that I can access x1 and x2 in
> both contexts.

Can you clarify what you mean by 'can access'? I'm guessing 'submit
kernel launches with that pointer as an argument'?

Andreas

_______________________________________________
PyCUDA mailing list
[hidden email]
https://lists.tiker.net/listinfo/pycuda
Reply | Threaded
Open this post in threaded view
|

Re: Create gpuarrays on different GPUs

Zhangsheng Lai
My 'can access' simply means that I'm able to access the values in the variable in python by typing x1 or x2. My understanding is that if the variables are stored on different GPUs, then I should be able to type x1 and get its values when ctx1 is active and similarly, I can type x2 and get the x2 values when ctx2 is active, not when ctx1 is active.

On 24 May 2018 at 18:56, Andreas Kloeckner <[hidden email]> wrote:
Zhangsheng Lai <[hidden email]> writes:
> with the setup above, I tried to check by poping ctx2 and pushing ctx1, can
> I access x1 and not x2 and vice versa, popping ctx1 and pushing ctx2, I can
> access x2 and not x1. However, I realise that I can access x1 and x2 in
> both contexts.

Can you clarify what you mean by 'can access'? I'm guessing 'submit
kernel launches with that pointer as an argument'?

Andreas


_______________________________________________
PyCUDA mailing list
[hidden email]
https://lists.tiker.net/listinfo/pycuda
Reply | Threaded
Open this post in threaded view
|

Re: Create gpuarrays on different GPUs

Andreas Kloeckner
Zhangsheng Lai <[hidden email]> writes:
> My 'can access' simply means that I'm able to access the values in the
> variable in python by typing x1 or x2. My understanding is that if the
> variables are stored on different GPUs, then I should be able to type x1
> and get its values when ctx1 is active and similarly, I can type x2 and get
> the x2 values when ctx2 is active, not when ctx1 is active.

You could measure bandwidths to between host/presumed gpu1/presumed gpu2
to ascertain where the data actually resides if you have doubts about
that/don't trust the API.

Andreas

_______________________________________________
PyCUDA mailing list
[hidden email]
https://lists.tiker.net/listinfo/pycuda