[PyCUDA] import pycuda.driver fails

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

[PyCUDA] import pycuda.driver fails

Harshit Suri
Hello Everyone,

I had a working installation of pycuda. However, after running updates on my Ubuntu machine; 
import pycuda.driver as cuda fails. 
( I had also updated my anaconda install and updated all packages that anaconda found that required updates )

When I try running " import pycuda.driver " through jupyter notebook 
I get the following error message. Followed by a kernel crash
--------------------------------------------------------------------------
[I 10:02:37.765 NotebookApp] Adapting to protocol v5.1 for kernel 5fee93d7-62cf-468a-bcb0-b48e6f1d0987
terminate called after throwing an instance of 'std::runtime_error'
  what():  numpy failed to initialize
[I 10:02:43.117 NotebookApp] KernelRestarter: restarting kernel (1/5), keep random ports
kernel 5fee93d7-62cf-468a-bcb0-b48e6f1d0987 restarted
--------------------------------------------------------------------------
It seems numpy fails to initialize.
However if I try to import numpy independently, it passes successfully. 

My guess is that when I ran the updates it updated something that caused this. The installation for pycuda was done by "pip install pycuda"

I even tried uninstalling via pip uninstall pycuda followed by pip install pycuda. But I still get the same error. I though this might force pycuda to rebuild with the newer versions of python/numpy. 

Current installation versions:
python is 3.6
numpy is 1.13.3
pycuda is 2017.1.1
gcc 4.8.4
ubuntu 14.04.4

I have tried searching through the archives for a solution to this. So I hope someone will be able to help.

Thanks
-Suri


_______________________________________________
PyCUDA mailing list
[hidden email]
https://lists.tiker.net/listinfo/pycuda
Reply | Threaded
Open this post in threaded view
|

Re: import pycuda.driver fails

Andreas Kloeckner
Harshit,

Harshit Suri <[hidden email]> writes:

> I had a working installation of pycuda. However, after running updates on
> my Ubuntu machine;
> import pycuda.driver as cuda fails.
> ( I had also updated my anaconda install and updated all packages that
> anaconda found that required updates )
>
> When I try running " import pycuda.driver " through jupyter notebook
> I get the following error message. Followed by a kernel crash
> --------------------------------------------------------------------------
> [I 10:02:37.765 NotebookApp] Adapting to protocol v5.1 for kernel
> 5fee93d7-62cf-468a-bcb0-b48e6f1d0987
> terminate called after throwing an instance of 'std::runtime_error'
>   what():  numpy failed to initialize
> [I 10:02:43.117 NotebookApp] KernelRestarter: restarting kernel (1/5), keep
> random ports
> kernel 5fee93d7-62cf-468a-bcb0-b48e6f1d0987 restarted
> --------------------------------------------------------------------------
> It seems numpy fails to initialize.
> However if I try to import numpy independently, it passes successfully.
>
> My guess is that when I ran the updates it updated something that caused
> this. The installation for pycuda was done by "pip install pycuda"

My guess would be Nvidia kernel/driver mismatch. Check the end of dmesg,
there might be a message there.

Andreas

_______________________________________________
PyCUDA mailing list
[hidden email]
https://lists.tiker.net/listinfo/pycuda
Reply | Threaded
Open this post in threaded view
|

Re: import pycuda.driver fails

Harshit Suri
Thank you for the reply Andreas,

The NVIDIA driver was definitely updated. (  it was on the list of items in the Ubuntu "software updater" prior to running the updates ).
I have included the lines of dmesg corresponding to NVIDIA below, after trying to rerun the import pycuda.driver in jupyter notebook.

-I am able to run my regular CUDA-C/C++ code binaries successfully after the update: Based on this am I correct in assuming that this is not necessarily a Nvidia kernel/driver mismatch issue? 

-Is there a way I can start from scratch and force pycuda to rebuild the installation? 

-Should I try to go to an older NVIDIA driver binary?

Thanks

output of dmesg
======================================================================================
[    2.523799] nvidia: loading out-of-tree module taints kernel.
[    2.523807] nvidia: module license 'NVIDIA' taints kernel.
[    2.523808] Disabling lock debugging due to kernel taint
[    2.534215] nvidia: module verification failed: signature and/or required key missing - tainting kernel
[    2.541075] systemd-udevd[682]: failed to execute '/bin/systemctl' '/bin/systemctl start --no-block nvidia-persistenced.service': No such file or directory
[    2.541164] nvidia-nvlink: Nvlink Core is being initialized, major device number 242
[    2.541358] nvidia 0000:05:00.0: enabling device (0100 -> 0103)
[    2.541415] vgaarb: device changed decodes: PCI:0000:05:00.0,olddecodes=io+mem,decodes=none:owns=none
[    2.541514] vgaarb: device changed decodes: PCI:0000:04:00.0,olddecodes=io+mem,decodes=none:owns=io+mem
[    2.541584] NVRM: loading NVIDIA UNIX x86_64 Kernel Module  384.130  Wed Mar 21 03:37:26 PDT 2018 (using threaded interrupts)
[    2.575938] nvidia-modeset: Loading NVIDIA Kernel Mode Setting Driver for UNIX platforms  384.130  Wed Mar 21 02:59:49 PDT 2018
[    2.576962] [drm] [nvidia-drm] [GPU ID 0x00000500] Loading driver
[    2.577030] [drm] [nvidia-drm] [GPU ID 0x00000400] Loading driver
[    2.590360] clocksource: Switched to clocksource tsc
[    2.606251] init: failsafe main process (693) killed by TERM signal
[    2.616069] random: dbus-daemon: uninitialized urandom read (12 bytes read, 124 bits of entropy available)
[    2.618449] random: dbus-daemon: uninitialized urandom read (12 bytes read, 124 bits of entropy available)
[    2.635743] nvidia-uvm: Loaded the UVM driver in 8 mode, major device number 241
[    2.674524] random: nonblocking pool is initialized
[    2.678916] systemd-udevd[776]: failed to execute '/bin/systemctl' '/bin/systemctl start --no-block nvidia-persistenced.service': No such file or directory

[    7.507516] nvidia-modeset: Allocated GPU:1 (GPU-debe2b79-7d7c-b48e-e93e-92f5881136ab) @ PCI:0000:05:00.0
[   12.010315] tty_warn_deprecated_flags: 'ModemManager' is using deprecated serial flags (with no effect): 00008000
[   32.726283] audit_printk_skb: 51 callbacks suppressed
[   32.726285] audit: type=1400 audit(1531765181.557:28): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="/usr/lib/cups/backend/cups-pdf" pid=2416 comm="apparmor_parser"
[   32.726290] audit: type=1400 audit(1531765181.557:29): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="/usr/sbin/cupsd" pid=2416 comm="apparmor_parser"
======================================================================================

On Mon, Jul 16, 2018 at 10:59 AM Andreas Kloeckner <[hidden email]> wrote:
Harshit,

Harshit Suri <[hidden email]> writes:
> I had a working installation of pycuda. However, after running updates on
> my Ubuntu machine;
> import pycuda.driver as cuda fails.
> ( I had also updated my anaconda install and updated all packages that
> anaconda found that required updates )
>
> When I try running " import pycuda.driver " through jupyter notebook
> I get the following error message. Followed by a kernel crash
> --------------------------------------------------------------------------
> [I 10:02:37.765 NotebookApp] Adapting to protocol v5.1 for kernel
> 5fee93d7-62cf-468a-bcb0-b48e6f1d0987
> terminate called after throwing an instance of 'std::runtime_error'
>   what():  numpy failed to initialize
> [I 10:02:43.117 NotebookApp] KernelRestarter: restarting kernel (1/5), keep
> random ports
> kernel 5fee93d7-62cf-468a-bcb0-b48e6f1d0987 restarted
> --------------------------------------------------------------------------
> It seems numpy fails to initialize.
> However if I try to import numpy independently, it passes successfully.
>
> My guess is that when I ran the updates it updated something that caused
> this. The installation for pycuda was done by "pip install pycuda"

My guess would be Nvidia kernel/driver mismatch. Check the end of dmesg,
there might be a message there.

Andreas

_______________________________________________
PyCUDA mailing list
[hidden email]
https://lists.tiker.net/listinfo/pycuda
Reply | Threaded
Open this post in threaded view
|

Re: import pycuda.driver fails

Harshit Suri
I was able to resolve this. I am writing the steps in case it is helpful to someone in the future...

Basically I uninstalled pycuda 
$ pip uninstall pycuda

I downloaded the sources from https://pypi.org/project/pycuda/#files (pycuda .tar file)

Then I ran steps 1 and 3 from https://wiki.tiker.net/PyCuda/Installation/Linux (skipped step 2, which installs numpy, because it was already installed)

I guess pycuda needed to be rebuilt from sources after all the updates to my Ubuntu machine were done.
Thanks for your time Andreas.


On Mon, Jul 16, 2018 at 11:50 AM Harshit Suri <[hidden email]> wrote:
Thank you for the reply Andreas,

The NVIDIA driver was definitely updated. (  it was on the list of items in the Ubuntu "software updater" prior to running the updates ).
I have included the lines of dmesg corresponding to NVIDIA below, after trying to rerun the import pycuda.driver in jupyter notebook.

-I am able to run my regular CUDA-C/C++ code binaries successfully after the update: Based on this am I correct in assuming that this is not necessarily a Nvidia kernel/driver mismatch issue? 

-Is there a way I can start from scratch and force pycuda to rebuild the installation? 

-Should I try to go to an older NVIDIA driver binary?

Thanks

output of dmesg
======================================================================================
[    2.523799] nvidia: loading out-of-tree module taints kernel.
[    2.523807] nvidia: module license 'NVIDIA' taints kernel.
[    2.523808] Disabling lock debugging due to kernel taint
[    2.534215] nvidia: module verification failed: signature and/or required key missing - tainting kernel
[    2.541075] systemd-udevd[682]: failed to execute '/bin/systemctl' '/bin/systemctl start --no-block nvidia-persistenced.service': No such file or directory
[    2.541164] nvidia-nvlink: Nvlink Core is being initialized, major device number 242
[    2.541358] nvidia 0000:05:00.0: enabling device (0100 -> 0103)
[    2.541415] vgaarb: device changed decodes: PCI:0000:05:00.0,olddecodes=io+mem,decodes=none:owns=none
[    2.541514] vgaarb: device changed decodes: PCI:0000:04:00.0,olddecodes=io+mem,decodes=none:owns=io+mem
[    2.541584] NVRM: loading NVIDIA UNIX x86_64 Kernel Module  384.130  Wed Mar 21 03:37:26 PDT 2018 (using threaded interrupts)
[    2.575938] nvidia-modeset: Loading NVIDIA Kernel Mode Setting Driver for UNIX platforms  384.130  Wed Mar 21 02:59:49 PDT 2018
[    2.576962] [drm] [nvidia-drm] [GPU ID 0x00000500] Loading driver
[    2.577030] [drm] [nvidia-drm] [GPU ID 0x00000400] Loading driver
[    2.590360] clocksource: Switched to clocksource tsc
[    2.606251] init: failsafe main process (693) killed by TERM signal
[    2.616069] random: dbus-daemon: uninitialized urandom read (12 bytes read, 124 bits of entropy available)
[    2.618449] random: dbus-daemon: uninitialized urandom read (12 bytes read, 124 bits of entropy available)
[    2.635743] nvidia-uvm: Loaded the UVM driver in 8 mode, major device number 241
[    2.674524] random: nonblocking pool is initialized
[    2.678916] systemd-udevd[776]: failed to execute '/bin/systemctl' '/bin/systemctl start --no-block nvidia-persistenced.service': No such file or directory

[    7.507516] nvidia-modeset: Allocated GPU:1 (GPU-debe2b79-7d7c-b48e-e93e-92f5881136ab) @ PCI:0000:05:00.0
[   12.010315] tty_warn_deprecated_flags: 'ModemManager' is using deprecated serial flags (with no effect): 00008000
[   32.726283] audit_printk_skb: 51 callbacks suppressed
[   32.726285] audit: type=1400 audit(1531765181.557:28): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="/usr/lib/cups/backend/cups-pdf" pid=2416 comm="apparmor_parser"
[   32.726290] audit: type=1400 audit(1531765181.557:29): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="/usr/sbin/cupsd" pid=2416 comm="apparmor_parser"
======================================================================================

On Mon, Jul 16, 2018 at 10:59 AM Andreas Kloeckner <[hidden email]> wrote:
Harshit,

Harshit Suri <[hidden email]> writes:
> I had a working installation of pycuda. However, after running updates on
> my Ubuntu machine;
> import pycuda.driver as cuda fails.
> ( I had also updated my anaconda install and updated all packages that
> anaconda found that required updates )
>
> When I try running " import pycuda.driver " through jupyter notebook
> I get the following error message. Followed by a kernel crash
> --------------------------------------------------------------------------
> [I 10:02:37.765 NotebookApp] Adapting to protocol v5.1 for kernel
> 5fee93d7-62cf-468a-bcb0-b48e6f1d0987
> terminate called after throwing an instance of 'std::runtime_error'
>   what():  numpy failed to initialize
> [I 10:02:43.117 NotebookApp] KernelRestarter: restarting kernel (1/5), keep
> random ports
> kernel 5fee93d7-62cf-468a-bcb0-b48e6f1d0987 restarted
> --------------------------------------------------------------------------
> It seems numpy fails to initialize.
> However if I try to import numpy independently, it passes successfully.
>
> My guess is that when I ran the updates it updated something that caused
> this. The installation for pycuda was done by "pip install pycuda"

My guess would be Nvidia kernel/driver mismatch. Check the end of dmesg,
there might be a message there.

Andreas

_______________________________________________
PyCUDA mailing list
[hidden email]
https://lists.tiker.net/listinfo/pycuda