In the previous post, we’ve proceeded with CUDA 9.1 installation on Ubuntu 16.04 LTS. As with other software that evolves, NVIDIA released CUDA 9.2 back in May. It is also safe to assume that CUDA 9.2 will not be final version. Newer version will may come soon or later and here we are left with the bogging question: “How can we upgrade safely without clobbering the currently working system?” Moreover, we may also wonder if there is a mechanism to rollback the change and live with current setup while recognizing that it’s not yet the time to upgrade.

This post will cover three scenarios of CUDA 9.2 installation: 1) fresh installation, 2) install to upgrade by removing old version, 3) install to upgrade and keep multiple versions.

Pre-Installation Checks

Before we proceed with the installation, we need to check and confirm the current state of installed software components by running a series of commands as outlined below.

Tip: The commands shown in this section are taken from the command cheatsheet written in this article.

1. Check the NVIDIA graphics driver that is currently installed.

We run this command to get the graphics driver version:

$ nvidia-smi | grep "Driver Version" | awk '{print $6}'

If you see the error message “bash: nvidia-smi: command not found”, it means that the graphics driver has not been installed. You can refer to this article for NVIDIA graphics driver installation.

2. Check if there is existing CUDA installed

We run this command to get the installed CUDA version:

$ nvcc --version | grep "release" | awk '{print $6}' | cut -c2-

If you see the error message “bash: nvcc: command not found”, it implies that CUDA has not been installed or previous installation was not successful.

3. Check if there is existing CUDA profiling tools installed

We run this command to get the installed CUDA profiling tools version:

$ locate cupti | grep "libcupti.so." | tail -n1 | sed -r 's/^.*\.so\.//'

Empty result means that the CUDA profiling tools library has not been installed.

Note: if you see the error message “bash: locate: command not found”, you may need to first install mlocate package. Use this command to install the package:

$ sudo apt-get install mlocate

Alternatively, we can substitute locate command with find command as follows:

$ find /usr -name "*libcupti.so.*" | head -n1 | sed -r 's/^.*\.so\.//'

4. Check if there is existing cuDNN library installed

We run this command the get the currently installed cuDNN library

$ locate cudnn | grep "libcudnn.so." | tail -n1 | sed -r 's/^.*\.so\.//'

Alternative command:

$ find /usr -name "*libcudnn.so.*" | head -n1 | sed -r 's/^.*\.so\.//'

If you get empty result, it can be concluded the cuDNN library is not present on the system.

5. Check NVIDIA and CUDA packages installed by the package manager

To double check the install statuses of NVIDIA and CUDA packages in the system, we run the following command and observe the result

– Checking install statuses of NVIDIA packages

$ dpkg -l | grep nvidia

Sample output:

... rc nvidia-384 384.111-0ubuntu0.16.04.1 amd64 NVIDIA binary driver - version 384.111 ii nvidia-390 390.30-0ubuntu1 amd64 NVIDIA binary driver - version 390.30 ii nvidia-390-dev 390.30-0ubuntu1 amd64 NVIDIA binary Xorg driver development files rc nvidia-opencl-icd-384 384.111-0ubuntu0.16.04.1 amd64 NVIDIA OpenCL ICD ii nvidia-opencl-icd-390 390.30-0ubuntu1 amd64 NVIDIA OpenCL ICD ...

Interpretation of the output:

The install status of a package is displayed as the value of the first column. It can consist of three letters:

The first letter: the expected / desired status of the package

The second letter: the current status of the package

The third letter: the error status of the package

What are the available states for expected, current, and error statuses? We can find them by tweaking how we parse dpkg command output:

$ dpkg -l | head -n3

... Desired=Unknown/Install/Remove/Purge/Hold | Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad) ...

Interpretation of the output:

From the output, we have two combinations of values: “rc” and “ii”. The value “rc” means that the package should be or should have been removed and only configuration files remain in the system without any error noticed. Alternatively, the value “ii” means that the package should be or should have been installed and is currently installed properly without error. Further examining the first and second rows, it can be seen that the currently active NVIDIA graphics driver is version 390.30. This driver replaces the older version 384.111 that has also been removed from the system.

– Checking install statuses of CUDA packages

$ dpkg -l | grep cuda

Sample output:

... ii cuda 9.1.85-1 amd64 CUDA meta-package ii cuda-9-1 9.1.85-1 amd64 CUDA 9.1 meta-package ii cuda-command-line-tools-9-1 9.1.85-1 amd64 CUDA command-line tools ii cuda-compiler-9-1 9.1.85-1 amd64 CUDA compiler ii cuda-cublas-9-1 9.1.85.3-1 amd64 CUBLAS native runtime libraries ii cuda-cublas-dev-9-1 9.1.85.3-1 amd64 CUBLAS native dev links, headers ii cuda-cudart-9-1 9.1.85-1 amd64 CUDA Runtime native Libraries ii cuda-cudart-dev-9-1 9.1.85-1 amd64 CUDA Runtime native dev links, headers ii cuda-cufft-9-1 9.1.85-1 amd64 CUFFT native runtime libraries ii cuda-cufft-dev-9-1 9.1.85-1 amd64 CUFFT native dev links, headers ii cuda-cuobjdump-9-1 9.1.85-1 amd64 CUDA cuobjdump ii cuda-cupti-9-1 9.1.85-1 amd64 CUDA profiling tools interface. ii cuda-curand-9-1 9.1.85-1 amd64 CURAND native runtime libraries ii cuda-curand-dev-9-1 9.1.85-1 amd64 CURAND native dev links, headers ii cuda-cusolver-9-1 9.1.85-1 amd64 CUDA solver native runtime libraries ii cuda-cusolver-dev-9-1 9.1.85-1 amd64 CUDA solver native dev links, headers ii cuda-cusparse-9-1 9.1.85-1 amd64 CUSPARSE native runtime libraries ii cuda-cusparse-dev-9-1 9.1.85-1 amd64 CUSPARSE native dev links, headers ii cuda-demo-suite-9-1 9.1.85-1 amd64 Demo suite for CUDA ii cuda-documentation-9-1 9.1.85-1 amd64 CUDA documentation ii cuda-driver-dev-9-1 9.1.85-1 amd64 CUDA Driver native dev stub library ii cuda-drivers 390.30-1 amd64 CUDA Driver meta-package ii cuda-gdb-9-1 9.1.85-1 amd64 CUDA-GDB ii cuda-gpu-library-advisor-9-1 9.1.85-1 amd64 CUDA GPU Library Advisor. ii cuda-libraries-9-1 9.1.85-1 amd64 CUDA Libraries 9.1 meta-package ii cuda-libraries-dev-9-1 9.1.85-1 amd64 CUDA Libraries 9.1 development meta-package ii cuda-license-9-1 9.1.85-1 amd64 CUDA licenses ii cuda-memcheck-9-1 9.1.85-1 amd64 CUDA-MEMCHECK ii cuda-misc-headers-9-1 9.1.85-1 amd64 CUDA miscellaneous headers ii cuda-npp-9-1 9.1.85-1 amd64 NPP native runtime libraries ii cuda-npp-dev-9-1 9.1.85-1 amd64 NPP native dev links, headers ii cuda-nsight-9-1 9.1.85-1 amd64 CUDA nsight ii cuda-nvcc-9-1 9.1.85.2-1 amd64 CUDA nvcc ii cuda-nvdisasm-9-1 9.1.85-1 amd64 CUDA disassembler ii cuda-nvgraph-9-1 9.1.85-1 amd64 NVGRAPH native runtime libraries ii cuda-nvgraph-dev-9-1 9.1.85-1 amd64 NVGRAPH native dev links, headers ii cuda-nvml-dev-9-1 9.1.85-1 amd64 NVML native dev links, headers ii cuda-nvprof-9-1 9.1.85-1 amd64 CUDA Profiler tools ii cuda-nvprune-9-1 9.1.85-1 amd64 CUDA nvprune ii cuda-nvrtc-9-1 9.1.85-1 amd64 NVRTC native runtime libraries ii cuda-nvrtc-dev-9-1 9.1.85-1 amd64 NVRTC native dev links, headers ii cuda-nvtx-9-1 9.1.85-1 amd64 NVIDIA Tools Extension ii cuda-nvvp-9-1 9.1.85-1 amd64 CUDA nvvp ii cuda-repo-ubuntu1604 9.2.148-1 amd64 cuda repository configuration files ii cuda-runtime-9-1 9.1.85-1 amd64 CUDA Runtime 9.1 meta-package ii cuda-samples-9-1 9.1.85-1 amd64 CUDA example applications ii cuda-toolkit-9-1 9.1.85-1 amd64 CUDA Toolkit 9.1 meta-package ii cuda-tools-9-1 9.1.85-1 amd64 CUDA Tools meta-package ii cuda-visual-tools-9-1 9.1.85-1 amd64 CUDA visual tools ii libcuda1-390 390.30-0ubuntu1 amd64 NVIDIA CUDA runtime library ii libcudnn7 7.1.2.21-1+cuda9.1 amd64 cuDNN runtime libraries ii libcudnn7-dev 7.1.2.21-1+cuda9.1 amd64 cuDNN development libraries and headers ...

Interpretation of the output:

The system currently has CUDA 9.1 installed. Additionally, cuDNN library version 7.1 is also installed. Installing CUDA 9.2 in the system simply means upgrading from or replacing the previous version, unless both are configured to be in active use.

6. Build the system summary table for CUDA 9.2 installation

After performing pre-installation step 1 to 5, we build a simple table to summarize the current state of the system.



No Component Installed Version 1 Graphics Driver TO BE FILLED 2 CUDA (via locate) TO BE FILLED 3 CUDA (via dpkg) TO BE FILLED 4 CUDA profiling tools TO BE FILLED 5 cuDNN (via locate) TO BE FILLED 6 cuDNN (via dpkg) TO BE FILLED

Based on the values we inputted to the table, we may use the following rule for CUDA 9.2 installation:

No graphics driver and/or CUDA installed: Install the graphics driver first, follow with CUDA 9.2 fresh install Graphics driver version < 396.x and CUDA version < 9.2: Update the graphics driver first, follow with CUDA 9.2 upgrade install Graphics driver version >= 396.x and CUDA version < 9.2: Upgrade installation to CUDA 9.2 cuDNN < 7.x and CUDA version < 9.2: Upgrade installation to CUDA 9.2 followed with cuDNN upgrade cuDNN >= 7.x and CUDA version >= 9.2: Thanks for reading even though you already have everything set up

Installation Preparation: CUDA 9.2 Package Download

For all the installation scenarios, we need to download CUDA 9.2 packages from NVIDIA developer website. The steps to download are as follows:

1. Go to the CUDA download page: https://developer.nvidia.com/cuda-downloads

2. Select the target platform and installer type. We will choose “deb (network)” for the installer type since it is the installer type that has been consistently working and is easier to work with for an upgrade.



3. Download the installer. Refrain from immediately executing the commands displayed in the installation instruction box. We will run different commands for different installation scenario.



Depending on the website policy, you may or may not need to login to download the installer. In the past, installer download required login. However, at the time this post is written, login is not required for downloading the packages.

Fresh Installation

After downloading CUDA 9.2 package, we now proceed with the installation. For a fresh install, we go through the remaining steps of the installation as follows:

1. Update apt package index and install the newest version of all currently installed packages

$ sudo apt-get update $ sudo apt-get upgrade

2. Use the base installer to install CUDA toolkit and driver packages

For fresh installation, we can religiously follow the installation instruction displayed on the download page:

– Install CUDA repository metada

$ sudo dpkg -i cuda-repo-ubuntu1604_9.2.148-1_amd64.deb

– Install CUDA repository GPG keys

$ sudo apt-key adv --fetch-keys http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/7fa2af80.pub

– Update the apt package index

$ sudo apt-get update

– Install CUDA toolkit and drivers

$ sudo apt-get install cuda

3. Add CUDA 9.2 bin directory to the path and CUDA 9.2 library folder to the library path

$ export PATH=/usr/local/cuda-9.2/bin${PATH:+:${PATH}} $ export LD_LIBRARY_PATH=/usr/local/cuda-9.2/lib64${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}

We can make this change permanent to the current user by modifying the .profile file

$ vi ~/.profile

... PATH="/usr/local/cuda-9.2/bin:$HOME/bin:$HOME/.local/bin:$PATH" LD_LIBRARY_PATH=/usr/local/cuda-9.2/lib64${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}} ...

4. Create the NVIDIA persistence service for optimizing the GPU performance

$ cd /usr/lib/systemd/system && sudo touch nvidia-persistenced $ sudo vi nvidia-persistenced

=========== nvidia-persistenced =============

[Unit] Description=NVIDIA Persistence Daemon Wants=syslog.target [Service] Type=forking PIDFile=/var/run/nvidia-persistenced/nvidia-persistenced.pid Restart=always ExecStart=/usr/bin/nvidia-persistenced --verbose ExecStopPost=/bin/rm -rf /var/run/nvidia-persistenced [Install] WantedBy=multi-user.target

5. Enable NVIDIA persistence service

$ sudo systemctl enable nvidia-persistenced

6. Disable the default udev rule so that NVIDIA software can initialize the memory with non-default settings

$ sudo vi /lib/udev/rules.d/40-vm-hotadd.rules

... #SUBSYSTEM=="memory", ACTION=="add", DEVPATH=="/devices/system/memory/memory[0-9]*", TEST=="state", ATTR{state}="online" ...

7. Reboot the system

$ sudo reboot

8. Verify the installation by checking the CUDA driver and toolkit version

– To check the CUDA driver version

$ cat /proc/driver/nvidia/version | grep NVRM | awk '{print $8}'

– To check the CUDA toolkit version

$ nvcc --version | grep "release" | awk '{print $6}' | cut -c2-

Confirm that the driver version is 396.xx and the toolkit version is 9.2.x

9. Done. CUDA has been successfully installed.

You can now proceed with follow-up installations, for example cuDNN, tensorflow, or Caffe2.

Install to Upgrade by Removing Old Version

It is important to note that upgrading will only work with the same installer type. Since we will be installing using the “deb (network)” installer, make sure that the previous installation was also performed with this installer type. If you installed with another installer type, for example “runfile”, we need to uninstall the version first before performing upgrade.

To uninstall previous CUDA version X.Y that was installed using “runfile”, we run the following commands:

– Uninstalling CUDA toolkit

$ sudo /usr/local/cuda-X.Y/bin/uninstall_cuda_X.Y.pl

– Uninstalling CUDA driver

$ sudo /usr/bin/nvidia-uninstall

After uninstalling the conflicting installed version, we start the upgrade process as follows:

1. Remove the currently installed CUDA packages

$ sudo apt-get purge cuda* $ sudo apt-get autoremove $ sudo apt-get autoclean

2. Remove the existing CUDA installation folder

$ sudo rm -Rf /usr/local/cuda*

3. Use the downloaded CUDA 9.2 base installer to install CUDA toolkit and driver packages

– Install CUDA repository metada

$ sudo dpkg -i cuda-repo-ubuntu1604_9.2.148-1_amd64.deb

– Install CUDA repository GPG keys (optional)

$ sudo apt-key adv --fetch-keys http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/7fa2af80.pub

– Update the apt package index

$ sudo apt-get update

– Install CUDA toolkit and drivers

$ sudo apt-get install cuda

4. Update the path and library path

$ export PATH=/usr/local/cuda-9.2/bin${PATH:+:${PATH}} $ export LD_LIBRARY_PATH=/usr/local/cuda-9.2/lib64${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}

We can make this change permanent to the current user by modifying the .profile file

$ vi ~/.profile

... PATH="/usr/local/cuda-9.2/bin:$HOME/bin:$HOME/.local/bin:$PATH" LD_LIBRARY_PATH=/usr/local/cuda-9.2/lib64${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}} ...

5. Reboot the system

$ sudo reboot

6. Verify the installation by checking the CUDA driver and toolkit version

– To check the CUDA driver version

$ cat /proc/driver/nvidia/version | grep NVRM | awk '{print $8}'

– To check the CUDA toolkit version

$ nvcc --version | grep "release" | awk '{print $6}' | cut -c2-

Confirm that the driver version is 396.xx and the toolkit version is 9.2.x

7. Done. CUDA has been successfully installed.

Install to Upgrade and Keep Multiple Versions

Starting from CUDA 8.0, it is possible to have multiple versions of CUDA installed in the system. So, if you currently have CUDA 8.0, 9.0, or 9.1 installed and want to install CUDA 9.2 without uninstalling the older version(s), you can opt for this kind of installation. This can be a desired setup if you want to compare the behavior and performance of different CUDA versions.

If you installed the older version via “runfile”, you need to uninstall it first. However, if the installation was performed using “deb (network)”, you can safely proceed with the installation. The steps are outlined as follows.

1. Use the downloaded CUDA 9.2 base installer to update CUDA toolkit and driver packages

– Install CUDA repository metada

$ sudo dpkg -i cuda-repo-ubuntu1604_9.2.148-1_amd64.deb

– Install CUDA repository GPG keys (optional)

$ sudo apt-key adv --fetch-keys http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/7fa2af80.pub

– Update the apt package index

$ sudo apt-get update

– Install CUDA 9.2 toolkit and drivers via its meta package

$ sudo apt-get install cuda-9-2

2. Create a new environment file for CUDA 9.2

$ vi ~/cuda92-env.sh

#!/bin/sh export PATH="/usr/local/cuda-9.2/bin:$HOME/bin:$HOME/.local/bin:$PATH" export LD_LIBRARY_PATH=/usr/local/cuda-9.2/lib64${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}

3. To switch to use CUDA 9.2, activate the environment

$ source ~/cuda92-env.sh

4. Verify that we are currently using CUDA 9.2

$ nvcc --version | grep "release" | awk '{print $6}' | cut -c2-

5. To switch back to the default (older) CUDA version, we simply activate the default environment again

$ source ~/.profile

– Verify that we are now back to the older version

$ nvcc --version | grep "release" | awk '{print $6}' | cut -c2-

6. Done. We now have multiple versions of CUDA installed in the system

Possible Side Effects

Please note that if you previously installed TensorFlow by building from sources, you may need to rebuild it to CUDA 9.2 environment. However, this is only valid if you install CUDA 9.2 by removing the old version.

Concluding Remark

CUDA installation process is becoming better and more streamlined in the latest version. If you have already installed CUDA in the past and is considering upgrading to a newer version but is afraid of the side effect, this post can be some help to prevent such issue from occurring.

How was your CUDA 9.2 installation experience? Share your finding and gotcha in the comment section below.