[PYTHON] From nothing on Ubuntu 18.04 to setting up a Deep Learning environment in Tensor
Introduction
Apart from my research, I decided to touch CNN as a part-time job, so I thought I couldn't run the GPU on my mac and couldn't easily turn it, so I'm going to run Linux, which I hadn't been able to handle! This is an article I wrote for the time being.
To put it simply, I got a user from a Linux server in the laboratory, so it is a reminder when I was able to run the GPU with Tensor etc. Long live your memo.
Basically, we will proceed while referring to this Running TensorFlow-gpu with ubuntu 18.04.
English but [A step by Step Guide to Install Tensorflow GPU on Ubuntu 18.04 LTS](https://medium.com/@kekayan/step-by-step-guide-to-install-tensorflow-gpu-on-ubuntu-18 -04-lts-6feceb0df5c0) was also helpful.
environment
- HP Z640 Workstation
- OS: Ubuntu 18.04LTS --Memory: 256GB
- GPU: GeForce GTX 1080 Ti
procedure
Change Password
Now that you have the user with sudo privileges, let's change the password
$ passwd [username]
(current) UNIX password:
Enter new UNIX password:
Retype new UNIX password:
Enter your current password and then enter your new password twice to complete.
passwd: password updated successfully
Nvidia driver installation
For the time being, check the current status with the following command.
$ nvidia-smi
Then,
Wed Nov 20 13:44:59 2019
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 390.129 Driver Version: 390.129 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Quadro K620 Off | 00000000:02:00.0 On | N/A |
| 34% 39C P8 1W / 30W | 124MiB / 1994MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 1 GeForce GTX 108... Off | 00000000:03:00.0 Off | N/A |
| 23% 40C P8 26W / 250W | 2MiB / 11178MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 2599 G /usr/lib/xorg/Xorg 49MiB |
| 0 2929 G /usr/bin/gnome-shell 71MiB |
+-----------------------------------------------------------------------------+
It seems that it is already installed. So next, I will introduce CUDA cuDNN.
Installation of CUDA 9.0 and cuDNN
This is also confirmed by the following command for the time being.
$ nvcc -V
Command 'nvcc' not found, but can be installed with:
apt install nvidia-cuda-toolkit
Please ask your administrator.
It seems that this needs to be installed.
It says Please ask your administrator, I'm glad I got the administrator authority.
Let's proceed with the installation.
This is for reference Run TensorFlow-gpu on ubuntu 18.04 Or Put Tensorflow with GPU in Ubuntu 18.04 Then, the latest CUDA 10.0 says that Tensorflow-gpu is not supported, so CIDA 9.0 was installed.
Both articles are about a year ago, so as of November 20, 2019, if you look at the Tensorflow official website,
The following NVIDIA® software must be installed on your system: NVIDIA® GPU drivers —CUDA 10.0 requires 410.x or higher. CUDA® Toolkit —TensorFlow supports CUDA 10.0 (TensorFlow >= 1.13.0) CUPTI ships with the CUDA Toolkit. cuDNN SDK (>= 7.4.1) (Optional) TensorRT 5.0 to improve latency and throughput for inference on some models.
Was written.
CUDA® Toolkit —TensorFlow supports CUDA 10.0 (TensorFlow >= 1.13.0)
So, it seems that CUDA 10.0 is okay if you use TensorFlow with 1.13.0 or higher.
However, it says NVIDIA® GPU drivers — CUDA 10.0 requires 410.x or higher..
What version of nvidia was that? I will return to confirm.
$ nvidia-smi
Wed Nov 20 13:44:59 2019
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 390.129 Driver Version: 390.129 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Quadro K620 Off | 00000000:02:00.0 On | N/A |
| 34% 39C P8 1W / 30W | 124MiB / 1994MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 1 GeForce GTX 108... Off | 00000000:03:00.0 Off | N/A |
| 23% 40C P8 26W / 250W | 2MiB / 11178MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 2599 G /usr/lib/xorg/Xorg 49MiB |
| 0 2929 G /usr/bin/gnome-shell 71MiB |
+-----------------------------------------------------------------------------+
It was 390.120. It seems to either update nvidia or set CUDA to 9.0.
Is it okay for the GPU I'm using in the first place? I thought, and confirmed it at here.
Enter the product name and click SEARCH.
It was recommended to install the 440.31 version of the driver. Apparently it's okay at all.
After updating nvidia and installing TensorFlow 1.13.0 and CUDA 10.0, or worrying about TensorFlow 1.12.0 or earlier and CUDA 9.0 for about 30 seconds, I decided to go with the latter.
Well, it's a shared server, so it would be a problem if there was an update, and it's safer to be one before the latest.
By the way, in order to investigate the version relationship of Nvidia, CUDA, TensorFlow, etc., this article → [Nvidia driver, CUDA, cuDNN, Tensorflow-gpu and Python version correspondence](https://qiita.com/konzo_ / items / a6f2e8818e5e8fcdb896) Was very helpful! Thank you.
Finally, proceed to install CUDA 9.0. Access the official NVIDIA installation page → CUDA Toolkit 9.0 Downloads Select to. Is it okay because there is no 18.04? I thought, but when I looked it up, it seems that 17.04 is okay.
Since I am working locally this time, I will download it and send it to the server with scp.
local
$ scp -P [port number] Downloads/cuda-repo-ubuntu1704_9.0.176-1_amd64.deb [username]@[IP address]:[Copy destination path]
Normally port specification -p, but when scp, it is -P.
$ wget http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/[deb file]
But it's okay.
After that, proceed with the installation according to the official website.
** In addition, it seems that the latest version will be installed with the last sudo apt-get install cuda, so it seems important to specify the version corresponding to the end.
This time, -toolkit-9-0 is added. ** **
$ sudo dpkg -i cuda-repo-ubuntu1704_9.0.176-1_amd64.deb
$ sudo apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1704/x86_64/7fa2af80.pub
$ sudo apt-get update
$ sudo apt-get install cuda-toolkit-9-0
Next, it seems that work to pass through the PATH is necessary. Enter the following command and edit ".bashrc".
$ nano ~/.bashrc
Enter the following character string at the end and save.
.bashrc
export PATH=/usr/local/cuda-9.0/bin${PATH:+:${PATH}}
export LD_LIBRARY_PATH=/usr/local/cuda-9.0/lib64${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}
Reboot
$ sudo reboot
After starting, check if CUDA is installed with the following command.
$ nvcc -V
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2017 NVIDIA Corporation
Built on Fri_Sep__1_21:08:03_CDT_2017
Cuda compilation tools, release 9.0, V9.0.176
Is displayed, so OK.
cuDNN installation
cuDNN is a multi-library installed with CUDA, and seems to be for high-speed calculation on GPU using CUDA.
You need to access the following site and register as a member. https://developer.nvidia.com/rdp/cudnn-download
After that, install the following 3 from Download cuDNN v7.6.4 for CUDA 9.0.
- cuDNN v7.6.4 Runtime Library for Ubuntu16.04 (Deb)
- cuDNN v7.6.4 Developer Library for Ubuntu16.04 (Deb)
- cuDNN v7.6.4 Code Samples and User Guide for Ubuntu16.04 (Deb)
Execute the following commands in order in the downloaded directory. Change the version to the downloaded version as appropriate.
$ sudo dpkg -i libcudnn7_7.6.4.38-1+cuda9.0_amd64.deb
$ sudo dpkg -i libcudnn7-dev_7.6.4.38-1+cuda9.0_amd64.deb
$ sudo dpkg -i libcudnn7-doc_7.6.4.38-1+cuda9.0_amd64.deb
This completes the installation of cuDNN.
Installation of pyenv and Anaconda, environment construction (Python environment construction)
I want to create an environment where I can install Anaconda from pyenv and create a virtual environment with Anaconda while making it possible to switch the version of Anaconda with pyenv at any time.
pyenv
$ git clone https://github.com/yyuu/pyenv.git ~/.pyenv
$ nano ~/.bashrc
After installing, add the following to .bashrc.
.bashrc
export PYENV_ROOT=$HOME/.pyenv
export PATH=$PYENV_ROOT/bin:$PATH
eval "$(pyenv init -)"
After rebooting and confirming that the pyenv path is in place, proceed to install Anaconda.
Anaconda
$ pyenv install --list | grep anaconda
Check the version of Anaconda that can be installed with. This time, install 5.2.0.
$ pyenv install anaconda3-5.2.0
Switch the version and check if it is reflected.
$ pyenv global anaconda3-5.2.0
$ pyenv versions
system
* anaconda3-5.2.0 (set by /home/okuda/.pyenv/version)
$ python --version
Python 3.6.5 :: Anaconda, Inc.
Looks good.
Next, create a virtual environment with anaconda.
$ conda -V
conda 4.5.4
Make sure it's working properly
$ conda create -n tensor pip python=3.6
Create a virtual environment named tensor. Install pip and python3.6 together.
And I would like to switch the virtual environment,
When using anaconda on the pyenv environment, conda activate cannot be used as it is.
$ conda activate tensor
CommandNotFoundError: Your shell has not been properly configured to use 'conda activate'.
If your shell is Bash or a Bourne variant, enable conda for the current user with
$ echo ". /home/okuda/.pyenv/versions/anaconda3-5.2.0/etc/profile.d/conda.sh" >> ~/.bashrc
or, for all users, enable conda with
$ sudo ln -s /home/okuda/.pyenv/versions/anaconda3-5.2.0/etc/profile.d/conda.sh /etc/profile.d/conda.sh
The options above will permanently enable the 'conda' command, but they do NOT
put conda's base (root) environment on PATH. To do so, run
$ conda activate
in your terminal, or to put the base environment on PATH permanently, run
$ echo "conda activate" >> ~/.bashrc
Previous to conda 4.4, the recommended way to activate conda was to modify PATH in
your ~/.bashrc file. You should manually remove the line that looks like
export PATH="/home/okuda/.pyenv/versions/anaconda3-5.2.0/bin:$PATH"
^^^ The above line should NO LONGER be in your ~/.bashrc file! ^^^
I get angry.
If your shell is Bash or a Bourne variant, enable conda for the current user with
$ echo ". /home/okuda/.pyenv/versions/anaconda3-5.2.0/etc/profile.d/conda.sh" >> ~/.bashrc
Because it is written as it is
$ echo ". /home/okuda/.pyenv/versions/anaconda3-5.2.0/etc/profile.d/conda.sh" >> ~/.bashrc
If you run and read bashrc again
$ source .bashrc
$ conda activate tensor
(tensor) :~$
It switched safely!
The rest is the installation of the necessary libraries. First, install the famous framework TensorFlow.
TensorFlow-GPU installation
The version is on the TensorFlow official website
So, this time, download tensorflow_gpu-1.12.0.
$ pip install tensorflow-gpu==1.12.0
When it is completed, check the operation to see if TensorFlow works on the GPU.
$ python
Python 3.6.9 |Anaconda, Inc.| (default, Jul 30 2019, 19:07:31)
[GCC 7.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>
>>>import tensorflow as tf
>>> tf.test.gpu_device_name()
2019-11-20 19:02:23.713902: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-11-20 19:02:24.115406: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found device 0 with properties:
name: GeForce GTX 1080 Ti major: 6 minor: 1 memoryClockRate(GHz): 1.582
pciBusID: 0000:03:00.0
totalMemory: 10.92GiB freeMemory: 10.76GiB
2019-11-20 19:02:24.278457: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found device 1 with properties:
name: Quadro K620 major: 5 minor: 0 memoryClockRate(GHz): 1.124
pciBusID: 0000:02:00.0
totalMemory: 1.95GiB freeMemory: 1.80GiB
2019-11-20 19:02:24.278929: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1496] Ignoring visible gpu device (device: 1, name: Quadro K620, pci bus id: 0000:02:00.0, compute capability: 5.0) with Cuda multiprocessor count: 3. The minimum required count is 8. You can adjust this requirement with the env var TF_MIN_GPU_MULTIPROCESSOR_COUNT.
2019-11-20 19:02:24.278972: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1511] Adding visible gpu devices: 0
2019-11-20 19:02:24.784359: I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-11-20 19:02:24.784436: I tensorflow/core/common_runtime/gpu/gpu_device.cc:988] 0 1
2019-11-20 19:02:24.784450: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 0: N N
2019-11-20 19:02:24.784459: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 1: N N
2019-11-20 19:02:24.784905: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/device:GPU:0 with 10403 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:03:00.0, compute capability: 6.1)
'/device:GPU:0'
Do you recognize another GPU? I will investigate it a little later. It's OK because it became'/ device: GPU: 0'.
Install Jupyter Lab
From here, I personally like Jupyter, so I will add it. I like it.
$ conda install jupyterlab
After that, do what you did with here so that you can access it remotely, and you're done.
Surprisingly, it is difficult to use GPU. Thank you for your hard work.
Recommended Posts