[PYTHON] Dealing with tensorflow suddenly stopped working using GPU in deep learning

Purpose

In the following environment, deep learning using GPU suddenly stopped working.

OS:windows10 GPU:GeForce GTX 1070 PC: (omitted) Gaming? laptop

Also, what tensorflow feels like is

tensorboard              1.14.0
tensorboardX             1.9
tensorflow               1.14.0
tensorflow-estimator     1.14.0
tensorflow-gpu           1.14.0
tensorflow-gpu-estimator 2.1.0

However, it is installed. The coping method is shown.

Workaround

The reason why it stopped working

When executing learning, Ctrl + C stopped the process.

State of being stuck

If you try to perform learning again,

2020-04-19 15:36:34.104585: I tensorflow/stream_executor/platform/default/dlopen_checker_stub.cc:25] GPU libraries are statically linked, skip dlopen check.
2020-04-19 15:36:34.111310: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0

C:\_work\_

** I get back to the prompt without any error. ** **

Workaround

I tried restarting the computer, but it didn't work.

From here, by intuition tensorflow-gpu 1.14.0 I was inspired to put it back in.

tensorflow-gpu 1.14.0 I got an access error or something when I tried to uninstall. (With pulse)

I tried to delete each folder, and after uninstalling all the tensorflow related items, tensorflow-gpu 1.14.0 I was able to uninstall it, or I got a response that I wanted to uninstall.

From here, install tenforflow etc. again As a result, it started to work. (Of course, the installation was restored with the version specified, like'tensorflow-gpu == 1.14.0'.)

It didn't stop at the previous point and started to move.

2020-04-19 16:04:06.169138: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0
2020-04-19 16:04:06.172271: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-04-19 16:04:06.176606: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187]      0
2020-04-19 16:04:06.179549: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0:   N

Arrangement of coping

I don't know if the "trigger that stopped working" was due to Ctrl + C. It is unclear whether the "countermeasure" was that uninstalling all the tensorflow-related items and then reinstalling was effective, or whether the computer was in a good mood over time while doing various things.

Other findings

** Python + GPU may quit without any error ** I feel. (I don't know how to get rid of the error.) I feel like I've experienced it before.

Summary

Nothing in particular. I hope this miscellaneous information will be useful to someone. (If you have any information such as stopping at the same place or being cured in the same way, please let me know if you feel like it.)

Recommended Posts

Dealing with tensorflow suddenly stopped working using GPU in deep learning
Try deep learning with TensorFlow
Dealing with Tensorflow error "Import Error: DLL load failed: Specified module not found" in deep learning
Image recognition model using deep learning in 2016
Dealing with Python error "Attribute Error: module'scipy.misc' has no attribute'imresize'" in deep learning
Until the Deep Learning environment (TensorFlow) using GPU is prepared for Ubuntu 14.04
Stock Price Forecast Using Deep Learning (TensorFlow) -Part 2-
[TensorFlow 2 / Keras] How to run learning with CTC Loss in Keras
Sine wave prediction using RNN in deep learning library Keras
(Now) Build a GPU Deep Learning environment with GeForce GTX 960
I tried hosting a TensorFlow deep learning model using TensorFlow Serving
Use "% tensorflow_version 2.x" when using TPU with Tensorflow 2.1.0 in Colaboratory
real-time-Personal-estimation (learning using GPU locally)
Deep Learning Gaiden ~ GPU Programming ~
Working with LibreOffice in Python
Deep Kernel Learning with Pyro
Generate Pokemon with Deep Learning
Dealing with Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR in Tensorflow v2.x
Under investigation about PYNQ-Let's do deep learning with FPGA using Python-
Ensure reproducibility with tf.keras in Tensorflow 2.3
Try Deep Learning with FPGA-Select Cucumbers
Cat breed identification with deep learning
Working with LibreOffice in Python: import
Make ASCII art with deep learning
Deep learning to start without GPU
Solve three-dimensional PDEs with deep learning.
Categorize news articles with deep learning
I tried deep learning using Theano
Forecasting Snack Sales with Deep Learning
Until dealing with python in Atom
Make people smile with Deep Learning
Working with DICOM images in Python
Dealing with tensorflow suddenly stopped working using GPU in deep learning
Until the Deep Learning environment (TensorFlow) using GPU is prepared for Ubuntu 14.04
The story that `while queue` did not work in python
Solve the Japanese problem when using the CSV module in Python.
Solution to the problem that the display is corrupted when the .exe command is included in the while loop in wsl2
Put your own image data in Deep Learning and play with it
Coursera's TensorFlow introductory course to get you started with Deep Learning implementations
Impressions of people with experience in other languages learning Python using PyQ
Try to predict FX with LSTM using Keras + Tensorflow Part 2 (Calculate with GPU)
Dealing with Python (bash, Windows10) error "bash: / c / Users / XYZZZ / AppData / Local / Microsoft / WindowsApps / python3: Permission denied" in deep learning