Debug by attaching to the Python process of the SSH destination


I often want to attach to a Python process that is already running and manipulate a living object.

――It's good to put out a graph that takes a long time to preprocess, but I forgot to write the process to save the data. ――The learning of the model, which takes several days, has stopped without any error. I want to know where it stopped. --Debugging Python scripts spawned from a huge service ――I want to attach to a process that has become a zombie or an orphan

It may be easy if the IDE works in a local environment, but it becomes more problematic if you are SSHing to a GPU machine or the like.

Here's how to solve these problems with a combination of lightweight and portable tools.

  1. Small is beautiful. Small is beautiful ――The UNIX philosophy――


As an example, let's target a program that keeps displaying the elapsed seconds every second. The point is that you don't have to import additional libraries or insert snippets in advance.

import time
n = 0
while True:
	n += 1

Let's experiment by creating a process that is intentionally isolated and does not have a control terminal.

$ nohup python & ; exit


First, take the standard input / output of the target and replace it with the prepared terminal.

# -s Target file descriptor 0,1,Take over even if 2 is not connected to the TTY.
$ reptyr -s $PID


Next, inject the following debugger startup code into the target.

$ pyrasite $PID


Rewrite the variable n from the internal shell of the debugger that was started last, and try stepping.

import pudb


output.gif By taking it by reptyr, I came to touch the log that had disappeared in the past. You can also see that it reflects the execution of each loop and the change to n = -100 by adding breakpoints.

Note: You will see no source code available on the way, but you can press n until you return from the C function.

Finally, I'll put a script that will do what I've done so far.


PID=`pgrep -a python|fzf|awk '{print $1}'`
[ -z "$PID" ] && exit 1

cat << EOF > $tmpfile
import pudb

pyrasite $PID $tmpfile &
reptyr -s $PID

Recommended Posts

Debug by attaching to the Python process of the SSH destination
Pandas of the beginner, by the beginner, for the beginner [Python]
Set the process name of the Python program
(Python Selenium) I want to check the settings of the download destination of WebDriver
Setting to debug test by entering the contents of the library with pytest
I tried to verify and analyze the acceleration of Python by Cython
I tried to output the rpm list of SSH login destination to an Excel sheet with Python + openpyxl.
Send Gmail at the end of the process [Python]
How to erase the characters output by Python
[python] How to sort by the Nth Mth element of a multidimensional array
the zen of Python
Easy way to check the source of Python modules
python beginners tried to predict the number of criminals
The wall of changing the Django service from Python 2.7 to Python 3
Template of python script to read the contents of the file
How to get the number of digits in Python
List of posts related to optimization by Python to docker
The process of installing Atom and getting Python running
[python] option to turn off the output of click.progressbar
[Python] Summary of how to specify the color of the figure
14 quizzes to understand the surprisingly confusing scope of Python
Get the URL of the HTTP redirect destination in Python
[Introduction to Python] Basic usage of the library matplotlib
To do the equivalent of Ruby's ObjectSpace._id2ref in Python
Completely translated the site of "The Hitchhiker's Guide to Python"
Python Note: The mystery of assigning a variable to a variable
I want to increase the security of ssh connections
I tried to summarize the string operations of Python
Add a function to tell the weather of today to slack bot (made by python)
I tried to find the entropy of the image with python
Try to get the function list of Python> os package
Towards the retirement of Python2
Output "Draw ferns programmatically" to the drawing process in Python
[Python] How to make a list of character strings character by character
How to switch the configuration file to be read by Python
How to test the attributes added by add_request_method of pyramid
How to debug the Python standard library in Visual Studio
Make the display of Python module exceptions easier to understand
The story of introducing jedi (python auto-completion package) to emacs
I want to use only the normalization process of SudachiPy
Settings to debug the contents of the library with VS Code
[Python3] Call by dynamically specifying the keyword argument of the function
How to check if the contents of the dictionary are the same in Python by hash value
[Python] I tried to visualize the follow relationship of Twitter
Try to automate the operation of network devices with Python
The process of making Python code object-oriented and improving it
I want to know the features of Python and pip
Find the diameter of the graph by breadth-first search (Python memory)
About the features of Python
Try to import to the database by manipulating ShapeFile of national land numerical information with Python
The Power of Pandas: Python
Get the source of the page to load infinitely with python.
Put the process to sleep for a certain period of time (seconds) or more in Python
I tried to verify the speaker identification by the Speaker Recognition API of Azure Cognitive Services with Python. # 1
I tried to verify the speaker identification by the Speaker Recognition API of Azure Cognitive Services with Python. # 2
Find the white Christmas rate by prefecture with Python and map it to a map of Japan
I tried to summarize the contents of each package saved by Python pip in one line
[Python] How to use the for statement. A method of extracting by specifying a range or conditions.
The story that the version of python 3.7.7 was not adapted to Heroku
How to determine the existence of a selenium element in Python
How to change the log level of Azure SDK for Python