Debug by attaching to the Python process of the SSH destination
Overview
I often want to attach to a Python process that is already running and manipulate a living object.
――It's good to put out a graph that takes a long time to preprocess, but I forgot to write the process to save the data. ――The learning of the model, which takes several days, has stopped without any error. I want to know where it stopped. --Debugging Python scripts spawned from a huge service ――I want to attach to a process that has become a zombie or an orphan
It may be easy if the IDE works in a local environment, but it becomes more problematic if you are SSHing to a GPU machine or the like.
Here's how to solve these problems with a combination of lightweight and portable tools.
- Small is beautiful. Small is beautiful ――The UNIX philosophy――
conditions
As an example, let's target a program that keeps displaying the elapsed seconds every second. The point is that you don't have to import additional libraries or insert snippets in advance.
roop.py
import time
n = 0
while True:
print(n)
time.sleep(1)
n += 1
Let's experiment by creating a process that is intentionally isolated and does not have a control terminal.
$ nohup python roop.py & ; exit
reptyr
First, take the standard input / output of the target and replace it with the prepared terminal.
# -s Target file descriptor 0,1,Take over even if 2 is not connected to the TTY.
$ reptyr -s $PID
pyrasite
Next, inject the following debugger startup code into the target.
$ pyrasite $PID set_pudb.py
PuDB
Rewrite the variable n from the internal shell of the debugger that was started last, and try stepping.
set_pudb.py
import pudb
pudb.set_trace()
result
By taking it by reptyr, I came to touch the log that had disappeared in the past.
You can also see that it reflects the execution of each loop and the change to n = -100 by adding breakpoints.
Note: You will see no source code available on the way, but you can press n until you return from the C function.
Finally, I'll put a script that will do what I've done so far.
injector.sh
#!/bin/bash
tmpfile=`mktemp`
PID=`pgrep -a python|fzf|awk '{print $1}'`
[ -z "$PID" ] && exit 1
cat << EOF > $tmpfile
import pudb
pudb.set_trace()
EOF
pyrasite $PID $tmpfile &
reptyr -s $PID
Recommended Posts