[PYTHON] Allow Slack to notify you of the end of a time-consuming program process


Note that Slack now notifies you when a process for a time-consuming program (such as learning a deep model) is complete.

As a feature,

  1. In the program execution environment, execute a shell script that takes the process ID of the program to be tracked as an argument.
  2. Confirm the start of tracking and display the program execution command in Slack.
  3. In Slack, display the end notification of the program and the process ID to end the tracking

to hold.

Tools used & environment

I've touched shell scripts for almost the first time, so please point out any redundant coding.

Installation procedure

1. Link Incoming Webhooks to Slack channels

Go to Incoming Webhook and link to the channel of your workspace.


I hope you can set it to your liking. You will need the Webhook URL here for further work.

2. Create a shell script to track

I created the following directory structure. I think it can be anywhere if it is easy to understand.

└ Documents/
      └ slack/
          └ (I'll create it here)

First, create a shell script in ~/Documents/slack / with the following command.

$ touch watch #I think the name can be anything.

Then open watch in your editor. I wrote the following shell script there.

$ vi watch



set -eu

URL="" ###Write the Webhook URL here!###
USERNAME="Aoi Kiriya"

if [ ! $PID ]; then
  echo "ERROR: PID required." >&2
  exit 1

#Process at the start of execution
START_MSG="`ps ho args $PID`"
if [ $ret -ne 0 ]; then
  echo "ERROR: Process not found." >&2
  exit 1
  ary=(`echo $START_MSG`)
  for i in `seq 1 ${#ary[@]}`
  set +u
    if [ "$i" -eq 1 ]
        PRETXT="${ary[$i-1]##*/}I ran"
        TEXT="$TEXT ${ary[$i-1]}"
set -u

TITLE="Execution command"
TEXT="\`\$ ${ary[0]##*/}$TEXT\`"
BEGIN_DATA="payload={\"username\": \"$USERNAME\", \"text\": \"Start tracking!(PID: \`$PID\` )\", \"attachments\": [{\"fallback\": \"Confirm execution command\",\"color\": \"#003399\",\"pretext\": \"$PRETXT\" ,\"title\": \"$TITLE\",\"text\": \"$TEXT\"}]}"

curl -s -X POST --data-urlencode "$BEGIN_DATA" ${URL} >/dev/null

#At the end of the process
END_MSG="${ary[0]##*/}Is finished\n Tracking finished!(PID: \`$PID\` )"
: "start watch ${PID}"
  while true
    if ! ps -p ${PID} >/dev/null ;then
      curl -s -X POST --data-urlencode "payload={\"username\": \"$USERNAME\", \"text\": \"${END_MSG}\"}" ${URL} >/dev/null
    sleep 1m
} &

I referred to this code. thank you very much.

Reference: https://gist.github.com/ohsuga/41a459792748e8c2afdb81d0b261c3de

3. 3. test

Once you've done this, let's test it once.

First, create a source code that counts for 30 seconds in Python. Print os.getpid () so that the process ID is also output at the same time.

$ vi test.py


import os
import sys
import time

def print_args(pid, args):
    print(os.getpid(), args)

if __name__ == "__main__":
    args = sys.argv
    print_args(os.getpid(), args)

    for i in range(0, 30, 1):
        print(i, "sec")

And the test command is: Let's add various things as a test of command line arguments. Here, the process ID is different for each process, so please respond flexibly. In the following cases, it is 17027.

$ python test.py  1 2 3 hello --args-conf
17027 ['test.py', '1', '2', '3', 'hello', '--args-conf']
0 sec
1 sec
2 sec
3 sec

Execute the following command within 30 seconds.

$ bash watch 17027

Then, the following posts will be posted on the channel linked with Slack's Incoming Webhook. Screenshot from 2021-01-20 23-12-45.png And when 30 seconds have passed. .. ..

26 sec
27 sec
28 sec
29 sec

I think there is a time lag, but I think the following posts will be posted. Screenshot from 2021-01-20 23-17-16.png


Now that you've created the command, let's put it in your PATH so that you can access it from anywhere on your PC.

First, write the PATH of the shell script in ~/.bashrc with the following command.

$ echo export "PATH=$HOME/Documents/slack:$PATH" >> ~/.bash_profile
$ source ~/.bash_profile

Let's also set an alias so that we can run it without hitting bash. I made it run with the command aoi.

$ echo alias aoi='bash watch' >> ~/.bash_aliases
$ source  ~/.bash_aliases

Now try typing aoi on the command line, and if you get the following error, you're in your PATH.

$ aoi
/home/user/Documents/slack/watch: line 11: $1: unbound variable

Execution example

Let's try the PyTorch tutorial MNIST Classification Task. Please note that this requires torch and torchvision.

When you bring the code, first run the main code with the following command.

$ python main.py --no-cuda
Train Epoch: 1 [0/60000 (0%)]   Loss: 2.305401
Train Epoch: 1 [640/60000 (1%)] Loss: 1.359780
Train Epoch: 1 [1280/60000 (2%)]        Loss: 0.830692
Train Epoch: 1 [1920/60000 (3%)]        Loss: 0.620273
Train Epoch: 1 [2560/60000 (4%)]        Loss: 0.354148
Train Epoch: 1 [3200/60000 (5%)]        Loss: 0.461558
Train Epoch: 1 [3840/60000 (6%)]        Loss: 0.278612

Then check the process ID to run aoi. You can check it with the following command.

$ ps x | grep "python main.py"
11313 pts/0    Rl+    0:10 /home/user/.pyenv/versions/3.6.7/bin/python main.py --no-cuda
11387 pts/5    S+     0:00 grep --color=auto python main.py

Probably the process ID is 11313. Now let's run aoi!

$ aoi 11313

Then. .. .. Screenshot from 2021-01-20 22-22-49.png It came to notify the start and end of learning like this.

in conclusion

I think it is still in the development stage, so please be careful when introducing it m (_ _) m

As a future issue

I want to do it.

If you don't mind, we look forward to your contribution to the GitHub repository!

Recommended Posts

Allow Slack to notify you of the end of a time-consuming program process
Write a script in Shell and Python to notify you in Slack when the process is finished
How to insert a specific process at the start and end of spider with scrapy
The story of writing a program
Created a Discord bot to notify you of updates to become a novelist
[Python] A program that rotates the contents of the list to the left
Set the process name of the Python program
Notice the completion of a time-consuming command
[Python] A program that calculates the number of socks to be paired
I made a program to notify you by LINE when switches arrive
I made a program to check the size of a file in Python
How to calculate the volatility of a brand
How to mention a user group in slack notification, how to check the id of the user group
I made a tool to notify Slack of Connpass events and made it Terraform
Creating a LINE BOT to notify you of additional AtCoder contests using AWS
Send Gmail at the end of the process [Python]
[Python] Throw a message to the slack channel
A program that notifies slack of the operating status of fully automatic botanical photography equipment
I tried to notify the update of "Become a novelist" using "IFTTT" and "Become a novelist API"
Add a function to tell the weather of today to slack bot (made by python)
[Python] A program that counts the number of valleys
A memo to visually understand the axis of pandas.Panel
Notify Slack of process state changes using Supervisor's EventListener
[Python] A program that compares the positions of kangaroos.
Python Note: The mystery of assigning a variable to a variable
Make a note of what you want to do in the future with Raspberry Pi
Create a 2D array by adding a row to the end of an empty array with numpy
A Python script that allows you to check the status of the server from your browser
[Python] A program to find the number of apples and oranges that can be harvested
[Ubuntu] How to delete the entire contents of a directory
I want to improve efficiency with Python even in the experimental system (5) I want to send a notification at the end of the experiment with the slack API
Write a python program to find the editing distance [python] [Levenshtein distance]
Write a program to solve the 4x4x4 Rubik's Cube! 1. Overview
I want to use only the normalization process of SudachiPy
Debug by attaching to the Python process of the SSH destination
How to find the scaling factor of a biorthogonal wavelet
[End of 2020] A memo to start using AWS CLI (Version 2)
Is there a secret to the frequency of pi numbers?
How to connect the contents of a list into a string
[Django] What to do if the model you want to create has a large number of fields
[Python] Let LINE notify you of the ranking of search results on your site on a daily basis.
Notify Slack of GitHub push
How to start the program
Process the contents of the file in order with a shell script
A story that struggled to handle the Python package of PocketSphinx
Conditional branch due to the existence of a shell script file
How to check the memory size of a variable in Python
[Introduction to StyleGAN] I played with "The Life of a Man" ♬
If you give a list with the default argument of the function ...
Don't you want to say that you made a face recognition program?
[Go] Create a CLI command to change the extension of the image
How to check the memory size of a dictionary in Python
How to output the output result of the Linux man command to a file
When you want to save the result of the callback function somewhere
Try to write a program that abuses the program and sends 100 emails
How to get the vertex coordinates of a feature in ArcPy
How to use Jupyter on the front end of supercomputer ITO
A command to easily check the speed of the network on the console
Node.js: How to kill offspring of a process started with child_process.fork ()
I made a slack bot that notifies me of the temperature
Create a function to get the contents of the database in Go