[PYTHON] A program that notifies slack of the operating status of fully automatic botanical photography equipment

The story of making a python program that notifies the operating status of the plant photography device via slack

It turned out to be something like this.

The shooting device stops with an error. ↓ A notice comes in slack asking me to do something about it.

Background

* Fully automatic plant phenotype analysis system "RIPPS" press release *

Papers: Miki Fujita, Takanari Tanabata, Kaoru Urano, Saya Kikuchi, Kazuo Shinozaki, "RIPPS: A Plant Phenotyping System for Quantitative Evaluation of Growth under Controlled Environmental Stress Conditions", Plant & Cell Physiology, 10.1093 / pcp / pcy122 //academic.oup.com/pcp/advance-article/doi/10.1093/pcp/pcy122/5043525)

An automatic plant growing and photographing device developed by RIKEN CSRS. For details, see the press (Japanese) above or the dissertation (English, open access).

It is a device that stores photos and growth data on a PC over time with a built-in camera while rotating plants on a conveyor belt, but it seems that it sometimes stops due to an error. We do not know the details of the cause, but it seems that there are various causes such as water dripping in the water tank due to hardware causes such as damage to parts and deterioration due to adhesion of soil and solution. Furthermore, since the research building is different between the room where the researcher in charge resides and the room where the device with a control PC is located, it was difficult to continue monitoring whether it is operating normally. At first, I thought that it would be better to see the latest update time of the save destination folder via the network, but there are folders for each camera type in one device, and the number of devices is further multiplied. Considering the increase, it was a very very situation. When I was talking about joint research as a user of the imaging device, I decided to make it in a hurry because this would hinder the acquisition of my own experimental data.

I decided that it was difficult to implement the error handling function in the control software (I do not want to touch the hardware control software that has been completed once), so I monitor the update status of the folder from the outside and notify slack of the status I made a simple program to do. The following memorandum and information sharing.

idea

-python -Specify the folder to monitor -Since subfolders may be created, recursive searches the specified folder. -Notify if there is no difference in the files in the folder with the specified interval in between. -Notify that the program is running normally about once a day. -The way to notify is twitter \ * or slack \ * I didn't know how long it would take to register the api on twitter, so I chose slack. --For automatic posting from python to slack, I searched for incoming webhook on qiita and decided to use the code that is often used.

program

1. Loading required libraries and automatic installation

When I had python executed remotely, I found that instructing pip install ~~~ was actually a high hurdle, so I included that part in the script as well.

from pip._internal import main as _main
import importlib

def _import(name, module, ver=None):
    try:
        globals()[name] = importlib.import_module(module)
    except ImportError:
        try:
            if ver is None:
                _main(['install', module])
            else:
                _main(['install', '{}=={}'.format(module, ver)])
            globals()[name] = importlib.import_module(module)
        except:
            print("can't import: {}".format(module))

_import('requests','requests')
_import('argparse','argparse')
_import('json','json')
_import('threading','threading')

reference https://vaaaaaanquish.hatenablog.com/entry/2018/12/02/210647 https://stackoverflow.com/questions/12332975/installing-python-module-within-code

If the library does not exist at startup and you get an import error, install it and try importing again. requests, argparse, json, threading were specified because they were not installed as standard in many cases.

2. Manage command line arguments with argparse

parser = argparse.ArgumentParser()
parser.add_argument('-i', '--interval',default=1)
parser.add_argument('-d', '--dir', nargs='+', help="pass abs directory pass as list", required = True)
parser.add_argument('-n', '--name',default="RIPPS monitoring program_First issue")

interval: File confirmation frequency (time) dir: Monitor folder (absolute path), multiple can be specified name: slack poster name

Use as below

python monitor.py --dir PATH1 PATH2 --interval 1.5 --name RIPPS_monitoring_robot_1

3. Other hard code variables

SLACK_WEBHOOK = "https://hooks.slack.com/services/XXXXXXXXXXXXX" start = time.time()

4. Initial monitoring folder scan and slack posting

def initiation(path,nfiles):    
	message = "%Start monitoring s. Currently in the monitored folder%There are d files.%Check for updates every s hours." % (path,nfiles,args.interval) 
	ellapsed = int(time.time() - start)
	payload_dic = {
		"icon_emoji": ':heartpulse:',
		"text": message,
		"username": args.name + "_" + str(int(ellapsed/(60*60))),
		"channel": "#general", # #Also needed
	}
	try:
		r = requests.post(SLACK_WEBHOOK, data=json.dumps(payload_dic))
	except requests.ConnectionError:
		print(requests.ConnectionError)
		print("Could not connect to slack.")

befores = []

for i, path_to_watch in enumerate(args.dir):
	print(path_to_watch)
	assert os.path.isdir(path_to_watch) == True, print("%s is not a valid directory" % path_to_watch)
	if path_to_watch[-1] != "/":
		path_to_watch += "/**"
	else:
		path_to_watch += "**"
	before = dict ([(f, None) for f in glob.glob(path_to_watch,recursive=True)])
	initiation(path_to_watch,len(before))
	args.dir[i] = path_to_watch
	befores.append(before)

--If you include the option to monitor multiple directories, args.dir will be a list, so process it with a for loop. A single folder is also available. --If it is not a directory, it will be rejected with an assertion error. --If recursive = True of glob.glob and \ * \ * is added to the end of the path, the subdirectory will be searched (should), so add \ * \ * to the end of the input argument. To be able to handle even if you pull the folder to terminal or command prompt and drop it. The instruction to add ~ ~ to the end of the path is quite complicated, and the user may drop it. --Make the file list obtained by glob.glob (although the number of folders is included) into a dictionary type, and pass the length of the dictionary to the initiation function. --Post to the slack channel with the initiation function. --The file name of the monitoring folder in the initial state is stored in the befores list and used later for difference comparison.

The following is the behavior of the bot when it is set to monitor two folders

5. Folder monitoring

def errorpostslack(path):
	error_message = "Monitored folder(%s)But%Not updated for more than s hours" % (path,args.interval)  #No update message
	ellapsed = int(time.time() - start)
	payload_dic = {
		"icon_emoji": ':cry:',
		"text": error_message,
		"username": args.name + "_" + str(int(ellapsed/(60*60))),
		"channel": "#general", # #Also needed
	}
	try:
		r = requests.post(SLACK_WEBHOOK, data=json.dumps(payload_dic))
	except requests.ConnectionError:
		print(requests.ConnectionError)
		print("Could not connect to slack.")

while 1:
	time.sleep(float(args.interval)*60*60)
	for i, (before, path_to_watch) in enumerate(zip(befores,args.directory)):
	    after = dict ([(f, None) for f in glob.glob(path_to_watch,recursive=True)])
	    added = [f for f in after if not f in before]
	    removed = [f for f in before if not f in after]
	    if added:
	        print ("Added: ", ", ".join (added))
	        #goodpostslack(added)
	        pass
	    elif removed:
	        print ("Removed: ", ", ".join (removed))
	        pass
	    else:
	        errorpostslack(path_to_watch)
	    befores[i] = after

--Executes a loop with an interval (seconds) based on the set interval (time). --Zip the befores list and directory obtained in the initial scan at the same time. The enumerate doesn't make sense here, but it's a remnant of bug fixing. --Use the updated befores from the second time onwards --Get the latest glob.glob file list for each loop. Compared to before, if the number of files has increased, it will be added, and if it has disappeared, it will be removed. If there is no change, both added and removed are empty, and else processing is executed. --added removed I don't need a pass for both, but I've left it for when I commented out the print. --If the file has not been updated, call the errorpostslack function to post an error message to slack.

result

6. Survival report

If the shooting device is working properly and the images continue to accumulate, there is no notification. On the other hand, it is troublesome to keep the normal notification every interval. Therefore, we will add a function to report survival once a day.

def dailynotice():
	message = "It is a regular report once a day."
	ellapsed = int(time.time() - start)
	for i, (before, path_to_watch) in enumerate(zip(befores,args.dir)):
		nfiles = len(glob.glob(path_to_watch,recursive=True))
		message += "%Under the s folder%There are d folders and files" % (path_to_watch,nfiles)
	payload_dic = {
		"icon_emoji": ':smile:',
		"text": message,
		"username": args.name + "_" + str(int(ellapsed/(60*60))),
		"channel": "#general", # #Also needed
	}
	try:
		r = requests.post(SLACK_WEBHOOK, data=json.dumps(payload_dic))
	except requests.ConnectionError:
		print(requests.ConnectionError)
		print("Could not connect to slack.")	
    
while 1:
	dailynotice()
	time.sleep(24*60*60)

result

7. Coexistence of error detection and survival report

Since there are two while loops, thread them at the same time.

pseudocode
def error_check():
Watched folder update check function
def daily_check():
Survival report

t1 = Thread(target = error_check)
t2 = Thread(target = daily_check)
t1.start()
t2.start()

8. Finally

--The code has been uploaded to github. https://github.com/totti0223/ripps_utility --I was wondering whether to use the url of slack's incoming webhook as a command line argument, but I made it hard code because it confuses the user. --There are probably mac autometers, free software, and other methods, but it was faster to make them in python. If you know it, please let me know regardless of language or os platform (even if it is not via slack). ――Even though the icon is changed for each survival report and error report, when the same contributor posts continuously, it is sad that the threads are not reflected together. ..

--It is a painstaking measure to deal with by merging the time from the start of the program at the end of the poster name (username). Not beautiful. Even if the poster name is the same, the icons may be displayed one by one, so I don't understand the rules.

	payload_dic = {
		"icon_emoji": ':heartpulse:',
		"text": message,
		"username": args.name + "_" + str(int(ellapsed/(60*60))),
		"channel": "#general", # #Also needed
	}