After all it is wrong to cat with python subprocess.

Yaharo.

wrap up

If you write subprocess.run ('cat path', shell = True) in python, repent with me and use ʻopen ()` from tomorrow.

What is subprocess?

If you are familiar with it, please skip it. subprocess is a module that executes Linux OS commands from python. There are os function and system function to execute OS commands on Python, but it seems that subprocess is currently recommended.

import subprocess

list = subprocess.run('ls') 
#hoge.py  hoge.sh  hoge.wav
print(list)
#CompletedProcess(args='ls', returncode=0)

#If you pass a command as an argument, the execution result (status) will be returned.

list = subprocess.run('ls', encoding='utf-8', stdout=subprocess.PIPE).stdout
print(list)
#hoge.py
#hoge.sh
#hoge.wav

#By specifying encoding, utf-Can be encoded with 8 etc.
#You can pass output to standard output by specifying the stdout attribute

list = subprocess.run('ls -a', shell=True, encoding='utf-8', stdout=subprocess.PIPE).stdout
print(list)
#.hoge
#hoge.py
#hoge.sh
#hoge.wav

#shell=You can include spaces by specifying True, but there is a risk of command injection etc.

You can execute various OS commands like this, but the process is slow because a subprocess is created each time. The image is that parents bother to give birth to a child to read a book and have the child read aloud. If you read it yourself, you can solve it, but it's understandable that it's heavy.

By the way, read aloud the option of stdout = subprocess.PIPE! It means. Parents cannot know what the child is reading (unless they can read their thoughts with the sixth sense), but they can do so by reading aloud.

Although it is dissed in this article, it is a very convenient module, so if you are new to it, please use it. ~~ Processing is not so heavy for normal use. ~~ (Plan collapse)

cat was stupid in subprocess

As mentioned above, you can get the text content of hogehoge.txt by doing subprocess.run ('cat /hogehoge.txt', shell = True), but you can start a subprocess just to pull the text. Since it is raised, the processing will be slow. You can use a built-in function called ʻopen ()` to open a file in Python, so let's use that. It's very rudimentary, but I didn't know how to use it easily.

with open('/hogehoge.txt') as f:
    hoge = f.read()
print(hoge)
#/hogehoge.The text of txt is displayed

Since anything opened with ʻopen ()must be closed withclose (), we use with to avoid forgetting to close it. After closing ʻopen (), the file can no longer be referenced, but the assigned variable can be referenced. See the official documentation (https://docs.python.org/ja/3/library/functions.html#open) for more information.

How fast is ʻopen ()`

I wrote the following script and verified it.

#!/usr/bin/python3.7

import subprocess
import time

#open()
start = time.time() #Get start time
for i in range(1000): #Repeat 1000 times
    with open('/sys/class/thermal/thermal_zone0/temp') as f: #Open file
        test = f.read() #Substitute the text of the file
elapsed_time = time.time() - start #Get the end time and find the difference from the start time
print("Open: {}s".format(elapsed_time)) #Write out elapsed time

#subprocess
start = time.time()
for i in range(1000):
    test = subprocess.run('cat /sys/class/thermal/thermal_zone0/temp', shell=True, stdout=subprocess.PIPE).stdout
elapsed_time = time.time() - start
print("Subprocess: {}s".format(elapsed_time))

#vcgencmd in subprocess (reference record for CPU temperature)
start = time.time()
for i in range(1000):
    test = subprocess.run('vcgencmd measure_temp', shell=True, stdout=subprocess.PIPE).stdout
elapsed_time = time.time() - start
print("Vcgencmd: {}s".format(elapsed_time))

The execution result is as follows.

Open: 0.08225083351135254s
Subprocess: 5.188024044036865s
Vcgencmd: 6.032892465591431s

What do you mean! Subprocess, which was so slow, is now more than 50 times faster.

I think that the breakdown of the elapsed time is probably the standby time, so I can not say simply about the CPU load, but it is certain that it is light as if I was monitoring from another window. If you execute it several times, there will be some variation, but if it is so different, you will want to use ʻopen ()`.

By the way, in subprocess.run, the processing of the parent process is stopped until the child process ends, but by using subprocess.Popen, the child process can be turned to the background and parallel processing can be performed. Even if you use subprocess.Popen, it took about 2s in the above verification, so there is no doubt that ʻopen ()` is fast, but the difference may be reduced depending on the situation. In any case, the purpose of this time is to get a text file, so I want to avoid parallelism ...

This verification is triggered by the improvement of the script dealt with in the article Measuring the CPU temperature of Raspberry Pi with Python, so check it with vcgencmd as a reference. I tried it. The output of vcgencmd is formatted so that the temperature value is more readable, so it is likely to be slower.

[Addition] Based on the contents of this article, I wrote Script to monitor CPU with Python.

Finally

I was surprised to see a clearer difference than I expected. I want to study more and more and be able to write programs that are conscious of "the right person in the right place".

Recommended Posts

After all it is wrong to cat with python subprocess.
Read CSV file with Python and convert it to DataFrame as it is
If you try to install Python2 pip after installing Python3 pip and it is rejected
It is easy to execute SQL with Python and output the result in Excel
The math of some entrance exam question is awkward to think about, so I left it to python after all
Which is the most popular python visualization tool after all?
When it is troublesome to set up an SMTP server locally when sending mail with Python.
Connect to BigQuery with Python
Is it possible to detect similar images only with ImageHash?
Connect to Wikipedia with Python
Post to slack with Python 3
After all, who is Embedding?
Switch python to 2.7 with alternatives
Write to csv with Python
It's Halloween so I'll try to hide it with Python
[Python] What is a tuple? Explains how to use without tuples and how to use it with examples.
[Confrontation! Human power vs Python] After all, which is faster, solving the mathematics of the center test with Python or solving it by yourself?
[Python] What is a slice? An easy-to-understand explanation of how to use it with a concrete example.
[Python] What is pip? Explain the command list and how to use it with actual examples
Microsoft's Deep Learning framework "CNTK" is now compatible with Python, making it much easier to use
Convert to a string while outputting standard output with Python subprocess
Made it possible to convert PNG to JPG with Pillow of Python
Don't write Python if you want to speed it up with Python
When it is troublesome to copy what you built with vue
Sample to use after OAuth authentication of BOX API with Python
How to use the asterisk (*) in Python. Maybe this is all? ..
I tried to find out if ReDoS is possible with Python
How to utilize Python with Jw_cad (Part 1 What is external transformation)
Python script written in PyTorch is converted to exe with PyInstaller
What should I do with the Python directory structure after all?
What is Python? What is it used for?
[Python] What is a with statement?
Python: How to use async with
Link to get started with python
[Python] Write to csv file with Python
Create folders from '01' to '12' with python
Nice to meet you with python
Try to operate Facebook with Python
Output to csv file with Python
After all, what is statistical modeling?
Convert list to DataFrame with python
MP3 to WAV conversion with Python
To do tail recursion with Python2
How to get started with Python
What to do with PYTHON release?
Unable to install Python with pyenv
How to use FTP with Python
How to calculate date with python
Easily post to twitter with Python 3
I want to debug with Python
Easy parallel execution with python subprocess
[TCP / IP] After studying, try to make an HTTP client-like with Python
I want to do it with Python lambda Django, but I will stop
I want to tweet on Twitter with Python, but I'm addicted to it
With PEP8 and PEP257, Python coding that is not embarrassing to show to people!
When you want to use it as it is when using it with lambda memo
It is better to use Weight Initializer when initializing with HeNormal with Chainer
Use Python from Java with Jython. I was also addicted to it.
If pip stops due to SSL, it is better to re-insert python itself
[Introduction to Python] What is the method of repeating with the continue statement?
[Zaif] I tried to make it easy to trade virtual currencies with Python