[LINUX] [Improved version] Script to monitor CPU with Python

When I execute temp.py in Measure the CPU temperature of Raspeye with Python I wrote earlier, the number of CPUs is counted every time it fires in a while loop. I was using a percentage. Since it is a script for monitoring, I would like to improve it because it should be as light as possible.

By the way, after the improvement, the output looks like this. From the left, CPU temperature, frequency, overall usage rate, and usage rate for each core.

Cause

What was heavy was the module called subprocess used to execute Linux OS commands. About this, It is wrong to cat with python subprocess. , so please have a look. Simply put, it's slow because it launches a child process just to get the text. If you use the built-in function for file operation called ʻopen ()`, it will be completed only in Python, so it will be about 50 times faster.

Improvement points

There are four main improvements.

1. Replace subprocess with ʻopen ()`
2. The loop of while is slow, so replace it with for
3. Rewrite the string operation in print with format
4. Abolished the operating voltage display and displayed the usage rate

The first is the highlight of this time. Previously, I used subprocess to execute a Raspberry Pi OS-made command called vcgencmd. This can only be run on the Raspberry Pi OS, and since vcgencmd is designed to run in the shell, it's not suitable for typing from Python. Monitoring in Python is a sub-process bomb, so let's stop it, so getting CPU information without using vcgencmd is also a sub-feature of this time.

The second one is insignificant, but I rewrote it in my mood. The third purpose is mainly to improve readability. Fourth, I didn't find much merit in checking the operating voltage, and I couldn't find the file that recorded the operating voltage. Changed to display CPU usage instead.

code

Here is the improved code.

temp.py

#!/usr/bin/python3.7

import time
import sys

#CPU usage time acquisition function
def get_data():
    with open('/proc/stat') as r: #Read CPU statistics file
        stats = r.readlines() #List by line
        stat = [line.strip().split() for line in stats if 'cpu' in line] #Pick up the line containing the cpu, remove the newline character, double list with whitespace delimiter
    rs = [] #Declare a list that contains usage time
    for data in stat: #Extract data for the entire CPU and each logical core from statistics
        busy = int(data[1]) + int(data[2]) + int(data[3]) #Find the time of Busy state
        all = busy + int(data[4]) #Ask for the whole time
        rs.append([all, busy]) #Add to list
    return rs #Returns a value

pre_data = get_data() #Get the first CPU usage time
ave = 0 #Variable declaration for average calculation

time.sleep(1) #Wait for 1 second

try: #Ctrl with the following except Keyboard Interrupt+Normal operation part for catching C
    for i in range(60): #Repeat 60 times
        with open('/sys/class/thermal/thermal_zone0/temp') as t: #CPU temperature read
            temp = int(t.read()) / 1000 #Type conversion and match units
        with open('/sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq') as f: #CPU frequency reading
            freq = int(f.read()) #Type conversion

        now_data = get_data() #Get current CPU usage time
        rates = [] #Declare a list containing CPU usage

        for j in range(5): #Whole CPU+Number of logical cores(4)Repeat times
            now = now_data[j] #Extract the CPU data obtained from the current usage time
            pre = pre_data[j] #Extract the CPU data obtained from the usage time 1 second ago
            rate = (now[1] - pre[1]) / (now[0] - pre[0]) * 100 #(Busy state/The entire) *Find CPU usage at 100
            rates.append(rate) #Add CPU usage to list

        #Format and export using format method
        print("Temp:{0:>6.2f}'C, Freq: {1:.2f}GHz, CPU:{2:>5.1f}% [{3:>3.0f},{4:>3.0f},{5:>3.0f},{6:>3.0f}]({7:>2})".format(temp, freq / 1000000, rates[0], rates[1], rates[2], rates[3], rates[4], i + 1))

        ave += temp #Add current temperature for average temperature
        pre_data = now_data #Save current data for next second previous data
        time.sleep(1) #Wait for 1 second
    print("Average: {:.2f}'C (60s)".format(ave / 60)) #Write the average after the loop ends
except KeyboardInterrupt: #Ctrl+Catch C
    sec = i + 1 #Get the elapsed time at the end
    print(" Aborted.\nAverage: {0:.2f}'C ({1}s)".format(ave / sec, sec)) #Write out the average temperature
    sys.exit() #Successful completion

CPU usage

Prior to the explanation, I would like to describe how to calculate the CPU usage rate separately. I referred to here.

The CPU usage rate is basically the ratio of the CPU usage time to the total time, so if you get the CPU usage time from the CPU statistics provided by Ubuntu, you can find it by the following principle.

The general flow of the code is "data acquisition (pre)" → "wait for 1 second" → "data acquisition (now and next pre)" → "calculation" → "data acquisition (next now)" → (repetition) ..

Ubuntu CPU statistics are listed in / proc / stat, and the following is an excerpt of the necessary information from the manual.

man

/proc/stat
    kernel/system statistics.  Varies with architecture.  Common entries include:
        cpu 10132153 290696 3084719 46828483 16683 0 25195 0 175628 0
        cpu0 1393280 32966 572056 13343292 6130 0 17875 0 23933 0
        #Format the above string so that it looks like the one on the right[[cpu, 10132153, 290696, 3084719, 46828483, 16683, 0, 25195, 0, 175628, 0], [cpu0, 1393280, 32966, 572056, 13343292, 6130, 0, 17875, 0, 23933, 0]]
            The amount of time, measured in units of USER_HZ (1/100ths of a second on most architectures, use sysconf(_SC_CLK_TCK) to obtain the right value), that the system ("cpu" line) or the specific CPU ("cpuN" line) spent in various states:
                user   (1) Time spent in user mode.　#Busy by user
                nice   (2) Time spent in user mode with low priority (nice). #Busy due to low priority processes by users
                system (3) Time spent in system mode. #Busy by system
                idle   (4) Time spent in the idle task.  This value should be USER_HZ times the second entry in the /proc/uptime pseudo-file. #Idol

We will format it so that you can retrieve the necessary data by referring to this. If you get it with subprocess, there is a convenient one called grep, but you can't use it this time. So use the readlines () included in ʻopen (). [Here](https://note.nkmk.me/python-grep-like/) Refer to the article, reproduce the same behavior as grep, and further process it to double CPU> item` Make a list. (See comments in the above manual)

From this double list, for extracts the data for the entire CPU and each logical core, calculates the Busy state and the total elapsed time, and returns it as a double list of entire CPU and logical core> elapsed time. Here, the sum of the 2nd, 3rd, and 4th items in the list is the sum of the Busy state, and the sum of the Busy state and the Idle state (the 5th item) is the total elapsed time. The above is the processing of the usage time acquisition function get_data (). After that, I would like to touch on it in the explanation.

Commentary

I wrote it in the comments, but I would like to deal with it in order from the top. I'm a beginner myself, so I'll explain it in excess.

import First, ʻimportthe required modules. This time I loaded thetime module for standby and the sysmodule for termination. In addition to this, the pre-improved code contained asubprocess`, but I removed it.

def get_data() I have already dealt with CPU usage in another section, but I have declared a function to get the cumulative usage time with def.

pre_data and ʻave, time.sleep (1) `

Before entering the loop, get the data for the initial calculation and declare a variable to find the average CPU temperature. Then wait for a second. If you proceed without waiting, now_data will be acquired earlier than the update interval of / proc / stat, so the difference will be zero and you will be angry if you do not divide by 0.

try and ʻexcept`

By enclosing the entire loop in try, you can catch the input of Ctrl + C as an exception instead of an error. When Ctrl + C is input, it jumps to ʻexcept Keyboard Interrupt`, calculates the average temperature, and then ends normally.

for i in range(60) It repeats the process while substituting numbers from 0 to 59 for i. It seems that repeating 10000000 times is faster than repeating with while for about 0.04 seconds. This time it's not within the margin of error, but I like the fact that I don't have to prepare a counter.

with open() A built-in function that opens a file introduced on behalf of subprocess. As mentioned above, please see here for details. Since the processing is completed inside Python, it contributes to weight reduction.

now_data and rates

You are getting the current cumulative CPU usage time. Declaring an empty list to assign CPU usage.

for j in range(5) Data is fetched and processed for the entire CPU and each core. Calculates the CPU usage by calculating the difference between the current cumulative usage time now_data and the cumulative usage time pre_data one second ago, and returns the result as a list. (What should I name an index that is not ʻi`?)

print It is written out so that it is easy to see using the format method. It's very convenient because you can do simple calculations such as dividing the average value inside the format.

ʻAve and pre_data, time.sleep (1)`

The current temperature is added to ʻaveto get the average temperature. Substitute the current data obtained earlier forpre_data` for the next CPU usage calculation. After waiting for one second, the process loops.

Get CPU information without using vcgencmd

As I mentioned earlier, getting information from Python with vcgencmd is very inefficient. Since the data is made into a file on Ubuntu like / proc / stat, I tried to get it from there.

CPU temperature

I think this is a fairly famous place.

cat /sys/class/thermal/thermal_zone0/temp
#1000 times the temperature is output

With vcgencmd, units and so on are attached, but since it is only numbers, it seems good to get it from here to embed it in the script. Division is not a pain if it is in the program, and for some reason it is good that the significant figures are larger.

CPU frequency

I had a hard time finding this, but in any case there are great men. Thank you.

Ubuntu CPUfreq Part 3-About the files in the cpufreq folder of the CPU

#/sys/devices/system/cpu/cpu*/cpufreq to cpu(core)Information is gathered
#cpuinfo_cur_freq is the value obtained from the hardware
#scaling_cur_freq is the value obtained from the Linux kernel

cat /sys/devices/system/cpu/cpu0/cpufreq/cpuinfo_cur_freq
#Permission denied
cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq
#The CPU frequency of cpu0 is displayed

It's nice to find a place, but I ran into two problems. The first is that you can only see root for the numbers obtained from the hardware, and the other is that the files are different for each cpu core. If possible, I would like to refer to the numerical value obtained from the hardware, but I want to execute it without root, and if the frequency is different for each cpu core, I want to take the average, but I want to avoid it because the processing will be quadrupled.

I tried to verify by brute force to solve these problems. If all eight values match, I think that there is no problem even if one value is used as a representative sample, so obtain and compare cpuinfo_cur_freq and scaling_cur_freq of all 4 cores 10000 times. As a result, they all matched at a rate of about 95%. It seems that there is a shift when changing the frequency, and if you put a sleep between the verifications, it was about 95%, and if you did nothing and repeated it, it was 100% consistent. Also, it seems that the deviation occurs mainly between the hardware and the kernel, and the match between the cores was about 99% at the worst. It doesn't have to be that strict, and sampling every second will have little effect, so I decided to use scaling_cur_freq from cpu0 as a representative value this time.

Linux (on Raspberry Pi?) Doesn't control power saving by changing the frequency for each core. I would appreciate it if you could let me know if anyone is familiar with this area.

CPU voltage

We couldn't find the location of the file. You may find it if you look for it seriously, but since it was a value that did not have to be originally, I decided to cut it at this time. If anyone knows, please let me know.

Summary

I started working on reducing the weight of heavy scripts, and since I monitored the CPU with Python, I added the purpose of completing it with Python alone, and in the end I succeeded in significantly reducing the weight. I am keenly aware of my studies every day because I can learn about super useful functions that I did not know. Also, while I was writing this article, I was able to rewrite the code and refine it a little. Review is important. It's been a long article, but thank you for reading to the end. If you have any suggestions, please feel free to contact us.