[LINUX] Information acquisition from IoT devices via pseudo USB memory

Introduction

Inexpensive IoT devices such as Raspberry PI, Arduino, and ESP32 that can be connected to a wired / wireless LAN and can change the processing contents programmatically have become widespread. Using these devices, there are many applications for business such as equipment operation monitoring in factories and real-time work instructions to workers.

As a method of monitoring the operation of a certain device, there is a method of collecting the observed values measured constantly on the server side and observing long-term time-series changes on a weekly, monthly, or yearly basis. You can predict the possibility of failure due to aging deterioration, and calculate the replacement time of regular maintenance parts according to the frequency of use.

Here, I will introduce a concrete example of finding a method with better performance by trial and error when I developed a mechanism to collect operation information of a certain device A. We will explain the procedure for proposing several methods, prototyping each method, verifying the operation, and measuring the performance to determine which method is better.

Thing you want to do

What I want to do this time is "I want to send the operation information from device A to the collection server on the Internet as soon as possible."

The following are the features of device A.

--Device A cannot connect to the Internet. Output the operation information file to the USB memory. --Output operation information several times a day --When device A outputs the operation information to a file, the file output is completed in 5 seconds.

System configuration and operation outline

Prepare ARM-based Linux device B and transfer the operation information of device A to the collection server on the Internet. Make the following settings.

--Connect device A and device B with a USB cable --Format the disk area of device B (/ dev / mmcblk2p9) with VFAT --Use the function of Mass Storage Gadget of Linux to make the area appear to device A as a USB memory.

Reference: https://www.kernel.org/doc/html/latest/usb/mass-storage.html During operation, it operates as follows.

--When device A writes a file to the pseudo USB memory, it is written to the disk area of device B. --When device B detects writing, it acquires the operation information file. --Device B sends the acquired operation information to the collection server via the Internet.

File write detection

By the way, I was able to design up to the point of making a pseudo USB memory using the function of Mass Storage Gadget of Linux. The next challenge is how to detect that device B has 1) written a new file. Copying a file in the middle of writing will result in inconsistent information, so it is also necessary to detect that the file has been written 2). Copying all the files on the USB memory every time wastes time and power consumption, so only new files are copied. 3) Differential copy is also required.

Here, I thought about three ways. A) Read only mount, B) mtools polling, C) inotify API. The outline of each method is shown in the table below, and the details are explained.

A) Read only mount

Repeat mounting and unmounting the disk area for the USB memory with Read only.

--The reason for setting it to readonly is to prevent the file system from being corrupted if you write from both device A and device B. --The reason for umounting each time is that the file list remains cached in the memory as it is mounted, and device B cannot detect that device A has added the file.

The shell script that achieves this is shown below. Each time you mount, it compares the file list with the previous one and copies the new file to / dest /.

#!/bin/bash
touch old.lst
while true; do
  mount -o ro /dev/mmcblk2p9 /mnt
  ls /mnt | sort > new.lst
  diff old.lst new.lst > /dev/null # 1.
  if [ $? -ne 0 ]; then
    sleep 5 # 2.
    comm -13 old.lst new.lst | xargs -i cp {} /dest/ # 3.
  fi
  umount /mnt
  cp new.lst old.lst
  sleep 5 # 1.
done

B) mtools polling Use mtools to detect new file additions and copy new files. --mtools are tools mainly used for reading and writing to floppy disks and magneto-optical disks (MO, Magneto-Optical Disk). --You can get the list of files in the USB memory by using the mdir command of mtools. --Detects the addition of new files by periodically executing the mdir command and seeing the difference in the results. --Copy the new file with the mcopy command.

Below is a shell script that accomplishes this.

#!/bin/bash
touch old.lst
while true; do
  mdir -b -i /dev/mmcblk2p9 | sort > new.lst
  diff old.lst new.lst > /dev/null # 2.
  if [ $? -ne 0 ]; then
    sleep 5 #Wait 5 seconds to finish writing
    comm -13 old.lst new.lst | xargs -i mcopy -i /dev/mmcblk2p9 {} /dest/ # 3.
  fi
  cp new.lst old.lst
  sleep 5 # 1.
done

C) inotify API It uses the Linux inotify API to detect when a file has been updated with an event. --Execute the inotify wait command. --When writing (modify) to the disk image of the USB memory is detected, the inotify wait command ends. --Detect the completion of writing with the inofity watch command. Specifically, wait until the last 5 seconds of writing disappear. --Copy the newly added file with the mcopy command.

You need inotify-tools to run it. For debian / ubuntu, run apt install inotify-tools in advance.

#!/bin/bash
touch old.lst
while true; do
  inotifywait -e modify /dev/mmcblk2p9 # 1.
  while inotifywatch -t 5 -e modify /dev/mmcblk2p9 \
    | grep -v 'No events occurred.' > /dev/null; do # 2.
    :
  done

  mdir -b -i /dev/mmcblk2p9 | sort > new.lst
  comm -13 old.lst new.lst | xargs -i mcopy -i /dev/mmcblk2p9 {} /dest/ # 3.
  cp new.lst old.lst
done

Performance comparison

Let's compare the processing performance of these three methods and decide which method is better. Performance was measured under the following conditions

• Hardware: ARM Cortex A53, 4core
• Kernel: Linux 4.4.194 ARM64 --Create 30000 files on a pseudo USB memory (because the maximum number of files that can be placed in one directory with FAT32 is 32767) --Execute the methods A to C and measure the load of device 2 with the vmstat command. --Performance is measured on average for 5 seconds. --In order to measure the load at regular times, the measurement results for the first 5 times are discarded and the average of the results for the next 10 times is measured.

$ vmstat 5
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 0  0      0 1249656  81340 461084    0    0     5     2   72   52  0  1 99  0  0
 1  0      0 1249392  81340 461084    0    0  1671    11 1001 1088  1  2 96  0  0
 0  0      0 1249376  81340 461092    0    0  1671    10  910  968  1  2 96  0  0

The results are as follows. The lower the number, the better the performance.

CPU usage is low for all three methods. Focusing on the amount of Disk read, Method C is the lowest at 0. Methods A and B perform high-load processing every 5 seconds, while method C waits without doing anything until a write system call occurs, so the load is minimal. It is desirable that the IO bandwidth usage is low in order to avoid the influence on other processing in terms of power consumption. From the above results, we decided to adopt method C that uses the inotify API.

Finally

When comparing the performance, I confirmed a strange phenomenon. When the file on the USB memory is read on the device A side, the inotify MODIFY event appears. It's strange to get a change event when I'm just reading a file. Next time, I will introduce the process of analyzing the cause of this strange phenomenon.

trademark

--Raspberry Pi is a registered trademark or trademark of RASPBERRY PI FOUNDATION in the United Kingdom and other countries. --Arduino is a trademark registration or trademark of Arduino AG in Italy or other countries or in other countries. --ESP32 is a registered trademark or trademark of Espressif Systems (Shanghai) Co., Ltd. in China or other countries. --Arm is a registered trademark or trademark of Arm Limited in the United Kingdom and other countries. --Linux is a registered trademark or trademark of Linus Torvalds in Japan and other countries. --The company names, product names, service names, etc. described are trademarks or registered trademarks of their respective companies.