[PYTHON] Periodically log the value of Omron environment sensor with Raspberry Pi

What is an environmental sensor?

Omron's environment sensors include temperature, humidity, illuminance, barometric pressure, noise, etc. A device that combines multiple sensors together The full range of functions that are typical of sensor manufacturers is attractive! オムロン環境センサ.png (Left: BAG type 2JCIE-BL01, Right: USB type 2JCIE-BU01)

In this article, we will get the value of the environmental sensor from Raspberry Pi. I uploaded it to a Google spreadsheet.

Try it out, from sensing to logging ** Can the functions up to this point be realized with only inexpensive general-purpose equipment? ** ** I am surprised. (Instead, there are many addictive points that make my heart break, but lol)

Things necessary

** ・ RaspberryPi ** (Pi3Model B is used this time) **-Python execution environment ** (This time, pyenv uses Python 3.7.6) ** ・ Google account ** (required to use spreadsheet) ** ・ Smartphone ** (for BLE Scanner setting) ** ・ Omron environment sensor (BAG type) ** * USB type will be created separately

procedure

** ① Check the Bluetooth connection between the Raspberry Pi and the sensor ** ** ② Change the environmental sensor mode to Broadcaster ** ** ③ Get the measured value of the environmental sensor with Python ** ** ④ Hit the GAS API from Python to write data to the spreadsheet ** ** ⑤ Script execution regularly **

Basically, Raspberry Pi is the starting point (central) of processing.

Method

① Check the Bluetooth connection between the Raspberry pi and the sensor

Confirmation of recognition of environmental sensor

**-Check the operation of the environmental sensor ** Insert the batteries and make sure the lamp lights up for a moment

**-Scan for Bluetooth devices ** Execute the following command on Raspberry Pi

sudo hcitool lescan
LE Scan ...
DD:CC:BB:AA:77:66 Env

If you see the name "Env", this is the MAC address of the environmental sensor. If it does not come out, check the battery contact and the Bluetooth enable of Raspberry Pi.

Recognition confirmation with bluepy

bluepy is a library for accessing Bluetooth Low Energy (BLE) in Python (class definition)

Here, we will check if bluepy can recognize the environmental sensor.

** ・ Installation of required packages ** Install the following

sudo apt install libglib2.0-dev

** ・ Install bluepy **

Install with pip with the following command

pip install bluepy

** ・ Grant permissions to bluepy ** You need to give bluepy Sudo privileges for scanning.

Go to the folder where bluepy is installed and

cd ~.pyenv/versions/3.7.6/lib/python3.7/site-packages/bluepy

Grant Sudo privileges to bluepy-helper with the following command

sudo setcap 'cap_net_raw,cap_net_admin+eip' bluepy-helper

**-Creating a scanning code ** Create the code below

ble_scan.py


from bluepy import btle

class ScanDelegate(btle.DefaultDelegate):
    def __init__(self): #constructor
        btle.DefaultDelegate.__init__(self)

    def handleDiscovery(self, dev, isNewDev, isNewData):  #Scan handler
        if isNewDev:  #When a new device is found
            print('found dev (%s)' % dev.addr)

scanner = btle.Scanner().withDelegate(ScanDelegate())
while True:
    scanner.scan(10.0) 

The above code works, What is important is ** Scanner class ** ** DefaultDelegate class ** ** ScanEntry class ** is.

In the above code **-Scanner.scan method: ** Set the argument (10 seconds in the above case) as the timeout value and execute the delegate specified by the Scanner.withDelegate method. **-DefaultDelegate class: ** Inheritance class for creating a delegate for the argument of Scanner.withDelegate. Describe specific processing in handleDiscovery method **-ScanEntry class: ** Holds device information acquired by Scanner.scan. The argument "dev" of handleDiscovery is equivalent Using,

** "Set 10 seconds as the timeout value and keep displaying the Mac address of the newly found device" **.

** ・ Execution of scanning code **

python ble_scan.py

As shown below, if the same MAC address as in LE Scan comes out, it is successful. (Note that the MAC address is lowercase)

found dev (dd:cc:bb:aa:77:66)

With the above, we have confirmed the environmental sensor recognition in bluepy.

② Change the mode of the environment sensor to Broadcaster

As described in this article, For data transmission (advertisement) of BLE devices ** ・ Connect mode: ** Get data by two-way communication **-Broadcaster: ** One-sided data transmission from device to central There are two types, ** Broadcaster is advantageous in terms of power consumption **. (Conversely, connect mode allows complicated operations to change device settings)

To increase the operating time of battery-powered environmental sensors Change to Broadcaster and proceed with the following operations

Install BLE Scanner

A tool called BLE Scannear used to change modes, Please install it on your smartphone from the following Android iOS

Change the advertisement setting of the environmental sensor

Change the advertisement settings and set the mode to Broadcaster. You can also change the advertisement transmission interval here (for applications that require speed)

**-Open BLE Scanner and CONNECT to the device labeled "Env" ** Screenshot_20200504-214958.png

** ・ "0C4C3042 ~" is the advertisement setting, so look for it ** Screenshot_20200504-215138.png Screenshot_20200504-214713.png

** ・ How to read the advertisement settings in (2) above ** If you read the OMRON Official User's Manual, it will look like the figure below. 環境センサADVSetting.png ** ・ If you want to change to broadcast mode: Set Beacon Mode to 02 or 04 ** ** ・ If you want to shorten the update interval: Reduce the advertisement transmission interval ** It is necessary to change.

Beacon Mode is wondering whether it is 02 or 04, but This time, we will focus on the environmental index Before:0808A0000A003200 08 00 After:0808A0000A003200 04 00 will do. (If you use an accelerometer, set it to 02)

** ・ Change advertisement settings ** Click ③ "W" in the above figure, change it as shown in the figure below, and press OK. Screenshot_20200504-233253.png

** ・ Confirmation of change ** If you go back to the top and the device name changes to "EP", you are successful. Screenshot_20200504-233429.png

Shorten the update interval of the environmental sensor

The sensor value update interval is 300 seconds (5 minutes) by default. This is too long! I think that there are many people, so I will change it short.

** ・ "0C4C3011 ~" is the update interval setting, so look for it ** Following the previous operation, connect to the BLE Scanner device "EP" and Look for "0C4C3011 ~" Screenshot_20200504-234845.png Screenshot_20200504-234917.png

** ・ How to read the update interval setting in (2) above ** It is the update interval in seconds converted to hexadecimal. However, because it is a beginner crying specification that the upper byte and the lower byte are reversed ** be careful. This time, enter "0A00" to change the interval to 10 seconds as shown in the figure below. 環境センサ更新間隔Setting.png

** ・ Confirmation of change of update interval ** If you press "R" and the contents of Hex are updated, it is successful. Screenshot_20200504-235058.png

If you click "RAW DATA" on the top screen, you can see the measured values of the sensor. Screenshot_20200504-234654.png

The specifications of the measured values are as shown in the figure below. ADVセンサ値仕様_EP.png

③ Get the measured value of the environmental sensor with Python

Since it was found that the measured value of the sensor can be obtained in (2), Increase the versatility of processing with Python.

Create a class for acquiring sensor measurements

Create a class to get sensor measurements with bluepy. In order to pass the processing at the time of scanning as a delegate as in ① and time, Create code like the one below.

The acquired sensor measurement value is stored as a dict for each sensor type in the variable "sensorValue".

omron_env_broadcast.py


from bluepy import btle
import time
import struct

class ScanDelegate(btle.DefaultDelegate):
    #constructor
    def __init__(self):
        btle.DefaultDelegate.__init__(self)
        #Variables for holding sensor data
        self.sensorValue = None

    #Scan handler
    def handleDiscovery(self, dev, isNewDev, isNewData):  
        #When a new device is found
        if isNewDev or isNewData:  
            #Extract advertisement data
            for (adtype, desc, value) in dev.getScanData():  
                #Data retrieval is executed when the environment sensor is used.
                if desc == 'Manufacturer' and value[0:4] == 'd502':
                    #Take out the sensor type (EP or IM)
                    sensorType = dev.scanData[dev.SHORT_LOCAL_NAME].decode(encoding='utf-8')
                    #Extraction of sensor data for EP
                    if sensorType == 'EP':
                        self.decodeSensorData_EP(value)
                    #Extraction of sensor data during IM
                    if sensorType == 'IM':
                        self.decodeSensorData_IM(value)

    #Take out sensor data and convert it to dict format (in EP mode)
    def decodeSensorData_EP(self, valueStr):
        #Sensor data from character strings(6th and subsequent characters)Only take out and convert to binary
        valueBinary = bytes.fromhex(valueStr[6:])
        #Convert binary sensor data to integer Tuple
        (temp, humid, light, uv, press, noise, discomf, wbgt, rfu, batt) = struct.unpack('<hhhhhhhhhB', valueBinary)
        #Store in dict type after unit conversion
        self.sensorValue = {
            'SensorType': 'EP',
            'Temperature': temp / 100,
            'Humidity': humid / 100,
            'Light': light,
            'UV': uv / 100,
            'Pressure': press / 10,
            'Noise': noise / 100,
            'Discomfort': discomf / 100,
            'WBGT': wbgt / 100,
            'BatteryVoltage': (batt + 100) / 100
        }
    
    #Take out sensor data and convert it to dict format (in IM mode)
    def decodeSensorData_IM(self, valueStr):
        #Sensor data from character strings(6th and subsequent characters)Only take out and convert to binary
        valueBinary = bytes.fromhex(valueStr[6:])
        #Convert binary sensor data to integer Tuple
        (temp, humid, light, uv, press, noise, accelX, accelY, accelZ, batt) = struct.unpack('<hhhhhhhhhB', valueBinary)
        #Store in dict type after unit conversion
        self.sensorValue = {
            'SensorType': 'IM',
            'Temperature': temp / 100,
            'Humidity': humid / 100,
            'Light': light,
            'UV': uv / 100,
            'Pressure': press / 10,
            'Noise': noise / 100,
            'AccelerationX': accelX / 10,
            'AccelerationY': accelY / 10,
            'AccelerationZ': accelZ / 10,
            'BatteryVoltage': (batt + 100) / 100
        }

In the following part, the sensor measurement value is acquired from the advertisement data.
                #Data retrieval is executed when the environment sensor is used.
                if desc == 'Manufacturer' and value[0:4] == 'd502':
                    #Take out the sensor type (EP or IM)
                    sensorType = dev.scanData[dev.SHORT_LOCAL_NAME].decode(encoding='utf-8')
                    #Extraction of sensor data for EP
                    if sensorType == 'EP':
                        self.decodeSensorData_EP(value)
                    #Extraction of sensor data during IM
                    if sensorType == 'IM':
                        self.decodeSensorData_IM(value)

The above process was created based on the following. ・ Environmental sensor can be identified by "desc ='Manufacturer' & value 0-4 characters is'd502'" -Sensor modes and measured values are held in the form shown in the following [Reference]. -Since the acquired measured value is binary data and the format differs depending on whether the mode is EP or IM, create a method "decodeSensorData" for each mode to convert it to the numerical Dict format for ease of use.

** [Reference] Results of debugging advertisement data in BroadCast mode with VS Code ** bluepy取得Broadcastアドバタイズデータ.png

Creating main code

Create the main code to execute the above sensor value acquisition class. As in (2), set the sensor value acquisition class in the Scanner delegate and set it. Scan and run.

omron_env_toSpreadSheet.py


from bluepy import btle
from omron_env_broadcast import ScanDelegate

#omron_env_broadcast.Set the sensor value acquisition delegate of py to execute at scan time
scanner = btle.Scanner().withDelegate(ScanDelegate())
#Scan to get sensor value (timeout 5 seconds)
scanner.scan(5.0)
#Display the temperature as a trial
print(scanner.delegate.sensorValue['Temperature'])

#Describe the process of uploading to Google Spreadsheet in ④

Try running from the console

python omron_env_toSpreadSheet.py
25.98

You have now obtained the sensor readings in Python. The acquired measurements are stored in scanner.delegate.sensorValue.

④ Hit the GAS API from Python to write data to the spreadsheet

Output the acquired data to a Google spreadsheet.

** * Most people want to output other than spreadsheets, so ** ** In that case, skip this chapter and move to ⑤ ** (There are many options such as CSV output with Pandas and visualization with Ambient)

Create GAS script

Perform this work on your PC.

Go to Google Spreadsheets and go to Create a spreadsheet like the one below Note that the sheet name should be the same as the device name ** SpreadSheet.png

Select "Tools"-> "Script Editor" and select Create a GAS script like the one below

postSensorData.gs


var spreadsheetId = '******'//← Enter the spreadsheet ID

//Receive the posted data
function doPost(e){
  var data = [      
    e.parameter.Date_Master, //Master date and time
    e.parameter.Date, //Measurement date and time
    e.parameter.Temperature, //temperature
    e.parameter.Humidity, //Humidity
    e.parameter.Light, //Illuminance
    e.parameter.UV, // UV    
    e.parameter.Pressure, //pressure
    e.parameter.Noise, //noise
    e.parameter.BatteryVoltage, //Battery voltage
    new Date(), //Upload end time
  ];
  //Describe the acquired data in the log
  Logger.log(new Date());
  //Write a row of data to a spreadsheet
  addData(e.parameter.DeviceName, data);
  return ContentService.createTextOutput("ok");
}

//Write a row of data to a spreadsheet
function addData(sheetName, data){
  var sheet = SpreadsheetApp.openById(spreadsheetId).getSheetByName(sheetName);
  sheet.appendRow(data);
}

** * Apparently, the Function name must be "doPost" to post using the API **

https://docs.google.com/spreadsheets/d/AAAAAAA/edit

If so, the "AAAAAAA" part corresponds

Publish the script as a Web API

"Publish" → "Introduced as a web application" webアプリ公開.png

The following screen will appear, so ・ Project version → Version you want to make into API (At first, select "New" to OK) ・ Execute the app as → Me (self) ・ Who has access to the app → Anyone, even anonimous (all (including anonymous users)) Select to update (Deploy) GASのAPI公開.png

If successful, the following screen will appear. Make a note of the API URL as you will use it later. API_URL.png

API external execution test

** * From here, the work will be done on the Raspberry Pi side ** Hit the API with curl to send the appropriate data and test if it can be POSTed from the outside. Execute the following command (install curl if it is not included)

curl -L [API URL] -F 'sheetName=[Sheet name]' -F "Date_Master=4.1" -F "Date=4.2" -F "SensorType=4.2" -F "Temperature=4.4" -F "Humidity=4.4" -F "Light=4.1" -F "UV=4.2" -F "Pressure=4.4" -F "Noise=4.4" -F "BatteryVoltage=4.4" -F "DeviceName=[Sheet name]"

Success if the values are entered in the spreadsheet spreadsheet_post_test.png

Install Requests

Install the Requests library to perform POST on the Python side

pip install requests

Added upload process to Python code

In the code "omron_env_toSpreadSheet.py" created in ③, Add the process of uploading data by hitting the API of the GAS script.

omron_env_toSpreadSheet.py


from bluepy import btle
from omron_env_broadcast import ScanDelegate
from datetime import datetime, timedelta
import requests

#Get the current time
date = datetime.today()
#Round the current time in minutes
masterDate = date.replace(second=0, microsecond=0)
if date.second >= 30:
    masterDate += timedelta(minutes=1)

#omron_env_broadcast.Set the sensor value acquisition delegate of py to execute at scan time
scanner = btle.Scanner().withDelegate(ScanDelegate())
#Scan to get sensor value
scanner.scan(5.0)

######Processing to upload to Google Spreadsheet######
#Upload only when the sensor value is not None
if scanner.delegate.sensorValue is not None:
    #Device name
    deviceName = '****'#← Make it the same name as the sheet name of the spreadsheet
    #Data to POST
    data = {
        'DeviceName': deviceName,
        'Date_Master': str(masterDate),
        'Date': str(date),
        'SensorType': str(scanner.delegate.sensorValue['SensorType']),
        'Temperature': str(scanner.delegate.sensorValue['Temperature']),
        'Humidity': str(scanner.delegate.sensorValue['Humidity']),
        'Light': str(scanner.delegate.sensorValue['Light']),
        'UV': str(scanner.delegate.sensorValue['UV']),
        'Pressure': str(scanner.delegate.sensorValue['Pressure']),
        'Noise': str(scanner.delegate.sensorValue['Noise']),
        'BatteryVoltage': str(scanner.delegate.sensorValue['BatteryVoltage'])
    }
    #API URL
    url = 'https://script.google.com/macros/s/******/exec'#← Enter the API URL
    #POST data to API
    response = requests.post(url, data=data)

Run the Python script to check the upload

Execute the above Python script with the following command

python omron_env_toSpreadSheet.py

As shown in the figure below, if the sensor value is entered in the spreadsheet, it is successful. sensor_spreadsheet.png

⑤ Periodic execution of script

With the above method, it is troublesome to execute the script every time, so Automate using the periodic package "cron". It looks like a Windows task scheduler.

Enable cron

It may be disabled by default, so enable it by referring to here.

**-Checkpoint 1: Check rsyslog.conf ** It will not work if "cron" in /etc/rsyslog.conf is commented out. In my case, it was commented out as follows, so

###############
#### RULES ####
###############

#
# First some standard log files.  Log by facility.
#
auth,authpriv.*                 /var/log/auth.log
*.*;auth,authpriv.none          -/var/log/syslog
#cron.*                         /var/log/cron.log
daemon.*                        -/var/log/daemon.log

After uncommenting as below,

cron.*                         /var/log/cron.log

Restart rsyslog with the following command.

sudo /etc/init.d/rsyslog restart

** ・ Checkpoint 2: Change log level ** In / etc / default / cron, specify the items to be described in the log when cron is executed. By default, it seems that the log is not output as shown below, so

# For quick reference, the currently available log levels are:
#   0   no logging (errors are logged regardless)
#   1   log start of jobs
#   2   log end of jobs
#   4   log jobs with exit status != 0
#   8   log the process identifier of child process (in all logs)
#
#EXTRA_OPTS=""

Uncomment EXTRA_OPTS as shown below

EXTRA_OPTS='-L 15'

will do. It means that all will be output with 1 + 2 + 4 + 8 = 15.

Restart cron with the following command.

sudo /etc/init.d/cron restart

If cron.log is generated in / var / log, it is successful. (Please check here if you want to see the execution history of cron)

Schedule execution with cron

Register periodic execution with cron

** ・ Editing crontab ** Open crontab with the following command

crontab -e

When asked which editor to open, select the editor you like (nano recommended for beginners)

# Edit this file to introduce tasks to be run by cron.
# 
# Each task to run has to be defined through a single line
# indicating with different fields when the task will be run
: Various things continue

There are various comments written as above, At the very end, describe the timing and command you want to execute. The timing format is [here](http://tech.junkpot.net/archives/721/crontab-%E3%81%A7%E7%B4%B0%E3%81%8B%E3%81%84%E3 % 82% B9% E3% 82% B1% E3% 82% B8% E3% 83% A5% E3% 83% BC% E3% 83% AB% E3% 81% AE% E8% A8% AD% E5% AE Please refer to% 9A.html)

This time, since it is executed every 5 minutes, the following contents are described.

*/5 * * * * [Python full path] [omron_env_toSpreadSheet.full path of py] >/dev/null 2>&1

** ・ Start cron ** Start cron with the following command

sudo /etc/init.d/cron start

Wait for a while above and you're done if the spreadsheet is updated every 5 minutes!

in conclusion

Increase the number of sensors and realize your dream IoT house! ** It looks like you can do it. Omron's sensors are multifunctional and of high quality, but they are not cheap enough to buy pompoms, so I will try even a cheaper sensor!

Recommended Posts

Periodically log the value of Omron environment sensor with Raspberry Pi
Logging the value of Omron environment sensor with Raspberry Pi (USB type)
Log the value of SwitchBot thermo-hygrometer with Raspberry Pi
Take the value of SwitchBot thermo-hygrometer with Raspberry Pi
Using the digital illuminance sensor TSL2561 with Raspberry Pi
I tweeted the illuminance of the room with Raspberry Pi, Arduino and optical sensor
Periodically notify the processing status of Raspberry Pi with python → Google Spreadsheet → LINE
I tried using the DS18B20 temperature sensor with Raspberry Pi
I tried to automate the watering of the planter with Raspberry Pi
Build a Tensorflow environment with Raspberry Pi [2020]
Use the Grove sensor on the Raspberry Pi
Improved motion sensor made with Raspberry Pi
Use PIR motion sensor with raspberry Pi
Inkbird IBS-TH1 value logged with Raspberry Pi
A memo to simply use the illuminance sensor TSL2561 with Raspberry Pi 2
Building a distributed environment with the Raspberry PI series (Part 1: Summary of availability of diskless clients by model)
Note: Prepare the environment of CmdStanPy with docker
Prepare the execution environment of Python3 with Docker
Observe the Geminids meteor shower with Raspberry Pi 4
Get CPU information of Raspberry Pi with Python
Measure CPU temperature of Raspberry Pi with Python
Realize a super IoT house by acquiring sensor data in the house with Raspberry Pi
GPGPU with Raspberry Pi
Display images taken with the Raspberry Pi camera module
Try to visualize the room with Raspberry Pi, part 1
[C language] [Linux] Get the value of environment variable
Preparing the execution environment of PyTorch with Docker November 2019
DigitalSignage with Raspberry Pi
Let's operate GPIO of Raspberry Pi with Python CGI
I want to be notified of the connection environment when the Raspberry Pi connects to the network
Make a note of what you want to do in the future with Raspberry Pi
Control the display of RGB LED matirix electric bulletin board freely with Raspberry Pi 3B +
I tried to move ROS (Melodic) with the first Raspberry Pi (Stretch) at the beginning of 2021
Switch the setting value of setting.py according to the development environment
Unify the environment of the Python development team starting with Poetry
Check! Get sensor data via Bluetooth with Raspberry Pi ~ Preparation
Production of temperature control system with Raspberry Pi and ESP32 (1)
Construction of Cortex-M development environment for TOPPERS using Raspberry Pi
The true value of Terraform automation starting with Oracle Cloud
I sent the data of Raspberry Pi to GCP (free)
Try to extract the features of the sensor data with CNN
Mutter plants with Raspberry Pi
Building a distributed environment with the Raspberry PI series (Part 2: PiServer analysis and alternative system design)
Real-time classification of multiple objects in the camera image with deep learning of Raspberry Pi 3 B + & PyTorch
Log in to Raspberry PI with ssh without password (key authentication)
Find the optimal value of a function with a genetic algorithm (Part 2)
March 14th is Pi Day. The story of calculating pi with python
I learned how the infrared remote control works with Raspberry Pi
Notify LINE of body temperature from BLE thermometer with Raspberry Pi # 1
Notify LINE of body temperature from BLE thermometer with Raspberry Pi # 2
Graph display of household power consumption with 3GPI and Raspberry Pi
Solve the initial value problem of ordinary differential equations with JModelica
Raspberry Pi --1 --First time (Connect a temperature sensor to display the temperature)
Tips: [Python] Calculate the average value of the specified area with bedgraph
I made an npm package to get the ID of the IC card with Raspberry Pi and PaSoRi
Read the data of the NFC reader connected to Raspberry Pi 3 with Python and send it to openFrameworks with OSC
[Raspberry Pi] Stepping motor control with Raspberry Pi
Find the definition of the value of errno
Use vl53l0x with Raspberry Pi (python)
Servo motor control with Raspberry Pi
Extract the maximum value with pandas.