[PYTHON] Scraping the rainfall data of the Japan Meteorological Agency and displaying it on M5Stack

Overview

• In an apartment house, I couldn't see the rain from the window, so I often noticed the rain after going out to the entrance and returned to pick up my umbrella.
• In some office buildings (30 stories and above), the elevator hall was notified of precipitation with an umbrella sign.
• There is also a method of installing a precipitation sensor on the balcony to notify the alarm, but it is difficult in terms of securing power and durability of the sensor.
• Display the result of scraping the observation data page of the Japan Meteorological Agency on the server (Raspberry Pi) that is always running on M5Stack. https://www.data.jma.go.jp/obd/stats/data/mdrr/pre_rct/alltable/pre1h00.html

What to prepare

• Raspberry Pi3 Model B (4 is probably okay)

• Case for Raspberry Pi

• Heat sink for Raspberry Pi

• Raspberry Pi3 Model B B + compatible power supply set (5V 3.0A)

• M5Stack

• Power supply for M5Stack

• PC running Arduino IDE (Mac / Windows)

• The server also serves as the visitor notification system (always started) created last time. https://qiita.com/cami_oshimo/items/b4e4002f7c47dd00ba84

Environment

(1) Install Arduino IDE https://www.arduino.cc/en/main/software (2) Install M5Stack related libraries https://mag.switch-science.com/2018/02/28/getting-started-with-m5stack/ (3) Install Apache (http server) on Raspberri Pi

• The setup on the Raspberry Pi side is omitted.

Preparation

• Open the precipitation page of the Japan Meteorological Agency with Chrome's Developer Tool and check the location where the precipitation in the area is output.

Example) Naka-ku, Hiroshima City * I am working remotely from Hiroshima City

programming

• Scraping with Python code on a Raspberry Pi that is always running and outputting to ʻindex.html`
• Display ʻindex.html` on M5Stack

Main program If you change Hiroshima to another area, you can get the rainfall of that land.

/var/www/html/rain.php

<?php

mb_language("Japanese");//Processing language setting
mb_internal_encoding("UTF-8");//Character code specification
date_default_timezone_set('Asia/Tokyo');//set timezone

$url = "http://www.data.jma.go.jp/obd/stats/data/mdrr/pre_rct/alltable/pre1h00.h
tml";
 
//Loading a web page
$contents= file_get_contents($url);
 
//Character code conversion
$contents_utf = mb_convert_encoding($contents,"UTF-8","auto");
 
//Extracts the part enclosed by the specified character string.
$startString = '<td style="white-space:nowrap">Hiroshima (Hiroshima)*</td><td style="
text-align:right;white-space:nowrap;">';
$endString = '</td>';
 
$startPoint = mb_strpos($contents_utf,$startString) ;//$Find out where startString appears
$startPoint = $startPoint + mb_strlen($startString);//Starting position
//echo $startPoint;
//echo"";
 
$endPoint = mb_strpos($contents_utf,$endString,$startPoint );//$endString is next
Find out where it appears
//echo $endPoint;
//echo"";
 
$length = $endPoint - $startPoint;//Find the length of the string.
$rainfallString = mb_substr($contents_utf,$startPoint,$length)." mm"."\n\n"."GET
 Time"."\n".date("H:i:s")."\n";
echo $rainfallString;
file_put_contents("/var/www/html/index.html", $rainfallString);
?>

Shell script for PHP execution

/var/www/html/cron.php

#!/bin/sh
/usr/bin/php /var/www/html/rain.php

PHP execution result

/var/www/html/index.html

0.0 mm

GET Time
09:30:02

Make sure you can see it from your browser.

Auto-start setting (in this example, get precipitation every hour) */60 * * * * root sudo sh /var/www/html/cron.sh

Programming on the M5Stack side

rain.ino

#include <M5Stack.h>
#include <WiFi.h>
#include <HTTPClient.h>

const char* ssid       = "XXXXXXXXXXXXX";  // Wifi SSID
const char* password   = "XXXXXXXXXXXXX";  // Wifi Password

const long  gmtOffset_sec = 3600;
const int   daylightOffset_sec = 3600;


void setup()
{
  M5.begin();
  M5.Lcd.setBrightness(10);
  M5.Lcd.printf("Connecting to %s ", ssid);
  WiFi.begin(ssid, password);
  while (WiFi.status() != WL_CONNECTED) {
      delay(500);
      M5.Lcd.printf(".");
  }

  M5.Lcd.println(" CONNECTED");
  M5.Lcd.fillScreen(BLACK);
}

void getrain()
{
  M5.Lcd.fillScreen(BLACK);
  HTTPClient http;

  http.begin("http://xxx.xxx.xxx.xxx/index.html");  //IP address of Raspberry Pi
  int httpCode = http.GET();

  String result = http.getString();
  M5.Lcd.setTextSize(5);
  M5.Lcd.setCursor(0,0);
  M5.Lcd.print(result);
  
  http.end();

}

void loop()
{
  getrain();
  delay(xxxxxxx);  //From raspberry Pi`index.html`Interval to get (milliseconds)
}

important point

• If you increase the frequency of scraping too much, it will load the server on the other side and cause problems, so be careful.

Try using

• Precipitation can be seen on the smartphone app, but it is convenient because it can be easily checked by placing it at the front door.
• Initially, it was operated on Sakura's VPS (Linux) for about 2 years, but it was abolished due to the fixed monthly cost and moved to Raspberry Pi (about 1 month has passed, but there is no problem)