[PYTHON] COVID-19 Hokkaido Data Edition (2) Toward open data + automatic update

Lottery

COVID-19 Hokkaido Data Edition ① Initial data creation by scraping etc. COVID-19 Hokkaido Data Edition (2) Toward open data + automatic update ← This article! COVID-19 Hokkaido Data Edition ③ Full Automation

In ①, we summarized V0 in the above figure, but after that, this article will summarize the utilization of external APIs such as open data portals.

Challenges in V0

In V0, data acquisition relied on (1) scraping from the road website and (2) static CSV files provided by Sapporo City. This has the following problems:

--I want to use data that is properly published as open data, not scraped data. --Scraping is vulnerable to changes in the table structure of a website --Operational effort to push every time the CSV file is updated

Then what happens after V1 in the above figure? It will be very easy if the lower left flow is realized.

Currently, (probably) Excel-controlled tabular data is converted to PDF for easy viewing by users and published, or the data is posted directly in HTML, so-called machine-readable data. (I understand the intention of making materials that anyone can read immediately). I think it would have been difficult to collect data on this website if even positive patient data was provided in PDF (at least it was an HTML table, so I could read it by force). I hope many people will recognize that there is such a good thing if various information is distributed as raw data (CSV etc.) as open data.

If my feelings are realized, it will be in the lower left shape. I hope that happens in the near future.

If I could afford it, it would be realized in about two days.

Migration to open data

Hokkaido

The data of Hokkaido will be updated not only on the website but also on the Hokkaido Open Data Portal from time to time. The person in charge in Hokkaido uploads the latest CSV file to this portal at any time. As the name suggests, the data published here is open data, and anyone can access CSV data for free.

Sapporo

In addition, Sapporo City also operates an open data portal site Sapporo City ICT Utilization Platform DATA SMART CITY SAPPORO, and the number of consultations at the counter is open data. It was published as. Creating data is the work of the person in charge as in Hokkaido. Thank you.

It was a tremendous sense of speed (I think it was possible to make a smooth transition because of the soil of such a portal). In spite of the sense of speed that the image of the government is overturned, we have modified the previous script to correspond to the open data portal.

https://github.com/codeforsapporo/covid19hokkaido_scraping/tree/f3923df7f6a3781e94ef5b514c9d9ec7fe5aa4b1

• It goes back to the commit at the time of writing this article.

When main.py is executed, the data input to REMOTE_SOURCES in settings.py is patrolled and json etc. are spit out. The CSV data released this time can access the file through the API, but since it was necessary to read the CSV data after all, it is not an access via the API but a direct link to the resource. Convert the read CSV to dict and dump it to json.

On this repository, the execution of main.py is scheduled every 15 minutes and spits out jsons to the gh-pages branch. So, if the data of each portal is updated, it will be reflected in json etc. within 15 minutes. These jsons can be accessed from the outside without CORS restrictions (like). In other words, it functions as a pseudo API server </ b>. If those jsons are read by asynchronous communication on the front side, data update will be fully automated.