[PYTHON] I made an original program guide using the NHK program guide API.

Motivation

• I'm looking for material, Qiita.com: Can be used by individuals! Recommended API list.
• In the comment section, "NHK Program Guide API is also good", so it seemed easy to take a peek.
• I think it's a good practice to get it from REST API (JSON format), process and format it, and create HTML.
• ** I made an original program guide using the NHK program API. ** (I'm sorry to see you, but ...)

API key acquisition flow

1. Register your account at NHK Program API.
2. After email authentication, register the application name: (your favorite name) and URL: (unregistered).
3. When registration is complete, you will be issued an ʻAPI key`. ** This is required for acquisition. ** **

API overview

• There are four types of APIs that are open to the public.

• The Program List API used this time has the following structure.

https://api.nhk.or.jp/v2/pg/list/{area}/{service}/{date}.json?key={apikey}

• The response format is json, and ** the usage limit is 300 times / day **.
• The following example is a part obtained in area: 130 (Tokyo), service: g1 (NHK General TV), date: 2020-04-07.

{
  "list": {
    "g1": [
      {
        "id": "2020040704706",
        "event_id": "04706",
        "start_time": "2020-04-07T04:02:00+09:00",
        "end_time": "2020-04-07T04:30:00+09:00",
        "area": {
          "id": "130",
          "name": "Tokyo"
        },
        "service": {
          "id": "g1",
          "name": "NHK General 1",
          "logo_s": {
            "url": "//www.nhk.or.jp/common/img/media/gtv-100x50.png ",
            "width": "100",
            "height": "50"
          },
          "logo_m": {
            "url": "//www.nhk.or.jp/common/img/media/gtv-200x100.png ",
            "width": "200",
            "height": "100"
          },
          "logo_l": {
            "url": "//www.nhk.or.jp/common/img/media/gtv-200x200.png ",
            "width": "200",
            "height": "200"
          }
        },
        "title": "Darwin is here! "Infiltrate the turbulent lion school! Train the king of beasts !!"",
        "subtitle": "Growing lion children learn hunting techniques and parenting techniques from adults who play the role of teachers in a herd. Unscrupulous students will be expelled! ?? Adhere to the turbulent days of the school drama!",
        "content": "In the first new year, we will introduce Lion's "school". Growing lion children learn everything from adult teachers to hunting and parenting to repelling rivals in a herd. However, rival hyenas have no teeth at all, and when hunting, they can be seen in full view and easily escape to prey. Even more unscrupulous students will be expelled from school! ?? Infiltrate a turbulent class that is as good as a school drama! Song: MISIA",
        "act": "[Talk] Mayuko Wakuda, Naoki Tatsuta, Machiko Toyoshima, Takayuki Yamada, Inori Minase",
        "genres": [
          "0802",
          "1000"
        ]
      },
{Omitted below}

Data acquisition

• Hit Program List API with the obtained API key.
• There are four acquisition services: g1: Comprehensive 1, e1: E-Tele, s1: BS1, and s3: BS Premium.
• Use pandas'json_normalize () to convert the json format to a dataframe.

import pandas as pd
import json
import requests
import datetime

#Set the obtained API key
apikey = 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'

# 130:Tokyo
area = '130'
# g1:NHK General 1,e1:E-tele,s1:BS,s3:BSP
service = ['g1','e1','s1','s3']
#Date and time
date = datetime.date.today()

all_results = pd.DataFrame(index=None, columns=[])
for i in range(len(service)):
    url = 'https://api.nhk.or.jp/v2/pg/list/{0}/{1}/{2}.json?key={3}'.format(area,service[i],date,apikey)
    request_get = requests.get(url)
    if request_get.status_code != 200:
        print('The data of NHK program guide API cannot be acquired.')
        break
    result = pd.json_normalize(request_get.json(), ['list',[service[i]]])
    all_results = pd.concat([all_results, result])

all_results = all_results[~all_results['title'].str.contains('Broadcast suspension')]

Summary of acquired data:

<class 'pandas.core.frame.DataFrame'>
Int64Index: 235 entries, 0 to 37
Data columns (total 22 columns):
 #   Column                 Non-Null Count  Dtype 
---  ------                 --------------  ----- 
 0   id                     235 non-null    object
 1   event_id               235 non-null    object
 2   start_time             235 non-null    object
 3   end_time               235 non-null    object
 4   title                  235 non-null    object
 5   subtitle               235 non-null    object
 6   content                235 non-null    object
 7   act                    235 non-null    object
 8   genres                 235 non-null    object
 9   area.id                235 non-null    object
 10  area.name              235 non-null    object
 11  service.id             235 non-null    object
 12  service.name           235 non-null    object
 13  service.logo_s.url     235 non-null    object
 14  service.logo_s.width   235 non-null    object
 15  service.logo_s.height  235 non-null    object
 16  service.logo_m.url     235 non-null    object
 17  service.logo_m.width   235 non-null    object
 18  service.logo_m.height  235 non-null    object
 19  service.logo_l.url     235 non-null    object
 20  service.logo_l.width   235 non-null    object
 21  service.logo_l.height  235 non-null    object
dtypes: object(22)
memory usage: 42.2+ KB

Data processing

• Replaced with date format, calculated broadcast time, added HTML code string for in-page link

#Convert to date format, calculate broadcast time
all_results['start_time'] = pd.to_datetime(all_results['start_time'], format='%Y/%m/%d %H:%M')
all_results['end_time'] = pd.to_datetime(all_results['end_time'], format='%Y/%m/%d %H:%M')
all_results['airtime'] = all_results['end_time'] - all_results['start_time']
all_results['link'] = all_results['id']
#Creating in-page link information
func = lambda x: x.replace(x, f'<div id="{x}">Program details</div>')
all_results['link'] = all_results['link'].apply(func)

#time. Get hours and minutes. In addition, add the time line name of the program guide
tmp = pd.concat([all_results['start_time'].dt.month ,all_results['start_time'].dt.day,
                 all_results['start_time'].dt.hour, all_results['start_time'].dt.minute,
                 (all_results['start_time'].dt.day.astype(str)+all_results['start_time'].dt.strftime('%H'))], axis=1)
tmp.columns = ['month', 'day', 'hour', 'minute','time_bins']

#Combine the retrieved data with the new column
all_results = pd.concat([all_results, tmp], axis=1)

#Aggregation for program listings
data = all_results.iloc[:,[23,0,1,2,3,4,5,6,7,8,9,10,11,12,22,24,25,26,27,28]]

Data shaping (1)

• Now that the necessary data for the program guide has been prepared, create the final table and save it in HTML format.

#Creating a data frame for the final program guide
tv_index = data.time_bins.sort_values().unique()
tv_table = pd.DataFrame(index=tv_index, columns=['H', 'g1', 'e1', 's1', 's3']).fillna('')

#Combine content and insert in time zone.
cell = ''
for s in range(len(service)):
    for i in range(len(tv_table.index)):
        tmp = data[(data['time_bins'] == tv_table.index[i]) & (data['service.id'] == service[s])]

        for c in range(len(tmp)):
            cell += '[' + str(tmp['minute'].iloc[c]) + '~]'
            cell += tmp['title'].iloc[c]
            cell += f'<a href="#{tmp["id"].iloc[c]}">▼</a><br>'
        tv_table[service[s]].iloc[i] = cell
        cell = ''

#Add time zone
for h in range(len(tv_index)):
    tv_table['H'].iloc[h] = int(tv_index[h][-2:])
tv_table.columns = ['H', 'NHK General', 'E-tele', 'BS', 'BSP']

Data shaping (2)

• From the program title, create a table to provide detailed program information as an in-page link.

detail_table = data.stack().reset_index()
detail_table.drop(columns='level_0', inplace=True)
detail_table.columns = ['Type', 'value']
detail_table.set_index('Type', drop=True, inplace=True)

Data shaping (3)

• Use pandas to_html to create an html file.

• Definition information of html file

html_template = '''
<!doctype html>
<html lang="ja">
  <head>
    <!-- Required meta tags -->
    <meta charset="utf-8">
    <meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">

    <!-- Bootstrap CSS -->
    <link rel="stylesheet" href="https://stackpath.bootstrapcdn.com/bootstrap/4.4.1/css/bootstrap.min.css" integrity="sha384-Vkoo8x4CGsO3+Hhxv8T/Q5PaXtkKtu6ug5TOeNV6gBiFeWPGFN9MuhOf23Q9Ifjh" crossorigin="anonymous">
    <title>Private program schedule</title>　<!-- #1 -->
  </head>
  <body>
    <h2>&nbsp;I made an original program guide using the NHK program guide API.{date}Edition</h2>　<!-- #2 -->

    <!-- Optional JavaScript -->
    <!-- jQuery first, then Popper.js, then Bootstrap JS -->
    <script src="https://code.jquery.com/jquery-3.4.1.slim.min.js" integrity="sha384-J6qa4849blE2+poT4WnyKhv5vZF5SrPo0iEjwBvKU7imGFAV0wwj1yYfoRSJoZ+n" crossorigin="anonymous"></script>
    <script src="https://cdn.jsdelivr.net/npm/[email protected]/dist/umd/popper.min.js" integrity="sha384-Q6E9RHvbIyZFJoft+2mJbHaEWldlvI9IOYy5n3zV9zzTtmI3UksdQRVvoxMfooAo" crossorigin="anonymous"></script>
    <script src="https://stackpath.bootstrapcdn.com/bootstrap/4.4.1/js/bootstrap.min.js" integrity="sha384-wfSDF2E50Y2D1uUdj0O3uMBJnjuUD4Ih7YwaYd1iqfktj0Uod8GCExl3Og8ifwB6" crossorigin="anonymous"></script>
    <style type="text/css"> th {{ text-align: center; background-color: #f5f5f5}}</style>
    <div class="container-fluid"> <!-- #3 -->
        {table}
    </div>
    <div class="container-fluid"> <!-- #4 -->
        {table1}
    </div>
    <div>Information provision:ＮＨＫ</div> <!-- #5 -->
</body>
</html>

The following contents have been added to the definition information.

1. title: Page title
2. h2: Heading + {date}
3. Program table of 4 stations table {table}-> table
4. Program details table {table1}-> table1
5. Credit display

• Save html file format

table = tv_table.to_html(classes=['table', 'table-bordered', 'table-hover'],escape=False)
table1 = detail_table.to_html(classes=['table', 'table-bordered', 'table-hover'],escape=False)
html = html_template.format(table=table, table1=table1, date=date)

with open('g1_table_' + str(date) + '.html', 'w') as f:
    f.write(html)

Click ▼ to scroll to the detailed program data with the link on the page.

Reference site

• Create a table with images based on pandas DataFrame
• Updated the bootstrap version from the above site information.

Summary

• I wanted to see how much pandas.read_json and pandas.to_html could be used, so it was a great practice.
• pandas.read_json ended up in a light entrance, so I'll look for materials again.
• It was the harvest that I could understand how to use pandas.to_html in the first place.
• However, even if I introduced Bootstrap, my ** Web power ** was low, so it was a disappointing result **.
• Since the news is a concern, if you execute it in the morning, you can check it without skipping to the program guide on the original site.
• Please check Notes on using the NHK Program Guide API and follow the usage and dosage.