Make JSON into CSV with Python from Splunk

Display the status of COVID19 infection in Japan with Splunk (Revised 2nd edition) uses python to download data from HP and process it into CSV.

Splunk doesn't have pandas, and it seems that the threshold is high for Windows people, so I tried my best with python in Splunk.

code

dl_json_to_csv2.py

#!/usr/bin/env python
# -*- coding: utf-8 -*-

import requests
import json
import csv

headers={'accept': 'application/json', 'content-type': 'application/json'}
url = 'https://www3.nhk.or.jp/news/special/coronavirus/data/47patients-data.json'

response=requests.get(url,headers=headers)
json_obj = response.json()

##Storage of date data
header=["pref"]+[i for i in list(json_obj['category'])]

##Storage of infected person number data
lst = [[] for _ in range(len(json_obj['data47']))] ##Initialization
for j,json in enumerate(json_obj['data47']):
    lst[j].append(json['name'])
    for json2 in json['data']:
        lst[j].append(json2)

##CSV write
with open('japan_covid19.csv', 'w', encoding='UTF-8',newline='') as f:
    writer = csv.writer(f)
    writer.writerow(header)
    writer.writerows(lst)

I managed to get the above code

splunk

$SPLUNK_HOME/bin/splunk cmd python3 ./dl_json_to_csv2.py

Operation confirmed with

Correction by pythonista

dl_json_to_csv2.py

#!/usr/bin/env python
# -*- coding: utf-8 -*-

import requests
import json
import csv

headers={'accept': 'application/json', 'content-type': 'application/json'}
url = 'https://www3.nhk.or.jp/news/special/coronavirus/data/47patients-data.json'

response=requests.get(url,headers=headers)
json_obj = response.json()

lst = [['pref']+json_obj['category']] + [[json['name']]+json['data'] for json in json_obj['data47']]

with open('japan_covid19.csv', 'w', encoding='UTF-8',newline='') as f:
        writer = csv.writer(f)
        writer.writerows(lst)

: astonished: Too advanced skills are magic.

Commentary

python

a=['1']
b=['2','3']
c=[a+b]
d=['4']
c+d

The result of [['1', '2', '3'], '4'] If you enclose list in[], it becomes a list in the list.

python

d={'data47':[{'name':'4','data':['5','6','7']},{'name':'8','data':['9','10','11']}]}
e=[[json['name']] + json['data'] for json in d['data47']]
e

The result of [['4', '5', '6', '7'], ['8', '9', '10', '11']]

Therefore

python

a=['1']
b=['2','3']
c=[a+b]
d={'data47':[{'name':'4','data':['5','6','7']},{'name':'8','data':['9','10','11']}]}
e=[[json['name']] + json['data'] for json in d['data47']]
c+e

[['1', '2', '3'], ['4', '5', '6', '7'], ['8', '9', '10', '11']]

Finer

ʻE` is a list

--Whole inclusion: Expression with variables for variables in list --Expression: [json ['name']] + json ['data'] --Variable: json --List: d ['data47']

For example

python

d={'data47':[{'name':'4','data':['5','6','7']},{'name':'8','data':['9','10','11']}]}
e=[json['name'] + json['data'] for json in d['data47']]
e

Then Type Error: can only concatenate str (not" list ") to str. The return of json ['name'] is str, whereasjson ['data']is list list You can't combine unless you are comrades.

: smiling_imp: I understand something.

Summary

I made it while checking which array was used with type (). I was addicted to making a list in the list, and it turned out to be awkward code. ~~ Currently being corrected. ~~ I had pythonista correct it.

~~ When it's clean, I'd like to use it for the ~~ Splunk dashboard.