[PYTHON] I tried the Naruro novel API

Trigger

https://qiita.com/dely13/items/5e949a384161c961d8ce If you read this article and try it yourself after practicing ~~ play ~~, the result will be different → This article is 2017 So I tried to put out the latest (as of 10:00 on June 29, 2020)

The first half remains as it is

I will use @ dely13's article as it is

dely13.py

import pandas as pd
import requests
import numpy as np
import seaborn as sns
from scipy import stats
import matplotlib.pyplot as plt
%matplotlib inline

url = "http://api.syosetu.com/novelapi/api/"
#Specify API parameters in the dictionary
#Under this condition, output json format data in the order of comprehensive evaluation
payload = {'of': 't-gp-gf', 'order': 'hyoka','out':'json'}

st = 1
lim = 500

data = []
while st < 2000:
    payload = {'of': 't-gp-gf-n', 'order': 'hyoka',
          'out':'json','lim':lim,'st':st}
    r = requests.get(url,params=payload)
    x = r.json()
    data.extend(x[1:])
    st = st + lim
df = pd.DataFrame(data)

#Preprocessing('year'Add column,'title_len'Add column)
df['general_firstup'] = pd.to_datetime(df['general_firstup'])
df['year'] = df['general_firstup'].apply(lambda x:x.year)

df['title_len'] = df['title'].apply(len)

Please read the original article for details as it is really as it is

Main subject

In 2017

Interesting numbers. The average value is 17 characters, which is the same as the number of characters in haiku. In other words, the title of Naruro was haiku! Frog Poem and the sound of water jumping into a frog ...

I was told, but in 2020 ...?

df['title_len'].hist()

df['title_len'].describe()

Histogram diagram df ['title_len'] .hist () Data df ['title_len'] .describe ()

count 2000.000000 mean 24.179500 std 15.528356 min 2.000000 25% 12.000000 50% 21.000000 75% 32.000000 max 100.000000 Name: title_len, dtype: float64

Wwwwwww which increases 7 characters on average

And what's really interesting is from here

per_year.py

title_by_year = df.groupby('year')['title_len'].agg(['mean','count','std']).reset_index()
#plot
title_by_year.plot(x='year',y='mean') 
#data
title_by_year

Plot title_by_year.plot (x ='year', y ='mean') * mean = average

Aggregate title_by_year

• Data for 2020 is data up to June 29

Conclusion

** The title of 2019 will be Tanka ** The person who guessed in the 2017 article is amazing. It's Don Pisha.

Digression 1

Since it's a big deal, I'll try to find the maximum and minimum

title_by_year = df.groupby('year')['title_len'].agg(['mean','min','max']).reset_index()
#plot
title_by_year.plot(x='year')
#data
title_by_year.plot

Plot title_by_year.plot (x ='year')

Data title_by_year

Isn't this 100-character data exceeding the number of characters?

max_100.py

df[['ncode','title','year','title_len']].set_index('ncode').query('title_len==100')

... this isn't over 100 characters ...?

When I looked it up after writing the article, it was exactly 100 characters

Is there a character limit? That's what I'm fighting at the limit.

Digression 2

On the contrary, I was interested in short titles

mini_len.py

df.groupby('title_len')['title_len'].agg(['count']).head(9).T

List of correspondence between the number of characters and the number of works Since it has become longer, it is placed horizontally

title2_4.py

df[['title','year','title_len']].set_index('title').sort_values('title_len').query('title_len<5')

4 characters are excerpts

Even a few letters are famous. I was impressed that the former Moba people had a "title" in the four letters.

Impressions

Is it the influence of the animation of mobile novels that many beginners enter, if not as much as Moba (currently Ebu)? I was trained by Moba, so even if it's a little difficult to read, I'll read it if the content is interesting, but even so, the title is long. I'm addicted to this and this, which are rather long titles. (Ebudato this * Stemmer)

I wanted to try various things because I can narrow down the search conditions with the Naro API. What if you want to extract more than 2000 items ...