[PYTHON] Instant visualization of Qiita Advent Calender 2019

Instant visualization of Qiita Advent Calender 2019

This article is the 20th day article of All About Group (All About Co., Ltd.) Advent Calendar 2019. It's Christmas when I sleep a few more times. I believe that Santa Claus will really come now as an adult.

Nice to meet you for the first time, this is Akidukin14. I am usually in charge of machine learning of All About's advertisement distribution system and visualization of analysis results.

This is my first time writing an Advent Calendar, so I'll try what I have come up with.

What I did (overview)

Let's visualize the article data inside each calendar of Qiita Advent Calender (Only for articles posted on Qiita ...)

Contents (Topics)

--Background --What you did (bulleted list) --Result --What you did (details) --Extract the article text of the Advent calendar --Process the article text in natural language (separate writing) --Visualization with WordCloud --Reference --Advent Calender Index

background

(As of 12 / n 2019) ** Number of calendars: 770 ** ** Number of participants: 13,414 **

This is my first time to participate in Advent Calender, but the first thing I thought about was.

Pane. The number of calendars. Participant Pane.

Where is the calendar that you are interested in pinpointing ... ?? I thought that was my motive.

What I did (bulleted)

1. Extract the article text of the Advent calendar
2. Process the article text in natural language (separate writing)
3. Visualize with WordCloud

result

I will post the result first. If the result is enough, please check here. (I've done so much light, so I think the results are biased, I'm sorry ...)

Image 1

Image 2

Image 3

Image 4

Image 5

Image 6

Image 7

Image 8

Image 9

Image 10

Image 11

Image 12

Image 13

Image 14

Image 15

Image 16

Image 17

Image 18

Image 19

Image 20

Image 21

Image 22

Image 23

Image 24

Image 25

Image 26

Image 27

Image 28

Image 29

Image 30

Image 31

Image 32

1. Extract the article text of the Advent calendar

1. Hit Qiita API from Python to get article data

call_qiita_api

def call_qiita_api(header, per_page = None, query = None, page = None):
    ##api designation
    get_items_api = 'https://qiita.com/api/v2/items'
    params = {'per_page' : per_page
        , 'query' : query
        , 'page' : page}
    datas = requests.get(get_items_api, params = params, headers = header)
    return datas

2. This is an Advent calendar! Get data by targeting articles that you think

regs_body_text

###The code is really dirty...
def regs_body_text(text):
    ##Normalization pattern
    reg_pattern = re.compile('(\n|\t| |　|-|~|-|`|:|;|_|\*|\!|\?|！|？|\+|\$|#|\[|\])')
    tmp = re.sub(reg_pattern, '', text.lower())
    target_type = re.compile('(Advent calendar|adventcalendar)')
    if not re.search(target_type, tmp):
        return None, None
    calender_type = re.compile('(This article)\w+?(adventcalendar2019)')
    if not re.search(calender_type, tmp):
        return None, None,
    url_strings = re.search(calender_type, tmp).group()
    get_calender_type = re.sub('(This article|adventcalendar2019)', '', url_strings)
    return get_calender_type, tmp

2. Process the article text in natural language (separate writing)

1. Specify the part of speech with MeCab and divide it into words

parse_text

mecab = MeCab.Tagger('-Owakati')
mecab.parse('')

def parse_text(text, parser = mecab):
    part = ['noun','動noun']
    parsed_text = []
    t = parser.parseToNode(text)
    while t:
        parts = t.feature.split(',')
        if parts[0] in part:
            parsed_text.append(t.surface)
        t = t.next
    return parsed_text

3. Visualize with WordCloud

1. Supports drawing in 9x9 area at any time

make_wordcloud

##Visualize with WordCloud
keys_len = len(dataset.keys())
plot_picture = int(keys_len / 9) + 1
plot_area = np.arange(0,9,1).reshape(3,3)
keys = sorted(dataset.keys())
fp = FontProperties(fname = fonts)
k = 0
for pp in range(plot_picture):
    fig,axes = plt.subplots(nrows = 3, ncols = 3, figsize = (10,10))
    for i in range(9):
        sys.stdout.write('\r {}/{}'.format(k, keys_len))
        target_key = keys[k]
        wc = wordcloud.WordCloud(
                font_path = fonts
                , prefer_horizontal = 1
                , max_words = 300
                , background_color = 'white'
                , colormap = 'RdYlBu'
                , contour_color='pink'
                , width = 750
                , height = 750)
        n,m = [x.item() for x in np.where(plot_area == i)]
        plot_data = ' '.join([y for x in dataset[target_key]['parsed_text'] for y in x if not check_word(y)])
        wc_gen = wc.generate(plot_data)
        axes[n,m].imshow(wc_gen, interpolation = 'bilinear')
        axes[n,m].set_title('AdventCalendar : {}'.format(target_key), FontProperties = fp, color = 'gray', fontsize = 10)
        axes[n,m].axis('off')
        k += 1
    plt.subplots_adjust(left=0.1, right=0.95, bottom=0.1, top=0.95)
    plt.savefig('{}_wordcloud.png'.format(pp))
    plt.close()

Reference URL

Using Qiita API: https://qiita.com/arai-qiita/items/94902fc0e686e59cb8c5

AdventCalender index

This is the type of calendar I used this time. I prepared it as an index.