[PYTHON] Looking back on the transition of the Qiita Advent calendar

Happy New Year.

No one may look at the Advent calendar anymore because the calendar has changed a little over a week after the Advent calendar 2020 is over, but let's see the transition of the trend by comparing it with the past Advent calendar.

What i did

1. Get the advent calendar for the past 6 years starting from 2015 and convert it to csv for each calendar
2. See the transition of trend technology

result

Table 1. Programming language

Rust seems to be popular from 2019. The number of companies developing with Go is increasing, and it is steadily increasing in popularity. Data analysis languages ​​such as MATLAB and Python are still popular.

Table 2. Library

Laravel is popular in language frameworks. Also, in 2020, VR engines like Unity and Unreal Engine were popular. To put it bluntly, the day when the platform will move from mobile to VR may be near.

Table 3. Database

PostgreSQL seems to be still popular. I think it's okay to include NoSQL articles.

Table 4. mobile

Google's Flutter and Facebook's React Native are popular for cross-platform development. Mobile app development will be even easier in combination with Google's Firebase.

Table 5. DevOps

Circle CI is popular for CI/CD tools, and Ansible , Terraform , and Kubernetes are popular for Infrastructure as Code. In the infrastructure area, is it proof that it is standardized to containerize with Docker, manage branches with GitHub, and automate with CI?

in conclusion

Looking back on Qiita's Advent calendar, I think we can somehow predict this year's trends. Personally, I speculate that in 2021, the VR development environment will improve dramatically, mobile application development will become more active and then saturated, and the demand for infrastructure engineers will increase as the environment becomes more automated. Technology is very popular and obsolete, so I would like to continue fixed-point observations.

Code

get_ad_cal.py

import requests
import pandas as pd
import numpy as np
from tqdm import tqdm
from bs4 import BeautifulSoup as bs

year = '2020'
url = 'https://qiita.com/advent-calendar/'+year+'/ranking/'
feed = 'feedbacks/categories/'
keyword_list = ['feedbacks', feed+'programming_languages', feed+'libraries', feed+'databases', feed+'web_technologies', feed+'mobile', feed+'devops', feed+'iot', feed+'os', feed+'editors', feed+'academic', feed+'services', feed+'company', feed+'miscellaneous']
columns_list = ['feedbacks', 'programming_languages', 'libraries', 'databases', 'web_technologies', 'mobile', 'devops', 'iot', 'os', 'editors', 'academic', 'services', 'company', 'miscellaneous']

def get_item_name(keyword):
    key_url = url+keyword
    res = requests.get(key_url)
    if res.status_code == 200:
        soup = bs(res.text, 'html.parser')
        items = soup.find_all('a', class_='ad-Item_name')
    else:
        pass
    for item in items:
        items_list.append(item.text)
    return items_list

df_list = []
for keyword in keyword_list:
    items_list = []
    items_list = get_item_name(keyword)
    df_items = pd.DataFrame(items_list)
    df_list.append(df_items)

np_list = np.arange(1, 21)
np_list = list(np_list)
df = pd.DataFrame(np_list)

for i in range(len(df_list)):
    df[columns_list[i]] = df_list[i]
df.to_csv('ad_cal/'+year+'.csv', encoding='utf-16', index=False, sep='\t')

sort_year.py

import pandas as pd

years = ['2020', '2019', '2018', '2017', '2016', '2015']
columns_list = ['feedbacks', 'programming_languages', 'libraries', 'databases', 'web_technologies', 'mobile', 'devops', 'iot', 'os', 'editors', 'academic', 'services', 'company', 'miscellaneous']

for column in columns_list:
    dic_year = {}
    for year in years:
        df = pd.read_csv('ad_cal/'+year+'.csv', encoding='utf-16', sep='\t')
        try:
            df = df[column]
            dic_year[year] = df
        except:
            continue
    df_cat = pd.DataFrame.from_dict(dic_year)
    df_cat.to_csv('ad_cal/'+column+'.csv', encoding='utf-16', sep='\t', index=False)