Consider what you can do with Python from the Qiita article

Introduction

As I continue to learn programming, what can I do after finishing the tutorial (I have)? I understand the syntax of the language, but I don't know what to do next. I think there is such a situation for beginners.

So, this time, I would like to get an idea of what to make next by acquiring and analyzing the top articles and tags of Qiita.

What i did

  1. Get the title of Qiita's top article from a specific word
  2. Get and analyze tags related to a specific word

    In short, I tried to consider from the tags what kind of genre there are many, as well as referring to what I make by referring to the top articles.

    • Basically, you just scrape

    environment

    Python 3.9.0

    1. Get the title of Qiita's top article from a specific word

    For example, I tried to get 10 pages of top articles when I searched for "Python".

    qiita_title.py

    

import requests
from bs4 import BeautifulSoup as bs

query = input('please input words : ')

with open(query + '_qiita.txt', 'wb') as outfile:
    for i in range(1,11):
        url = 'https://qiita.com/search?page=' + str(i) + '&q=body%3A' + query + '+tag%3A' + query + '&sort=like'
        res = requests.get(url)
        soup = bs(res.text, 'html.parser')
        titles = soup.find_all('h1', class_="searchResult_itemTitle")

        for i in titles:
            s = i.get_text()
            outfile.write(s.encode('utf-8'))
            outfile.write(b'\n \n')

    Execution result: Search word = python

    Summary of English words often used in programming [Updated from time to time]

    [Summary] Over 100 general-purpose tools that programmers who do not know this will lose

    How to write JavaScript of Imadoki 2018

    I want new programmers to know The trajectory of humankind to get object-oriented

    After registering with AtCoder, what to do next-If you solve this much, you can fight enough! Past question selection 10 questions ~

    ~ The following is omitted.

    You can extract only the titles of the top articles of qiita and refer to what to make next. By the way, it was around here that I personally found it interesting.

    Master Django fastest part1 Learn by making a blockchain ~ The fastest way to learn how a blockchain works is to make it ~ The antonym of "red stranger" is "Shiroi Koibito" A story that wants to automatically generate this If you have deep learning, you can exceed 100% recovery rate in horse racing When I made a technical book ranking site from the total of Qiita articles, about 4000 technical books lined up nicely

    I also want to make a ranking site ~

    2. Get and analyze tags related to a specific word

    This time, by acquiring and analyzing the related tags of the top articles, it is judged what kind of thing is appropriate for the search word.

    qiita_tag.py

    

import requests
from bs4 import BeautifulSoup as bs
import csv

query = input('please input words : ')
lists = []

for i in range(1,11):
    url = 'https://qiita.com/search?page=' + str(i) + '&q=body%3A' + query + '+tag%3A' + query + '&sort=like'
    # url = 'https://qiita.com/search?page=1' + '&q=body%3A' + query + '+tag%3A' + query + '&sort=like'
    res = requests.get(url)
    soup = bs(res.text, 'html.parser')
    tags = soup.find_all('li', class_="tagList_item")
    for i in tags:
        lists.append(i.get_text())

print("\n /// result is below /// \n")
print(lists, '\n')

unique_tags = set(lists)

with open('./csv/'+query+'_tags.csv', 'w', newline='') as f:
    for i in unique_tags:
        writer = csv.writer(f)
        writer.writerow([str(i), lists.count(i)])

    As before, I extracted the tags of the articles searched by python into csv. The following is a sort of it (the first column is the tag, the second column is the number of appearances (total number of articles = 100 articles))

    qiita_tags_python.png

    Looking at it in this way, it can be seen that there are many "machine learning", "deep learning", and "data processing", and that it is also a language that processes data. Next, since many "Django", "scraping", "Ruby", etc. are seen, it can be seen that there is also demand for Web systems. This is because python is not used for the Web system, but because the population parameter of Web development people is large, I think that the number of Web system tags is increasing accordingly (I do not know).

    in conclusion

    So it turns out that python is a strong language for data processing like machine learning (which I knew before). Anyway, even though I'm learning python myself, I'm still a machine learning virgin, so I'd like to touch on it in the next article.

    Recommended Posts