I tried scraping the ranking of Qiita Advent Calendar with Python
Qiita's Advent Calender has a public ranking. Overall calendar ranking
At first, I took a look at it for the purpose of finding an interesting calendar, but when I looked closely, we were also ranked! : clap :: clap ::
However, even if the number of likes is 1, it is ranked, so it is natural.
It was the third day for the first time since I knew that it was ranked ** "What is the ranking today?" **. This is a hassle ...
I decided to automate the ranking acquisition, saying, "Since Qiita has an API, you can get the ranking with it."
Scraping if you don't have an API!
Qiita has an API. Here → Qiita API v2 specifications However, even if you look at the documentation, there is no Advent Calender API: sob:
If there is no API, I decided to scrape with Python.
When I searched for Python scraping, I found many Beautiful Soup, so I decided to use ** Beautiful Soup **.
Scraping is easy
The purpose is clear. To get the ranking of your company. I looked up the HTML tags.
To get the ranking numbers, go up two from the calendar link <a class="adventCalendarRankingListItem_calendarName"> and get the text.
I tried to make a code using Beautiful Soup.
from urllib import request
from bs4 import BeautifulSoup
targethref = '/advent-calendar/2019/fork'
def main():
url = 'https://qiita.com/advent-calendar/2019/ranking/feedbacks/all'
targetclass = 'adventCalendarRankingListItem_calendarName'
response = request.urlopen(url)
soup = BeautifulSoup(response,features="html.parser")
ranking = soup.find('a',class_=targetclass,href=targethref).parent.parent.contents[0].text
response.close()
print(ranking)
if __name__ == "__main__":
main()
That's it, it's that easy! ** BeautifulSoup ** Great! (It's my feeling because I don't know other libraries that can be scraped)
- If you want to get the ranking of your favorite calendar, rewrite
target href =on the 4th line. *
in conclusion
I run this logic on AWS Lambda once a day. The result of the execution was thrown to the in-house Chat tool to share the information.
It was a day when I realized that I could scrape even if the API wasn't published: sunny:
:fork_and_knife: FORK Advent Calendar 2019 The Advent Calender article I wrote is here
Recommended Posts