[PYTHON] I scraped the Organization member team and made a ranking

SDT's Qiita Organization is getting hotter and hotter! So I tried to make a ranking somehow ~ (What does that mean !!)

Source code

nnsnodnb/sdt-qiita-ranking | GitHub

Public page sample

It's almost like this. Approximately ... I can only see the ones in SDT yet, but in the future, I'm thinking or not thinking about creating an API so that ranking can be taken in any organization when I have free time !!

Scraping

package

requirements.txt

beautifulsoup4==4.5.0
bottle==0.12.9
Jinja2==2.8
MarkupSafe==0.23

Don't ask if you used Jinja2 or something :; (∩´﹏`∩) ;:

Import the package ♪

from urllib.request import urlopen
from bs4 import BeautifulSoup

Put together the parts to put together

get_soup()

url = 'http://qiita.com/organizations/smartdt/members'
html = urlopen(url)
return = BeautifulSoup(html, 'html.parser')

Member list and number of contributions

get_contribution()

soup = get_soup()

contribution_array = {}
# organizationMemberList_Get all li tags of item class
members = soup.findAll('li', class_='organizationMemberList_item')

for member in members:
    #Get the first a tag for each member
    a = member.find('a')
    #Organization Member List for each member_Get the first div tag of the memberProfile class
    profiles = member.find('div', 'organizationMemberList_memberProfile')
    #OrganizationMemberList from the div tags obtained above_Get all div tags of memberStats class and specify the second one
    contribute = profiles.findAll('div', 'organizationMemberList_memberStats')[1]

    # a.attrs['href'] -> nnsnodnb
    name = a.attrs['href'].split('/')[1]
    # contribute.text -> 50 Contribution
    number = int(contribute.text.split(' Con')[0])
    
    contribution_array.update({name: number})

Member icon

get_membericon()

soup = get_soup()

icons = {}
# organizationMemberList_Get all img tags of icon class
member_icons = soup.findAll('img', 'organizationMemberList_icon')
# organizationMemberList_Get all li tags of item class
members = soup.findAll('li', class_='organizationMemberList_item')

for index in range(len(members)):
    a = members[index].find('a')
    name = a.attrs['href'].split('/')[1]
    #Extract the src attribute from each member of the member icon list
    icon = member_icons[index].attrs['src']

    icons.update({name: icon})

This time, name is set as the key in order to relate the ** member list and the number of contributions ** and the ** member icon **.

Sort by ranking

I'm using a dictionary because I can't have a relationship between key and value in a list. However, since the dictionary is just a dictionary, it is output out of order. Want to cry So it takes the form of taking out in order and typing in the list. Also, don't forget reverse () because the data is added in ascending order when you hit it.

ranking()

contribution = get_contribution()

names, numbers = [], []
for key, value in sorted(contribution.items(), key=lambda x:x[1]):
    names.append(key)
    numbers.append(value)
    
user_icons = get_membericon()

names.reverse()
numbers.reverse()

Pour into template

This time I used Bottle as the web framework. There are many ways to use it if you google! Should be!

from bottle import template, get, run

Basically this is fine

Flow to template

return template('hoge')
return template('foo', name = name)

If you do it like this, it will be manageable ~ For hoge and foo, create a views folder in the application directory and create hoge.tpl and foo.tpl below! Same as HTML

If you want to run Python code in hoge.tpl, you can run it with% . Also, if you like template ('foo', name = name), you can replace it with {{name}} in hoge.tpl. ~~ Mustache It's the same. ~~

Also, if you want to end the loop etc. using for or ʻif`, you can do as follows

for sample

% for i in range(10):
% #Output to terminal
% print(i)
% end

if sample

% if name == 'nnsnodnb':
% <p>Yaho ~!</p>
% else:
% <p>Who are you wwwww</p>
% end

After all it feels bad without indentation! !! Of course, you can also write hoge.tpl with indentation.

If you code loosely like this, a page like a public page sample will be created.

Bottle execution

main()

run(host = '0.0.0.0')

$ python hoge.py

Summary

Just by the existence of such a thing, everyone's fighting power can be visualized and the target has been visualized loosely!

If you have any questions, please leave a comment or edit request!