Introduction

I'll write it first, but I'm not good at writing sentences, so if it's hard to read, I'm sorry. Well, I'm very upset to know that I can't see the list of likes because I used likes instead of stock. It seems that many people are worried about twitter and Qiita. [^ 1] [^ 2] So, scrape with python to get a list of likes.

Introduction

After installing python3, use the pip command to request, BeautifulSoup4, progressbar2 Please install

code

Get the article list and save it in the results.json file.

# -*- coding:utf-8 -*-
from bs4 import BeautifulSoup
import requests
from time import sleep
import json
from progressbar import ProgressBar

#Login to Qiita
payload = {
    'utf8': '✓',
    'identity': 'lovemuffim114',  #username
    'password': 'tkhr2783'  #password
}

# authenticity_Get token
s = requests.Session()
r = s.get('https://qiita.com')
soup = BeautifulSoup(r.text, "html.parser")
auth_token = soup.find(attrs={'name': 'authenticity_token'}).get('value')
payload['authenticity_token'] = auth_token

#Login
s.post('https://qiita.com/login', data=payload)

#Dictionary for saving results
results = dict()

searchId = 1000000
param = {"before": searchId, "type": "id"}
#Get the number of articles
maxId = s.get("http://qiita.com/api/public", params=param).json()[0]["id"]
bar = ProgressBar(min_value=0, max_value=maxId)  #Set maximum value

try:  #Error when there are no more articles to load
    while True:
        param = {"before": searchId, "type": "id"}
        #Get article list
        j = s.get("http://qiita.com/api/public", params=param).json()

        #Add articles that you like and do not stock to results.
        # "url"other than,"uuid"Can also be specified.
        results.update({i["title"]: i["url"] for i in j if i["liked"] and not i["stocked"]})
        
        searchId = j[-1]["id"]
        bar.update(maxId - searchId)  #Progress bar update
        sleep(1)  # sleep(1 second)You can use a decimal point.
finally:
    print(results)

    # results.Save to json
    with open("results.json", "a") as f:
        json.dump(results, f, ensure_ascii=False, indent=4, separators=(',', ': '))

that's all. Result is,

{
    "[Narou4j] I made a novel acquisition library to become a novelist in Java": "863aa22a29db16463e52",
    "Accelerate ListView without using ViewHolder": "28f8be64d39b20e69552",
    "[Narou4j] Created Java wrapper library for Narurou API": "6c050593f45174056005",
    "I made my own hydroponics set.": "5d60c14d560ecf518a4e",
    "RxJava + Flux (+ Kotlin)Android app design by": "cbf304891daec87ba5b7",
    "I made EventBus with RxJava": "a4ece37834446c9a39c8"
}

It will be saved as. For logging in to Qiita, I used Login to website in Python.

Annotation

[^ 1]: [[Caution! ] Articles will not be stocked if you just press the "Like" button on Qiita! I wrote an extension! ]](Http://qiita.com/gimupop/items/be53044143a9a3e90a4b) [^ 2]: [Twitter search results for "qiita Likes-! -Breakthrough"](https://twitter.com/search?f=tweets&vertical=default&q=qiita%20%E3%81%84%E3% 81% 84% E3% 81% AD% E3% 80% 80% E4% B8% 80% E8% A6% A7% E3% 80% 80% 20-% EF% BC% 81% 20-% E7% AA% 81% E7% A0% B4 & src = typd)

[PYTHON] Get a list of Qiita likes by scraping

Introduction

Introduction

code

Annotation