[PYTHON] I dare to fill out the form without using selenium

Introduction

It's not so much as "dare", but when I'm investigating form editing in Python, selenium often appears, so I implemented it with Requests + Beautiful Soup.

Posted before is just right as a development, so I edited Qiita's profile text.

Editing function

Many editable items were found to be of type ['text','url','checkbox'] from the source, so I was able to get them with little effort. If you add'user [desctiption]]', which is an acquisition omission, and edit it, the post data is completed. For some reason, the check box for publishing email was checked by default, so I am fixing it to 0. Once you have created the data, post it using the login session and you're done.

def edit_profile(session):
    profile_page_data = get_page_data(session, profile_url)
    bs_profile_page = BeautifulSoup(profile_page_data.text, 'html.parser')
    authenticity_token = str(bs_profile_page.find(attrs={'name':'authenticity_token'}).get('value'))
    post_data = {
        'utf-8':'✓',
        '_method': 'put',
        'authenticity_token':authenticity_token
    }
    response = bs_profile_page.find_all(attrs={'type':['text', 'url', 'checkbox']})
    for i in response:
        value = i.get('value')
        post_data[i.get('name')] = value
    post_data['user[public_email]'] = 0 #1 by default for some reason
    post_data['user[description]'] = 'I edited it! !! !! !!'
    print(post_data)
    response = session.post(profile_url, post_data)
    print(response)

Execute

Add the edit_profile () described above to the program created last time to complete the program.

import requests
import os
from bs4 import BeautifulSoup
import json


user_name = 'user_name'
user_password = 'user_password'
login_url = 'https://qiita.com/login'
profile_url = 'https://qiita.com/settings/profile'
    

def get_authenticity_token(session, login_url):
    page_data = get_page_data(session, login_url)
    bs_page_data = BeautifulSoup(page_data.text, 'html.parser')
    authenticity_token = str(bs_login_page.find(attrs={'name':'authenticity_token'}).get('value'))
    return bs_page_data, authenticity_token


def get_page_data(session, url):
    response = session.get(url)
    response.encoding = response.apparent_encoding
    return response


def login(session):
    login_form = {
        'utf-8':'✓',
        'authenticity_token':'token',
        'identity':user_name,
        'password':user_password
    }
    bs_login_page, authenticity_token = get_authenticity_token(session, login_url)
    login_form['authenticity_token'] = authenticity_token
    session.post(login_url, login_form)
    

def edit_profile(session):
    bs_profile_page, authenticity_token = get_authenticity_token(session, profile_url)
    post_data = {
        'utf-8':'✓',
        '_method': 'put',
        'authenticity_token':authenticity_token
    }
    response = bs_profile_page.find_all(attrs={'type':['text', 'url', 'checkbox', 'textarea']})
    for i in response:
        value = i.get('value')
        post_data[i.get('name')] = value
    post_data['user[public_email]'] = 0 #1 by default for some reason
    post_data['user[description]'] = 'I edited it! !! !! !!'
    print(post_data)
    response = session.post(profile_url, post_data)
    print(response)
    
if __name__ == '__main__':
    session = requests.Session()
    login(session)
    edit_profile(session)

If it is executed and Response [200] is returned, it is successful.