Basic summary of scraping with Requests that beginners can absolutely understand [Python]

Requests basics

import

import requests

You will definitely need this import.

Get the source from the website

--Get with GET method --Get with the POST method

You should remember these two.

Get with GET method (requests.get)

import requests

url = 'https://www.yahoo.co.jp/'
response = requests.get(url)
print(response) # →<Response [200]>

html = response.text
print(html) #→ HTML source string

The return value of requests.get (url) is the HTTP status code. If successful, 200 will be returned.

You can get the HTML source string you are looking for in response.text.

Get by POST method (requests.post)

You may not get the source you are looking for without the POST method.

data =  {'username':'tarouyamada', 'password':'4r8q99fiad'}

response = requests.post(url, data=data)

Now you can send the request including the request body.

How to add request header

headers = {'user-agent':'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.116 Safari/537.36', 
'accept': 'application/json'}

response = requests.get(url, headers=headers)

You can now send the request with the request header attached. The writing method is common for get and post.

Get an image

You can get binary data using .content. Images are also a type of binary data.

response = requests.get(url)

img_data = response.content

print(img_data)
#b'\xff\xd8\xff\xe0\x00\x10JFIF\x00\x01\x01\x00\x00\x01\x00\x01\x00\x00\xff\xdb\x00C\x00\x03\x02\x02\x03\x02\x02\x03\x03\x03\x03\x04\x03\x03\x04\x05\x08\x05\x05\x04\x04\x05\n\x07\x07\x06\x08\x0c\n\x0………

print(type(img_data))
# <class 'bytes'>

--The output is bytes type

Save image

--By the way, when you want to save the acquired image data --Add'b'to read / write binary files

with open('test.jpg', 'wb') as f:
    f.write(response.content)

Parameter specification

params = {'q':'qiita', 'date':'2020-7-3'}

response = requests.get(url, params=params)

View response header

--Content-type can be used to judge whether it is text, json, or image.

response = requests.get(
    'https://www.pakutaso.com/shared/img/thumb/nekocyan458A3541_TP_V.jpg')

print(response.headers)

# {'Server': 'nginx', 'Date': 'Tue, 07 Jul 2020 22:39:37 GMT', 'Content-Type': 'image/jpeg', 'Content-Length': '239027', 'Last-Modified': 'Sun, 05 Jul 2020 01:51:48 GMT', 'Connection': 'keep-alive', 'ETag': '"5f013234-3a5b3"', 'Expires': 'Thu, 06 Aug 2020 22:39:37 GMT', 'Cache-Control': 'max-age=2592000', 'X-Powered-By': 'PleskLin', 'Strict-Transport-Security': 'max-age=31536000;  includeSubDomains; preload', 'Accept-Ranges': 'bytes'}

If there is a redirect

Get the redirected response

If you want to use the history in the middle of redirect, use .history

Encoding check

response = requests.get(
    'https://qiita.com/')

print(response.encoding)

# utf-8

Get json data

--Can be obtained as a dictionary with response.json ()

response = requests.get(url)

json_dict = response.json()