[PYTHON] Fix garbled characters when handling Japanese in Requests

environment

Python: 3.5 Requests: 2.11.1

Overview

Garbled characters occur when displaying a Japanese page when it is acquired using Requests. This time, in my case, it happened when the encoding on the page side was Shift-JIS.

import requests

response = requests.get('Appropriate Japanese page')
print(response.encoding)

When I looked it up in, ISO-8859-1 was returned. Apparently, if you can't get the character code well, do you change it properly? ??

Solution

import requests

response = requests.get('Appropriate Japanese page')
response.encoding = response.apparent_encoding  #Add this line

When apparent_encoding is called, it seems that the library is used to properly determine the character code. In this case, the characters are no longer garbled.

reference

https://blog.aoshiman.org/entry/118/

Recommended Posts

Fix garbled characters when handling Japanese in Requests
Eliminate garbled Japanese characters in JSON data acquired by API.
Eliminate garbled Japanese characters in Python library matplotlib and NetworkX
Resolve garbled Japanese characters in matplotlib of Jupyter Notebook on Docker
[Note] Japanese characters are garbled with atom-runner
Eliminate garbled Japanese characters in matplotlib graphs in Cloud Pak for Data Notebook
Windows Qt5.4 Python3.4 QProcess Japanese garbled characters
Handles UTF-8 Japanese characters in Python's MySQLdb.
Create an image with characters in python (Japanese)
[EC2] Prevent garbled characters when capturing with selenium
[Python] Read Japanese csv with pandas without garbled characters (and extract columns written in Japanese)
Forcible response when Warning appears in Python's requests library
How to eliminate garbled characters in matplotlib output image
Japanese output when dealing with python in visual studio
When you get an error in python scraping (requests)
[EC2] Prevent garbled Japanese characters on selenium (noto compatible)