[PYTHON] Fix garbled characters when handling Japanese in Requests

environment

Python: 3.5 Requests: 2.11.1

Overview

Garbled characters occur when displaying a Japanese page when it is acquired using Requests. This time, in my case, it happened when the encoding on the page side was Shift-JIS.

import requests

response = requests.get('Appropriate Japanese page')
print(response.encoding)

When I looked it up in, ISO-8859-1 was returned. Apparently, if you can't get the character code well, do you change it properly? ??

Solution

import requests

response = requests.get('Appropriate Japanese page')
response.encoding = response.apparent_encoding  #Add this line

When apparent_encoding is called, it seems that the library is used to properly determine the character code. In this case, the characters are no longer garbled.

reference

https://blog.aoshiman.org/entry/118/