[PYTHON] UnicodeDecodeError: What to do when'shift_jis' codec can't decode byte

--Environment --Windows10 Pro version 1909 - Python 3.8.5 - Pandas 1.0.5

Event: I got angry when I read a CSV file with Pandas

Traceback (most recent call last):
  File "C:/path/to/my_code.py", line 258, in <module>
    csv = read_files(target_dir)
  File "C:/path/to/my_code.py", line 74, in read_files
    data = pd.read_csv(file, encoding="shift_jis")
  File "C:\path\to\venv\lib\site-packages\pandas\io\parsers.py", line 676, in parser_f
    return _read(filepath_or_buffer, kwds)
  File "C:\path\to\venv\lib\site-packages\pandas\io\parsers.py", line 448, in _read
    parser = TextFileReader(fp_or_buf, **kwds)
  File "C:\path\to\venv\lib\site-packages\pandas\io\parsers.py", line 880, in __init__
    self._make_engine(self.engine)
  File "C:\path\to\venv\lib\site-packages\pandas\io\parsers.py", line 1114, in _make_engine
    self._engine = CParserWrapper(self.f, **self.options)
  File "C:\path\to\venv\lib\site-packages\pandas\io\parsers.py", line 1891, in __init__
    self._reader = parsers.TextReader(src, **kwds)
  File "pandas\_libs\parsers.pyx", line 529, in pandas._libs.parsers.TextReader.__cinit__
  File "pandas\_libs\parsers.pyx", line 720, in pandas._libs.parsers.TextReader._get_header
  File "pandas\_libs\parsers.pyx", line 916, in pandas._libs.parsers.TextReader._tokenize_rows
  File "pandas\_libs\parsers.pyx", line 2063, in pandas._libs.parsers.raise_parser_error
UnicodeDecodeError: 'shift_jis' codec can't decode byte 0xee in position 4225: illegal multibyte sequence

my_code.py

data = pd.read_csv(file, encoding="shift_jis")

Cause: The file has some extended characters in CP932 that are not in SJIS.

This is in test2.csv, ・ Hashigodaka "Taka" ・ Tachisaki "Saki" It is caused by the mixture of windows extension strings such as>. Points to note when letting pandas read csv of excel output --Qiita

I'm crazy so I tried the combination. *. I tried using "Takasaki" as the extended character.

shift with cp932_Error when reading a file with extended characters with jis

# ...abridgement...
UnicodeDecodeError: 'cp932' codec can't decode byte 0x86 in position 5: illegal multibyte sequence

If you use a file with Japanese on Windows, it is better to set it to cp932. I also studied CP932 and SJIS.

Action: Read the file with CP932

my_code.py

data = pd.read_csv(file, encoding="cp932")