[PYTHON] pickle To read what was made in 2 series with 3 series

I want to pickle.dump the 2nd system and pickle.load the 3rd system.

• In the case of characters, it remains byte, so add a method to convert it to characters.
• On the contrary, add a method to read what was made in 3 series in 2 series.

Various specifications are listed: http://docs.python.jp/3.4/library/pickle.html

A little after moving to the 3rd system, I wanted to read the data dumped by the previously created pickle, but an error occurred.

error

pickle-load-error.

# test_w_2.pkl is 2 system[1]File dumped
In [25]: fin = open('test_w_2.pkl', 'r') 
In [26]: pickle.load(fin)    
TypeError: a bytes-like object is required, not 'str'       

I get an error like this, isn't it str? It seems that I have to make it bytes

Solution

It seems that you should add'rb' to the option at the time of ʻopen` (binary mode)

piclkle-load.

In [38]: fin = open('test_wb_2.pkl', 'rb')
In [39]: pickle.load(fin)                                                                              
Out[39]: [1]   

[Addition] When dumped is string

In the above example, only [1] was dumped. It seems that it may not be possible to load if the list contains a character string. When doing pickle.load, change the encoding method with ʻencoding ='bytes'. It is necessary to convert the contents of the received list to string using .decode ('utf8') `.

python2.7.9

pickle.dump

>>> pickle.dump(['AIUEO'], open('test.pkl', 'w')  )

python3.5.0

pickle.load

#Read normally and error
>>> pickle.load(open('test.pkl', 'r') )                                                                
Traceback (most recent call last):                                                                     
  File "<stdin>", line 1, in <module>                                                                  
TypeError: a bytes-like object is required, not 'str'                                                  

#Even if you read it in binary'ascii'Error because it does not correspond to
>>> pickle.load(open('test.pkl', 'rb') )                                                               
Traceback (most recent call last):                                                                     
  File "<stdin>", line 1, in <module>                                                                  
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe3 in position 0: ordinal not in range(128)      

#I could read it after changing the encoding method, but I can't read it as bytes.
>>> pickle.load(open('test.pkl', 'rb'), encoding='bytes' )                                             
 [b'\xe3\x81\x82\xe3\x81\x84\xe3\x81\x86\xe3\x81\x88\xe3\x81\x8a']                                      

# decode('utf8')Use to decode each element in the list
>>> list(map(lambda x: x.decode('utf8'), pickle.load(open('test.pkl', 'rb'), encoding='bytes' ) ) )    
 ['AIUEO']  

[Addition] On the contrary, to read what was made in 3 series in 2 series

If it is normally made with 3 series (dump), it cannot be read (load) with 2 series It is written in the URL I wrote at the beginning, but it seems that you can specify protocol. Below is a description of each protocol version.

-Protocol version 0 is the original "human readable" protocol, backwards compatible with earlier versions of Python. --Protocol version 1 is an older binary format, which is also compatible with earlier versions of Python. --Protocol version 2 was introduced in Python 2.3. This version provides a more efficient pickle version of the new method of class. See PEP 307 for information on improvements made with Protocol 2. --Protocol version 3 was added in Python 3.0. This version supports bytes objects. It cannot be non-pickle in Python 2.x. This is the default protocol and is the recommended protocol if compatibility with other Python 3 versions is required. --Protocol version 4 was added in Python 3.4. This version supports huge objects, pickles more types of objects, and optimizes some data formats. See PEP 3154 for information on improvements made with Protocol 4.

In python2 series, as a matter of course, there is only up to 2 protocol, so it seems better to match this with python3 as well. (Because people around me don't use python2 system very much) As far as I can see, 4 seems to be the best, so it seems better to specify this when only you use it. By the way, the default is set to 3.

python3.5

pickle.dump

>>> pickle.dump(['AIUEO'], open('test.pkl','wb'), protocol=2 ) 

python2.7

pickle.load

>>> pickle.load(open('test.pkl') ) 
[u'\u3042\u3044\u3046\u3048\u304a']  
>>> print(pickle.load(open('test.pkl') )[0] )
AIUEO

Summary

If you add'rb', it's quite so (for now) There are other options for fix_imports and ʻerrors`, but for now it works. I will add it when a scene that does not move appears.