About Python pickle (cPickle) and marshal

Introduction

Python has a serialization module for objects called pickle (cPickle). I think pickle is very famous. In fact, there is also a similar module called marshal. This is not very famous. It's not a persistent module, and compatibility between versions isn't guaranteed, so there's no reason to use marshal. Not surprisingly, it's not famous.

Marshal is full of drawbacks compared to pickle, but I found that marshal also has some good points, so this time I will focus on that. The good thing about marshal is "speed".

Check speed

Let's check the export and import speeds immediately. This time, the speeds were compared between "cPickle" and "marshal".

First, write it out.

#!/usr/bin/env python
# -- coding:utf-8 -*-
import marshal
import cPickle as pickle
import time
import numpy as np


def main():
    a = np.ndarray(shape=(10000, 10000))
    start = time.time()
    pickle.dump(a, open('output.pkl', 'wb'))
    p_time = time.time() - start
    start = time.time()
    marshal.dump(a, open('output.msl', 'wb'))
    m_time = time.time() - start
    print p_time, m_time

if __name__ == '__main__':
    main()

The code looks like this. I'm generating a multidimensional array and writing it out.

Let's take a look at the output.

143.123441935 5.09839010239

The left is cPickle, the right is marshal, and the unit is seconds. It made a difference more than I expected, it was unexpected.

Next is reading.

#!/usr/bin/env python
# -- coding:utf-8 -*-
import marshal
import cPickle as pickle
import time
import numpy as np


def main():
    start = time.time()
    a = pickle.load(open('output.pkl', 'rb'))
    p_time = time.time() - start
    start = time.time()
    b = marshal.load(open('output.msl', 'rb'))
    m_time = time.time() - start
    print p_time, m_time

if __name__ == '__main__':
    main()

The code is this. I just set dump to load.

Click here for results

445.698551893 1.64994597435

Similarly, the left is cPickle, the right is marshal, and the unit is seconds. Well, it's a huge difference ... I'm pretty surprised.

Summary

Yes. So it turns out that marshal is faster in speed. I haven't investigated the reason why.

However, although marshal is superior in speed, pickle that makes it permanent is overwhelmingly convenient, so if you use it, it is still pickle. Then why did you compare them? The only reason is that I was curious.

So I spotlighted marshal, which I would rarely use! That is all for the story.