Compare the speed of Python append and map

I will reprint what I wrote for the company

Purpose

Check what the difference between the processing speed and memory usage of the code you wrote first (using append) and the code pointed out in the review (using map) is

Verification code

Prepare a program that merges the two arrays and outputs them. (Originally it was a program that merges fields and rows fetched from DB and outputs it as json)

The tools used for measurement are as follows.

--memory_profiler: You can measure the increase or decrease of memory in units of lines. You can also check the current usage and increment --line_profiler: Processing speed can be measured line by line.

Verification 1

from memory_profiler import profile
from line_profiler import LineProfiler


@profile(precision=8)
def main():
    a = [1, 2, 3]
    b = ['1', '2', '3']

    results = []
    for _ in range(1000):
        results.append(dict(zip(a, b)))

    for r in results:
        print(r)　# {1: '1', 2: '2', 3: '3'}


if __name__ == "__main__":
    #When measuring memory
    #    main()
    
    #When comparing processing speeds
    prof = LineProfiler()
    prof.add_function(main)
    prof.runcall(main)
    prof.print_stats(output_unit=1e-9)

Verification 2

Except for the contents of the method, it is the same as validation 1, so it will be omitted.

def main():
    a = [1, 2, 3]
    b = ['1', '2', '3']

    results = map(dict, (zip(a, b) for _ in range(1000)))

    for r in results:
        print(r) # {1: '1', 2: '2', 3: '3'}

result

The amount of data was small in the first place, but the difference was noticeable.

processing speed

• Verification 2 was about 0.02s faster.
• It seems that the reason why verification 2 was longer in the place of for r in results: is that the iterator went to access the data each time at this point. However, it was still faster than the code using append.

Verification 1

Total time: 0.087002 s
File: append.py
Function: main at line 6

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
     6                                           def main():
     7         1       4000.0   4000.0      0.0      a = [1, 2, 3]
     8         1       1000.0   1000.0      0.0      b = ['1', '2', '3']
     9                                           
    10         1          0.0      0.0      0.0      results = []
    11      1001     395000.0    394.6      0.5      for _ in range(1000):
    12      1000    1973000.0   1973.0      2.3          results.append(dict(zip(a, b)))
    13                                           
    14      1001    5854000.0   5848.2      6.7      for r in results:
    15      1000   78775000.0  78775.0     90.5          print(r)

Verification 2

Total time: 0.069483 s
File: map.py
Function: main at line 7

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
     7                                           def main():
     8         1       4000.0   4000.0      0.0      a = [1, 2, 3]
     9         1       1000.0   1000.0      0.0      b = ['1', '2', '3']
    10                                           
    11         1       2000.0   2000.0      0.0      results = map(dict, (zip(a, b) for _ in range(1000)))
    12                                           
    13      1001    8476000.0   8467.5     12.2      for r in results:
    14      1000   61000000.0  61000.0     87.8          print(r)

memory usage

• Compared in total, map was less than 0.16MB.
• It turned out that the memory is not used more than necessary in the writing method of verification 2. (Important here!)

Verification 1

Line #    Mem usage    Increment   Line Contents
================================================
     5  38.28125000 MiB  38.28125000 MiB   @profile(precision=8)
     6                             def main():
     7  38.28515625 MiB   0.00390625 MiB       a = [1, 2, 3]
     8  38.28515625 MiB   0.00000000 MiB       b = ['1', '2', '3']
     9                             
    10  38.28515625 MiB   0.00000000 MiB       results = []
    11  38.38671875 MiB   0.00390625 MiB       for _ in range(1000):
    12  38.38671875 MiB   0.00390625 MiB           results.append(dict(zip(a, b)))
    13                             
    14  38.39453125 MiB   0.00000000 MiB       for r in results:
    15  38.39453125 MiB   0.00781250 MiB           print(r)

Verification 2

Line #    Mem usage    Increment   Line Contents
================================================
     5  38.22656250 MiB  38.22656250 MiB   @profile(precision=8)
     6                             def main():
     7  38.23046875 MiB   0.00390625 MiB       a = [1, 2, 3]
     8  38.23046875 MiB   0.00000000 MiB       b = ['1', '2', '3']
     9                             
    10  38.23828125 MiB   0.00000000 MiB       results = map(dict, (zip(a, b) for _ in range(1000)))
    11                             
    12  38.23828125 MiB   0.00000000 MiB       for r in results:
    13  38.23828125 MiB   0.00781250 MiB           print(r)

in conclusion

• I first learned that there is such a verification tool. It's fun to see each line of code ...!
• Not only is it fast and slow, but in the case of a large amount of data that rows are fetched from the DB like the case I wanted to do this time, if you do not write using a map that does not use memory more than necessary, it will be an obstacle at the time of production execution I felt that I had to be careful because it could lead to a connection.

Please let me know if there is something strange.