[PYTHON] numpy memory reuse cheat sheet

Introduction

For those who work with large matrices, matrix malloc / realloc is a life-and-death problem. Evil people who want to recycle memory should take a look at this short cheat sheet.

Array initialization

There may be times when you want to zero an array, such as at the beginning of a loop. Use numpy.ndarray.fill to fill an already allocated array with a certain value.

import numpy as np

a = np.ones(5)

for i in range(5):
    a.fill(0)
    a[i] = 1
    print(a)

Copy of array

Use numpy.copyto if you want to copy the contents of an array to another array. Note that numpy.copy will generate a new array.

import numpy as np

m, n = 2000, 1000
A = np.ones((m, n))
B = np.zeros(A.shape)

#A copy that does not generate a new array
np.copyto(B, A)

If you want to secure a large array and fill it from the beginning with a small array, you can use slices. If you want to fill the remaining part with 0, process it with the above fill.

import numpy as np

a = np.array([0, 1, 2, 3, 4])
b = np.array([5, 5, 5])

np.copyto(a[0:3], b)  #Fill with small array values from the beginning
a[3:5].fill(0)        #Fill the remaining part with 0

Mathematical manipulation

For mathematical operations, the storage destination can be specified for those that can take ʻout` as an argument. Even if you can't get it, there are times when you're not messing with the array itself.

Four arithmetic operations / others

Operations that can specify the ʻoutparameter in [Mathematical functions](https://numpy.org/doc/stable/reference/routines.math.html) use memory. Theshapeof the array may be different as long as it is within the range that broadcasting can handle. If the assignment destination matches the original matrix, you can use an assignment operator such as+ =`.

import numpy as np

A = np.array([[1, 2, 3],
              [4, 5, 6]])
B = np.array([[7, 8, 9],
              [8, 7, 6]])

C = np.zeros(A.shape)

#Four arithmetic operations that do not generate a new array
np.add(A, B, out=C)
np.multiply(A, B, out=C)
np.subtract(A, B, out=C)
np.divide(A, B, out=C)

#Patterns that use assignment operators
A += B
A -= B
A *= B
A /= B

Note that ʻA = A + B creates a new matrix and assigns it to ʻA.

Transpose

Transposition is just messing with the parameter called strides It's fast because the array itself isn't messed up.

import numpy as np

A = np.array([[1, 2, 3],
              [4, 5, 6]])

#Transpose does not generate new sequences
B = A.T

B[0,1] = 9

print(A)

output


array([[1, 2, 3],
       [9, 5, 6]])

Although not introduced in this article, reshape is also fast because the arrangement order itself has not changed.

Scalar times

You can use the * = operator.

import numpy as np

A = np.array([[1, 2, 3],
              [4, 5, 6]])

A *= 5

Matrix product

You can also specify ʻout for numpy.dot`.

import numpy as np

A = np.array([[1, 2, 3],
              [4, 5, 6]])
B = np.array([[7, 8, 9],
              [8, 7, 6]])

C = np.zeros((2, 2))

#Matrix product that does not generate a new array
np.dot(A, B.T, out=C)

It's a complete digression, but when multiplying multiple matrices, using numpy.linalg.multi_dot will take the matrix product in the most efficient order. However, ʻout` cannot be specified here.

Recommended Posts

numpy memory reuse cheat sheet
Curry cheat sheet
SQLite3 cheat sheet
pyenv cheat sheet
conda command cheat sheet
PIL / Pillow cheat sheet
Linux command cheat sheet
ps command cheat sheet
Spark API cheat sheet
Python3 cheat sheet (basic)
PySpark Cheat Sheet [Python]
Python sort cheat sheet
Go language cheat sheet
tox configuration file cheat sheet
[Python3] Standard input [Cheat sheet]
Data Science Cheat Sheet (Python)
Slack API attachments cheat sheet
Python Django Tutorial Cheat Sheet
scikit learn algorithm cheat sheet
Apache Beam Cheat Sheet [Python]
Google Test / Mock personal cheat sheet
Continuation Passing Style (CPS) Cheat Sheet
Python cheat sheet (for C ++ experienced)
Python Computation Library Cheat Sheet ~ itertools ~