[python] Checking the memory consumption of variables

This article is the 8th day article of Furukawa Lab Advent_calendar. This article was written by a student at Furukawa Lab as part of his studies. The content may be ambiguous or the expression may be slightly different.


As axes for comparing machine learning algorithms of the same task, learning stability (dependency of initial value), whether it can be explained, (accuracy in supervised learning), precision, recall ( There is also an index called Recall)) And there is "calculation amount".

It is difficult to adopt an algorithm that tends to increase the amount of calculation so that it does not fit in memory depending on the situation (time / calculation resource). (Ex: $ \ mathcal {O} (number of data ^ 3) $ once The number of training data must be limited to 1000 ($ 1000 ^ 3 = 10 Billion \ fallingdotseq 10GB $) at most.

Also, even if the amount of calculation is small, it may not be implemented in that calculation.

This article describes how to check the memory consumption of a python program.

No package @shiracamus has specified how to know the memory size without importing anything, so I will add it to the text as well.

import numpy as np
N = 1000
D = 3
X = np.random.randn(N,D)
# 24112

Extra: How to check memory consumption other than variables

X = [X]
# 48

def f():
# 112

class A():
#   File "<stdin>", line 1, in <module>
# TypeError: descriptor '__sizeof__' of 'object' object needs an argument
#Sys described later.getsizeof()Is possible


import sys
import numpy as np

N = 1000
D = 3
X = np.random.randn(N,D)
# 24112

By the way, in the case of pytorch, it will not be brought properly unless .storage () is added after the variable (one loss)

import sys
import torch

N = 1000
D = 3
X = torch.randn(N,D)
# 72
# 12064


Calling sys.getsizeof for each variable you want to see is tedious. So memory_profiler. It is not a standard package, so you need to install it.

By the way, the contents of the target method in @profile do not stop in pycharm debug mode (one loss)

from memory_profiler import profile
import numpy as np

def test():
    N = 10000000
    D = 3
    X = np.random.randn(N, D)


Then in the output

Line #    Mem usage    Increment   Line Contents
     5     62.3 MiB     62.3 MiB   @profile()
     6                             def test():
     7     62.3 MiB      0.0 MiB       N = 10000000
     8     62.3 MiB      0.0 MiB       D = 3
     9    291.2 MiB    228.9 MiB       X = np.random.randn(N, D)

Will give you.

Mem usage is Total and Increment is the amount of memory added on that line. It seems that I can not trust it very much depending on the implementation, but I use this for light confirmation.

Recommended Posts

[python] Checking the memory consumption of variables
The story of manipulating python global variables
the zen of Python
Code for checking the operation of Python Matplotlib
Towards the retirement of Python2
About the ease of Python
About the features of Python
The Power of Pandas: Python
Check the scope of local variables with the Python locals function.
Consideration for Python decorators of the type that passes variables
Find the diameter of the graph by breadth-first search (Python memory)
What beginners learned from the basics of variables in python
The story of Python and the story of NaN
[Python] The stumbling block of import
First Python 3 ~ The beginning of repetition ~
Existence from the viewpoint of Python
pyenv-change the python version of virtualenv
Change the Python version of Homebrew
[Python] Understanding the potential_field_planning of Python Robotics
Review of the basics of Python (FizzBuzz)
Checking the NAOqi Python development environment
About the basics list of Python basics
Learn the basics of Python ① Beginners
How to check the memory size of a variable in Python
[Python] Variables
How to check the memory size of a dictionary in Python
Change the length of Python csv strings
Check the behavior of destructor in Python
[Python3] Understand the basics of Beautiful Soup
Learn the basics while touching python Variables
The story of making Python an exe
Learning notes from the beginning of Python 1
Check the existence of the file with python
About the virtual environment of python version 3.7
[Python] Understand the content of error messages
[Python3] Rewrite the code object of the function
I didn't know the basics of Python
The result of installing python in Anaconda
Check the path of the Python imported module
[python] Get a list of instance variables
[python] [meta] Is the type of python a type?
The basics of running NoxPlayer in Python
Pandas of the beginner, by the beginner, for the beginner [Python]
The Python project template I think of.
In search of the fastest FizzBuzz in Python
Python Basic Course (at the end of 15)
Set the process name of the Python program
[Python] Get the character code of the file
The story of blackjack A processing (python)
Intuitively learn the reshape of Python np
Python Note: The secret role of commas
Reading comprehension of "The Go Memory Model"
Learning notes from the beginning of Python 2
Japanese translation: PEP 20 --The Zen of Python
[Python3] Understand the basics of file operations
Get the contents of git diff from python
Output the number of CPU cores in Python
Try the python version of emacs-org parser orgparse
[python] Check the elements of the list all, any
[Python] Sort the list of pathlib.Path in natural sort
Prepare the execution environment of Python3 with Docker