Memory leak in Python Jupyter Lab (Notebook)?



I'm using Jupyter Lab, and after defining variables, del and gc.collect () doesn't free memory.

Reproduction procedure

Write the following code in one cell and execute it.

import numpy as np
import pandas as pd
import psutil
print(psutil.Process().memory_info().rss / 1024**2)
df = pd.DataFrame(np.arange(50000000).reshape(-1,5))


Note that it is df.head () at the end.

In the next cell

print(psutil.Process().memory_info().rss / 1024**2)
del df
print(psutil.Process().memory_info().rss / 1024**2)


The memory status is not displayed immediately, so check it in the next cell just in case.

print(psutil.Process().memory_info().rss / 1024**2)




I thought it was a problem with pandas.DataFrame, but it's not. When the output is displayed by the Jupyter Lab function at the end of the "cell", it seems that it will not be released even if you del in another cell after that.


Do not df.head () at the end of the cell. Print if you want to display it.



――It's a part / behavior that makes you feel uneasy, but is it something you are careful about when you get started with Jupyter properly (I don't know). ――I was able to investigate and confirm it at hand, but the reason why it is not subject to gc is whether the reference is retained in the screen display function of jupyter or something. It seems that IPython is the cause. If the search term is Jupyter memory leak, will appear, and it seems that there is an explanation in the future (I have not seen it). --If the search term is pandas memory leak, etc. will appear and you will continue to get lost (get lost).

Recommended Posts

Memory leak in Python Jupyter Lab (Notebook)?
Generate Jupyter notebook ".ipynb" in Python
Install Python Jupyter lab
3 Jupyter notebook (Python) tricks
Display HTML in Jupyter notebook
Multiprocessing error in Jupyter Notebook
Setting up Jupyter Lab in a Python 3.9 venv environment
python3.8 venv environment jupyter notebook
Check for memory leaks in Python
Snippet settings for python jupyter notebook
Python memo Anaconda x Jupyter Notebook
[Python] Generate QR code in memory
View graphs inline in Jupyter Notebook
Easy to use Jupyter notebook (Python3.5)
Jupyter Notebook 6.0.2 cannot be installed in the Python 2.7 environment created in Anaconda
Handle multiple python versions in one jupyter
Build Jupyter Lab (Python) environment with Docker
Use Jupyter Lab and Jupyter Notebook with EC2
How to execute commands in jupyter notebook
When looking at memory usage in Python 3
Linking python and JavaScript with jupyter notebook
Quadtree in Python --2
Python in optimization
CURL in python
Metaprogramming in Python
Python 3.3 in Anaconda
SendKeys in Python
Jupyter Notebook memo
[Jupyter Notebook / Lab] 3 ways to debug on Jupyter [Pdb]
Introducing Jupyter Notebook
Meta-analysis in Python
Unittest in python
Jupyter Lab begins
How to implement shared memory in Python (mmap.mmap)
Browser specification of Jupyter Notebook in Windows environment
Epoch in Python
Discord in Python
Powerful Jupyter Notebook
Sudoku in Python
DCI in Python
quicksort in python
nCr in python
N-Gram in Python
Programming in python
Bash in Jupyter
Plink in Python
Constant in python
Lifegame in Python.
FizzBuzz in Python
Sqlite in python
StepAIC in Python
Jupyter notebook password
Jupyter Notebook memo
N-gram in python
LINE-Bot [0] in Python
Csv in python
Disassemble in Python
Reflection in Python
Constant in python
nCr in Python.
format in python