Memory leak in Python Jupyter Lab (Notebook)?

environment

Kaggle Docker Python v86
https://console.cloud.google.com/gcr/images/kaggle-images/GLOBAL/python

phenomenon

I'm using Jupyter Lab, and after defining variables, del and gc.collect () doesn't free memory.

Reproduction procedure

Write the following code in one cell and execute it.

import numpy as np
import pandas as pd
import psutil
print(psutil.Process().memory_info().rss / 1024**2)
df = pd.DataFrame(np.arange(50000000).reshape(-1,5))
df.head()

Note that it is df.head () at the end.

In the next cell

print(psutil.Process().memory_info().rss / 1024**2)
del df
print(psutil.Process().memory_info().rss / 1024**2)

The memory status is not displayed immediately, so check it in the next cell just in case.

print(psutil.Process().memory_info().rss / 1024**2)

By the way, it is the same even if gc.collect () is subsequently performed.

Cause

I thought it was a problem with pandas.DataFrame, but it's not. When the output is displayed by the Jupyter Lab function at the end of the "cell", it seems that it will not be released even if you del in another cell after that.

solution

Do not df.head () at the end of the cell. Print if you want to display it.

comment

――It's a part / behavior that makes you feel uneasy, but is it something you are careful about when you get started with Jupyter properly (I don't know). ――I was able to investigate and confirm it at hand, but the reason why it is not subject to gc is whether the reference is retained in the screen display function of jupyter or something. It seems that IPython is the cause. If the search term is Jupyter memory leak, https://github.com/jupyter/notebook/issues/3713 will appear, and it seems that there is an explanation in the future (I have not seen it). --If the search term is pandas memory leak, https://stackoverflow.com/questions/14224068/memory-leak-using-pandas-dataframe etc. will appear and you will continue to get lost (get lost).