What to do if the inode is exhausted on EC2 Linux

There appears to be insufficient space on your system to finish and an error occurred!

One day, the company's service stopped, and when I investigated the cause, the above error occurred and I could not create a new file. When I checked the disk space, I found that there was no problem with the free space, but the inode was exhausted. Here is a memorandum of dealing with this problem.

Inode Check free space

Check the free space of the inode with the df -i command

/dev/xvda1     217227 217227 217227   100% /

Upon confirmation, the usage rate is 100%. With this, a new file cannot be created, and the service stops when the log is written.

File system check

Check the file system with the df -T command

/dev/xvda1 ext4 abbreviation

This time it was ext4. Since ext4 cannot (probably) change the size of the inode table dynamically, rebuild the file system or increase the disk size and at the same time take the method of changing the size of the inode table. Since EC2 can easily expand the disk size, we will expand the disk size in consideration of the future.

Volume size expansion

$ lsblk
NAME    MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
xvda    202:0    0  8G  0 disk 
└─xvda1 202:1    0  8G  0 part /

Check the volume size. Then, on the EC2 dashboard, click on the EBS ID from the instance in question to go to the EBS volume page. スクリーンショット 2020-07-10 2.26.59.png

Make a snapshot of the target volume and change the volume. スクリーンショット 2020-07-10 2.31.20.png

Here, specify a value larger than the volume size confirmed earlier and make a change request. This change can take up to 5 minutes.

If you can change it, it will be as follows.

$ lsblk
NAME    MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
xvda    202:0    0  20G  0 disk 
└─xvda1 202:1    0  8G  0 part /

If it does not change even after waiting for a while, it will be changed by restarting the instance. However, since the root partition is still 8G and has not been changed, extend it with the following command.

$ sudo growpart /dev/xvda 1

mkdir: cannot create directory '/tmp/growpart.1959': No space left on device If FAILED: failed to make temp dir is displayed, there is no space and the tmp file cannot be created, so clean it before executing.)

Check again

$ lsblk
NAME    MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
xvda    202:0    0  20G  0 disk 
└─xvda1 202:1    0  20G  0 part /

This has expanded the size to the maximum.

File system extension

Finally, extend the file system

$ sudo resize2fs /dev/xvda1
$ df -i
/dev/xvda1     1310720 217227 1093493   17% /

The inode usage has been reduced to 17%. This is the end.

bonus

$ dpkg --get-selections | grep linux-
linux-base					install
linux-headers-4.4.0-121				install
linux-headers-4.4.0-121-generic			install
linux-headers-4.4.0-124				install
linux-headers-4.4.0-124-generic			install
linux-headers-4.4.0-127				install
linux-headers-4.4.0-127-generic			install
linux-headers-4.4.0-128				install
linux-headers-4.4.0-128-generic			install
linux-headers-4.4.0-130				install
linux-headers-4.4.0-130-generic			install
linux-headers-4.4.0-133				install
linux-headers-4.4.0-133-generic			install
linux-headers-4.4.0-134				install
linux-headers-4.4.0-134-generic			install
linux-headers-4.4.0-137				install
linux-headers-4.4.0-137-generic			install
linux-headers-4.4.0-138				install
linux-headers-4.4.0-138-generic			install
linux-headers-4.4.0-139				install
linux-headers-4.4.0-139-generic			install
linux-headers-4.4.0-151				install
linux-headers-4.4.0-151-generic			install
linux-headers-4.4.0-154				install
linux-headers-4.4.0-154-generic			install
linux-headers-4.4.0-157				install
linux-headers-4.4.0-157-generic			install
linux-headers-4.4.0-159-generic			install
linux-headers-generic				install
linux-headers-virtual				install
linux-image-4.4.0-101-generic			deinstall
linux-image-4.4.0-104-generic			deinstall
linux-image-4.4.0-108-generic			deinstall
linux-image-4.4.0-109-generic			deinstall
linux-image-4.4.0-112-generic			deinstall
linux-image-4.4.0-116-generic			deinstall
linux-image-4.4.0-141-generic			deinstall
linux-image-4.4.0-142-generic			deinstall
linux-image-4.4.0-143-generic			deinstall
linux-image-4.4.0-151-generic			install
linux-image-4.4.0-154-generic			install
linux-image-4.4.0-157-generic			install
linux-image-4.4.0-159-generic			install
linux-image-4.4.0-92-generic			deinstall
linux-image-4.4.0-93-generic			deinstall
linux-image-4.4.0-96-generic			deinstall
linux-image-4.4.0-97-generic			deinstall
linux-image-4.4.0-98-generic			deinstall
linux-image-virtual				install
linux-libc-dev:amd64				install
linux-modules-4.4.0-143-generic			deinstall
linux-modules-4.4.0-151-generic			install
linux-modules-4.4.0-154-generic			install
linux-modules-4.4.0-157-generic			install
linux-modules-4.4.0-159-generic			install
linux-virtual					install

When I checked it, there were a lot of old kernels left. If you clean this, the space will increase, so if there is no space, you should check it. Remove the kernel that you will definitely not use

Perform a deletion test with the following option --dry-run, and delete if there is no problem.

$ apt-get autoremove --purge linux-*-4.4.0-{121,124,127,128,130,133,134,137,138,139}* --dry-run

This completes the procedure. Thank you for your support!

Recommended Posts

What to do if the inode is exhausted on EC2 Linux
What to do if you can't use WiFi on Linux
What to do if you get the error ʻERR_FEATURE_UNAVAILABLE_ON_PLATFORM` when using ts-node-dev on Linux
What to do if the Pipenv environment is corrupted by updating Homebrew
What to do if you forget your login password on Manjaro Linux
What to do if you get stuck during Anaconda installation on Linux
What is the Linux kernel?
What to do if SciPy installation fails on CentOS
Check if the LAN cable is disconnected on Linux
What to do if pyenv is not enabled (zsh)
What to do if Japanese language support is not completely installed on Ubuntu 16.04
What to do if (base) is displayed at the beginning of the Mac terminal
What to do if the Microsoft Store opens even if you run python on Windows
What to do if the progress bar is not displayed in tqdm of python
What to do if Python IntelliSense is not displayed in VS Code on Windows
What to do when Japanese is not displayed on matplotlib
What to do if the package dependency cannot be repaired
[AWS EC2] Settings you want to do on Amazon Linux 2
What to do if `pip install matplotlib` fails on Mac
What to do if Linux VLC can no longer rotate
What to do if pip install mysqlclient fails on MacOS
What to do if you lose your EC2 key pair
What to do if yum breaks
[* CentOS 6.10] What to do if you cannot add the IUS Community Project repository on CentOS 6.5 (VirtualBox)
What to do if the image is not displayed using matplotlib etc. in the Docker container
What to do if Linux Modem Manager is interfering with writing keymaps to your own keyboard
What to do if Python doesn't work on Git for Windows
What if RHEL on Azure needs to extend the file system?
What to do if "Unnamed: 0" is added in to_csv-> read_csv in pandas
What to do if PyAudio cannot be installed on Python 3.7, 3.8, 3.9 on Windows
What to do if you can't use the trash in Lubuntu 18.04.
What to do when the value type is ambiguous in Python?
What to do if grep: empty (sub) expression appears on Mac grep
What to do if you cat or tail a binary file and the terminal is garbled
What to do after installing Linux (Ubuntu)
What to do if the user name is changed and the pyenv library path does not pass
What to do if there is a decimal in python json .dumps
What to do if the yum command fails to set locale, defaulting to C
What to do if the Chainer (Windows) sample mnist terminates with WinError 183.
What to do if the server doesn't start with python manage.py runserver
What to do when the warning "The environment is in consistent ..." appears in the Anaconda environment
What to do if you get "The session could not be opened" when installing CentOS on VirtualBox
What is Linux
What is Linux?
What is Linux
What is Linux
What to do if you get the message "" ~ .pkg "is corrupted and cannot be opened" when installing wxPython on Mac OS X
What to do if you can't pipenv shell
Note: What to do if pip install fails
What to do if mod_fcgid cannot resolve UnicodeEncodeError
What to do if rails s doesn't work
What to do if pip cannot be installed
It's a Mac. What is the Linux command Linux?
What to do if atom autocomplete-python doesn't work
(Linux beginner) What is the magic word aux?
What to do if Docker-sync suddenly stops working
What to do if "amazon-linux-extras" → "No module named amazon_linux_extras"
What is the difference between Unix and Linux?
What to do if the latest Jupyter Notebook and nb extensions don't work
Let's Encrypt updated! What to do if the certbot renew command doesn't work
[Note] What to do if the Qt library conflicts between pyqt and opencv