What to do if the inode is exhausted on EC2 Linux
There appears to be insufficient space on your system to finish and an error occurred!
One day, the company's service stopped, and when I investigated the cause, the above error occurred and I could not create a new file. When I checked the disk space, I found that there was no problem with the free space, but the inode was exhausted. Here is a memorandum of dealing with this problem.
Inode Check free space
Check the free space of the inode with the df -i command
/dev/xvda1 217227 217227 217227 100% /
Upon confirmation, the usage rate is 100%. With this, a new file cannot be created, and the service stops when the log is written.
File system check
Check the file system with the df -T command
/dev/xvda1 ext4 abbreviation
This time it was ext4. Since ext4 cannot (probably) change the size of the inode table dynamically, rebuild the file system or increase the disk size and at the same time take the method of changing the size of the inode table. Since EC2 can easily expand the disk size, we will expand the disk size in consideration of the future.
Volume size expansion
$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
xvda 202:0 0 8G 0 disk
└─xvda1 202:1 0 8G 0 part /
Check the volume size.
Then, on the EC2 dashboard, click on the EBS ID from the instance in question to go to the EBS volume page.
Make a snapshot of the target volume and change the volume.
Here, specify a value larger than the volume size confirmed earlier and make a change request. This change can take up to 5 minutes.
If you can change it, it will be as follows.
$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
xvda 202:0 0 20G 0 disk
└─xvda1 202:1 0 8G 0 part /
If it does not change even after waiting for a while, it will be changed by restarting the instance. However, since the root partition is still 8G and has not been changed, extend it with the following command.
$ sudo growpart /dev/xvda 1
（mkdir: cannot create directory '/tmp/growpart.1959': No space left on device If FAILED: failed to make temp dir is displayed, there is no space and the tmp file cannot be created, so clean it before executing.)
Check again
$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
xvda 202:0 0 20G 0 disk
└─xvda1 202:1 0 20G 0 part /
This has expanded the size to the maximum.
File system extension
Finally, extend the file system
$ sudo resize2fs /dev/xvda1
$ df -i
/dev/xvda1 1310720 217227 1093493 17% /
The inode usage has been reduced to 17%. This is the end.
bonus
$ dpkg --get-selections | grep linux-
linux-base install
linux-headers-4.4.0-121 install
linux-headers-4.4.0-121-generic install
linux-headers-4.4.0-124 install
linux-headers-4.4.0-124-generic install
linux-headers-4.4.0-127 install
linux-headers-4.4.0-127-generic install
linux-headers-4.4.0-128 install
linux-headers-4.4.0-128-generic install
linux-headers-4.4.0-130 install
linux-headers-4.4.0-130-generic install
linux-headers-4.4.0-133 install
linux-headers-4.4.0-133-generic install
linux-headers-4.4.0-134 install
linux-headers-4.4.0-134-generic install
linux-headers-4.4.0-137 install
linux-headers-4.4.0-137-generic install
linux-headers-4.4.0-138 install
linux-headers-4.4.0-138-generic install
linux-headers-4.4.0-139 install
linux-headers-4.4.0-139-generic install
linux-headers-4.4.0-151 install
linux-headers-4.4.0-151-generic install
linux-headers-4.4.0-154 install
linux-headers-4.4.0-154-generic install
linux-headers-4.4.0-157 install
linux-headers-4.4.0-157-generic install
linux-headers-4.4.0-159-generic install
linux-headers-generic install
linux-headers-virtual install
linux-image-4.4.0-101-generic deinstall
linux-image-4.4.0-104-generic deinstall
linux-image-4.4.0-108-generic deinstall
linux-image-4.4.0-109-generic deinstall
linux-image-4.4.0-112-generic deinstall
linux-image-4.4.0-116-generic deinstall
linux-image-4.4.0-141-generic deinstall
linux-image-4.4.0-142-generic deinstall
linux-image-4.4.0-143-generic deinstall
linux-image-4.4.0-151-generic install
linux-image-4.4.0-154-generic install
linux-image-4.4.0-157-generic install
linux-image-4.4.0-159-generic install
linux-image-4.4.0-92-generic deinstall
linux-image-4.4.0-93-generic deinstall
linux-image-4.4.0-96-generic deinstall
linux-image-4.4.0-97-generic deinstall
linux-image-4.4.0-98-generic deinstall
linux-image-virtual install
linux-libc-dev:amd64 install
linux-modules-4.4.0-143-generic deinstall
linux-modules-4.4.0-151-generic install
linux-modules-4.4.0-154-generic install
linux-modules-4.4.0-157-generic install
linux-modules-4.4.0-159-generic install
linux-virtual install
When I checked it, there were a lot of old kernels left. If you clean this, the space will increase, so if there is no space, you should check it. Remove the kernel that you will definitely not use
Perform a deletion test with the following option --dry-run, and delete if there is no problem.
$ apt-get autoremove --purge linux-*-4.4.0-{121,124,127,128,130,133,134,137,138,139}* --dry-run
This completes the procedure. Thank you for your support!
Recommended Posts