Have you ever used Linux and experienced "I installed a new kernel module and it didn't boot" or "I changed the boot options I passed to the kernel and it didn't boot"?
The ** grub2-reboot ** command, which changes the kernel only at the next boot, is useful in this situation. It's especially useful in the cloud, where console access is restricted, as a subsequent reboot will revert to the original kernel. It is also useful when switching between multiple kernels for testing purposes.
It was a command that was unexpectedly unknown, so I was thinking of introducing it, but as I investigated it, I ended up writing the following.
I explained it at the beginning, but I will explain it in a little more detail. When a server that has been normally started up to now suddenly stops starting, the following causes / triggers are possible.
When such a trouble occurs, it is necessary to troubleshoot the following procedure.
** How to troubleshoot when not starting **
There is no problem when the server itself is at hand, but when it is in a remote location such as the cloud or data center, the console screen cannot be displayed and the old kernel may not be selected.
** Public Cloud: ** Some vendors do not offer console login functionality ** Data center: ** For servers that do not have a BMC / IPMI board or KVM device (PC switch) set, you need to go to the data center.
The grub2-reboot command only temporarily changes the boot kernel, so recovery is possible as long as you can reboot. However, be aware that the default kernel will also be updated when you install with Yum / DNF.
The GRUB configuration file is basically /etc/grub.conf or /boot/grub/grub.conf for GRUB up to 6 series, and /etc/grub2.cfg for GRUB2 after 7 series. That's right. However, there is a difference between BIOS mode and UEFI mode. Especially in the case of physical servers, most of them are used in UEFI. I will explain about that.
GRUB2 has many files, the following files are typical.
/etc/grub2.cfg/boot/grub2/grub.cfg/boot/loader/entries/*/boot/grub2/grubenv/etc/default/grub/etc/sysconfig/kernelOf these, /etc/grub2.cfg is referenced when changing the default kernel. The general procedure for changing the default kernel in 7 series is as follows. 8 series uses grubby.
python
awk -F\' '$1=="menuentry " {print i++ " : " $2}' /etc/grub2.cfg
grub2-set-default entry number
/boot/grub2.cfg is a symbolic link, so to ensure reliability, specify the actual /boot/grub2/grub.cfg and rebuild.grub2-mkconfig -o /boot/grub2/grub.cfg
** Tips </ font> ** 8 series requires grubby operation, but 7 series can also use grubby. However, older versions may not be available. </ font>
** Caution </ font> ** For GRUB up to 6 series, modified /boot/grub/grub.conf with an editor to change the default kernel. However, in GRUB2 you should not change /boot/grub2/grub.cfg by hand. Be sure to change it using commands such as grub2-mkconfig and grubby.
In BIOS mode, the explanation so far is fine, but in UEFI mode it is different. Generally, it is explained as follows, but in addition to the substance, the symbolic link is also different. explain in detail.
** BIOS boot **: /boot/grub2/grub.cfg ** UEFI boot **: /boot/efi/EFI/*/grub.cfg
You can determine if the server or virtual machine you are currently using is in UEFI mode with the following command: When I examined the cloud, AWS and Azure were in BIOS mode, and Oracle Cloud was in UEFI mode. I haven't checked all instance types, so it's just for reference.
In UEFI mode
$ ls -l /sys/firmware/efi
total 0
-r--r--r--. 1 root root 4096 Aug 13 10:00 config_table
drwxr-xr-x. 2 root root 0 Aug 13 09:06 efivars
-r--r--r--. 1 root root 4096 Aug 13 10:00 fw_platform_size
-r--r--r--. 1 root root 4096 Aug 13 10:00 fw_vendor
-r--r--r--. 1 root root 4096 Aug 13 10:00 runtime
drwxr-xr-x. 9 root root 0 Aug 13 10:00 runtime-map
-r--------. 1 root root 4096 Aug 13 10:00 systab
drwxr-xr-x. 29 root root 0 Aug 13 10:00 vars
In BIOS mode
$ ls -l /sys/firmware/efi
ls: cannot access /sys/firmware/efi: No such file or directory
The configuration file in UEFI mode was introduced as /boot/efi/EFI/*/grub.cfg. The asterisk part depends on the distribution.
| Distribution | file name |
|---|---|
| RHEL | /boot/efi/EFI/redhat/grub.cfg |
| CentOS | /boot/efi/EFI/centos/grub.cfg |
| Oracle Linux | /boot/efi/EFI/redhat/grub.cfg |
| Amazon Linux | /boot/efi/EFI/amzn/grub.cfg |
| Ubuntu | /boot/grub/grub.cfg(UEFI/Common to BIOS) |
Perhaps the confusing thing is symlinks. In UEFI mode, /etc/grub2-efi.cfg has been added.
BIOS mode
$ ls -l /etc/grub2*
lrwxrwxrwx. 1 root root 22 Sep 3 13:31 /etc/grub2.cfg -> ../boot/grub2/grub.cfg
UEFI mode
$ ls -l /etc/grub2*
lrwxrwxrwx. 1 root root 22 Sep 3 13:31 /etc/grub2.cfg -> ../boot/grub2/grub.cfg
lrwxrwxrwx. 1 root root 31 Sep 3 13:31 /etc/grub2-efi.cfg -> ../boot/efi/EFI/
centos/grub.cfg
If you're using symlinks, you're likely to do it. However, /etc/grub2.cfg refers to /boot/grub2/grub.cfg, but the entity does not exist.
$ ls -l /boot/efi/EFI/centos/grub.cfg
-rwx------. 1 root root 10223 Sep 3 13:35 /boot/efi/EFI/centos/grub.cfg
$ ls -l /boot/grub2/grub.cfg ★ does not exist
ls: cannot access /boot/grub2/grub.cfg: No such file or directory
So the next command will fail in UEFI mode.
python
awk -F\' '$1=="menuentry " {print i++ " : " $2}' /etc/grub2.cfg
It is necessary to recognize that it is UEFI and specify /etc/grub2-efi.cfg or /boot/efi/EFI/*/grub.cfg.
python
awk -F\' '$1=="menuentry " {print i++ " : " $2}' /etc/grub2-efi.cfg
The following files, which have not been explained so far, are explained. This time, the contents of CentOS 8 are displayed, but it is almost the same for 7 series.
3-3-1. /etc/default/grub This file contains the GRUB2 overall settings. Related to the default kernel is the ** GRUB_DEFAULT ** directive. The key is "** saved **".
GRUB_TIMEOUT=5 ★ Display time of kernel selection menu at startup
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved ★ The name of the directive used by the default kernel
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="console"
GRUB_CMDLINE_LINUX="resume=/dev/mapper/cl-swap rd.lvm.lv=cl/root rd.lvm.lv=cl/swap rhgb quiet"
GRUB_DISABLE_RECOVERY="true"
GRUB_ENABLE_BLSCFG=true
3-3-2. /boot/grub2/grubenv This file contains the default kernel settings. Since it was "** saved **" earlier, the kernel specified by the ** saved_entry ** directive becomes the default kernel.
In RHEL7 series, "menu number" or "menu name" is set, but in RHEL8 series, "ID + kernel version" is set.
7 series
# GRUB Environment Block
saved_entry=CentOS Linux (3.10.0-1062.el7.x86_64) 7 (Core)
8 series
# GRUB Environment Block
saved_entry=0aa24a7da6284581821b2215d0757580-4.18.0-147.el8.x86_64
kernelopts=root=/dev/mapper/cl-root ro resume=/dev/mapper/cl-swap rd.lvm.lv=cl/root rd.lvm.lv=cl/swap rhgb quiet
boot_success=0
boot_indeterminate=0
In RHEL8 series, the matching configuration file exists here. This is a specification called BootLoaderSpecByDefault (BLS).
RHEL8 series only
# ls /boot/loader/entries/
0aa24a7da6284581821b2215d0757580-0-rescue.conf
0aa24a7da6284581821b2215d0757580-4.18.0-147.el8.x86_64.conf
0aa24a7da6284581821b2215d0757580-4.18.0-193.14.2.el8_2.x86_64.conf
3-3-3. /etc/sysconfig/kernel This file contains instructions on how to update the kernel.
# UPDATEDEFAULT specifies if new-kernel-pkg should make
# new kernels the default
UPDATEDEFAULT=yes
# DEFAULTKERNEL specifies the default kernel package type
DEFAULTKERNEL=kernel-core
** UPDATEDEFAULT ** specifies whether to make the default kernel when the kernel is updated. If yes, it will be the default kernel, but if no, it will not be the default kernel.
For example, specify no when you do not want to use a new kernel even if you install an additional kernel with yum update.
In the first place, if you want to suppress the kernel installation itself, you should exclude it in the Yum / DNF configuration file as follows.
shell-session:/etc/yum.conf（/etc/dnf/dnf.conf）
exclude=kernel
Set the default kernel type in ** DEFAULTKERNEL **. The normal kernel is kernel and kernel-core, but the debug kernel is kernel-debug. Also, in UEK of Oracle Linux, it is kernel-uek.
I'm going to explain how to use grub2-reboot, but I've taken a detour. The method of changing the default kernel is almost the same, so I will explain it together. The main procedure is as follows.
The command to change the kernel differs depending on the version, so each will be explained. If you want to use grubby even in 7 series, please see the explanation of 8 series.
** 7 series: ** grub2-mkconfig / grub2-set-default or grubby ** 8 series: ** grubby
The current default can be displayed with the grub2-editenv command. The value of the saved_entry directive is the default kernel.
# grub2-editenv list
saved_entry=CentOS Linux (3.10.0-1127.19.1.el7.x86_64) 7 (Core)
# uname -a
Linux hostname 3.10.0-1127.19.1.el7.x86_64 #1 SMP Tue Mar 17 23:49:17 UTC 2020
x86_64 x86_64 x86_64 GNU/Linux
Display a list of installed kernels. Combined with the previous results, index number 0 is the default kernel.
** BIOS mode **
# awk -F\' '$1=="menuentry " {print i++ " : " $2}' /etc/grub2.cfg
** UEFI mode **
# awk -F\' '$1=="menuentry " {print i++ " : " $2}' /etc/grub2-efi.cfg
Output example
0 : CentOS Linux (3.10.0-1127.19.1.el7.x86_64) 7 (Core)
1 : CentOS Linux (3.10.0-1062.18.1.el7.x86_64) 7 (Core)
2 : CentOS Linux (3.10.0-1062.el7.x86_64) 7 (Core)
3 : CentOS Linux (0-rescue-b5a029229fd6464ca40d2e93351d5c5f) 7 (Core)
In the following example, one version specifies an older entry number 1 kernel.
# grub2-set-default 1
# grub2-editenv list
saved_entry=1
Reboot for the modified kernel to take effect. That's it.
# systemctl reboot
grubby can display kernel filenames and indexes.
# grubby --default-kernel
/boot/vmlinuz-4.18.0-193.14.2.el8_2.x86_64
# grubby --default-index
0
Display a list of installed kernels. This example also shows the next line that matches grep with the -A 2 option.
# grubby --info=ALL | grep -A 2 index
index=0
kernel="/boot/vmlinuz-4.18.0-193.14.2.el8_2.x86_64"
args="ro crashkernel=auto resume=/dev/mapper/cl-swap rd.lvm.lv=cl/root rd.lvm.lv=cl/swap rhgb quiet $tuned_params"
--
index=1
kernel="/boot/vmlinuz-4.18.0-147.el8.x86_64"
args="ro crashkernel=auto resume=/dev/mapper/cl-swap rd.lvm.lv=cl/root rd.lvm.lv=cl/swap rhgb quiet $tuned_params"
--
index=2
kernel="/boot/vmlinuz-0-rescue-0aa24a7da6284581821b2215d0757580"
args="ro crashkernel=auto resume=/dev/mapper/cl-swap rd.lvm.lv=cl/root rd.lvm.lv=cl/swap rhgb quiet"
To permanently change the default kernel, use the grubby command as follows: In this example, an equal (=) is specified between the --set-default option and the value, but a blank space may be used.
# grubby --set-default=/boot/vmlinuz-4.18.0-147.el8.x86_64
You can also specify the index number as follows.
# grubby --set-default-index=1
Confirm that it has been changed.
# grubby --default-kernel
/boot/vmlinuz-4.18.0-147.el8.x86_64
You can use --info to display the entire entry for the specified kernel. In this example, the kernel file name is specified, but the index can also be specified.
# grubby --info=/boot/vmlinuz-4.18.0-147.el8.x86_64
index=1
kernel="/boot/vmlinuz-4.18.0-147.el8.x86_64"
args="ro crashkernel=auto resume=/dev/mapper/cl-swap rd.lvm.lv=cl/root rd.lvm.lv
=cl/swap rhgb quiet $tuned_params"
root="/dev/mapper/cl-root"
initrd="/boot/initramfs-4.18.0-147.el8.x86_64.img $tuned_initrd"
title="CentOS Linux (4.18.0-147.el8.x86_64) 8 (Core)"
id="0aa24a7da6284581821b2215d0757580-4.18.0-147.el8.x86_64"
Reboot for the modified kernel to take effect. That's it.
# systemctl reboot
I changed the kernel permanently earlier, but you can change the kernel only at the next boot. Rebooting again will restore the original kernel, which is useful for the following purposes:
Use ** grub2-reboot ** to make changes. Specify the menu entry number (index number) as follows.
grub2-reboot 1
In order to confirm the operation, a series of steps will be described.
# uname -r
4.18.0-147.el8.x86_64
# grubby --default-kernel
/boot/vmlinuz-4.18.0-147.el8.x86_64
# grubby --default-index
1
# grub2-reboot 0
# grubby --default-index
1
# systemctl reboot
# uname -r
4.18.0-193.14.2.el8_2.x86_64
# grubby --info=/boot/vmlinuz-4.18.0-193.14.2.el8_2.x86_64
index=0
kernel="/boot/vmlinuz-4.18.0-193.14.2.el8_2.x86_64"
args="ro crashkernel=auto resume=/dev/mapper/cl-swap rd.lvm.lv=cl/root rd.lvm.lv=cl/swap rhgb quiet $tuned_params"
root="/dev/mapper/cl-root"
initrd="/boot/initramfs-4.18.0-193.14.2.el8_2.x86_64.img $tuned_initrd"
title="CentOS Linux (4.18.0-193.14.2.el8_2.x86_64) 8 (Core)"
id="0aa24a7da6284581821b2215d0757580-4.18.0-193.14.2.el8_2.x86_64"
# systemctl reboot
# uname -r
4.18.0-147.el8.x86_64