If the Linux server has an LVM VG containing an iSCSI device, the system shutdown will be delayed or fail.
Occurs in a configuration that puts data in remote storage using open-iscsi + lvmcache on Ubuntu 20.04. The target is attached to sda by auto login with open-iscsi, and it is deferred mounted at startup with the _netdev mount option. sda is formatted as LVM PV and mixed with some of the local NVMe to make LVM VG. (Because it is necessary for lvmcache)
# pvs
PV VG Fmt Attr PSize PFree
/dev/sda remote lvm2 a-- <200.00g 0
/dev/vda2 vg lvm2 a-- <40.00g 520.00m
/dev/vg/cachepv remote lvm2 a-- <5.00g 0
# lvs
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
data remote Cwi-aoC--- <200.00g [cache_cvol] [data_corig] 100.00 12.40 0.00
boot vg -wi-ao---- 500.00m
cachepv vg -wi-ao---- 5.00g
root vg -wi-ao---- 30.00g
swap vg -wi-ao---- 4.00g
If you try to shut down as it is, the shutdown is frequently blocked in the middle, the kernel stack trace is displayed, and the power does not turn off.
The iscsi device is managed by the systemd unit open-iscsi.service.
This unit is started by After = network-online.target after the network is built, and is terminated before the network is stopped during shutdown.
At the end, blk deactivate automatically unmounts the dependent file system and deactivates the LVM.
This is fine in most cases, but it doesn't necessarily cover everything involved in complex lvm configurations. If you do not unmount the filesystem, LVM, or both in the correct order, you will be permanently blocked trying to access the iscsi device after the sda or network stack disappears.
Create a systemd service that works only at shutdown so that it becomes After of open-iscsi.service and Before of other dependent services, and deactivate it with umount/lvchange -an in an appropriate order.
[Unit]
Wants=network-online.target remote-fs-pre.target
# open-Add iscsi
After=network-online.target iscsid.service open-iscsi.service
# remote/List all services that require data
Before=snap.lxd.daemon.service snap.lxd.activate.service
DefaultDependencies=no
Conflicts=shutdown.target
Before=shutdown.target
[Service]
Type=oneshot
RemainAfterExit=true
ExecStart=-/bin/true
#Unmount / deactivate in order of dependency.
#ExecStop as it may have been unmounted on another unit=-Ignoring the error
ExecStop=-/bin/umount -R /data
ExecStop=-/sbin/vgchange -an remote -vv
ExecStop=-/sbin/lvchange -an vg/cachepv -vv
[Install]
WantedBy=sysinit.target
As for the dependency, after/Before is added after copying open-iscsi.service. Units that depend on remote/data are listed in Before. I searched for a suitable target so that I didn't have to write it one by one, but I couldn't find a suitable one because there were few services that specified such remote-fs.target in After.