Symptoms
After the node update, multiple vstorage-csd/vstorage-mdsd services are failing to start at the node boot. The node may either reboot during the system boot and fall into an endless reboot loop, or boot with the root filesystem in read-only mode, and /vz and /vstoragepartitions not mounted.
Cause
This may happen on a Virtuozzo Storage cluster with configured High Availability after updating from any version below Virtuozzo 7 Update 11 to any other version.
2 product issues have been recognized (PSBM-93495 and PSBM-97071), which may result in empty /etc/fstab file after the update.
Resolution
The /etc/fstab file should be restored; this may require booting into the previous kernel, recovery mode, or single-user mode. The contents of /etc/fstab will vary between installations, but when using default installation options, the fstab will consist of four entries:
[root@vz7 ~]# cat /etc/fstab /dev/mapper/virtuozzo-root / ext4 defaults 1 1 UUID=d6003737-ca8c-437f-a1d0-a44fe391d13d /boot ext4 defaults 1 2 /dev/mapper/virtuozzo-vz /vz ext4 defaults,noatime,lazytime 1 2 UUID=929c09f1-0272-4bca-8a02-8176be1173a3 swap swap defaults 0 0
and in general, these entries can easily be deduced from the outputs of the following commands:
[root@vz7 ~]# awk '/executing.*mounted at/{print$16,$13}' /var/log/anaconda/storage.log
sda2 /boot
virtuozzo-vz /vz
virtuozzo-root /
[root@vz7 ~]# blkid | grep sda
/dev/sda4: UUID="soBtrw-ii7D-ZkL2-6kRz-XeKu-L0xN-PntYec" TYPE="LVM2_member" PARTUUID="f9aac970-f635-42f5-9e57-15f7cc8ea649"
/dev/sda2: UUID="d6003737-ca8c-437f-a1d0-a44fe391d13d" TYPE="ext4" PARTUUID="958a94bf-6adb-4524-b640-227d0ac9cb30"
/dev/sda3: UUID="929c09f1-0272-4bca-8a02-8176be1173a3" TYPE="swap" PARTUUID="17e8f09a-e2cf-46a9-8f72-a63ca9b79c1d"
/dev/sda1: PARTUUID="90613d3c-44d5-4cc1-b2c0-707f3faa1a95"
[root@vz7 ~]# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 100G 0 disk |-sda1 8:1 0 1M 0 part |-sda2 8:2 0 1G 0 part /boot |-sda3 8:3 0 3.9G 0 part [SWAP] `-sda4 8:4 0 95.1G 0 part |-virtuozzo-root 253:0 0 27.2G 0 lvm / `-virtuozzo-vz 253:1 0 68G 0 lvm /vz sdb 8:16 0 100G 0 disk `-sdb1 8:17 0 100G 0 part /vstorage/32c8f26f sdc 8:32 0 100G 0 disk `-sdc1 8:33 0 100G 0 part /vstorage/8a4ea2b7 sr0 11:0 1 2G 0 rom
[root@vz7 ~]# ls -l /dev/mapper crw------- 1 root root 10, 236 Sep 9 12:03 control lrwxrwxrwx 1 root root 7 Sep 9 12:03 virtuozzo-root - ../dm-0 lrwxrwxrwx 1 root root 7 Sep 9 12:03 virtuozzo-vz - ../dm-1
Note 1: vstorage CS mounts should not be written to /etc/fstab, as they are stored in a separate file — /etc/vstorage/fstab.
[root@vz7 ~]# cat /etc/vstorage/fstab UUID=8a4ea2b7-a70f-4f29-9d4b-214be884c980 /vstorage/8a4ea2b7 ext4 defaults,nofail 0 0 UUID=32c8f26f-8085-4cdf-a261-359af1ecdf8c /vstorage/32c8f26f ext4 defaults,nofail 0 0
Note 2: vstorage mount should not be written to /etc/fstab, as it is added by vstorage-fs.service systemctl service.
[root@vz7 ~]# systemctl status vstorage-fs.service vstorage-fs.service - Mount vstorage filesystems Loaded: loaded (/usr/lib/systemd/system/vstorage-fs.service; enabled; vendor preset: disabled) Active: active (exited) since Tue 2019-09-10 03:40:14 EDT; 2h 53min ago Main PID: 3821 (code=exited, status=0/SUCCESS) CGroup: /vstorage.slice/vstorage-services.slice/vstorage-fs.service └─4416 /usr/bin/vstorage-mount -c vz7 /mnt/vstorage
After /etc/fstab is repopulated, it is also advised to check that /boot/initramfs-*.imgof the proper version has been generated. In case the file is missing, it is necessary to recreate it by running the following command:
[root@vz7 ~]# yum reinstall vzkernel
PSBM-97071 is fixed and will be available in the next product updates
PSBM-93495 is fixed in Virtuozzo 7 Update 11.