Symptoms

1. Ploop snapshot fails to be deleted with the following error message:

   ~# prlctl snapshot-delete 333 -i f941b9bd-bc78-4b8f-a846-c91ec61499d2
   Delete the snapshot...
   Failed to delete snapshot: Operation failed. Failed to delete snapshot: Error in ioctl(PLOOP_IOC_MERGE): Device or resource busy [3]
   Failed to delete snapshot {f941b9bd-bc78-4b8f-a846-c91ec61499d2}
   

2. It is not possible to create a new backup of the affected ploop container:

   ~# vzabackup -F localhost -e 333
   Starting backup operation for node 'pcs.container.org'...
   * Operation with the Container pcs.container.org is started
   * Preparing for backup operation
   * Operation with the Container pcs.container.org is finished successfully.
   Backup operation for node 'pcs.container.org' failed:
   Backup failed
   

Cause

Previous backup of the container was not finished properly - it was terminated or it crashed. As a result, container's image is mounted twice.

Resolution

1. Clarify who locks root.hds:

   ~# grep /333/ /sys/block/ploop*/pdelta/*/image
   /sys/block/ploop32064/pdelta/0/image:/vz/private/333/root.hdd/root.hds
   /sys/block/ploop32064/pdelta/1/image:/vz/private/333/root.hdd/root.hds.{f37e6b00-7fcb-49b8-8942-58179ba3900d}
   /sys/block/ploop43803/pdelta/0/image:/vz/private/333/root.hdd/root.hds
   

   So you can see that root.hds is mounted twice.

2. Check what is regular mount and what mount is related to terminated backup:

   ~# cat /proc/mounts | grep 333 | grep ploop
   /dev/ploop32064p1 /vz/root/333 ext4 rw,relatime,barrier=1,data=ordered,balloon_ino=12,pfcache_csum,pfcache=/vz/pfcache,jqfmt=vfsv0,usrjquota=aquota.user,grpjquota=aquota.group 0 0
   /dev/ploop43803p1 /vz/backup/333/tmpidyHKK/fs ext4 ro,relatime,barrier=1,data=ordered,balloon_ino=12,pfcache_csum 0 0
   

   Most probably the second mount is related to a backup. You need to double check this assumption:

   ~# cat /sys/block/ploop43803/pstate/cookie
   vzbackup
   

   So it is obviously a backup-related mount that should be unmounted.

3. Confirm that the mount is not used by any process:

   ~# lsof 2 /dev/null | grep /vz/backup/333/tmpidyHKK/fs
   ~#
   

   In case the mount is still in use by a backup process:

   ~# ps aux | grep vzlpl
   root      429273  0.0  0.0 124800  7552 ?        S     2013   0:00 /opt/pva/agent/bin/vzlpl /var/opt/pva/agent/tmp.JJbakK
   root      520460  0.0  0.0 103256   848 pts/6    S+   07:08   0:00 grep vzlpl
   root      568003  0.0  0.0 123968  4168 ?        S     2013   0:00 /opt/pva/agent/bin/vzlpl /var/opt/pva/agent/tmp.QxeOqd
   

   Check if the process is actually operational and not stuck. For example, confirm that the temporary files in /var/opt/pva/agent/ are actually present. In case /var/opt/pva/agent/tmp.JJbakK and /var/opt/pva/agent/tmp.QxeOqd do not exist, kill both vzlpl processes and run lsof once again to confirm that the mount is not used.

4. Unmount the device:

   ~# umount /vz/backup/333/tmpidyHKK/fs
   ~# ploop umount -d /dev/ploop43803
   Unmounting device /dev/ploop43803