Symptoms:
While creating the image from a snapshot of volume having 1.95 TiB of file size, it shows uploading and the spinner (in VHI web panel) keeps loading for a long time for more than 5 hours and then it exits without prompting any error messages nor creates the image.
No direct error logs except this in beholder log:
/var/log/hci/beholder/beholder.log:2023-08-23 22:17:28.486 INFO [-] 23f778f8-fad1-4862-8a23-c9d09fcbe545 volume.vz14-den.vstoragedomain@vstorage volume.copy_to_image.start (info) /var/log/hci/beholder/beholder.log:2023-08-23 22:17:28.487 INFO [-] 23f778f8-fad1-4862-8a23-c9d09fcbe545 'updated' event recorded /var/log/hci/beholder/beholder.log:2023-08-24 10:29:57.795 INFO [-] 23f778f8-fad1-4862-8a23-c9d09fcbe545 volume.vz14-den.vstoragedomain@vstorage volume.copy_to_image.error (error) /var/log/hci/beholder/beholder.log:2023-08-24 10:29:57.796 INFO [-] 23f778f8-fad1-4862-8a23-c9d09fcbe545 'failed' event recorded
Cause:
The customer uses images to do an offline local backup of volume snapshots.
Support suggested customer create a volume out of the snapshot: https://docs.virtuozzo.com/virtuozzo_hybrid_infrastructure_5_4_admins_guide/index.html#managing-volume-snapshots.html?Highlight=Create%20a%20new%20volume%20from%20the%20snapshot.
However, the customer use case is that they want specifically to make an offline backup of a snapshot. They usually create an image from a snapshot and then download it via web panel as a means of offline local backup. However, since VHI image size has a 500GB hard-coded limitation. Their 1.95TB snapshot cannot be converted to an image. Having a volume of a snapshot does not give them the same ability to download as an image in the VHI web panel.
Resolution:
The workaround is to back up volume data using rsync after creating a volume from a snapshot. You can follow the steps from here on how to create a volume from a snapshot: https://docs.virtuozzo.com/virtuozzo_hybrid_infrastructure_5_4_admins_guide/index.html#managing-volume-snapshots.html?Highlight=Create%20a%20new%20volume%20from%20the%20snapshot.
Next you may use the steps below to backup that volume via rsync,
The following is an example of how to directly backup volume data using rsync. Basically, you need to keep a separate offline record of the size of the Source volume and the storage policy of the Source volume.
Important Note: As a precaution please note this workaround can only be used specifically for single volumes that are created from a snapshot. Normal volumes that have their own multiple snapshots cannot be backed up using rsync as they have a base file and other additional files.
Rsync steps to backup a volume:
1. let us use this example volume below as the Source:
[root@fra3-vhc-prd1-03 ~]# openstack volume show a6b536ea-008a-4ba2-8d87-f83424e054c1
+--------------------------------+---------------------------------------------------+
| Field | Value |
+--------------------------------+---------------------------------------------------+
| attachments | [] |
| availability_zone | nova |
| bootable | false |
| consistencygroup_id | None |
| created_at | 2022-07-03T07:29:16.714262 |
| description | None |
| encrypted | False |
| group_id | None |
| id | a6b536ea-008a-4ba2-8d87-f83424e054c1 |
| migration_status | None |
| multiattach | False |
| name | transfer-to-ams |
| os-vol-host-attr:host | fra3-vhc-prd1-02.vstoragedomain@vstorage#vstorage |
| os-vol-mig-status-attr:migstat | None |
| os-vol-mig-status-attr:name_id | None |
| os-vol-tenant-attr:tenant_id | eb41ec5c52b64427b123ae51016a48d6 |
| properties | |
| provider_id | None |
| replication_status | None |
| size | 1400 |
| snapshot_id | 8bb87bfe-741f-4da9-919b-28f99f935cbe |
| source_volid | None |
| status | creating |
| type | lowlatency3k |
| updated_at | 2022-07-03T07:29:16.947940 |
| user_id | ab9b005d7366431f899f77118a882f7d |
+--------------------------------+---------------------------------------------------+
Keep a record of its size and storage policy in this case lowlatency3k
2. Locate the volume in /mnt/vstorage
Source
[root@fra3-vhc-prd1-03 ~]# ll /mnt/vstorage/vols/datastores/cinder/volume-a6b536ea-008a-4ba2-8d87-f83424e054c1/volume-a6b536ea-008a-4ba2-8d87-f83424e054c1
-rwxrwx--- 1 root vstorage-users 1135467298816 Jul 3 09:29 /mnt/vstorage/vols/datastores/cinder/volume-a6b536ea-008a-4ba2-8d87-f83424e054c1/volume-a6b536ea-008a-4ba2-8d87-f83424e054c1
3. Rsync the volume data to a local/destination/local-file as a backup
[root@fra3-vhc-prd1-03 ~]# rsync -av --progress --inplace --append /mnt/vstorage/vols/datastores/cinder/volume-a6b536ea-008a-4ba2-8d87-f83424e054c1/volume-a6b536ea-008a-4ba2-8d87-f83424e054c1 /some/local/destination/local-file
Rsync steps to restore a volume:
4. To restore the volume. Create the Destination volume with the same size and policy as the Source volume. In the example below the new volume is in a new cluster and project called as chainstack/prod-chainstack-ams as an example: (for your own purposes you may create inside the same cluster and project)
[root@vhc-ams-man01 ~]# openstack volume show dec4c0c4-0c47-4d06-bbba-5bfe32393ccd
+--------------------------------+------------------------------------------------+
| Field | Value |
+--------------------------------+------------------------------------------------+
| attachments | [] |
| availability_zone | nova |
| bootable | false |
| consistencygroup_id | None |
| created_at | 2022-07-03T07:50:02.339483 |
| description | None |
| encrypted | False |
| group_id | None |
| id | dec4c0c4-0c47-4d06-bbba-5bfe32393ccd |
| migration_status | None |
| multiattach | False |
| name | transferring-do-not-touch |
| os-vol-host-attr:host | vhc-ams-man04.vstoragedomain@vstorage#vstorage |
| os-vol-mig-status-attr:migstat | None |
| os-vol-mig-status-attr:name_id | None |
| os-vol-tenant-attr:tenant_id | a2298131cf1c4ae19d267ea731a459b7 |
| properties | |
| provider_id | None |
| replication_status | None |
| size | 1400 |
| snapshot_id | None |
| source_volid | None |
| status | available |
| type | lowlatency3k |
| updated_at | 2022-07-03T07:50:04.839628 |
| user_id | 50172c6e76894c4aba87f54b25ba50f9 |
+--------------------------------+------------------------------------------------+
5. Locate the Destination volume path
Destination
[root@vhc-ams-man01 ~]# ll /mnt/vstorage/vols/datastores/cinder/volume-dec4c0c4-0c47-4d06-bbba-5bfe32393ccd/volume-dec4c0c4-0c47-4d06-bbba-5bfe32393ccd
-rwxrwx--- 1 root vstorage-users 3145816 Jul 3 07:50 /mnt/vstorage/vols/datastores/cinder/volume-dec4c0c4-0c47-4d06-bbba-5bfe32393ccd/volume-dec4c0c4-0c47-4d06-bbba-5bfe32393ccd
6. Do rsync to transfer data from the local file containing the volume data into the Destination volume
[root@fra3-vhc-prd1-03 ~]# rsync -av --progress --inplace --append /some/local/destination/local-file root@172.255.106.204:/mnt/vstorage/vols/datastores/cinder/volume-dec4c0c4-0c47-4d06-bbba-5bfe32393ccd/volume-dec4c0c4-0c47-4d06-bbba-5bfe32393ccd
sending incremental file list
volume-a6b536ea-008a-4ba2-8d87-f83424e054c1
291012696 0% 91.56MB/s 4:26:52
1501575512064 100% 102.69MB/s 3:52:24 (xfer#1, to-check=0/1)
sent 1501755663783 bytes received 31 bytes 107687473.65 bytes/sec
total size is 1501575512064 speedup is 1.00
[root@fra3-vhc-prd1-03 ~]#7. Wait till complete now it is done
Important Note: As a precaution please note this workaround can only be used specifically for single volumes that are created from a snapshot. Normal volumes that have their own multiple snapshots cannot be backed up using rsync as they have a base file and other additional files.