Symptoms:

Attempting to update a VHI cluster results in errro message:

[
    {
        "passed": false,
        "message": "Please contact support team for assistance. There are third-party RPM packages that should be removed before the upgrade from the following nodes:",
        "details": [
            "devvz01.vstoragedomain: imake, mlnx-en-doc, mlnx-en-sources, mlnx-en-utils, mlnx-fw-updater, mlnx-tools, mlnx_en, mstflint, ofed-scripts, sfx3xdriver-src",
            "devvz02.vstoragedomain: imake, mlnx-en-doc, mlnx-en-sources, mlnx-en-utils, mlnx-fw-updater, mlnx-tools, mlnx_en, mstflint, ofed-scripts, sfx3xdriver-src",
            "devvz03.vstoragedomain: certbot, gpg-pubkey, imake, mlnx-en-doc, mlnx-en-sources, mlnx-en-utils, mlnx-fw-updater, mlnx-tools, mlnx_en, mstflint, ofed-scripts, python-ndg_httpsclient, python-requests-toolbelt, python-zope-component, python-zope-event, python2-acme, python2-certbot, python2-configargparse, python2-josepy, python2-parsedatetime, python2-pyrfc3339, sfx3xdriver-src"
        ],
        "exception": null,
        "severity": "critical"
    }
]

Cause:

There exists 3rd party RPMS or rpm repositories installed on the VHI nodes that need to be removed before the update can be allowed to avoid any issues with the update process.

Resolution:

Sometimes there are important 3rd party drivers that are used by the node and cannot be removed however for other RPMs kindly remove them by:

# rpm -e <rpm_name>

check if  gpg-pubkey is part of virtuozzo/acronis:

[root@devvz03 ~]# rpm -q --qf '%{name}-%{version}-%{release} %{packager}\n' gpg-pubkey | grep -vP "Virtuozzo|Acronis"
gpg-pubkey-352c64e5-52ae6884 Fedora EPEL (7) <epel@fedoraproject.org>
[root@devvz03 ~]#

Therefore in gpg-pubkey above is safe to remove:

# rpm -e gpg-pubkey-352c64e5-52ae6884

Sometimes there are important 3rd party drivers that are used by the node and cannot be removed:

To overcome this issue had to add a workaround by editing a file on the primary management node to add exclusions to driver RPMs:

[root@devvz03 ~]# cat /usr/libexec/vstorage-ui-updater/etc/backend_updater.cfg                                                                                                           
#UPDATE_MAINTENANCE_TIMEOUT = 60 * 60 

#UPDATE_FORCE_REBOOT = False 

# Compute update mode: 
# - 'legacy': Update compute during control plane update 
# - 'rolling': Per-node compute update 
#COMPUTE_UPDATE_MODE = 'legacy' 

# snapshots count limit 
#SNAPSHOTS_COUNT_LIMIT = 150 

# RPM packages with bad signature which should be ignored 
#BAD_SIGN_RPMS_EXCLUDE = [] 

# Third-party RPM packages which should be ignored 
THIRDS_EXCLUDE = [ 
    'imake', 
    'mlnx-en-doc', 
    'mlnx-en-sources', 
    'mlnx-en-utils', 
    'mlnx-fw-updater', 
    'mlnx-tools', 
    'mlnx_en', 
    'mstflint', 
    'ofed-scripts', 
    'sfx3xdriver-src', 
] 

DUMMY_UPDATER_CONFIG = True

 
The list of packages in THIRDS_EXCLUDE = [   section is from the error message, also please add an extra statement at the end:

DUMMY_UPDATER_CONFIG = True

 

After editing make sure file permissions are as follows:

[root@devvz03 ~]# ll /usr/libexec/vstorage-ui-updater/etc/backend_updater.cfg 
-rw-r--r-- 1 vstoradmin vstoradmin 434 Dec 22 18:40 /usr/libexec/vstorage-ui-updater/etc/backend_updater.cfg 
[root@devvz03 ~]#

then make sure to :

systemctl reload vstorage-ui-backend-celery

Then run:

 tail -f /var/log/vstorage-ui-backend/celery.log

Observe the log to make sure the reload is finished after a few minutes you should attempt the update via web panel.

There might be a false alarm of "failed to start update: network error", but be patient and the system will start to update after a few minutes.

Related Ticket(s):

#416619