Symptoms
Object Strorage is not manageable:
- The Strorage Services > S3 screen is not loading
- The
vinfraCLI is timing out when attempting to get S3 configuration :
$ vinfra service s3 show timeout error: HTTPSConnectionPool(host='backend-api.svc.vstoragedomain', port=8888): Read timed out. (read timeout=100)
Cause
This issue can occur after a node is removed from the S3 cluster. In some cases, the cache file /var/lib/ostor/cluster_data_cl on the remaining nodes may still contain outdated references to the removed and now unreachable node. The behavior is considered as as software issue with ID VSTOR-108907 and planned to be fixed in upcoming releases of Virtuozzo Hybrid Infrastructure.
Resolution
To resolve an issue it is needed to remove the
/var/lib/ostor/cluster_data_cl file from all nodes in cluster.Additional diagnostic steps
- Verify that
vinfra service s3 showtimes out withread timeout=100 - Check if
/var/lib/ostor/cluster_data_clis out of sync. That is if it does not contain actual nodes addresses included in the current S3 config, and/or has different contents across all S3 nodes.