Skip to main content

S3 cluster may become unmanageable via UI/cli after the S3 node was removed from the S3 cluster

  • Updated
Applies to:
  • S3
  • object storage
  • Virtuozzo Infrastructure

Symptoms

Object Strorage is not manageable:

  • The  Strorage Services > S3 screen is not loading
  • The vinfra CLI is timing out when attempting to get S3 configuration :
$ vinfra service s3 show
timeout error: HTTPSConnectionPool(host='backend-api.svc.vstoragedomain', port=8888): Read timed out. (read timeout=100)
 

Cause

 

This issue can occur after a node is removed from the S3 cluster. In some cases, the cache file /var/lib/ostor/cluster_data_cl on the remaining nodes may still contain outdated references to the removed and now unreachable node. The behavior is considered as as software issue with ID VSTOR-108907 and planned to be fixed in upcoming releases of Virtuozzo Hybrid Infrastructure.

 

Resolution

To resolve an issue it is needed to remove the /var/lib/ostor/cluster_data_cl file from all nodes in cluster.
 

Additional diagnostic steps

  1. Verify that vinfra service s3 show times out with read timeout=100
  2. Check if /var/lib/ostor/cluster_data_cl is out of sync. That is if it does not contain actual nodes addresses included in the current S3 config, and/or has different contents across all S3 nodes.

 

Related to

Related articles