Customers may experience problems with S3 operations leaving backup chains locked, preventing further operation on that chain.
Theses issues are especially prevalent in v2.4.x, please upgrade to the latest version ASAP to significantly reduce probability of such incidents.
S3 Backup may be skipped because a previous S3 Backup operation is stuck in the "aborting" state.
The following messages may be printed in the CPM Server logs:
WARNING: is_copy_required(s3_copy.py:43) Backup copy for policy PolicyName not initiated: previous copy is in progress
INFO: start_backup_copy_if_needed(s3_copy.py:434) Copy is not required for Policy: PolicyName, Time: 2019-04-30 08:00:00. Skipping
The following message may be printed in the Backup Logs:
Info - Skipped S3 copy - previous copy is in progress...
S3 Backup may be skipped a previous S3 Backup operation didn't finish successfully and unlocked the backup chain:
The following messages may be printed in the CPM Server logs:
WARNING: create_backup_copy_objects(s3_copy.py:262) failed to lock policy PolicyName for copy - already locked for Copy since 2019-01-29 23:03:08
ERROR: create_backup_copy_objects(s3_copy.py:393) Failed to create Backup Copy objects
Traceback (most recent call last):
File "./cpmserver/cpm/backup_copy/s3_copy.py", line 265, in create_backup_copy_objects
Exception: Chain is busy
The following message may be printed in the Backup Logs:
Warning - Copy operation is currently in progress for policy PolicyName. Copy cannot run.
Cleanup may fail because a previous Cleanup operation didn't finish successfully and unlocked the backup chain:
The following messages may be printed in the CPM Server logs:
WARNING:
create_cleanup_objects(s3_cleanup.py:86) failed to lock policy
PolicyName for cleanup - already locked for Cleanup since 2019-05-15
16:30:05
WARNING: add_log(models.py:297) Cleanup operation is currently in progress for policy PolicyName. Cleanup cannot run
The following message may be printed in the Backup Logs:
Warning - Cleanup operation is currently in progress for policy PolicyName. Cleanup cannot run
Error - Cleanup for policy PolicyName failed
Cleanup may fail because a previous S3 Backup operation didn't finish successfully and unlocked the backup chain.
The following messages may be printed in the Backup Logs:
Error - A stuck task was detected - terminating operation
Warning - Failed to remove old snapshots of instance i-1234567890abcdef1 from S3 repository
Error - Cleanup for policy PolicyName failed