S3 backups may be skipped and S3 Cleanup may fail due to a locked policy

S3 backups may be skipped and S3 Cleanup may fail due to a locked policy

Customers may experience problems with S3 operations leaving backup chains locked, preventing further operation on that chain.
Theses issues are especially prevalent in v2.4.x, please upgrade to the latest version ASAP to significantly reduce probability of such incidents.


S3 Backup may be skipped because a previous S3 Backup operation is stuck in the "aborting" state.
The following messages may be printed in the CPM Server logs:
WARNING:  is_copy_required(s3_copy.py:43)  Backup copy for policy PolicyName not initiated: previous copy is in progress
INFO:  start_backup_copy_if_needed(s3_copy.py:434)  Copy is not required for Policy: PolicyName, Time: 2019-04-30 08:00:00. Skipping

The following message may be printed in the Backup Logs:
Info - Skipped S3 copy - previous copy is in progress...


S3 Backup may be skipped a previous S3 Backup operation didn't finish successfully and unlocked the backup chain:
The following messages may be printed in the CPM Server logs:
WARNING:  create_backup_copy_objects(s3_copy.py:262)  failed to lock policy PolicyName for copy - already locked for Copy since 2019-01-29 23:03:08
ERROR:  create_backup_copy_objects(s3_copy.py:393)  Failed to create Backup Copy objects
Traceback (most recent call last):
  File "./cpmserver/cpm/backup_copy/s3_copy.py", line 265, in create_backup_copy_objects
Exception: Chain is busy

The following message may be printed in the Backup Logs:
Warning - Copy operation is currently in progress for policy PolicyName. Copy cannot run.


Cleanup may fail because a previous Cleanup operation didn't finish successfully and unlocked the backup chain:
The following messages may be printed in the CPM Server logs:
WARNING: create_cleanup_objects(s3_cleanup.py:86) failed to lock policy PolicyName for cleanup - already locked for Cleanup since 2019-05-15 16:30:05
WARNING: add_log(models.py:297) Cleanup operation is currently in progress for policy PolicyName. Cleanup cannot run
The following message may be printed in the Backup Logs:
Warning - Cleanup operation is currently in progress for policy PolicyName. Cleanup cannot run
Error - Cleanup for policy
PolicyName failed

Cleanup may fail because a previous S3 Backup operation didn't finish successfully and unlocked the backup chain.
The following messages may be printed in the Backup Logs:
Error - A stuck task was detected - terminating operation
Warning - Failed to remove old snapshots of instance i-1234567890abcdef1 from S3 repository
Error - Cleanup for policy PolicyName failed


If you are experiencing any of the above problems, please open a support ticket, and provide the required information as requested at https://support.n2ws.com/portal/kb/articles/mustgather-information-for-cpm-collecting-data-for-a-support-ticket