CPM in Recovery Mode error - S3 worker fails with worker i-12345678912345678 stopped communicating - terminating operation

CPM in Recovery Mode error - S3 worker fails with worker i-12345678912345678 stopped communicating - terminating operation

 

Issue:  When CPM is in recovery mode being used for DR drills, the original CPM’s process to detect “zombie” worker’s can be triggered and it terminates the workers being launched from the CPM DR instance. This only occurs in DR drills where the original CPM instance is still up and running in the original location.

There are two solutions:

1. Enable termination protection on the workers being launched to do the recovery from the CPM DR Instance.  You will need to launch the recovery, and then go into AWS and enable termination protection on any workers that launch.

2. Change the CPM DR Server’s (not the original server) UUID by checking the “Generate New UUID” option shown below.

 

If you enable termination protection, then the original CPM instance will no longer terminate the DR CPM Instances workers.

If you change the UUID of the DR CPM instance then it will also stop terminating the workers, so either option will work to correct the issue.

• Related Articles

• S3 Cleanup fails with error on worker - No progress was reported - terminating operation

  Issue: S3 cleanup fails and reports error from worker - No progress was reported - terminating operation This occurs in version 4.2.2 and the error occurs when the worker reaches its 1 hour timeout while attempting to cleanup large snapshots. To fix ...

• EBS/RDS DR & Recovery with KMS key

  When copying cross account an EBS/RDS Volume encrypted with custom KMS, a KMS key should also be available in the other account. There are 2 ways that CPM uses for checking KMS key - Alias & Tag KMS Tag When using custom tag, you are telling CPM to ...

• DR of encrypted snapshots may fail with the “No matching KMS alias” error

  DR of encrypted snapshots may fail with the following error in the Backup log: ERROR: start_copy_region(.\cpmserver\cpm\dr_volume.py:<line number>) Volume DR copy snapshot failed (in Backup account). No matching KMS alias on target region (source ...

• Instance recovery may fail with the error "You are not authorized to perform this operation"

  Instance recovery may fail with the error "You are not authorized to perform this operation" : ERROR: recover_ebs_based_instance(.\cpmserver\cpm\recovery.py:749) Run_instance failed. Exception: EC2ResponseError: 403 Forbidden <?xml version="1.0" ...

• N2WS-22712 - Unable to select S3 source for recovery when N2WS is in recovery mode

  Issue: When N2WS Server is in recovery mode, User can't select S3 as the source for the recovery Solution: v4.2.0 - upgrade to 4.2.2 and install patch 4.2.1 & v4.2.2 - install attached patch