CPM in Recovery Mode error - S3 worker fails with worker i-12345678912345678 stopped communicating - terminating operation

CPM in Recovery Mode error - S3 worker fails with worker i-12345678912345678 stopped communicating - terminating operation

 

Issue:  When CPM is in recovery mode being used for DR drills, the original CPM’s process to detect “zombie” worker’s can be triggered and it terminates the workers being launched from the CPM DR instance. This only occurs in DR drills where the original CPM instance is still up and running in the original location.

There are two solutions:

1. Enable termination protection on the workers being launched to do the recovery from the CPM DR Instance.  You will need to launch the recovery, and then go into AWS and enable termination protection on any workers that launch.

2. Change the CPM DR Server’s (not the original server) UUID by checking the “Generate New UUID” option shown below.

 

If you enable termination protection, then the original CPM instance will no longer terminate the DR CPM Instances workers.

If you change the UUID of the DR CPM instance then it will also stop terminating the workers, so either option will work to correct the issue.

• Related Articles

• S3 Cleanup fails with error on worker - No progress was reported - terminating operation

  Issue: S3 cleanup fails and reports error from worker - No progress was reported - terminating operation This occurs in version 4.2.2 and the error occurs when the worker reaches its 1 hour timeout while attempting to cleanup large snapshots. To fix ...

• EBS/RDS DR & Recovery with KMS key

  When copying cross account an EBS/RDS Volume encrypted with custom KMS, a KMS key should also be available in the other account. There are 2 ways that CPM uses for checking KMS key - Alias & Tag KMS Tag When using custom tag, you are telling CPM to ...

• N2WS-22712 - Unable to select S3 source for recovery when N2WS is in recovery mode

  Issue: When N2WS Server is in recovery mode, User can't select S3 as the source for the recovery Solution: v4.2.0 - upgrade to 4.2.2 and install patch 4.2.1 & v4.2.2 - install attached patch

• Backup of RDS may fail with the error “can not take snapshot, db is in stopped state" error

  Backup of RDS may fail with the below error: "(database: >DB  name>) can not take snapshot, db is in stopped state" This is due to an AWS limitation. AWS automatically backs up a stopped RDS, so there is no need for manual backups: ...

• How to retrieve logs from a CPM AWS Worker instance

  Linux & AWS knowledge is required Please read the entire KB before starting. N2WS uses temporary EC2 worker instances for several operations (copy to S3, FLR, etc), In cases where a worker is failing before it could communicate with the main server, ...