Troubleshooting
RDS DR may fail with the error "Option group whateveroptiongroupgoeshere does not exist."
RDS DR may fail with the following error printed in the log: Fri 04/26/2019 09:34:52 AM - Error - RDS DR copy snapshot failed for region Anyregion (region), snapshot cpm-policy-40-db7-2019-4-26-12-30. Reason: Option group whateveroptiongroupgoeshere ...
Copy backup to S3 may fail with the error message "Workers could not be launched"
During backup copy to S3 a worker creation may fail with the error (backup.log): - Error - Workers could not be launched The worker usually fails right after it was started. There are several possible issues that gives you this error. 1) The was an ...
Cleanup from S3 or S3 copy fails with error: Worker configuration not found
Issue: When running a Copy to S3 policy, you may receive an error that states Worker configuration not found Or when S3 cleanup runs, you may see similar error in S3 cleanup logs. Solution The error will specify the region and/or account that a ...
An error may be recieved stating “A Problem Occurred (500)” when restoring an instance with multiple NIC’s
When trying to restore an AWS instance that utilizes multiple NIC’s, sometimes an error will return stating the below: “A Problem Occurred (500) Sorry, but there was an error while processing your request” Below is a screenshot of the error being ...
"Worker did not establish connection" and " worker did not complete initializing" errors during S3 and FLR
During S3 operations, you may encounter the message "Worker i-... did not establish connection" in the log of an S3 copy or S3 restore operation. Error - Worker i-1234567890abcdef did not establish connection - terminating operation During File Level ...
S3 copy fails with error: "This policy contains marketplace instances. The root volume of these instances cannot not be copied to S3, and will be excluded"
You may receive red letter warning when setting up "Copy to S3" if the root volume has a AWS Marketplace code associated with it. This policy contains marketplace instances. The root volume of these instances cannot not be copied to S3, and will be ...
Error when trying to configure S3 Copy "The cpmdata policy must be configured before 'Copy to S3' can be used".
When trying to create an S3 Repository in the CPM Manager screen you will get an error if the cmpdata policy is not configured and enabled like this: "The cpmdata policy must be configured before 'Copy to S3' can be used". Resolution: Please create a ...
CPM may mark a backup as successful although not all snapshots succeeded
In case CPM marks a backup as successful but not all snapshots succeed, please upgrade to the latest 2.X.X version Reference defect number – N2WS-3578, fixed in v2.4.0 Link to Release Notes and the latest patch – ...
CPM may abort Tag scanning with error "New policy xyz has more than one template”
In case CPM tag scanning is aborted, please open the Scan Log and search for the error “New policy xyz has more than one template tag scanning is running”. If you encounter this error, please upgrade to the latest 2.X.X version Reference defect ...
Instance may be removed from the policy once a tag scan is performed
An instance may be removed from the policy once a tag scan is performed Issue: When there is an instance which is added to two policies - by tag and via web-console, an instance will be removed from the policy in which it was added via GUI. In logs ...
A wrong account may be listed in the Snapshot View Report for cross account replicated snapshots
If CPM prints the backup account's name for all the snapshots in the snapshot view report, although some snapshots were replicated to another account, please upgrade to the latest 2.x.x version. Reference defect number – N2WS-3659, fixed in v2.4.0 ...
DR may fail with "DR encountered an internal problem and failed" instead of printing the actual permissions issue
DR may fail with "DR encountered an internal problem and failed" instead of printing the actual permissions issue with CPM role. You may find the error message below in backup log. This message is not informative. Error - DR encountered an internal ...
Permission check may fail with the error "User not found", if CPM was deployed using CloudFormation
Permission check may fail with the error "User not found", if CPM was deployed using CloudFormation. This could be seen in policy permissions check CSV file: (log bundle/policy_check_AWSDEVTEST.csv example) Error Evaluating Permissions for account ...
Cross-account DR backup of encrypted snapshots of Aurora cluster (RDS) may fail with the “No matching KMS alias” error
DR of encrypted snapshots of AuroraDB cluster (DRS) may fail with the “No matching KMS alias” error in the Backup log: Error - Aurora DR copy snapshot failed (in Backup account). No matching KMS alias on target region (source <source_region>, target ...
How to improve CPM’s performance or overcome high memory consumption by creating a swap partition
***IMPORTANT*** Please make sure your CPM complies with the recommended instance sizes regardless of creating the swap file. ...
AD FS may produce “An error occurred” page when logging in or performing Test Connection
Problem: AD FS may produce “An error occurred” page when logging in or performing Test Connection Symptom: you may find an error like this in the Windows Event Logs for AD FS on the AD FS server: Microsoft.IdentityServer.Web.InvalidScopeException: ...
Encrypted Snapshot Recovery from DR account may fail with the “Given key ID is not accessible” error
Problem: DR recovery may fail with the error “Given key ID is not accessible” The following messages may be printed in the CPM Server log: INFO: recover_volumes_helper(additional_views.py:515) from account: accountA, to account: accountB ... ...
Encrypted Snapshot may fail DR with the “Given key ID is not accessible” error
Problem: DR copy may fail with errors like “Given key ID is not accessible” The following error may be printed in the CPM Backup log and the CPM Server log: Error - Volume DR copy snapshot failed for region US East (Ohio) (in DR account), ...
DR of an encrypted snapshot may fail with the “Not Authorized to use key” error
Problem: DR copy may fail with errors like “Not authorized to perform” or “not authorized to use key” Example in CPM backup log: 10-24-18 10:11:04 - Error - Volume DR copy snapshot failed for region US East (N. Virginia), snapshot ...
Cross-Region DR RDS backup may fail with "Copy failed due to an AWS limitation regarding default encryption key” error
RDS DR backup of encrypted snapshots between different regions may fail with the following error in the Backup log: Error - RDS DR, copy snapshot failed (to DR account) due to an AWS limitation regarding default encryption key (source US East (Ohio), ...
Some backups may not run on schedule while other backups still run using the same schedule
Some backups may stop running on their scheduled time although other backups which use the same schedule do run and Run ASAP works. If you face this issue, please upgrade to the latest 2.3.x version. Reference defect number – N2WS-1633, fixed in ...
CPM configuration may fail with the error: "Could not attach volume %vol-id% to instance %Id% Reason: Invalid value '/dev/sdf' for unixDevice. Attachment point /dev/sdf is already in use"
Issue: CPM configuration may fail with the error: "Could not attach volume %vol-id% to instance %Id% Reason: Invalid value '/dev/sdf' for unixDevice. Attachment point /dev/sdf is already in use" ERROR: done(views.py:181) Could not attach volume ...
CPM server may display error "Signature validation failed. SAML Response rejected"
During login via Identity Provider on the sign on page or during the Test Connection, CPM server may display error message: login failed. reason: Signature validation failed. SAML Response rejected (invalid_response) You may find corresponding errors ...
CPM Server may display error "Problem Occurred (500)" during the Identity Provider configuration
CPM Server may display error "Problem Occurred (500)" during IdP configuration: Error message: A Problem Occurred(500) Sorry, but there was an error while processing your request This error appears after you hit the 'Apply' button. The most likely ...
Failed to open IdP login page or to test connection with the errors: "HTTP Error 503. The service is unavailable." or "404 - File or directory not found."
Issue: When trying to connect to IdP from CPM, you may receive errors: "HTTP Error 503. The service is unavailable." or "404 - File or directory not found." There are no errors in the AD FS event log or OKTA System logs, as well as CPM logs. ...
Logging into CPM using Identity Provider may fail with an error "An error occurred"
Issue: When trying to log in to CPM, you may receive the following error: In the CPM log files, there will be no related error messages, but in the Windows Event viewer you can find these errors: AD FS errors 261 and/or 364: AD FS error 364: AD FS ...
Login to the CPM using an IDP may fail with the "Invalid issuer" error message
If you're using an IDP, login to the CPM may fail with the following error message: login failed. reason: Invalid issuer in the Assertion/Response (invalid_response): It also may be found in the cpm_server.log: ERROR: ...
DR copy may fail with "Failed looking for source KMS. Error: Request limit exceeded” error
In case of DR failure, if you find the below error in the server log, please upgrade to the latest 2.x.x version. ERROR: _get_volume_snap_kms_func(aws_utils.py:<line_number>) failed getting snap snapshot-id' kms alias (region region_name,Backup). ...
DR snapshots may get encrypted with default encryption key on target region instead of custom key
This issue may occur if Encryption Keys list contains over a hundred of encryption keys. The below warning may appear in the backup log: Warning - Volume snapshot <snapshot_id> is encrypted with key from another account. Using default encryption on ...
Restoring an instance may not restore tags
If a restored instance is created without its tags, please search the server log for the following print: ERROR: add_tags_to_resource(aws_utils.py:<line_number>) Add tag failed for Instance (<instance_id>). Exception: EC2ResponseError: 400 Bad ...
Add support for Cross account DR of RDS and Aurora snapshots which are encrypted with custom shared key
In version 2.3.0 a new feature was added- Support for Cross account DR of RDS and Aurora snapshots which are encrypted with custom encryption key. This feature was available only for EC2 resources. ...
Cross account Recovery of an encrypted snapshot with shared custom encryption key may fail
Cross account recovery of an encrypted snapshot which is encrypted with a shared custom encryption key may fail when trying to recover a resource either from DR to backup account or from backup to DR account (same account recovery works). CPM server ...
Explore may fail with "Invalid value '/dev/xvdp' for unixDevice"
If File Explore fails with below error: ERROR: recover_volume(.\cpmserver\cpm\recovery.py:<line_number>) Volume <volume_id> created (original: vol-0d746a37a79a611fe). However attach to instance <instance_id> (device: <path>) failed. Reason: Invalid ...
CPM may not retry Tag Scanning if scanning fails due to RequestLimitExceeded
If Tag scanning is triggered and fails due to RequestLimitExceeded, an alert is generated and no retry is scheduled. Please search for the below print in the server log: ERROR: scan_resources(backup_tag.py:<line_number)) Failed getting <resources> ...
Backups may fail with "CPM Server identified this backup as orphaned: marking as failed" after upgrade to v2.3.x
Backups may fail with "CPM Server identified this backup as orphaned: marking as failed” error after upgrading to 2.3.0X and logs may show the below prints: update_request(agentapi_requests.py:<line_number>) failed calling url ...
CPM may generate an error in the logs when creating snapshots with tags that begin with “aws:”
When CPM creates snapshots, some snapshots may have tags that begin with “aws:”. These tags are AWS’s tags and are only allowed to be used by AWS. Therefore, when CPM will create a snapshot of an EC2 resource with such a tag, the snapshot will ...
CPM supports custom encryption keys for DR
To support the usage of a custom encryption key for DR, you will need to perform the following: In the account where the custom key resides: Go to KMS and browse to the key you wish to share. Go to the "Other AWS accounts" at the bottom of the page ...
Basic API example for CPM login and retrieving the list of CPM users using powershell
The below script provides a basic example for the usage of CPM API with Powershell. Copy the below text into a .ps1 file, replace the text in blue with your relevant text and run the script. ...
Cross-region DR copy of encrypted snapshot may fail with "Copy failed due to an AWS limitation regarding default encryption key” error
DR of encrypted snapshots may fail with the following error in the Backup log: Error - <DR Policy>, copy snapshot failed (in Backup account) due to an AWS limitation regarding default encryption key source <source region>, target <target region>, ...
Scan tags operation may hang if there are many Aurora clusters in an account
If "Scan Resources" is getting stuck (progress indicator doesn't stop spinning) when scanning an account with many Aurora clusters (more than 20), please upgrade to latest 2.X.X version Reference defect number – N2WS-1878, fixed in v2.3.0c Link to ...
Next page