Troubleshooting
Scripts may fail when executed by policy
If scripts fail when executed by CPM, but succeed when executed via terminal, please download CPM logs and search for the following message: INFO: backup_function_internal(.\cpmagent\agent.py:<line number>) After pre snapshot. script_logs: [{'log': ...
CPM may fail to communicate with AWS gateways with the "timed out" error
CPM may fail to communicate with AWS gateways with the "timed out" error during various operations: ERROR: run_snapshots(.\cpmagent\agent.py:1462) snapshot of policy PolicyName, volume: vol-0123456789abcdef failed (independent). Reason: timed out ...
Complete_VSS operation may fail if snapshot takes more than 45 minutes to complete
Complete_VSS operation may fail if snapshot takes more than 45 minutes to complete, due to exceeding the default VSS timeout: In order to verify this, you should check the Agent's logs - VSS script should fail with return code "3" immediately after ...
Backup of cpmdata may fail after migration/upgrade with the "The volume '....' does not exist" error
Backup of cpmdata may fail after migration/upgrade with the "The volume '....' does not exist" error: Error - snapshot of independent volume vol-0123456789abcdef failed to start. Reason: The volume 'vol-0123456789abcdef' does not exist This error may ...
DR may fail to copy a snapshot with the "The specified keyId .... is not accessible" error
DR may fail to copy a snapshot with the "The specified keyId .... is not accessible" error logged in the Backup Log and CPM Server log: Error - Volume DR copy snapshot failed for region EU (Ireland), snapshot snap-0123456789abcdef. Reason: The ...
Cross-account DR may fail with the "Encrypted snapshots with EBS default key cannot be shared (OperationNotPermitted)" error
DR may fail with the "Encrypted snapshots with EBS default key cannot be shared (OperationNotPermitted)" error: Error - Volume DR copy snapshot failed for region Asia Pacific (Sydney), snapshot snap-0123456789abcdef (to DR account). Reason: Failed ...
Instance recovery may fail with the error "You are not authorized to perform this operation"
Instance recovery may fail with the error "You are not authorized to perform this operation" : ERROR: recover_ebs_based_instance(.\cpmserver\cpm\recovery.py:749) Run_instance failed. Exception: EC2ResponseError: 403 Forbidden <?xml version="1.0" ...
File-level restore may fail with the "Could not get CPM instance info" error
File-level restore may fail with the "Could not get CPM instance info" error: The following error is printed in the CPM server log: ERROR: explore_backup_init(.\cpmserver\cpm\filemanager\views.py:317) Could not get CPM instance info ...
Troubleshooting Linux scripts in CPM
In order to troubleshoot scripts, you need to connect to the CPM instance over SSH (using user "cpmuser") and execute the scripts from command line. Please do not use "su" or "sudo", as CPM doesn't elevate permissions. If when you run the script you ...
Linux scripts may start failing after upgrading to a new CPM version (using AMI) or migrating CPM to a different account/region
Linux scripts may start failing after upgrading to a new CPM version (using AMI) or migrating CPM to a different account/region. In order to restore scripts' functionality, please run a script from command line, and approve the SSH key by answering ...
Better detection of dead remote Agents (Windows)
If a policy that is configured with app-aware targets fails, please open Backup log. If all script/VSS operations (before_, after_ and complete_), fail after reaching the script/VSS timeout, please upgrade to latest v2.x.x to resolve the issue. ...
Remote Agent functionality and scripts functionality may be disabled in v2.1.3
The following symptoms may occur when using v2.1.3 only: app-aware backups or backup scripts are failing. The link at the bottom of the CPM GUI “download thin backup agent" disappeared. The view of the Agents cannot be changed to show more than 20 ...
Cleanup may not delete out of retention backups (v2.1.3 only)
If cleanup does not delete out of retention backups, please search the Cleanup log for the following print: Warning - No backups were marked for deletion in policy , because out of X completed backups, only 0 were marked as successful If you found ...
Backups may fail due to the "Value (None) for parameter snapshotId is invalid" error
If backups are failing, please download the CPM logs and search for the string: <Code>InvalidParameterValue</Code><Message>Value (None) for parameter snapshotId is invalid. Expected: 'snap-...'.</Message> If you found the above string please update ...
CPM may fail to add RDS to a policy, if any of the displayed RDS instances have an "inaccessible-encryption-credentials" status
CPM may fail to add RDS to a policy, if any of the displayed RDS instances has a "inaccessible-encryption-credentials" status The "inaccessible-encryption-credentials" status is visible in the EC2 Console, in the CPM GUI it's displayed as "Ensure ...
Cleanup may fail with the “UnboundLocalError: local variable 'res' referenced before assignment” error
Cleanup process may fail to delete out of retention backups with the following error: cleanup_function UnboundLocalError: local variable 'res' referenced before assignment If you have found this error in the logs, please upgrade to the latest 2.1.X ...
Volume/instance restore may fail with Exception "The volume vol-12345678 does not exist"
If Volume/instance restore of an existing volume fails with the following exception in the Recovery log: "Exception: The volume '<volume-id>' does not exist", please upgrade to latest 2.X.X to resolve the issue. Reference defect number - ff4e8ae447, ...
Backup may fail after approximately 30 minutes
If backup fails after approximately 30 minutes, please check the Backup Log for the following prints: ERROR: check_keep_alive_delinquents(.\cpmagent\record_management.py:<line_number>) Found stuck backup of policy <policy_name>, has not responded for ...
DR may fail due to the "snap-0123456789abcdef failed, has status error" error
If Cross-Region DR fails with the following error in the Backup Log: Error - Volume DR copy snapshot failed for region <Region Name>, snapshot snap-<snapshot_ID>e (status: error).” Please download the logs from CPM GUI and search for the following: ...
DR may fail due to the “Volume DR Failed getting volumes info” and “RequestLimitExceeded” errors
If DR fails with the below errors in Backup log ERROR: check_region_copy_state(.\cpmserver\cpm\dr_volume.py:<line_number>) Volume DR Failed getting volumes info in region <Region name> (policy <Policy Name>,in Backup account), Exception: ...
Cross-Region recovery of encrypted RDS may fail
Cross-Region recovery of encrypted RDS may fail with following error in the Recovery Log: Reason: Must specify new KMS key for cross region encrypted snapshot copy If you found the above error, please upgrade to the latest 2.X.X. version to resolve ...
File level restore may fail with the "Failed adding share permission" error
File level restore may fail with the "Failed adding share permission" error: ERROR: share_snapshot(.\cpmserver\cpm\aws_utils.py:265) Failed adding share permission to snapshot snap-1234567890abcdef (original volume: vol-abcdef1234567890) Failed. ...
Cross account recovery of encrypted volumes may fail
A cross account recovery of an encrypted snapshot may fail with this error: Encrypted snapshots can only create volumes in the owner's account Please upgrade to latest 2.X.X version to resolve the issue. Reference defect number - 6676511295, fixed in ...
Failed backups may not be retried if schedule is limited to certain days of the week
A failed backup may not be retried if the retry is scheduled on a day different from the start day. Issue may occur if the day on which retry is scheduled is not enabled in the schedule. If you face this issue, please upgrade to the latest 2.X.X ...
Backup and snapshot view csv reports may not show managed users’ data when retrieved by CPM root user
When retrieving the “backup view csv report” or “snapshot view csv report” with CPM root user, the reports may contain data related to the logged in root user and not the managed users as well. To resolve this issue, please upgrade to the latest ...
Adding Redshift Clusters to a policy may fail
Adding Redshift Clusters to a policy may fail if the Node Type field is longer than 10 characters. If you face this issue, please upgrade to latest 2.X.X version. Reference defect number - 4ca2fa2b0d, fixed in v2.1.1 Link to the release notes and ...
Cross-account DR may fail with the "RequestLimitExceeded" error
The Cross-Account DR may fail with the following error in the Backup Log: Error,"Volume DR copy snapshot failed for region <Region Name>, snapshot snap-<Snapshot Number> (to DR account). Reason: Failed adding share permission to snapshot ...
Snapshot view csv report creation may crash (error 500) if there are non-ascii characters in volume or instance names
“Snapshot view csv report” which contains non-ASCii characters may cause “500 Internal Error” when trying to download it. To resolve this issue, please upgrade to the latest 2.X.X version. Link to Release Notes and the latest patch - ...
Snapshots may not get deleted if cleanup is running for more than 1 hour
If snapshots which are out of retention are not deleted by cleanup, please download CPM logs from GUI. Please check for the “Last Cleanup” time in “General Settings”, next to the “Run Now” button. Please search the logs for these 3 errors ...
Delegates may become “Zombie” users if a user is deleted without deleting its delegates first
Delegates may become “Zombie” users with User Type “Undetermined” if a user is deleted without deleting its delegates first. Trying to delete the “Zombie” users will result in Error 500. To resolve this issue, please upgrade to latest 2.X.X. fix. The ...
CPM may fail to add more than one Aurora cluster to a policy using “cpm backup” tags
If you have multiple Aurora clusters and you are using the “cpm backup” tags to add your resources to CPM policies – CPM may add only a single Aurora cluster to your policies. In order to resolve this problem, please upgrade to latest 2.X.X patch ...
CPM Server may display error "Problem Occurred (500)" during initial configuration
CPM Server may display error "Problem Occurred (500)" during initial configuration: A Problem Occurred(500) Sorry, but there was an error while processing your request The most likely reason for this is lack of some of the permissions required for ...
DR and backup may fail due to "RequestExpired" errors
If a Cross-region, Cross-account DR fails, please download the logs and search for the following error: ERROR: start_copy_region(.\cpmserver\cpm\dr_volume.py:<Line Number>) Volume DR copy snapshot failed for region <Region Name>, snapshot ...
DR process may time out due to exceeding the snapshot limit per region
In case Backup finished successfully but DR failed after 24 hours due to timeout as seen in the “Backup Log” example below: Error - DR process failed: Timed out after 24 hours Please download logs and search the CPM logs for: RDS DR We reached the ...
“cpm backup” tag values may not be updated for instances already included in a policy
When updating “cpm backup” tags on an instance which is already added to a policy, CPM may not add the tag to this policy during “Scan Tag” action. If you update the “cpm backup” tag of an instance that already exists in a policy but the change isn’t ...
Backups may fail without being retried
If a Backup policy has failed and the following conditions are met: Number of retries defined in the policy is not zero. Start time of the backup is the original scheduled start time. Retry Interval (“Wait between Retries (in minutes)”) has passed ...
CPM Agent may fail to connect to the CPM Server with errors 10060 or 10061
CPM Agent may fail to connect to the CPM Server with errors 10060 or 10061: failed calling url https://cpmserver/agentapi/XYZ/. Exception: <urlopen error [Errno 10060] A connection attempt failed because the connected party did not properly respond ...
CPM stops working with the "Lost connection to license server" error
Background: When running CPM "Free Trial & BYOL Edition", It needs to connect to our license server and you may experience the below error: The system identified issues and will stop backing up. This could be due to a downgrade of an existing server ...
VSS pre-snapshot operations on an Agent may fail almost immediately with "vss script crashed or failed. return code: 1"
VSS pre-snapshot operations on an Agent may fail almost immediately with "vss script crashed or failed. return code: 1" listed in the Agent log: ERROR: start_vss_quiesce(remote_agent.pyo:909) policy ihpa_daily, vss failed (reason: vss script ...
Browser fails to load CPM GUI with the "Bad Request" error
Browser may fail to load CPM GUI with the "Bad Request" error, with reason listed as "You're speaking plain HTTP to an SSL-enabled server port": The reason is that you are trying to access port 443 over HTTP instead of HTTPS, like this: ...
Next page