Backups/DR/tag scans may start to fail because of clock issues or connectivity issues.
Problem descriptions:
If you start seeing failures in the logs of AWS API calls saying that "AWS was not able to validate the provided access credentials", this is likely caused by either lack of connectivity with AWS endpoints or time synchronization issues between the CPM instance and AWS.
If you start seeing failures in the logs of AWS API calls saying that "AWS was not able to validate the provided access credentials", this is likely caused by either lack of connectivity with AWS endpoints or time synchronization issues between the CPM instance and AWS.
Critical Error - Failed getting
volumes in region EU (Frankfurt), exception: AWS was not able to
validate the provided access credentials
Critical Error - Failed getting
instances in region Asia Pacific (Seoul), exception: AWS was not able to
validate the provided access credentials
or
ERROR: start_copy_region(.\cpmserver\cpm\dr_volume.py:335) Volume DR copy_snapshot failed from region US East (N. Virginia) to region US West (Oregon). snapshot snap-0123456789012345, policy Daily_Backup (in Backup account). Exception AWS was not able to validate the provided access credentials
or
ERROR: scan_resources(.\cpmserver\cpm\backup_tag.py:643) Failed getting instances in region Asia Pacific (Seoul), exception: AWS was not able to validate the provided access credentials
ERROR: scan_resources(.\cpmserver\cpm\backup_tag.py:683) Failed getting volumes in region Asia Pacific (Seoul), exception: AWS was not able to validate the provided access credentials
ERROR: scan_resources(.\cpmserver\cpm\backup_tag.py:683) Failed getting volumes in region Asia Pacific (Seoul), exception: AWS was not able to validate the provided access credentials
Troubleshooting connectivity:
To test this outside of the CPM application itself, please
connect to the instance using SSH (username: cpmuser and your assigned
private key) and try this command:
See what it returns - there must be a failure with either resolving the URL or connecting to it.
If you don't see HTTP response 200 "OK" like in the screenshot below, there is a problem with either DNS resolution or a proxy refusing connections.
Troubleshooting time synchronization:
If you have verified that CPM Server has the required connectivity, you need to synchronize CPM Server instance's time with AWS
(This happens because more and more regions support only Signature Version 4 to authenticate requests (http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingAWSSDK.html#specify-signature-version), which is time-sensitive)
Solution:
(This happens because more and more regions support only Signature Version 4 to authenticate requests (http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingAWSSDK.html#specify-signature-version), which is time-sensitive)
Solution:
CPM v2.3 and newer has NTP daemon installed (please upgrade if you are using an older version).
In order for NTP to function, you must enable in your firewall full unrestricted access to UDP port 123 in both directions between the CPM instance and Amazon's NTP servers:
server 0.amazon.pool.ntp.org
server 1.amazon.pool.ntp.org
server 2.amazon.pool.ntp.org
server 3.amazon.pool.ntp.org
in version 3.x/4.x CPM uses AWS NTP by default and ntpd
Workaround:
Stop and then start the CPM instance or run "sudo ntpdate 0.amazon.pool.ntp.org" over SSH (using the "cpmuser" login).
Related Articles
Installation or upgrade of CPM may fail with the error "Can't connect to AWS: Wrong credentials or connectivity issue: "AccessKeyID""
Installation or upgrade of CPM may fail with the error "Can't connect to AWS: Wrong credentials or connectivity issue: "AccessKeyID"" After ensuring that the JSON permissions files have been updated, confirm that you have connectivity to the AWS ...File-level restore may fail with the "Could not get CPM instance info" error
File-level restore may fail with the "Could not get CPM instance info" error: The following error is printed in the CPM server log: ERROR: explore_backup_init(.\cpmserver\cpm\filemanager\views.py:317) Could not get CPM instance info ...CPM may fail to set the "cpm_deleted" tag to RDS Snapshots
Symptom: When performing a cross-account DR Copy of RDS snapshots, using a DR account with “Allow Deleting Snapshots” as False, the RDS snapshots that have exceeded the retention policy, will not have the “cpm_deleted” tag. You will also see the ...DR process may time out due to exceeding the snapshot limit per region
In case Backup finished successfully but DR failed after 24 hours due to timeout as seen in the “Backup Log” example below: Error - DR process failed: Timed out after 24 hours Please download logs and search the CPM logs for: RDS DR We reached the ...CPM may fail to add RDS to a policy, if any of the displayed RDS instances have an "inaccessible-encryption-credentials" status
CPM may fail to add RDS to a policy, if any of the displayed RDS instances has a "inaccessible-encryption-credentials" status The "inaccessible-encryption-credentials" status is visible in the EC2 Console, in the CPM GUI it's displayed as "Ensure ...