Backups/DR/tag scans may start to fail because of clock issues or connectivity issues.

Backups/DR/tag scans may start to fail because of clock issues or connectivity issues.

Problem descriptions:

If you start seeing failures in the logs of AWS API calls saying that "AWS was not able to validate the provided access credentials", this is likely caused by either lack of connectivity with AWS endpoints or time synchronization issues between the CPM instance and AWS.

Critical Error - Failed getting volumes in region EU (Frankfurt), exception: AWS was not able to validate the provided access credentials

Critical Error - Failed getting instances in region Asia Pacific (Seoul), exception: AWS was not able to validate the provided access credentials
or
ERROR:  start_copy_region(.\cpmserver\cpm\dr_volume.py:335)  Volume DR copy_snapshot failed from region US East (N. Virginia) to region US West (Oregon). snapshot snap-0123456789012345, policy Daily_Backup (in Backup account). Exception AWS was not able to validate the provided access credentials
or
ERROR:  scan_resources(.\cpmserver\cpm\backup_tag.py:643)  Failed getting instances in region Asia Pacific (Seoul), exception: AWS was not able to validate the provided access credentials
ERROR:  scan_resources(.\cpmserver\cpm\backup_tag.py:683)  Failed getting volumes in region Asia Pacific (Seoul), exception: AWS was not able to validate the provided access credentials

Troubleshooting connectivity:

To test this outside of the CPM application itself, please connect to the instance using SSH (username: cpmuser and your assigned private key) and try this command:

See what it returns - there must be a failure with either resolving the URL or connecting to it.
If you don't see HTTP response 200 "OK" like in the screenshot below,  there is a problem with either DNS resolution or a proxy refusing connections.




Troubleshooting time synchronization:
If you have verified that CPM Server has the required connectivity, you need to synchronize CPM Server instance's time with AWS
(This happens because more and more  regions support only Signature Version 4 to authenticate requests (http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingAWSSDK.html#specify-signature-version), which is time-sensitive)

Solution:
CPM v2.3 and newer has NTP daemon installed (please upgrade if you are using an older version).

In order for NTP to function, you must enable in your firewall full unrestricted access to UDP port 123 in both directions between the CPM instance and Amazon's NTP servers:
server 0.amazon.pool.ntp.org
server 1.amazon.pool.ntp.org
server 2.amazon.pool.ntp.org
server 3.amazon.pool.ntp.org
in version 3.x/4.x CPM uses AWS NTP by default and ntpd

Workaround:
Stop and then start the CPM instance.