Copy backup to S3 may fail with the error message Workers could not be launched

Copy backup to S3 may fail with the error message "Workers could not be launched"

During backup copy to S3 a worker creation may fail with the error (backup.log):

- Error - Workers could not be launched

The worker usually fails right after it was started.
There are several possible issues that gives you this error.

1) The was an internal issue at the worker creation and it wasn't started properly.
When a CPM server triggers S3 copy to start it creates worker with the actual data copied from the CPM instance.
It creates a script and put this script into the user-data parameter for AWS.
The content of this script could be shown on the worker instance with curl command:
curl http://169.254.169.254/latest/user-data


It is possible to check the script output in the cloud-init-output.log within cat command:
cat /var/log/cloud-init-output.log

You can check the latest match of the line in this log:
Cloud-init v. 0.7.9 finished at Tue, 19 Feb 2019 18:48:06 +0000. Datasource DataSourceEc2.  Up 17.28 seconds

The worker's userdata script should provide output like:
/var/log/cpm/
/var/log/cpm/n2w_connector_error.log
/var/log/cpm/date.log
/var/log/cpm/n2w_connector.log

The resulting command that shows the output of the script:
cat /var/log/cloud-init-output.log | grep -A 100 -B 14 '/var/log/cpm' -m -1

2) The time zone (and Linux time set) issue on the CPM server side.
At the same time in the cpm_server.log you may see this error: 

Traceback (most recent call last):

  File "./cpmserver/cpm/backup_copy/proxy_utils.py", line 141, in launch_proxy

  File ".\cpmserver\cpm\backup_copy\utils.py", line 292, in get_n2ws_connector_launch_script

  File "./cpmserver/cpm/utils.py", line 89, in get_tz_name

 OSError: [Errno 22] Invalid argument: '/etc/localtime'

Basically, description of the error says that software cannot use the path provided due to permission or access issues.

It could happen after manual changes in the CPM ssh console or installing a new software.

Policy fails just at the very beginning of a worker creation and no CloudTrail events are created in this case.

You can obtain more information by going to the CPM shell over SSH.
You should be logged in as "cpmuser".

Please change it to sudo mode before going any further:

sudo -i

Check the status of the file /etc/localtime:

ls -l  /etc/localtime

It should be similar to the line above:

lrwxrwxrwx 1 root www-data 33 Nov 20 13:14 /etc/localtime -> /usr/share/zoneinfo/GMT


If it's not then move the old file:

mv /etc/localtime /etc/localtime.old


Create a new symbolic link:

ln -s /usr/share/zoneinfo/GMT /etc/localtime

A time zone for your location could be different.

You can use the command below to list all possible zones:
 ls /usr/share/zoneinfo/ -R 

Change the owner group:

chmod :www-data /etc/localtime

And restart the server:

reboot


After that the worker should work fine.