During backup copy to S3 a worker
creation may fail with the error (backup.log):
- Error - Workers could not be launched
The worker usually fails right after it was started.
There are several possible issues that gives you this error.
1) The was an internal issue at the worker creation and it wasn't started properly.
When a CPM server triggers S3 copy to start it creates worker with the actual data copied from the CPM instance.
It creates a script and put this script into the user-data parameter for AWS.
The content of this script could be shown on the worker instance with curl command:
curl http://169.254.169.254/latest/user-data
It is possible to check the script output in the cloud-init-output.log within cat command:
cat /var/log/cloud-init-output.log
You can check the latest match of the line in this log:
Cloud-init v. 0.7.9 finished at Tue, 19 Feb 2019 18:48:06 +0000. Datasource DataSourceEc2. Up 17.28 seconds
The worker's userdata script should provide output like:
/var/log/cpm/
/var/log/cpm/n2w_connector_error.log
/var/log/cpm/date.log
/var/log/cpm/n2w_connector.log
The resulting command that shows the output of the script:
cat /var/log/cloud-init-output.log | grep -A 100 -B 14 '/var/log/cpm' -m -1
2) The time zone (and Linux time set) issue on the CPM server side.
At the same time in the cpm_server.log you may see this error:
Traceback (most recent call last):
File
"./cpmserver/cpm/backup_copy/proxy_utils.py", line 141, in
launch_proxy
File
".\cpmserver\cpm\backup_copy\utils.py", line 292, in
get_n2ws_connector_launch_script
File "./cpmserver/cpm/utils.py",
line 89, in get_tz_name
OSError: [Errno 22] Invalid argument:
'/etc/localtime'
Basically, description of the error says that software cannot use the path provided due to permission or access issues.
It could happen after manual changes in the CPM ssh console or installing a new software.
Policy fails just at
the very beginning of a worker creation and no CloudTrail events are created in this case.
You can obtain more information by going to the CPM shell over SSH.
You should be logged in as "cpmuser".
Please change it to
sudo mode before going any further:
sudo -i
Check the status of the file
/etc/localtime:
ls -l
/etc/localtime
It should be similar
to the line above:
lrwxrwxrwx 1 root www-data 33 Nov 20 13:14
/etc/localtime -> /usr/share/zoneinfo/GMT
If it's not then
move the old file:
mv /etc/localtime /etc/localtime.old
Create a new
symbolic link:
ln -s
/usr/share/zoneinfo/GMT /etc/localtime
A time zone for your location could be different.
You can use the command below to list all possible zones:
ls /usr/share/zoneinfo/ -R
Change the owner
group:
chmod
:www-data /etc/localtime
And restart the
server:
reboot
After that the
worker should work fine.