Issue: S3 cleanup fails and reports error from worker - No progress was reported - terminating operation
This occurs in version 4.2.2 and the error occurs when the worker reaches its 1 hour timeout while attempting to cleanup large snapshots.
To fix this you need increase the timeout on the workers that gets launched, by editing the cpmserver.cfg file.
Please see detailed steps on how to increase the worker timeout below. Its best to edit this file and restart apache when no polcies are running.
Solution: Increase the timeout setting for the workers using the steps below.
1. Connect to the N2WS server, user is cpmuser and your selected KeyPair.
2. Edit or create the file /cpmdata/conf/cpmserver.cfg
3. Add or edit the following section in cpmserver.cfg
[backup_copy]
operation_record_no_progress_time_out_in_seconds=7200
4. Restart the apache server:
sudo service apache2 restart
In addition to the above timeout setting, you can also change or add the following configuration options under the [backup_copy] section of cmpserver.cfg to increase the resources that the cleanup workers have, in the case of very large cleanups.
In the example section below you can add worker instance sizes - you can add any instance size you want to determine small, medium and large instance sizes for different size cleanup jobs.
[backup_copy]
worker_types_for_small_cleanup_worker=["t3.small", "t2.small"]
worker_types_for_medium_cleanup_worker=["t3.small", "t2.small"]
worker_types_for_large_cleanup_worker=["t3.small", "t2.small"]
The below line will increase the number of workers used per policy, the default is 2 but this can be increased if needed to speed up the job.
max_num_of_cleanup_workers=2