How to extend the size of a worker

Issue

One of the workers is running out resources or is close to the defined resource threshold and you need to extend the disk size.

Environment

Resolution

Depending on the type of worker, you will have to follow different steps

Extending Master workers

1) If you are running Elasticsearch on the master worker and only on that one worker, and want to keep the Elasticsearch data, you will have to perform the next operations. If you are running at least three ElasticSearch workers, as recommended, then no backup is necessary and you can skip this step and jump to 2)

  • Execute the cluster elasticsearch-backup operation to backup all ES data
$ cje prepare elasticsearch-backup
$ cje apply
  • The backup process can take some time. To verify if the backup finished successfully or no, execute the next commands
export ES_PASSWD=$(awk '/elasticsearch_password/ {print $3}' .dna/secrets)
export DOMAIN=$(awk '/domain_name/ {print $3}' .dna/project.config)
export ES_CREDS="admin:$ES_PASSWD"
export ES_URL="https://elasticsearch.$DOMAIN"
curl -f -s -XGET -u $ES_CREDS $ES_URL/_snapshot/tiger-backup/_all?pretty  > snapshots_backup.json

Check the fields state and end_time in the json returned to confirm if the backup finished. Wait until the backup is completed to continue.

2) Create a new worker with the desired configuration.

$ cje prepare worker-add
(edit the file worker-add.config and ensure that you configure the proper values (especially the workload_type to master) )
$ cje apply

3) Remove the old worker.

$ cje prepare worker-remove
(edit the file worker-remove.config setting the worker you want to remove)
$ cje apply

4) If you backed up ElasticSearch, then wait for any CJOC and/or Masters that were running on the removed worker to get rescheduled. Once you have verified that they are back up and running follow these steps:

  • Restart Elasticsearch
dna stop elasticsearch
dna start elasticsearch
  • Restore the Elasticsearch data
$ cje prepare elasticsearch-restore
$ cje apply

Extending Build workers

1) Create a new worker with the desired configuration.

$ cje prepare worker-add
(edit the file worker-add.config and ensure that you configure the proper values (especially the workload_type to build) )
$ cje apply

2) Remove the old worker

$ cje prepare worker-remove
(edit the file worker-remove.config setting the worker you want to remove)
$ cje apply

Extending Elasticsearch workers

1) Create a new worker with the desired configuration.

$ cje prepare worker-add
(edit the file worker-add.config and ensure that you configure the proper values (especially the workload_type to elasticsearch) )
$ cje apply

2) If you are only running one ElasticSearch worker then you must execute the cluster elasticsearch-backup operation to backup all ES data. If you are running at least three ElasticSearch workers, as recommended, then no backup is necessary.

$ cje prepare elasticsearch-backup
$ cje apply

3) The backup process can take some time. To verify if the backup finished successfully or no, execute the next commands

export ES_PASSWD=$(awk '/elasticsearch_password/ {print $3}' .dna/secrets)
export DOMAIN=$(awk '/domain_name/ {print $3}' .dna/project.config)
export ES_CREDS="admin:$ES_PASSWD"
export ES_URL="https://elasticsearch.$DOMAIN"
curl -f -s -XGET -u $ES_CREDS $ES_URL/_snapshot/tiger-backup/_all?pretty  > snapshots_backup.json

Check the fields state and end_time in the json returned to confirm if the backup finished. Wait until the backup is completed to continue.

6) Remove the old worker

$ cje prepare worker-remove
(edit the file worker-remove.config setting the worker you want to remove)
$ cje apply

4) Ithen restart it:

dna stop elasticsearch
dna start elasticsearch

5) If you backed up ElasticSearch, Restore the Elasticsearch data

$ cje prepare elasticsearch-restore
$ cje apply

Wait until the process is finished.

Have more questions? Submit a request

0 Comments

Please sign in to leave a comment.