How to set-up a Multi AZ environment in an existing CloudBees Core v1 environment

Issue

A Multi AZ environment is requested after CloudBees Core v1 is already deployed in AWS.

Environment

Resolution

Backup and shutdown operations

1) Take a backup of your CJE project
2) Take a backup of both the CJE-OC and Masters using a CJE-OC Cluster Operations job as a precaution. Use CloudBees Backup plugin.
3) Stop all Managed Masters from the Operations Center UI to ensure the last status of the instance is saved - no jobs are running.
4) Stop Operations Center from the CJE CLI: dna stop cjoc
5) Destroy the cluster and ensure that you preserve the data

$ cje prepare cluster-destroy
cluster-destroy is staged - review cluster-destroy.config and edit as needed - then run 'cje apply' to perform the operation.
  • To preserve the data ensure that destroy_storage_bucket and destroy_ebs_resources are commented in the cluster-destroy.config file. Only cluster_name should be uncommented in order to delete all the resources except the volumes which contains the data we would like to preserve.
[pse]

## Cluster name
# Uncomment the next line after checking this is really the cluster you want to destroy
# cluster_name = pse-fbelzunc-azs

## Destroy storage bucket
# Uncomment to destroy
# destroy_storage_bucket = yes

## Destroy EBS resources (This option destroys long-term storage of the cluster. It CANNOT be recovered)
# This is done as a best effort. Some resources may not be deletable (pending snapshots, in-use volumes) and their ids
# will be reported.
# Uncomment to destroy
# destroy_ebs_resources = yes

Finally, we cje apply to go ahead with the operation.

6) Create a new cluster (cje init-project) with the same cluster_name and similar configuration as the previous one.

$ cje init-project aws
project [/path/to/multiple-azs-example] is already initialized
$ cje prepare cluster-init
cluster-init is staged - review cluster-init.config and cluster-init.secrets and edit as needed - then run 'cje apply' to perform the operation.

cluster-init.config configuration

  • To configure Multi AZs you just need to specify the availability zones under availability_zone variable.

Note: Most of the times, users are requesting to have the controllers deployed on different availability zones, but keep the workers on the same one. When we start the cluster, the controllers will be deployed on each availability zone each one - in case we configured three controllers instances for three zones. By default, the workers created by the cluster-init will be randomly created on the three availability zones. Once the cluster is created, we can just delete the ones which are in the availability zone we don’t want and create the new ones in the availability zone we want.

## AWS availability zone
#
# The availability zone(s) where CJE will run, separated by comma.
# The number of availability zones must be either 1, or 3 for fault tolerance.
# If not set it will pick a random one.
#
availability_zone = us-east-1a, us-east-1b, us-east-1c

cluster-init.secrets configuration

At very the least, for the cluster-init to be successful the first time we will need to retrieve the CJOC username and password from the CJE project backup. You can find it under $CJE_PROJECT/.dna/secrets. Copy/paste it in cluster-init.secrets.

# CJOC Username (optional - defaults to admin)
#
# The username of the administrative user created during initialization.
#
cjoc_username = USER

# CJOC Password (optional - defaults to a random string)
#
# The password of the administrative user created during initialization.
#
cjoc_password = PASSWORD

Once we apply the operation (‘cje apply’), and it finishes, the cluster should be up and running. Even if the volumes created from the existing cluster were in a different availability zone, the new volumes are created in the same availability zone of the worker where the instance is deployed retrieving the previous information.

7) Have all the workers in the same availability zones

In case you would like to have all the workers in the same availability zone and only have the different controllers on different availability zones, then you should remove the workers which were automatically created in the availability zone we don’t want. You must have at least one worker of type master already created on the availability zone you would like to use.

Use cje prepare worker-remove and cje prepare worker-add to perform the previous actions. Notice the the operations cje prepare worker-add has an availability_zone variable where you can specify the availability zone you want to use.

Have more questions? Submit a request

0 Comments

Please sign in to leave a comment.