Issue

How we can list servers?
How we can init a servers?
How we can stop a servers?
How we can start a servers?
How we can enter on a worker/controller?
How we can copy files to/from a worker/controller?
How we can run scripts on worker/controller?
How we can run tests against a CJE environment?
How we can show CJE cluster status?

Environment

CloudBees Jenkins Enterprise - AWS/OpenStack/Anywhere

Resolution

Check CJE Cluster status

These two commands allow to obtains a basic test of CJE cluster proper behaviour.

cje check

cje status

Servers are cluster objects that provide a service(s) to the cluster, they can be a service , a group of services, or a Docker container running on a Controllers/Workers. They are the infrastructure basics of a CJE Cluster.

list servers

We can list the servers running on our CJE cluster with dna servers the output will be something like this it depends on the platform where we deployed CJE.

Servers

controller-* - Services running on a Controller DANGEROUS use instead Management Operations
worker-* - Services running on a Worker DANGEROUS use instead Management Operations
castle - Service to provide volumes to CJE-OC and CJE-MM
palace - Service to provision Agents
cjoc - Operations Center server (CJE-OC)
elasticsearch - Elasticsearch Service DANGEROUS see Backup before ES operations
tiger - This server group most of the services of the CJE CLuster DANGEROUS
vpc - Virtual Private Cloud in AWS template DANGEROUS
elb* - Load balancer servers
route53 - Domain Name services in AWS template DANGEROUS
security-groups - Security Groups in AWS template DANGEROUS
storage-bucket - 3S Bucket in AWS template DANGEROUS
We can init a server, this command will initialize the server from scratch

dna init <server>

stop a servers, it will stop the server and related servers in some cases, if we stop a controller we will stop all servers running in that controller, if we stop cjoc we only will stop cjoc.

dna stop <server>

start a servers it will start the server and related servers in some cases, if we start a controller we will start all servers running in that controller, if we start cjoc we only will start cjoc.

dna start <server>

Enter on a worker/controller

You can enter on a worker/controller by SSH with dna in a easy way only using the name

dna connect <worker/controller>

Copy files to/from a worker/controller

You can copy files from/to a worker/controller

dna copy <localFile> <remoteFile> <workers/controllers>

dna copy -rs <remoteFile> <localFile> <workers/controllers>

Run scripts on worker/controller

It is possible to execute scripts in a worker/controller or in all workers and controllers at the same time, some of those scripts are part of CJE but you can also run your own scripts.

dna <script> <workers/controllers>

dna <full_path_to_script> <workers/controllers>

Obtain the worker where a Docker container (tenant) is running - cje run find-worker-for cjoc
SSH into a Docker container (tenant) - cje run ssh-into-tenant controller-1
List workers in the CJE cluster - cje run list-workers
List resources used in the CJE cluster - cje run list-resources
List applications running in the CJE cluster and in which worker they are - cje run list-applications
Get marathon username and password cje run echo-secrets marathon_username and cje run echo-secrets marathon_password
Get mesos username and password cje run echo-secrets mesos_web_ui_username and cje run echo-secrets mesos_web_ui_password
Get cjoc username and password cje run echo-secrets cjoc_username and cje run echo-secrets cjoc_password
Get elasticsearch username and password cje run echo-secrets elasticsearch_username and cje run echo-secrets elasticsearch_password
Update project config after change the .dna/project.config - cje upgrade --config-only --force
Update templates and scripts - cje upgrade-project --apply-template
Recreate the certs docker container in a worker/controller - dna run certificates-update
Restart Mesos service in a worker - dna run restart-mesos-slave worker-X
Restart Logstash service in a worker - dna run restart-logstash worker-X
Restart Topbeat service in a worker - dna run restart-topbeat worker-X
Tool to grab the information need to diagnose a performance issue in a CJE-MM - cje run support-performance

 usage : cje run support-performance <TENANT> [DURATION] [FREQUENCY]

Manage Mesos by REST API calls - cje run support-mesos

  usage : cje run support-mesos <command> [parameters]

  ms-slaves : get info about Mesos slaves (workers)
  ms-state : Information about state of master (running/killed)
  ms-state-summary : Summary of agents, tasks, and registered frameworks in cluster.
  ms-tasks : Lists tasks from all active frameworks.
  ms-browse-mesos-stderr <slave_id> <framework_id> <mesos_id> : Returns a file listing for a directory.
  ms-download-mesos-stderr <slave_id> <framework_id> <mesos_id> : Returns the raw file contents for a given path.
  ms-read-mesos-stderr <slave_id> <framework_id> <mesos_id> : Reads data from a file.

Manage Marathon by REST API calls - cje run support-marathon

  usage : cje run support-marathon <command> [parameters]

  mt-ping : make a ping to the marathon service
  mt-info : Get info about the Marathon Instance
  mt-metrics : Get metrics data from this Marathon instance
  mt-get-jce-info : Get the application with id jce. The response includes some status information besides the current configuration of the app. You can specify optional embed arguments, to get more embedded information.
  mt-get-masters-info : Get the application with id masters. The response includes some status information besides the current configuration of the app. You can specify optional embed arguments, to get more embedded information.
  mt-running-apps : Get the list of running applications. Several filters can be applied via the following query parameters.
  mt-get-app-info <palace|castle|cjoc|elasticsearch> : Get the application with id jce/<palace|castle|cjoc|elasticsearch>. The response includes some status information besides the current configuration of the app. You can specify optional embed arguments, to get more embedded information.
  mt-running-tasks : List all running tasks.
  mt-search-dupliate-taks <tasks_json_file> : check for duplicate taks in the ouput of mt-running-tasks, required  installed.

Tools to manage/grab logs in workers/controllers - cje run support-log

  usage : cje run support-log <command> [parameters]
  get-logs [WORKERS|CONTROLLERS|worker-*] : get logs from worker/s/controller/s
  get-tenant-logs [TENANT|MESOS_TASK_ID] [WORKERS|worker-*] : get the logs of a tenant by searching in a worker or in all workers.
  tail-tenant-logs [MESOS_TASK_ID] [WORKERS|worker-*] : tail the stderr of a tenant by searching in a worker or in all workers.
  cleanup-support-files [WORKERS|CONTROLLERS|worker-* controller-*] : cleanup temporal files from support bundles and scripts.
  truncate-syslog [WORKERS|CONTROLLERS] : truncate the /var/log/syslog or /var/log/messages files, /var/log/syslog*.gz or /var/log/messages*.gz remove the  to free space.

Tools to check router system - cje run support-router

  usage : cje run support-router <command> [parameters]
  check-router : run every router check
  check-router-dns : try to access to mesos and marathon through the configured DNS
  check-router-lb : try to access to mesos and marathon through the Load Balancer
  check-router-controllers-external : try to access to mesos and marathon through controllers
  check-router-controllers-local : try to access to mesos and marathon locally through router (80) in controllers
  check-listen-ports : list the LISTEN ports on controllers
  check-marathon-controller : try to access to marathon locally (8080) in controllers
  check-mesos-master : try to access to mesos locally (5050) in controllers
  check-docker-router : try to access to mesos and marathon runing in controllers from inside of the router docker container
  check-config-router : test the nginx configuration and write the routes files to the standard output

Tools to manage Elasticsearch - cje run support-elasticsearch

  usage : cje run support-elasticsearch <command> [parameters]

  es-rest <GET|POST|PUT|DELETE> <ES_API> : make a REST call to http://ES_URL/ES_API by using the method pased (<GET|POST|PUT|DELETE>)
  es-rest-data <POST|PUT> <ES_API> <DATA> : make a REST call to http://ES_URL/ES_API by using the method pased (<POST|PUT>) and send the data (@filename or "{some_json_content}")
  es-get-status <full_path_workdir> : get the staus of ES cluster and write the ouput in the <full_path_workdir> folder
  es-assign-shards <full_path_workdir> : fix the unassigned shards and write the ouput in the <full_path_workdir> folder
  es-repo-create [repo_name] : create a new snapshot repository, the default name is "cloudbees-analytics"
  es-snapshot-take [repo_name] [snapshot_name] : create a new snapshot, the default repository is "cloudbees-analytics", the default name is snapshot%Y%m%d%H%M%s
  es-repo-list : list existing snapshot repositories
  es-snapshot-list <full_path_workdir> [repo_name] : list snapshots in a snapshot repository
  es-snapshot-status [repo_name] :
  es-snapshot-status-brief [repo_name] :
  es-snapshot-stop <repo_name> <snapshot_name> :
  es-snapshot-restore <full_path_workdir> <repo_name> <snapshot_name> <indice,indice> :
  es-status : return the status os the ES cluster
  es-snapshot-status-last [repo_name] : check the status of the [repo_name] repository
  es-snapshot-delete-old [repo_name] [limit] : delete snapshots from a [repo_name] and keep [limit] snapshots of all
  es-tasks-list-pending <full_path_workdir> : liste the pending tasks and write the ouput in the <full_path_workdir> folder
  es-delete-index <index> : delete the <index>
  help : this usage information.

Tested product/plugin versions

CloudBees Jenkins Enterprise 1.11.3

How to use cje command line tools