Re-Enable Elasticsearch in CJE

Issue

  • Elasticsearch was previously disabled
  • Re-enable Elasticsearch in CJE

Environment

Resolution

There are two different ways you can run CloudBees Analytics:

  • [RECOMMENDED] Using Elasticsearch Reporter, which will allow you to use Elasticsearch 5.x/6.x, although you will need to take care of the ES deployments and maintenance.
  • Re-enable Elasticsearch within the product, but Elasticsearch 1.7 will be used instead. The advantage of this approach is that CJE takes care of the Elasticsearch deployments through Mesos/Marathon.

Elasticsearch Reporter

This is documented in the CJE official documentation.

Re-enable Elasticsearch within the product

Step 1 - Create dedicated Elasticsearch workers

Ensure that there are at least three dedicated workers created for running the Elasticsearch cluster. You can use the command cje prepare worker-add to create the dedicated Elasticseach workers.

Step 2 - Enable CloudBees Jenkins Analytics plugins (If disabled)

Execute the following Groovy script in CloudBees Jenkins Operations Center through Operation Center -> Manage Jenkins -> Script Console to enable CloudBees Jenkins Analytics.

def jenkins = Jenkins.getInstance();
def plugins = jenkins.getPluginManager().getPlugins().each{ plugin ->
    if (!plugin.isEnabled()) {
    	if (plugin.getShortName().startsWith("operations-center-analytics") ||
            plugin.getShortName() == "pse-config-info" ||
            plugin.getShortName() == "pse-analytics-dashboards"
           ) {
          plugin.enable()
          println "Enabling the plugin " + plugin.getShortName()
        }
    }
}

After this, a restart of Operations Center is needed. Operations Center can be restarted by either, using the CJE CLI with the command dna stop cjoc / dna init cjoc, or through the Operations Center UI itself adding /restart at the end of the Operations Center URL.

This action will enable all the Analytics plugins in Operations Center and they should be enabling the Elasticsearch reporter on all the Managed Masters connected with Operations Center.

Step 3 - Enable CloudBees Jenkins Analytics

Go to Manage Jenkins -> Configure Analytics and ensure that the correct elasticsearch endpoint is configured. The remote URL should be something like https://<DOMAIN>/elasticsearch/

Step 4 - Re-enable Elasticsearch, Topbeat and Logstash in the CJE CLI

With the version 1.11.7 (and later) of CloudBees Jenkins Enterprise, you can permanently enable the Elasticsearch Reporter stack composed of Elasticsearch, Topbeat, and Logstash.

Prepare the elasticsearch-disable operation

cje prepare elasticsearch-disable

Edit the elasticsearch-disable.config and set disable to no

[elasticsearch]
disable = no

apply the operation

cje apply

Those changes will permanently enable Elasticsearch, Topbeat, and Logstash.

  • New controllers/workers will have these services automatically installed and started.
  • For existing workers/controllers, Topbeat and Logstash must be restarted (and may need to be installed for recent workers).

Step 5 - Manually start topbeat and logstash in all the controllers and workers

The following commands can be used to restarted Logstash and Topbeat on workers/controllers:

dna run restart-logstash worker-*
dna run restart-logstash controller-*
dna run restart-topbeat worker-*
dna run restart-topbeat controller-*

HOWEVER, this may fails for some workers/controllers depending on when they have been initialized for the last time (i.e. were added, restarted, or initialized using cje or dna commands).

  • If the worker/controller have been reinitialized after Elasticsearch was disabled, then the Logstash and Topbeat services are not installed nor configured.
  • If the worker/controller have been reinitialized before Elasticsearch was disabled, then the Logstash and Topbeat services are installed and configured. Though the configuration might be obsolete in regards to the version of the CJE project - if for example the cluster has been upgraded since Elasticsearch was disabled.

To reinitialize Logstash and Topbeat on all workers / controllers, create a script called start-topbeat-logstash.sh with the following content:

#!/bin/bash -eu

set -o pipefail

{% sync "config/" => "dna-config/" %}
find ~/dna-config -type f -exec chmod go= {} +
{% render "config/logstash.conf" => "~/dna-config/logstash.conf" %}
{% render "config/logstash-redhat.conf" => "~/dna-config/logstash-redhat.conf" %}
{% render "config/topbeat.yml" => "~/dna-config/topbeat.yml" %}
{% render "logstash-topbeat-check" => "~/dna-config/logstash-topbeat-check" %}

{% include "jce-system-support" %}

config_home=~/dna-config

echo "Reinitializaing logstash service on {{server.id}}"
jce-logstash-init

echo "Reinitializaing topbeat service on {{server.id}}"
jce-topbeat-init

echo "Restarting logstash service on {{server.id}}"
jce-logstash-stop
jce-logstash-start

echo "Restarting topbeat service on {{server.id}}"
jce-topbeat-stop
jce-topbeat-start

Make this script executable with:

chmod a+x start-topbeat-logstash.sh

Then run it on all workers and controllers:

dna run $FULL_PATH_CJE_PROJECT/start-topbeat-logstash.sh worker-* controller-*

(Note: another way to re-enable and properly reconfigure those services is to re-initialize the workers / controllers. For AWS, use cje prepare worker-restart for workers and cje prepare controller-restart for controllers. For Anywhere, follow Recycling CJE 1 Anywhere servers)

Step 6 - Start elasticsearch

dna start elasticsearch

Step 6 - Re-init CJOC

dna stop cjoc
dna start cjoc
Have more questions? Submit a request

2 Comments

  • 0
    Avatar
    Shinta Smith

    1) Step 4. Re-Enable

    > Those changes will permanently enable Elasticsearch, Topbeat, and Logstash. New controllers/workers will not have these services anymore. The services are also stopped on the current workers/controllers.

    Do you mean "New controllers/workers WILL have these services"?

    2) In Step 5. Manually enable

    #!/bin/sh
    sudo update-rc.d topbeat enable
    if [ -f /etc/init/tiger-logstash.conf ]; then
      sudo service tiger-logstash stop || echo
    else
      sudo service cloudbees-logstash stop || echo
    fi
    sudo service topbeat stop || echo

    Shouldn't the script run a "start" instead of a "stop"?

  • 0
    Avatar
    Allan Burdajewicz

    Shinta,

    You are right in 1) and 2). I have created an internal issue to fix the article.

    In the meantime, feel free to open a support request if you need help with this.

    Regards,

Please sign in to leave a comment.