Issue

• Elasticsearch was previously disabled

• Re-enable Elasticsearch in CJE

Environment

• CloudBees Jenkins Enterprise >= 1.11.7

Resolution

There are two different ways you can run CloudBees Analytics:

• [RECOMMENDED] Using Elasticsearch Reporter, which will allow you to use Elasticsearch 5.x/6.x, although you will need to take care of the Elasticsearch deployments and maintenance.
• Re-enable Elasticsearch within the product, but Elasticsearch 1.7 will be used instead. The advantage of this approach is that CJE takes care of the Elasticsearch deployments through Mesos/Marathon.

Elasticsearch Reporter

This is documented in the CJE official documentation.

Re-enable Elasticsearch within the product

Step 1 - Create dedicated Elasticsearch workers

Ensure that there are at least three dedicated workers created for running the Elasticsearch cluster. You can use the command cje prepare worker-add to create the dedicated Elasticsearch workers.

Step 2 - Enable CloudBees Jenkins Analytics plugins (If disabled)

Execute the following Groovy script on the Operations Center through Manage Jenkins -> Script Console to enable CloudBees Jenkins Analytics.

def jenkins = Jenkins.getInstance();
def plugins = jenkins.getPluginManager().getPlugins().each{ plugin ->
    if (!plugin.isEnabled()) {
    	if (plugin.getShortName().startsWith("operations-center-analytics") ||
            plugin.getShortName() == "pse-config-info" ||
            plugin.getShortName() == "pse-analytics-dashboards"
           ) {
          plugin.enable()
          println "Enabling the plugin " + plugin.getShortName()
        }
    }
}

After this, a restart of Operations Center is needed. Operations Center can be restarted by either, using the CJE CLI with the command dna stop cjoc / dna init cjoc, or through the Operations Center UI itself adding /restart at the end of the Operations Center URL.

This action will enable all the Analytics plugins in Operations Center.

You will also need to run the same script on all the existing Managed Masters connected to the Operations Center. You can do this either one by one or via a Cluster Operation on the Operations Center.

Step 3 - Enable CloudBees Jenkins Analytics

Go to Manage Jenkins -> Configure Analytics and ensure that the correct Elasticsearch endpoint is configured. The remote URL should be something like https://<DOMAIN>/elasticsearch/

Step 4 - Re-enable Elasticsearch, Topbeat, and Logstash in the CJE CLI

Prepare the elasticsearch-disable operation

cje prepare elasticsearch-disable

Edit the elasticsearch-disable.config and set disable to no

[elasticsearch]
disable = no

apply the operation

cje apply

When this operation is applied, it will permanently enable Elasticsearch, Topbeat, and Logstash.

• New controllers/workers will have these services automatically installed and started.
• For existing workers/controllers, Topbeat and Logstash must be restarted (and may need to be installed for recent workers).

Step 5 - Manually start topbeat and logstash in all the controllers and workers

The following commands can be used to restarted Logstash and Topbeat on workers/controllers:

dna run restart-logstash worker-*
dna run restart-logstash controller-*
dna run restart-topbeat worker-*
dna run restart-topbeat controller-*

HOWEVER, this may fail for some workers/controllers depending on when they have been initialized for the last time (i.e. were added, restarted, or initialized using cje or dna commands).

• If the worker/controller have been reinitialized after Elasticsearch was disabled, then the Logstash and Topbeat services are not installed nor configured.
• If the worker/controller have been reinitialized before Elasticsearch was disabled, then the Logstash and Topbeat services are installed and configured. Though the configuration might be obsolete in regards to the version of the CJE project - if for example the cluster has been upgraded since Elasticsearch was disabled.

To reinitialize Logstash and Topbeat on all workers / controllers, create a script called start-topbeat-logstash.sh with the following content:

#!/bin/bash -eu

set -o pipefail

{% sync "config/" => "dna-config/" %}
find ~/dna-config -type f -exec chmod go= {} +
{% render "config/logstash.conf" => "~/dna-config/logstash.conf" %}
{% render "config/logstash-redhat.conf" => "~/dna-config/logstash-redhat.conf" %}
{% render "config/topbeat.yml" => "~/dna-config/topbeat.yml" %}
{% render "logstash-topbeat-check" => "~/dna-config/logstash-topbeat-check" %}

{% include "jce-system-support" %}

config_home=~/dna-config

echo "Reinitializaing logstash service on {{server.id}}"
jce-logstash-init

echo "Reinitializaing topbeat service on {{server.id}}"
jce-topbeat-init

echo "Restarting logstash service on {{server.id}}"
jce-logstash-stop
jce-logstash-start

echo "Restarting topbeat service on {{server.id}}"
jce-topbeat-stop
jce-topbeat-start

Make this script executable with:

chmod a+x start-topbeat-logstash.sh

Then run it on all workers and controllers:

dna run $FULL_PATH_CJE_PROJECT/start-topbeat-logstash.sh worker-* controller-*

(Note: another way to re-enable and properly reconfigure those services is to re-initialize the workers / controllers. For AWS, use cje prepare worker-restart for workers and cje prepare controller-restart for controllers. For Anywhere, follow Recycling CJE 1 Anywhere servers)

Step 6 - Start Elasticsearch

dna start elasticsearch

Step 6 - Re-init Operations Centers

dna stop cjoc
dna start cjoc