Issue
-
Elasticsearch was previously disabled
-
Re-enable Elasticsearch in CJE
Environment
- CloudBees Jenkins Enterprise >= 1.11.7
Resolution
There are two different ways you can run CloudBees Analytics:
- [RECOMMENDED] Using Elasticsearch Reporter, which will allow you to use Elasticsearch 5.x/6.x, although you will need to take care of the Elasticsearch deployments and maintenance.
- Re-enable Elasticsearch within the product, but Elasticsearch 1.7 will be used instead. The advantage of this approach is that CJE takes care of the Elasticsearch deployments through Mesos/Marathon.
Elasticsearch Reporter
This is documented in the CJE official documentation.
Re-enable Elasticsearch within the product
Step 1 - Create dedicated Elasticsearch workers
Ensure that there are at least three dedicated workers created for running the Elasticsearch cluster. You can use the command cje prepare worker-add
to create the dedicated Elasticsearch workers.
Step 2 - Enable CloudBees Jenkins Analytics plugins (If disabled)
Execute the following Groovy script on the Operations Center through Manage Jenkins -> Script Console to enable CloudBees Jenkins Analytics.
def jenkins = Jenkins.getInstance();
def plugins = jenkins.getPluginManager().getPlugins().each{ plugin ->
if (!plugin.isEnabled()) {
if (plugin.getShortName().startsWith("operations-center-analytics") ||
plugin.getShortName() == "pse-config-info" ||
plugin.getShortName() == "pse-analytics-dashboards"
) {
plugin.enable()
println "Enabling the plugin " + plugin.getShortName()
}
}
}
After this, a restart of Operations Center is needed. Operations Center can be restarted by either, using the CJE CLI with the command dna stop cjoc
/ dna init cjoc
, or through the Operations Center UI itself adding /restart
at the end of the Operations Center URL.
This action will enable all the Analytics plugins in Operations Center.
You will also need to run the same script on all the existing Managed Masters connected to the Operations Center. You can do this either one by one or via a Cluster Operation on the Operations Center.
Step 3 - Enable CloudBees Jenkins Analytics
Go to Manage Jenkins -> Configure Analytics and ensure that the correct Elasticsearch endpoint is configured. The remote URL should be something like https://<DOMAIN>/elasticsearch/
Step 4 - Re-enable Elasticsearch, Topbeat, and Logstash in the CJE CLI
Prepare the elasticsearch-disable
operation
cje prepare elasticsearch-disable
Edit the elasticsearch-disable.config
and set disable to no
[elasticsearch]
disable = no
apply the operation
cje apply
When this operation is applied, it will permanently enable Elasticsearch, Topbeat, and Logstash.
- New controllers/workers will have these services automatically installed and started.
- For existing workers/controllers, Topbeat and Logstash must be restarted (and may need to be installed for recent workers).
Step 5 - Manually start topbeat and logstash in all the controllers and workers
The following commands can be used to restarted Logstash and Topbeat on workers/controllers:
dna run restart-logstash worker-*
dna run restart-logstash controller-*
dna run restart-topbeat worker-*
dna run restart-topbeat controller-*
HOWEVER, this may fail for some workers/controllers depending on when they have been initialized for the last time (i.e. were added, restarted, or initialized using cje
or dna
commands).
- If the worker/controller have been reinitialized after Elasticsearch was disabled, then the Logstash and Topbeat services are not installed nor configured.
- If the worker/controller have been reinitialized before Elasticsearch was disabled, then the Logstash and Topbeat services are installed and configured. Though the configuration might be obsolete in regards to the version of the CJE project - if for example the cluster has been upgraded since Elasticsearch was disabled.
To reinitialize Logstash and Topbeat on all workers / controllers, create a script called start-topbeat-logstash.sh
with the following content:
#!/bin/bash -eu
set -o pipefail
{% sync "config/" => "dna-config/" %}
find ~/dna-config -type f -exec chmod go= {} +
{% render "config/logstash.conf" => "~/dna-config/logstash.conf" %}
{% render "config/logstash-redhat.conf" => "~/dna-config/logstash-redhat.conf" %}
{% render "config/topbeat.yml" => "~/dna-config/topbeat.yml" %}
{% render "logstash-topbeat-check" => "~/dna-config/logstash-topbeat-check" %}
{% include "jce-system-support" %}
config_home=~/dna-config
echo "Reinitializaing logstash service on {{server.id}}"
jce-logstash-init
echo "Reinitializaing topbeat service on {{server.id}}"
jce-topbeat-init
echo "Restarting logstash service on {{server.id}}"
jce-logstash-stop
jce-logstash-start
echo "Restarting topbeat service on {{server.id}}"
jce-topbeat-stop
jce-topbeat-start
Make this script executable with:
chmod a+x start-topbeat-logstash.sh
Then run it on all workers and controllers:
dna run $FULL_PATH_CJE_PROJECT/start-topbeat-logstash.sh worker-* controller-*
(Note: another way to re-enable and properly reconfigure those services is to re-initialize the workers / controllers. For AWS, use cje prepare worker-restart
for workers and cje prepare controller-restart
for controllers. For Anywhere, follow Recycling CJE 1 Anywhere servers)
Step 6 - Start Elasticsearch
dna start elasticsearch
Step 6 - Re-init Operations Centers
dna stop cjoc
dna start cjoc
2 Comments