Issue

How to configure Elasticsearch on ha mode

Environment

• CloudBees Jenkins Operations Center
• CloudBees Analytics
• Remote Elasticsearch
• ha-proxy

Resolution

Highly-available Elastic Search Configuration based on haproxy

The image below represents the architecture of configuring Elasticsearh on ha mode for an Analytics environment.

On CJOC side under Configure Analytics you are asked to configure a multi-URL instance, however it actually does not create a cluster (tracking this misunderstanding internally in CJP-2554). In the latest CJOC versions such multi-URL specification does nothing, because Elasticsearch has its own clustering features. The cluster setup process is described in the Elasticsearch 1.7 installation guide.

On CJOC side you must always use the URL of the ha-proxy being the ha-proxy the one in charge of forwarding the data to the right instance/s.

Per our documentation it is critical that there is an odd number of nodes in your elastic search cluster due the split brain risk if a cluster gets network issues (especially on 2-node cluster). Otherwise cluster corruption is likely in the event of a failover

The way of configuring HA should be done through an ha-proxy

ha-proxy configuration with non security enabled on ElasticSearch

Below ha-proxy configuration of the architecture presented on this KB article with non authentication method enabled on ElasticSearch.

#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global
    maxconn     4000
    log 127.0.0.1 local0
    log 127.0.0.1 local1 notice
    daemon
defaults
  log     global
  mode    http
  retries 3
  timeout client 50s
  timeout connect 5s
  timeout server
  timeout http-keep-alive 10s

  listen elasticsearch-TCP-9200
      bind localhost:9200
      mode tcp
      option tcpka
      option tcplog
      option redispatch
      option httpchk GET /
      timeout server 15m
      balance           roundrobin
      server            elasticsearch-1 elasticsearch-1.beehive.com:9200 check
      server            elasticsearch-2 elasticsearch-2.beehive.com:9200 check
      server            elasticsearch-3 elasticsearch-3.beehive.com:9200 check

# monitor port
listen status
    bind 0.0.0.0:8088
    stats enable
    stats uri /

ha-proxy configuration with HTTP Basic authentication enabled on ElasticSearch

Below ha-proxy configuration of the architecture presented on this KB article with HTTP Basic Authentication enabled on ElasticSearch.

#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global
    maxconn     4000
    log 127.0.0.1 local0
    log 127.0.0.1 local1 notice
    daemon
defaults
  log     global
  mode    http
  retries 3
  timeout client 50s
  timeout connect 5s
  timeout server
  timeout http-keep-alive 10s

  listen elasticsearch-TCP-9200
      bind localhost:9200
      mode tcp
      option tcpka
      option tcplog
      option redispatch
      option httpchk GET / HTTP/1.0\r\nAuthorization:\ Basic\ <MY_PASSWORD>
      timeout server 15m
      balance           roundrobin
      server            elasticsearch-1 elasticsearch-1.beehive.com:9200 check
      server            elasticsearch-2 elasticsearch-2.beehive.com:9200 check
      server            elasticsearch-3 elasticsearch-3.beehive.com:9200 check

# monitor port
listen status
    bind 0.0.0.0:8088
    stats enable
    stats uri /

To generate <MY_PASSWORD> you can run:

echo -n "user:pass" | base64