Issue
How to configure Elasticsearch on ha mode
Environment
- CloudBees Jenkins Operations Center
- CloudBees Analytics
- Remote Elasticsearch
- ha-proxy
Resolution
Highly-available Elastic Search Configuration based on haproxy
The image below represents the architecture of configuring Elasticsearh on ha mode for an Analytics environment.
On CJOC side under Configure Analytics you are asked to configure a multi-URL instance, however it actually does not create a cluster (tracking this misunderstanding internally in CJP-2554). In the latest CJOC versions such multi-URL specification does nothing, because Elasticsearch has its own clustering features. The cluster setup process is described in the Elasticsearch 1.7 installation guide.
On CJOC side you must always use the URL of the ha-proxy being the ha-proxy the one in charge of forwarding the data to the right instance/s.
Per our documentation it is critical that there is an odd number of nodes in your elastic search cluster due the split brain risk if a cluster gets network issues (especially on 2-node cluster). Otherwise cluster corruption is likely in the event of a failover
The way of configuring HA should be done through an ha-proxy
ha-proxy configuration with non security enabled on ElasticSearch
Below ha-proxy configuration of the architecture presented on this KB article with non authentication method enabled on ElasticSearch.
#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global
maxconn 4000
log 127.0.0.1 local0
log 127.0.0.1 local1 notice
daemon
defaults
log global
mode http
retries 3
timeout client 50s
timeout connect 5s
timeout server
timeout http-keep-alive 10s
listen elasticsearch-TCP-9200
bind localhost:9200
mode tcp
option tcpka
option tcplog
option redispatch
option httpchk GET /
timeout server 15m
balance roundrobin
server elasticsearch-1 elasticsearch-1.beehive.com:9200 check
server elasticsearch-2 elasticsearch-2.beehive.com:9200 check
server elasticsearch-3 elasticsearch-3.beehive.com:9200 check
# monitor port
listen status
bind 0.0.0.0:8088
stats enable
stats uri /
ha-proxy configuration with HTTP Basic authentication enabled on ElasticSearch
Below ha-proxy configuration of the architecture presented on this KB article with HTTP Basic Authentication enabled on ElasticSearch.
#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global
maxconn 4000
log 127.0.0.1 local0
log 127.0.0.1 local1 notice
daemon
defaults
log global
mode http
retries 3
timeout client 50s
timeout connect 5s
timeout server
timeout http-keep-alive 10s
listen elasticsearch-TCP-9200
bind localhost:9200
mode tcp
option tcpka
option tcplog
option redispatch
option httpchk GET / HTTP/1.0\r\nAuthorization:\ Basic\ <MY_PASSWORD>
timeout server 15m
balance roundrobin
server elasticsearch-1 elasticsearch-1.beehive.com:9200 check
server elasticsearch-2 elasticsearch-2.beehive.com:9200 check
server elasticsearch-3 elasticsearch-3.beehive.com:9200 check
# monitor port
listen status
bind 0.0.0.0:8088
stats enable
stats uri /
To generate
<MY_PASSWORD>
you can run:
echo -n "user:pass" | base64
0 Comments