Sending email notification when a Jenkins failover occurs

Email notifications can be very useful when your highly available Jenkins master(1) fails-over from one node to the other.
Such notification can help to review the list of aborted builds or to look at the failed node.

To implement email notification on Jenkins master failover, please follow the steps:

  • Read the “CloudBees Jenkins Enterprise HA monitor tool” section of the “CloudBees Jenkins Enterprise » High Availability” user guide
  • On each Jenkins master of the highly available cluster
    • Download the “jenkins-ha-monitor-1.2-jar-with-dependencies.jar” jar from the Jenkins HA download section (where “1.2” is the version of the tool, you should pick the latest one)
    • Install a shell notification script “send-notification-email-on-jenkins-master-promotion.sh” similar tomail -s “[Jenkins HA failover] Master `hostname` has been promoted” \
      jenkins-admin@example.com «< “Jenkins master `hostname` has been promoted at `date`”
    • Launch the processjava -jar jenkins-ha-monitor-4.3-jar-with-dependencies.jar \
        -home /mnt/nfs/jenkins-home/ \
        -host-promotion /path/to/send-notification-email-on-jenkins-master-promotion.sh
      • The linux user used to start this process must have read & write access to the shared JENKINS_HOME
      • -home is the path to the JENKINS_HOME
      • -host-promotion is the path of the script invoked when the node is promoted

Sample of logs of the HA Monitor Tool on the Jenkins master node that gets promoted:

Aug 05, 2014 6:22:19 PM com.cloudbees.jenkins.ha.singleton.HASingleton$3 viewAccepted
INFO: Cluster membership has changed to: [ip-10-0-0-12-45063 (309 bytes)|7] [ip-10-0-0-12-45063 (309 bytes), ip-10-0-0-11-33579 (272 bytes), ip-10-0-0-11-46466 (272 bytes), ip-10-0-0-12-54079 (272 bytes)]
Aug 05, 2014 6:22:19 PM com.cloudbees.jenkins.ha.singleton.HASingleton$3 viewAccepted
INFO: New primary node is JenkinsClusterMemberIdentity[member=ip-10-0-0-12-45063 (309 bytes),weight=0,min=0]
Aug 05, 2014 6:22:19 PM com.cloudbees.jenkins.ha.monitor.GraveDigger$1 run
INFO: Processing the fail over from JenkinsClusterMemberIdentity[member=ip-10-0-0-11-8467 (309 bytes),weight=0,min=0] to JenkinsClusterMemberIdentity[member=ip-10-0-0-12-45063 (309 bytes),weight=0,min=0]
Aug 05, 2014 6:22:19 PM com.cloudbees.jenkins.ha.monitor.GraveDigger promoteHost
WARNING: This host is now hosting the primary node
Aug 05, 2014 6:22:19 PM com.cloudbees.jenkins.ha.monitor.GraveDigger run
INFO: Executing /home/ec2-user/send-notification-email-on-jenkins-master-promotion.sh
Tue Aug  5 18:22:20 UTC 2014 host promotion
Aug 05, 2014 6:22:20 PM com.cloudbees.jenkins.ha.monitor.GraveDigger run
INFO: Executed /home/ec2-user/send-notification-email-on-jenkins-master-promotion.sh: exit code=0

 

Notes:

  • The jenkins-ha-monitor allows to attach scripts to two events: “on-promotion” when the master gets promoted as the active master and “on-demotion” when the master becomes passive
  • In very specific cases, the “on-promotion” and “on-demotion” events can be invoked on a jenkins master that is respectively already active or already standing-by. For example, the on-promotion event is triggered on the active node if the monitor-tool restarts

(1) High Availability is a feature of CloudBees Jenkins Enterprise by CloudBees.

Have more questions? Submit a request

0 Comments

Please sign in to leave a comment.