Issue

• There is a Pipeline stuck executing and I am unable to stop it from executing, or a Pipeline is misbehaving and I would like to stop it

Environment

• CloudBees CI (CloudBees Core)
• CloudBees Jenkins Enterprise - Managed Master (CJE-MM)
• CloudBees Jenkins Platform - Client Master (CJP-CM)
• CloudBees Jenkins Distribution
• Jenkins LTS
• Pipeline Plugin
• Durable Task Plugin

Cause

This issue can be caused a number of ways but the most likely one is that an older version of the Durable Task plugin is installed and it has not picked up the fix released in JENKINS-47791

Resolution

In order to resolve this in the most efficient way possible, this is the correct order of operations:

1 . If your Jenkins instance is down, and it keeps being shut down by a misbehaving Pipeline, first we need to determine which Pipeline could be causing the issue by running:

grep -ri 'system.exit' $JENKINS_HOME

If you can find a Pipeline using that command, then the way to resolve the issue is to move that build directory (such as JENKINS_HOME/jobs/$JOB, or just one specific run of that build $JENKINS_HOME/jobs/$JOB/builds/$BUILD_NUMBER) outside of the JENKINS_HOME (or delete it) so that the build will no longer exist in Jenkins when you start it up again. You should also go to Manage Jenkins -> In process script approval and clear the system.exit that was approved to be run, as this method should never be allowed to be run inside of a Pipeline job, as it will cause the JVM that is running Jenkins to shut down.

This method of removing the build directory can also be used to prevent any Pipeline build from resuming when Jenkins is starting up, if you are able to tell that a specific Pipeline job is causing issues on your instance.

2 . If your Jenkins instance is online, Hit the red X in the UI to stop the build:

3 . After 30 seconds of the build not stopping that should generate a link in the Console of the build which says “Click here to forcibly terminate running steps” or on the sidebar link like shown here:

If you see the link and click it but that does not stop the build, move on to 4. If you don’t see the link after a refresh of the page then move to 5.


4 . If the previous step didn’t work then a new link should appear in that same console and on the sidebar after another 30 seconds which says “Click here to forcibly kill entire build” and clicking that should kill the build.


5 . If you are not seeing that link or number 4 does not work then try adding /kill to the URL of the job. This would mean your job URL looks like $JENKINS_URL/job/PipelineKillTest/1/ and you just add kill to the end of the url. That will perform the same function as Click here to forcibly kill entire build link from before.


6 . If none of the above work, or if you would like to use a groovy script to stop Pipeline builds, we recommend running ProperlyStopRunningPipelines.groovy under Manage Jenkins -> Script Console. In case that you only want to stop the pipelines that are running on a executor use this version of the script ProperlyStopOnlyRunningPipelines.groovy

7 . If none of the above work, then the next step will require a restart. You will need to stop Jenkins and move that build away from the filesystem which will be inside of $JENKINS_HOME/jobs/$JOB/builds/$BUILD_NUMBER. Once that has been removed from your $JENKINS_HOME you can restart Jenkins and the build should be gone.

8 . If none of this works then please follow the suggestions from Required data in Pipeline hangs article.

If the issue was in fact the Durable Task issue then you should ensure that your Durable Task plugin has been updated to the latest version.