Dead Executors in Nodes

Issue

  • Viewing the agent node I see “Dead(!)” in red text
  • Clicking on the “Dead(!)” text I see the following error message:
hudson.model.Executor#run: Unexpected executor death
java.lang.IllegalStateException: cannot create a build with number NUMBER since that (or higher) is already in use among [NUMBER]

Environment

Resolution

This is caused when the next build number is lower than the build number in the builds folder.

To resolve this issue please go to Manage Jenkins > Script Console and run the following script:

Jenkins.instance.getAllItems(Job.class).each { it ->
    // This does not work with matrix projects
    if (!(it instanceof hudson.matrix.MatrixConfiguration)) {
        def nextNumber;

        // If the next build number file does not exist then set it to one.
        try {
            nextNumber =  it.getNextBuildNumberFile().read().trim();
        } catch (java.io.FileNotFoundException e) {
            nextNumber = 1;
        }

        println("Job: " + it.getFullName() + ". Next Build Number: " + nextNumber + ". ");
        println "Content of file: " + nextNumber;

        def largest = 1;
        def current;
        // For each build directory in the job builds dir
        it.getBuildDir().list().each { builds ->
            try {
                // Get the file name and try to convert to an integer.
                current = Integer.valueOf(new File(builds).getName());
                // if the current directory largest is > largest from memory
                if (current > largest) {
                    // replace largest from memory
                    largest = current;
                }
            } catch (e) {
                // If it fails to convert, move on.
            }
        }

        println "Largest: " + (largest+1) + " Next Number: " + nextNumber;
        // If the next build number is > largest + 1
        if (!nextNumber.equals(largest+1)) {
            // Update and save the next build number as the new largest
            it.updateNextBuildNumber(largest + 1);
            println "Wrote content " + it.nextBuildNumber + " to the job: " + it.getFullName();
        }
    } else {
        println "Ignore matrix projects";
    }
};

This will cleanup all of the next build numbers, and will fix the node executors dying.

Reference

Dead Executor - Jenkins Wiki

Have more questions? Submit a request

3 Comments

  • 0
    Avatar
    Dhanush Panicker

    Even after running the script from the script console. Still see the same error. Builder executor status is dead. Is there any other resolution. The current version of jenkins which we are using are Jenkins Version-
    1.625.3.2

  • 0
    Avatar
    Dhanush Panicker

    Error form the script console:

     

    java.lang.NullPointerException: Cannot invoke method getAllItems() on null object
    	at org.codehaus.groovy.runtime.NullObject.invokeMethod(NullObject.java:77)
    	at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:45)
    	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:42)
    	at org.codehaus.groovy.runtime.callsite.NullCallSite.call(NullCallSite.java:32)
    	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:42)
    	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108)
    	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116)
    	at Script1.run(Script1.groovy:1)
    	at groovy.lang.GroovyShell.evaluate(GroovyShell.java:580)
    	at groovy.lang.GroovyShell.evaluate(GroovyShell.java:618)
    	at groovy.lang.GroovyShell.evaluate(GroovyShell.java:589)
    	at hudson.util.RemotingDiagnostics$Script.call(RemotingDiagnostics.java:142)
    	at hudson.util.RemotingDiagnostics$Script.call(RemotingDiagnostics.java:114)
    	at hudson.remoting.UserRequest.perform(UserRequest.java:120)
    	at hudson.remoting.UserRequest.perform(UserRequest.java:48)
    	at hudson.remoting.Request$2.run(Request.java:326)
    	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
    	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    	at hudson.remoting.Engine$1$1.run(Engine.java:62)
    	at java.lang.Thread.run(Thread.java:745)
  • 0
    Avatar
    Denys Digtiar

    Hi Dhanush

    Please consider upgrading, your release is more than two an a half years old and out of the support. Any release newer than 2.13 is no longer affected by this issue.

    Please raise a support ticket and we can assist with updating the script to support your old version.

Please sign in to leave a comment.