Issue

• Viewing the agent node I see “Dead(!)” in red text
• Clicking on the “Dead(!)” text I see the following error message:

hudson.model.Executor#run: Unexpected executor death
java.lang.IllegalStateException: cannot create a build with number NUMBER since that (or higher) is already in use among [NUMBER]

Environment

• CloudBees Jenkins Platform - Client controller (CJP-CM) < 2.19.3.1
• CloudBees Jenkins Platform - Operations Center (CJP-OC) < 2.19.3.1
• Jenkins LTS < 2.19.1

Resolution

This is caused when the next build number is lower than the build number in the builds folder.

To resolve this issue please go to Manage Jenkins > Script Console and run the following script:

Jenkins.instance.getAllItems(Job.class).each { it ->
    // This does not work with matrix projects
    if (!(it instanceof hudson.matrix.MatrixConfiguration)) {
        def nextNumber;

        // If the next build number file does not exist then set it to one.
        try {
            nextNumber =  it.getNextBuildNumberFile().read().trim();
        } catch (java.io.FileNotFoundException e) {
            nextNumber = 1;
        }

        println("Job: " + it.getFullName() + ". Next Build Number: " + nextNumber + ". ");
        println "Content of file: " + nextNumber;

        def largest = 1;
        def current;
        // For each build directory in the job builds dir
        it.getBuildDir().list().each { builds ->
            try {
                // Get the file name and try to convert to an integer.
                current = Integer.valueOf(new File(builds).getName());
                // if the current directory largest is > largest from memory
                if (current > largest) {
                    // replace largest from memory
                    largest = current;
                }
            } catch (e) {
                // If it fails to convert, move on.
            }
        }

        println "Largest: " + (largest+1) + " Next Number: " + nextNumber;
        // If the next build number is > largest + 1
        if (!nextNumber.equals(largest+1)) {
            // Update and save the next build number as the new largest
            it.updateNextBuildNumber(largest + 1);
            println "Wrote content " + it.nextBuildNumber + " to the job: " + it.getFullName();
        }
    } else {
        println "Ignore matrix projects";
    }
};

This will cleanup all of the next build numbers, and will fix the node executors dying.

Reference

Dead Executor - Jenkins Wiki