Issue

• Viewing the slave node I see “Dead(!)” in red text
• Clicking on the “Dead(!)” text I see the following error message:

hudson.model.Executor#run: Unexpected executor death
java.lang.IllegalStateException: cannot create a build with number NUMBER since that (or higher) is already in use among [NUMBER]

Environment

• CloudBees Jenkins Enterprise
• CloudBees Jenkins Operations Center

Resolution

This is caused when the next build number is lower than the build number in the builds folder.

To resolve this issue please go to Manage Jenkins > Script Console and run the following script:

Jenkins.instance.getAllItems(Job.class).each { it ->
    // This does not work with matrix projects
    if (!(it instanceof hudson.matrix.MatrixConfiguration)) {
        def nextNumber;

        // If the next build number file does not exist then set it to one.
        try {
            nextNumber =  it.getNextBuildNumberFile().read().trim();
        } catch (java.io.FileNotFoundException e) {
            nextNumber = 1;
        }

        println("Job: " + it.getFullName() + ". Next Build Number: " + nextNumber + ". ");
        println "Content of file: " + nextNumber;

        def largest = 1;
        def current;
        // For each build directory in the job builds dir
        it.getBuildDir().list().each { builds ->
            try {
                // Get the file name and try to convert to an integer.
                current = Integer.valueOf(new File(builds).getName());
                // if the current directory largest is > largest from memory
                if (current > largest) {
                    // replace largest from memory
                    largest = current;
                }
            } catch (e) {
                // If it fails to convert, move on.
            }
        }

        println "Largest: " + (largest+1) + " Next Number: " + nextNumber;
        // If the next build number is > largest + 1
        if (!nextNumber.equals(largest+1)) {
            // Update and save the next build number as the new largest
            it.updateNextBuildNumber(largest + 1);
            println "Wrote content " + it.nextBuildNumber + " to the job: " + it.getFullName();
        }
    } else {
        println "Ignore matrix projects";
    }
};

This will cleanup all of the next build numbers, and will fix the node executors dying.