Symptoms

• Jenkins takes a long time to start
• Jenkins hangs on startup

Diagnostic/Treatment

The following recommendation and techniques help to improve and troubleshoot performance issues on startup. They are based on experience with CloudBees customers.

Preconditions

Before troubleshooting any further, we recommend going through the following recommendations that address common causes of slowness on startup.

Apply JVM Recommendation

The speed of Jenkins startup depends on the amount of initial memory allocated. It is important that you follow the recommendation:

• Ensure that the recommended settings are applied to the process running Jenkins, see Prepare Jenkins for Support.
• Ensure that an appropriate initial heap size of memory is allocated directly on startup (i.e. -Xms). If this is not specified, the process starts with a low portion of the available physical memory (that can vary depending on OS distribution and properties as explained in Garbage Collector Ergonomics)

Quiet Down mode on startup

Any build that were in the queue when Jenkins was stopped as well as any resumable builds (i.e. Pipeline) will be restarted on startup. This can contribute to the startup time and load, especially in environments where nodes are provisioned on demand. To avoid this:

• Set the controller in Quiet Down mode on startup as explained in How can I prevent jenkins from starting new jobs after a restart?

Note: You may cancel the quiet down mode anytime when the controller has started.

Remove Custom Loggers

Loggers are a good troubleshooting tool but they can impact the performances of your instance if they do not target a specific component or if they are specified at a high level package like for example hudson.model or hudson.security.

• If you have created custom loggers, remove them.

Review Hook scripts

Check whether there is any post-initialization scripts under $JENKINS_HOME/init.groovy.d/ that could impact the startup of your instance. See Groovy Hook Scripts.

Note: Hook scripts time are actually captured by the startup log mentioned above.

Adjust jenkins.InitReactorRunner.concurrency

During start of Jenkins, loading of jobs in parallel have a fixed number of threads by default (twice the CPU). To make Jenkins load time decrease you can use tune the System Property jenkins.InitReactorRunner.concurrency.

RunIdMigrator occurrences

If seeing jenkins.model.RunIdMigrator#migrate logs on start up, please have a look at How to Troubleshoot the Root Cause of RunIdMigrator occurrences.

Data Collection

In order to troubleshoot the startup performance further:

• Activate the startup performance logging by adding the system property -Djenkins.model.Jenkins.logStartupPerformance=true on startup
• Restart Jenkins

While Jenkins is starting:

• Collect performance data as explained in Required Data: Jenkins Hang Issue On Linux that should contain output of the script collectPerformanceData.sh

Once Jenkins has started:

• Collect a Support Bundle
• Collect the Jenkins logs since the beginning of the startup
• Collect the GC logs

If Jenkins does not start and hang:

• Collect the latest auto-generated support bundle (see Generating a support bundle when Jenkins will not start)
• Collect the Jenkins logs since the beginning of the startup
• Collect the GC logs

Note: You may have a look at How to analyze Jenkins Startup Performance logs to understand how to analyse the startup logs

Considerations

Please also consider the following scenario in which the Jenkins startup can be impacted.

Upgrades

After an upgrade, it is not unusual to experience slower startup caused by plugins upgrades and migration processes. Look at the logs to understand if Jenkins is doing something and what it is doing.

Restore Operations

Restore operations will show a different behavior depending on whether you are using modern platforms: CloudBees CI (CloudBees Core) or traditional platforms: CloudBees CI (CloudBees Core) on traditional platforms - Client controller and * CloudBees CI (CloudBees Core) on traditional platforms - Operations Center.

While in traditional platforms, the restore process will only impact startup slowness and will not generally show any other symptoms, in modern platforms it can cause some additional errors that will potentially cause the instance to fail provisioning due to different failures in the liveness / readiness probes.

You can get additional details on how to solve these issues in Why is my restored controller not coming up?.

Cleanup Large Instances

The startup time is proportional to the size of your instance (number of items, nodes, installed plugins). Check our recommendations for Controller Sizing.

Cleanup Old Builds

Ensure you have a strategy for cleanup as explained in the article Deleting Old Builds - Best Strategy for Cleanup and disk space management

Obsolete plugins

An instance with a large amount of plugins may take a lot of time to load all the components. It is a good practice to review plugin usage and remove obsolete ones. The CloudBees Plugin Usage can help to identify plugin usages.

Storage File System

Jenkins configuration files are stored on disk. Startup performances - and also overall performances for that matter - can be severely impacted if the storage File System holding $JENKINS_HOME is not tuned properly. If the $JENKINS_HOME does not resides in a local File System, ensure that the File System mount is tuned for performances. For NFS, please have a look at the recommendation in the article NFS Guide