- Jenkins takes a long time to start
- Jenkins hangs on startup
The following recommendation and techniques help to improve and troubleshoot performance issues on startup. They are based on experience with CloudBees customers.
Before troubleshooting any further, we recommend to go through the following recommendations that address common causes of slowness on startup.
The speed of Jenkins startup depends on the amount of initial memory allocated. It is important that you follow the recommendation:
- Ensure that the recommended settings are applied to the process running Jenkins, see Prepare Jenkins for Support.
- Ensure that an appropriate initial heap size of memory is allocated directly on startup (i.e.
-Xms). If this is not specified, the process starts with a low portion of the available physical memory (that can vary depending on OS distribution and properties as explained in Garbage Collector Ergonomics)
Any build that were in the queue when Jenkins was stopped as well as any resumable builds (i.e. Pipeline) will be restarted on startup. This can contribute to the startup time and load, especially in environments where nodes are provisioned on demand. To avoid this:
- Set the master in Quiet Down mode on startup as explained in How can I prevent jenkins from starting new jobs after a restart?
Note: You may cancel the quiet down mode anytime when the master has started.
Loggers are a good troubleshooting tool but they can impact the performances of your instance if they do not target a specific component or if they are specified at a high level package like for example
- If you have created custom loggers, remove them.
Check whether there is any post-initialization scripts under
$JENKINS_HOME/init.groovy.d/ that could impact the startup of your instance. See Groovy Hook Scripts.
Note: Hook scripts time are actually captured by the startup log mentioned above.
In order to troubleshoot the startup performance further:
- Activate the startup performance logging by adding the system property
- Restart Jenkins
And then on startup:
- Collect performance data as explained in Required Data: Jenkins Hang Issue On Linux
Note: You may have a look at How to analyze Jenkins Startup Performance logs to understand how to analyse the startup logs
Please also consider the following scenario in which the Jenkins startup can be impacted.
After an upgrade, it is not unusual to experience slower startup caused by plugins upgrades and migration processes. Look at the logs to understand if Jenkins is doing something and what it is doing.
Restore operations will show a different behavior depending on whether you are using modern platforms: CloudBees CI (CloudBees Core) or traditional platforms: CloudBees CI (CloudBees Core) on traditional platforms - Client Master and * CloudBees CI (CloudBees Core) on traditional platforms - Operations Center.
While in traditional platforms, the restore process will only impact startup slowness and will not generally show any other symptoms, in modern platforms it can cause some additional errors that will potentially cause the instance to fail provisioning due to different failures in the liveness / readiness probes.
You can get additional details on how to solve these issues in Why is my restored controller not coming up?.
The startup time is proportional to the size of your instance (number of items, nodes, installed plugins). Ensure you have a strategy for cleanup as explained in the article Deleting Old Builds - Best Strategy for Cleanup and disk space management
An instance with a large amount of plugins may take a lot of time to load all the components. It is a good practice to review plugin usage and remove obsolete ones. The CloudBees Plugin Usage can help to identify plugin usages.
Jenkins configuration files are stored on disk. Startup performances - and also overall performances for that matter - can be severely impacted if the storage File System holding
$JENKINS_HOME is not tuned properly. If the
$JENKINS_HOME does not resides in a local File System, ensure that the File System mount is tuned for performances. For NFS, please have a look at the recommendation in the article NFS Guide