How to Troubleshoot and Address Jenkins Startup Performances

Symptoms

  • Jenkins takes a long time to start
  • Jenkins hangs on startup

Diagnostic/Treatment

The following recommendation and techniques help to improve and troubleshoot performance issues on startup. They are based on experience with CloudBees customers.

Preconditions

Before troubleshooting any further, we recommend to go through the following recommendations that address common causes of slowness on startup.

Apply JVM Recommendation

The speed of Jenkins startup depends on the amount of initial memory allocated. It is important that you follow the recommendation:

  • Ensure that the recommended settings are applied to the process running Jenkins, see Prepare Jenkins for Support.
  • Ensure that an appropriate initial heap size of memory is allocated directly on startup (i.e. -Xms). If this is not specified, the process starts with a low portion of the available physical memory (that can vary depending on OS distribution and properties as explained in Garbage Collector Ergonomics)

Quiet Down mode on startup

Any build that were in the queue when Jenkins was stopped as well as any resumable builds (i.e. Pipeline) will be restarted on startup. This can contribute to the startup time and load, especially in environments where nodes are provisioned on demand. To avoid this:

Note: You may cancel the quiet down mode anytime when the master has started.

Remove Custom Loggers

Loggers are a good troubleshooting tool but they can impact the performances of your instance if they do not target a specific component or if they are specified at a high level package like for example hudson.model or hudson.security.

Review Hook scripts

Check whether there is any post-initialization scripts under $JENKINS_HOME/init.groovy.d/ that could impact the startup of your instance. See Groovy Hook Scripts.

Note: Hook scripts time are actually captured by the startup log mentioned above.

Data Collection

In order to troubleshoot the startup performance further:

  • Activate the startup performance logging by adding the system property -Djenkins.model.Jenkins.logStartupPerformance=true on startup
  • Restart Jenkins

And then on startup:

Note: You may have a look at How to analyze Jenkins Startup Performance logs to understand how to analyse the startup logs

Considerations

Please also consider the following scenario in which the Jenkins startup can be impacted.

Upgrades

After an upgrade, it is not unusual to experience slower startup caused by plugins upgrades and migration processes. Look at the logs to understand if Jenkins is doing something and what it is doing.

Restore Operations

Restore operations will show a different behavior depending on whether you are using modern platforms: CloudBees CI (CloudBees Core) or traditional platforms: CloudBees CI (CloudBees Core) on traditional platforms - Client Master and * CloudBees CI (CloudBees Core) on traditional platforms - Operations Center.

While in traditional platforms, the restore process will only impact startup slowness and will not generally show any other symptoms, in modern platforms it can cause some additional errors that will potentially cause the instance to fail provisioning due to different failures in the liveness / readiness probes.

You can get additional details on how to solve these issues in Why is my restored controller not coming up?.

Cleanup Large Instances

Cleanup Strategy
The startup time is proportional to the size of your instance (number of items, nodes, installed plugins). Ensure you have a strategy for cleanup as explained in the article Deleting Old Builds - Best Strategy for Cleanup and disk space management

Obsolete plugins
An instance with a large amount of plugins may take a lot of time to load all the components. It is a good practice to review plugin usage and remove obsolete ones. The CloudBees Plugin Usage can help to identify plugin usages.

Storage File System

Jenkins configuration files are stored on disk. Startup performances - and also overall performances for that matter - can be severely impacted if the storage File System holding $JENKINS_HOME is not tuned properly. If the $JENKINS_HOME does not resides in a local File System, ensure that the File System mount is tuned for performances. For NFS, please have a look at the recommendation in the article NFS Guide

Was this article helpful?
0 out of 0 found this helpful

Have more questions?

2 Comments

Date Votes
  • 0
    Avatar
    Jonathan Block

    I believe the thing that causes the biggest difference between a 20 second jenkins startup and a 90+ second start is the amount of files in the JENKINS_HOME directory.

    Jenkins does not use a database - it uses the filesystem. In order to take a tally of what you see on the Jenkins homepage, the server appears to do a giant directory traversal of everything in JENKINS_HOME in order to both discover which jobs exist but also which have artifacts and how many at the moment the Jenkins master service starts.

    In order to keep Jenkins working as quickly as possible, do not keep huge numbers of job runs lying around. Set Jenkins jobs to only hold the last 100-150 job runs and keep the number of total jobs as low as possible in order to avoid the startup sequence being forced to do an insane directory scan across your JENKINS_HOME directory.

  • 0
    Avatar
    Carlos Rodriguez Lopez

    @Jonathan Block 

    Thanks for your feedback, there is a link in the "parent article" of this one How to Troubleshoot and Address Liveness / Readiness probe failure which is completely aligned about what you are saying Master Sizing Guidelines

Please sign in to leave a comment.