Jenkins’ builds and navigation is slowed down, and a thread dump shows several threads
[...] at jenkins.fingerprints.FileFingerprintStorage.save(FileFingerprintStorage.java:125) - waiting to lock <0x00007f9b65846a98> (a jenkins.fingerprints.FileFingerprintStorage) [...]
- CloudBees CI (CloudBees Core) >= 22.214.171.124
- CloudBees CI (CloudBees Core) on modern cloud platforms - Managed Master >= 126.96.36.199
- CloudBees CI (CloudBees Core) on modern cloud platforms - Operations Center >= 188.8.131.52
- CloudBees CI (CloudBees Core) on traditional platforms - Client Master >= 184.108.40.206
- CloudBees CI (CloudBees Core) on traditional platforms - Operations Center >= 220.127.116.11
- CloudBees Jenkins Platform - Client Master >= 18.104.22.168
- CloudBees Jenkins Platform - Operations Center >= 22.214.171.124
- CloudBees Jenkins Distribution >= 126.96.36.199
- Jenkins LTS >= 2.249.1
- JENKINS-62345: Introduce External Fingerprint Storage API in core
Jenkins LTS 2.249.1 introduces an API to enable pluggable storage for Jenkins Fingerprints.
The default implementation of that API uses the File System under
$JENKINS_HOME/fingerprints like previous version. However, a bug - JENKINS-64670 - was introduced that prevent several Fingerprints to be saved in parallel. All Fingerprint saves are blocked by a single object lock on the
This can cause severe performances issues when used with plugin that produce lots of fingerprints. For example the Maven Plugin and the Credentials plugin when tracking credentials usage.
A solution has already been merged in Jenkins Core and will likely be available in the next Jenkins LTS. The March 2021 release of CloudBees CI will be based on the new LTS.
For impacted instances, it is recommended to stay up to date and prepare for this release.
There is no workaround for the lock contention.
However, if impacted it is likely that the fingerprinting for Credentials usage contributes greatly to the problem. In which case it is recommended to disable it, see Performance issues caused by the Fingerprinting of Credentials Usage.