Issue
- Jenkins is getting slow and eventually freezes
- Many slow-requests and / or threads that are WAITING trying to get user information from the Security Realm
"Handling GET /job/myJob/job/develop/changes from X.X.X.X : RequestHandlerThread[#13]" [...]
[...]
.*.loadUserByUsername(.*)
hudson.model.User$UserIDCanonicalIdResolver.resolveCanonicalId(User.java:1049)
hudson.model.User.get(User.java:395)
hudson.model.User.get(User.java:364)
[...]
- Many slow-requests and / or threads that are WAITING trying to get user information from Security Realm (Pipeline Stage View)
"Handling GET /job/myJob/job/develop/1/wfapi/changesets from X.X.X.X : RequestHandlerThread[#13]" [...]
[...]
.*.loadUserByUsername(.*)
hudson.model.User$UserIDCanonicalIdResolver.resolveCanonicalId(User.java:1049)
hudson.model.User.get(User.java:395)
hudson.model.User.get(User.java:364)
hudson.plugins.git.GitChangeSet.findOrCreateUser(GitChangeSet.java:374)
hudson.plugins.git.GitChangeSet.getAuthor(GitChangeSet.java:435)
com.cloudbees.workflow.rest.external.ChangeSetExt.mapFields(ChangeSetExt.java:176)
com.cloudbees.workflow.rest.external.ChangeSetExt.create(ChangeSetExt.java:151)
com.cloudbees.workflow.rest.endpoints.RunAPI.doChangesets(RunAPI.java:112)
[...]
Environment
- CloudBees Jenkins Enterprise - Managed Master (CJEMM) versions between 1.642.18.x and 2.7.19.x (before CJE 1.3.0)
- CloudBees Jenkins Platform - Client Master (CJPCM) versions between 1.642.18.x and 2.7.19.x
- Jenkins LTS versions between 1.651.2 and 2.7.3
Related Issue(s)
Explanation
Since the resolution of SECURITY-243, all the users in the commits of an SCM changeset need to be looked up in the Security Realm to see if they are valid authentication “users”. For more details, please have look at JENKINS-35493.
This can cause performance issues, especially for large instances. Symptoms are a large number of threads requesting user information to the Security Realm, instance unresponsive and users unable to login. When the Security Realm server is overloaded, Jenkins eventually freezes and needs to be restarted.
Note: In general, any process interacting with SCM changesets is impacted (for example viewing a page with changelog). In particular, Pipeline jobs are impacted as the Pipeline Stage View Plugin attempts to resolve authors of ChangeLog entries of the ChangeSet to specific users, and this API is invoked with each page view, so it is hit frequently. For more details and a potential workaround, please have look at JENKINS-35484.
Resolution
Therefore the solution is to upgrade: A cache has been implemented for User/Groups in the version 2.7.3 of Jenkins Core. For more details, please have a look at the pull request of Jenkins core PR#2446
- CJE: upgrade to version 1.3.0 or later
- CJP / CJT: upgrade to version 2.7.19.x or later
- Jenkins LTS: upgrade to version 2.7.3 or later
Before Jenkins 2.7.3 / CJP 2.7.19.x
If upgrading Jenkins is not an option, the workaround is to disable the user lookup behavior using the following system properties:
-Dhudson.model.User.SECURITY_243_FULL_DEFENSE=false
0 Comments