User Lookup Can Cause Performances Issues

Issue

  • Jenkins is getting slow and eventually freezes
  • Many threads WAITING trying to get user information from Security Realm
    hudson.model.User$UserIDCanonicalIdResolver.resolveCanonicalId(User.java:1049)
    hudson.model.User.get(User.java:395)
    hudson.model.User.get(User.java:364)
  • Many threads WAITING trying to get user information from Security Realm (Pipeline Stage View)
    hudson.model.User.getOrCreate(User.java:425)
    hudson.model.User.get(User.java:403)
    hudson.model.User.get(User.java:362)
    hudson.plugins.git.GitChangeSet.findOrCreateUser(GitChangeSet.java:374)
    hudson.plugins.git.hudson.plugins.git.GitChangeSet.getAuthor.getAuthor(GitChangeSet.java:435)
    com.cloudbees.workflow.rest.external.ChangeSetExt.mapFields(ChangeSetExt.java:176)
    com.cloudbees.workflow.rest.external.ChangeSetExt.create(ChangeSetExt.java:151)
    com.cloudbees.workflow.rest.endpoints.RunAPI.doChangesets(RunAPI.java:112)

Environment

  • CloudBees Jenkins Enterprise
  • Jenkins

Related Issue(s)

Explanation

Since the resolution of SECURITY-243, all the users in the commits of an SCM changeset need to be looked up in the Security Realm to see if they are valid authentication “users”. For more details, please have look at JENKINS-35493.

In general, any process interacting with SCM changesets is impacted (for example viewing a page with changelog). In particular, Pipeline jobs are impacted as the Pipeline Stage View Plugin attempts to resolve authors of ChangeLog entries of the ChangeSet to specific users, and this API is invoked with each page view, so it is hit frequently. For more details, please have look at JENKINS-35484.

This can cause performance issues, especially for large instances. Symptoms are a large number of threads requesting user information to the Security Realm, instance unresponsive, users unable to login. When the Security Realm server is overloaded, Jenkins eventually freezes and needs to be restarted.

Resolution

The solution is to upgrade to CJP 2.7.19.x or later. A cache has been implemented for User/Groups in the version 2.7.3 of Jenkins Core. For more details, please have a look at the pull request of Jenkins core PR#2446

Before Jenkins 2.7.3 / CJP 2.7.19.x

If upgrading Jenkins is not an option, the workaround is to disable the user lookup behavior using the following system properties:

-Dhudson.model.User.SECURITY_243_FULL_DEFENSE=false

Have more questions? Submit a request

0 Comments

Please sign in to leave a comment.