Symptoms

• How to troubleshoot Move/Copy/Promote issues on CloudBees Jenkins

Diagnosis/Treatment

This article gives simple steps for troubleshooting a Move/Copy/Promote operations. It also gives steps to gather the most relevant information so that the Support Team can resolve your issue as quickly as possible.

What to do if a Move/Copy/Promote operation is getting stuck at validating stage

If you are performing a Move/Copy/Promote operation between two different controllers and the build is stucked in this window

probably the messaging database of the controller which is trying to perform the M/C/P operation is corrupted and the CJOC is not able to pick up the request. To troubleshoot such situations We have a script to be executed in the jenkins groovy console in order to confirm it, so please follow the next steps in you are facing this problem:

1. Try to reproduce the issue copying a job from a controller (MASTER_SOURCE) to another Master
2. When the screen is waiting on Validating
   1. Create a new support bundle of CJOC and the Master
   2. Go to ${CJOC_URL}/script and execute the attached cjoc-status.groovy script and save the output in cjoc-status.output
   3. Go to ${CJOC_URL}/script and execute the attached messaging-status.groovy script and save the output in cjoc-messaging-status.output
   4. Go to ${MASTER_SOURCE_URL}/script and execute the attached messaging-status.groovy script and save the output in controller-messaging-status.output
3. Create a zendesk case and attach the support bundles and the *-status.output files.

Get the Move/Copy/Promote History

For each Move/Copy/Promote operation, there is a history log created and you can access it via in Jenkins via Manage Jenkins > Move/Copy/Promote History. On this page you can locate the Move/copy operations that failed and check the log by selecting the small screen on the right.

This displays the detailed logs of the operation:

Add Loggers and Reproduce

You can enhance the logging to track the Queue and improve our visibility during the failing operation:

1. Create a Logger in the instance(s) involved in the operation. Name it for example move-copy-issue.
2. Add the following packages:
   • hudson.model.Executor
   • hudson.model.Queue
   • com.cloudbees.opscenter.dumbwaiter
   • com.cloudbees.opscenter.replication
     
3. Reproduce the issue
4. Get the log at $JENKINS_HOME/logs/custom/move-copy-issue.log

(If you generate a support bundle, this custom logger is included in it)

Check the Jenkins User Permissions

Ensure that the Jenkins user has the required permissions - Jenkins user should have read/write permissions - in both the source and the destination locations. You can for example run the following command (replace <path-to-item> with the path of your job or folder):

ls -ltR $JENKINS_HOME/jobs// > /tmp/permissions.log

This creates the file /tmp/permissions.log in which you can check the user’s permissions. For example:

FolderB:
total 4
drwxr-xr-x 3 jenkins jenkins  102 May 19 17:10 jobs
-rw-r--r-- 1 jenkins jenkins 2539 May 19 16:57 config.xml

FolderB/jobs:
total 0
drwxr-xr-x 7 jenkins jenkins 238 May 19 17:10 jobA

FolderB/jobs/jobA:
total 12
drwxr-xr-x 9 jenkins jenkins 306 May 19 17:10 builds
-rw-r--r-- 1 jenkins jenkins 481 May 19 17:10 config.xml
drwxr-xr-x 2 jenkins jenkins  68 May 19 17:10 workspace
-rw-r--r-- 1 jenkins jenkins   2 May 19 17:10 nextBuildNumber
-rw-r--r-- 1 jenkins jenkins  30 May 19 16:57 unique-id.txt

Check the User Permissions in Jenkins

Ensure that the user performing the operation has the required permission in both the source and the destination location.

1. Check the permissions at $JENKINS_SERVER/whoAmI
2. For RBAC, check the roles/permission at $JENKINS_SERVER/roles/whoAmI

When using CJOC SSO to login on controllers it is required to use trusted controllers configuration in CJOC, this requires to set “Trusted controller with equivalent security realm” in “Configure Global Security” - “Security Setting Enforcement” in CJOC general configuration. This configuration is needed to properly identify the user performing the move/copy operation in the target controller, so security policies are properly applied. Otherwise the user will be identified as anonymous in the target controller and the operations will fail (unless anonymous has write permissions in the destination location).

Check the Active Directory/LDAP Plugin configuration

If you are using Active Directory Plugin or LDAP Plugin to manage your security realm, you have to ensure that these plugins are properly configured as the MCP operation might need to perform a large quantity of permission validations and if these plugins are not configured correctly, this can cause slowness on the operation. You can even see slowness when you are trying to select the destination folder.

If that is the case, please check our recommendations in the corresponding document from the ones listed below:

• The log-in with LDAP is very slow
• The log-in with Active Directory is very slow

Recommendations

Many fixed/improvement have been added to this feature and we recommend that you update the operations-center-* plugins to the latest version.

More Information

• Operations Center Context Plugin - Release Notes
• Operations Center plugins versions