Issue

• When configuring a job in a Client Master, connected to an Operations Center, if we try to set a value for credentials on any given configuration section, when we click the dropdown to open it, it takes more time than expected.

• We see a lot of thread contention in threads related to one thread showing the trace below:

at hudson.XmlFile.write(XmlFile.java:181)
	at com.cloudbees.opscenter.client.plugin.OperationsCenterCredentialsProvider.saveCache(OperationsCenterCredentialsProvider.java:234)
	at com.cloudbees.opscenter.client.plugin.OperationsCenterCredentialsProvider.cachePut(OperationsCenterCredentialsProvider.java:170)
	at com.cloudbees.opscenter.client.plugin.OperationsCenterCredentialsProvider.getCredentials(OperationsCenterCredentialsProvider.java:153)

Environment

• CloudBees CI (CloudBees Core).
• CloudBees CI (CloudBees Core) Traditional Platforms - Client Master.
• CloudBees CI (CloudBees Core) Traditional Platforms - Operations Center.

Resolution

The problem comes from a serialization issue in the credentials cache provided by the Operations Center. This cache was meant to provide a set of credentials to the Masters even if the Master is temporarily disconnected from the Operations Center.

Being the main use case for the cache, the Shared Agent reconnection after a Master restart so that durable tasks (i.e. pipeline steps) could continue.

• If you are not using Shared Agent feature, then you can disable it by adding the following startup parameter to your Master.

• -Dcom.cloudbees.opscenter.client.plugin.OperationsCenterCredentialsProvider.cache.disabled=true

If you are using Shared Agent feature and you are using a Master version which includes operations-center-client Plugin version lower than 2.107.1.5, then in order to solve the issue, you should upgrade OC and Master until you have available to install operations-center-client Plugin version 2.107.1.5 or higher.

Mitigation Workaround

Due to the fact that upgrading an instance requires a lot of planning and testing, in the meantime and in order to reduce the impact of this issue you should execute the following script in the Jenkins Script Console when the issue is exposed or periodically in order to reduce the probability of the issue from happening.

import com.cloudbees.plugins.credentials.Credentials
import java.util.HashMap
import com.cloudbees.opscenter.client.plugin.OperationsCenterCredentialsProvider

provider = ExtensionList.lookup(OperationsCenterCredentialsProvider.class).get(0)
provider.lock.writeLock().lock();
println 'Cleaning credentials cache. Lock acquired...'
try {
  provider.cache = new HashMap<String, List<? extends Credentials>>();
  provider.saveCache()
  println 'Cache cleaned'
} finally {
  provider.lock.writeLock().unlock();
  println 'Lock released'
}

The script will flush the cache and you should be able to see the performance improvement immediately.