Clouds defined in the Operations Center are not pushed to the Connected Masters

Issue

  • A Shared cloud is defined in the Operations Center but it is not recognized by the Connected Masters.
  • Updates on an existing Shared Cloud item are no propagated/pushed to the Connected Masters.
  • A stacktrace similar to the one below appears in the Operations Center logs.
2020-11-19 09:39:04.444+0000 [id=32]	WARNING	c.c.o.s.model.ConnectedMaster#rebuildRemoteContext: Uncaught exception from com.cloudbees.opscenter.server.config.ConnectedMasterConfigurationOptOutProperty$RemoteContextContributorImpl@7fe1c4b4 on ManagedMaster{id=30, name='master-jenkins', encodedName='master-jenkins', idName='30-master-jenkins', timeStamp=0, error=null, grantId='ddf4ae57-8516-4ee4-b30e-78946b17c854', approved=true, persistedState=PersistedStateImpl{state=APPROVED, futureFilesystem=null, filesystem=null, futureResource=null, resource=KubernetesMasterResource [namespace=master-jenkins, fsGroup=1000, getName()=master-jenkins, getEndpoint()=https://master.cloudbees.example.com/master-jenkins/, getCpus()=1.0, getMemory()=3072.0, getRatio()=0.5, getImage()=DockerImageDefinition{imageTag='cloudbees/cloudbees-core-mm:2.190.3.2', name='CloudBees Core - Managed Master - 2.190.3.2'}]}, localHome='null', localEndpoint=https://master.cloudbees.example.com/master-jenkins/, channel=hudson.remoting.Channel@262ee43e:OperationsCenter2 connection from master-jenkins-0.master-jenkins.master-jenkins.svc.cluster.local/10.200.46.217:39068, identity=X.509, RSA}
java.lang.NullPointerException
	at com.cloudbees.opscenter.context.config.CloudConfigurationSnippetHandler.populate(CloudConfigurationSnippetHandler.java:111)
	at com.cloudbees.opscenter.context.config.ConfigurationSnippetHandler.contribute(ConfigurationSnippetHandler.java:128)
	at com.cloudbees.opscenter.server.config.ConnectedMasterConfigurationOptOutProperty$RemoteContextContributorImpl.contribute(ConnectedMasterConfigurationOptOutProperty.java:213)
	at com.cloudbees.opscenter.server.config.ConnectedMasterConfigurationOptOutProperty$RemoteContextContributorImpl.contribute(ConnectedMasterConfigurationOptOutProperty.java:314)
	at com.cloudbees.opscenter.server.config.ConnectedMasterConfigurationOptOutProperty$RemoteContextContributorImpl.contribute(ConnectedMasterConfigurationOptOutProperty.java:197)
	at com.cloudbees.opscenter.server.model.ConnectedMaster.rebuildRemoteContext(ConnectedMaster.java:800)
	at com.cloudbees.opscenter.server.model.ConnectedMaster.updateRemoteContext(ConnectedMaster.java:757)
	at com.cloudbees.opscenter.server.model.ConnectedMaster$UpdateRemoteContexts.doRun(ConnectedMaster.java:2354)
	at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:91)
	at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

Environment

Resolution

Expected Behaviour

Changes on the Operation Center Shared Cloud (e.g. Kubernetes Shared Cloud)

2020-12-18 15:26:57.439+0000 [id=132655] INFO c.c.o.s.model.ConnectedMaster#updateRemoteContext: Updating context for ManagedMaster{id=49, name='master-demo', encodedName='master-demo', ...... as it was out of date
2020-12-18 15:27:36.885+0000 [id=132779] INFO c.c.o.c.c.CloudConfigurationSnippetHandler#consume: Applied/updated Clouds configuration [kubernetes, kubernetes]

are propagated to the Connected Master

2020-12-18 15:24:51.510+0000 [id=56] INFO c.c.o.c.c.CloudConfigurationSnippetHandler#consume: Applied/updated Clouds configuration [kubernetes, Operations Center Agent Provisioning Service]

Most likely Root Cause

In most cases, this issue is produced by a Shared Configuration item (e.g: AWS EC2 Cloud Configuration, Miscellaneous Configuration Container o Kubernetes Cloud Configuration) in the Operation Center which configuration is not filled (empty fields). As a result, the Operations Center fails trying to push the updates towards the Masters.

Workaround

The following steps aim to identify which Shared Configuration item is empty in order to amend or delete it.

1.- At the UI level, run the following script in the Operation Center Manage Jenkins -> Script Console in the Operations Center to list all the Shared Configuration items and their full paths

Jenkins.instance.getAllItems(com.cloudbees.opscenter.server.config.SharedConfiguration).each {
    println it.getName()
}
return null

Example output

ec2-cloud-configuration-shared
kubernetes-shared-cloud
new_cloud
email
Result: [com.cloudbees.opscenter.clouds.ec2.EC2Configuration@141d831[ec2-cloud-configuration-shared], com.cloudbees.opscenter.clouds.kubernetes.KubernetesConfiguration@41339d90[kubernetes-shared-cloud], com.cloudbees.opscenter.clouds.kubernetes.KubernetesConfiguration@3467e53b[new_cloud], com.cloudbees.opscenter.server.config.SharedConfiguration@1a78fa7a[email]]

2.- At File System level, navigate to the Operation Center jobs folders and run the following

grep -R -s -e "</com.cloudbees.opscenter.server.config.SharedConfiguration>" -e "</com.cloudbees.opscenter.clouds.ec2.EC2Configuration>" -e "</com.cloudbees.opscenter.clouds.kubernetes.KubernetesConfiguration>" . --exclude-dir=build

Example output

./jobs/kubernetes-shared-cloud/config.xml:</com.cloudbees.opscenter.clouds.kubernetes.KubernetesConfiguration>
./jobs/new_cloud/config.xml:</com.cloudbees.opscenter.clouds.kubernetes.KubernetesConfiguration>
./jobs/email/config.xml:</com.cloudbees.opscenter.server.config.SharedConfiguration>
./jobs/ec2-cloud-configuration-shared/config.xml:</com.cloudbees.opscenter.clouds.ec2.EC2Configuration>
grep -R -s -h -e "</com.cloudbees.opscenter.server.config.SharedConfiguration>" -e "</com.cloudbees.opscenter.clouds.ec2.EC2Configuration>" . -e "</com.cloudbees.opscenter.clouds.kubernetes.KubernetesConfiguration>" --exclude-dir=build -C 200`

Example output

<?xml version='1.1' encoding='UTF-8'?>
<com.cloudbees.opscenter.clouds.kubernetes.KubernetesConfiguration plugin="operations-center-kubernetes-cloud@2.263.0.1">
  <actions/>
  <description></description>
  <snippets>
    <com.cloudbees.opscenter.clouds.kubernetes.KubernetesCloudConfigurationSnippet>
      <value>
        <string>&lt;org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud plugin=&quot;kubernetes@1.28.3&quot;&gt;
  &lt;name&gt;kubernetes-oc&lt;/name&gt;
  &lt;defaultsProviderTemplate&gt;&lt;/defaultsProviderTemplate&gt;
 ...
</com.cloudbees.opscenter.clouds.kubernetes.KubernetesConfiguration>
--
<?xml version='1.1' encoding='UTF-8'?>
<com.cloudbees.opscenter.clouds.kubernetes.KubernetesConfiguration plugin="operations-center-kubernetes-cloud@2.263.0.1">
  <actions/>
  <description></description>
  <snippets>
    <com.cloudbees.opscenter.clouds.kubernetes.KubernetesCloudConfigurationSnippet>
      <value>
        <string>&lt;org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud plugin=&quot;kubernetes@1.28.3&quot;&gt;
  &lt;name&gt;kubernetes&lt;/name&gt;
  &lt;defaultsProviderTemplate&gt;&lt;/defaultsProviderTemplate&gt;
 ...
</com.cloudbees.opscenter.clouds.kubernetes.KubernetesConfiguration>
--
<?xml version='1.1' encoding='UTF-8'?>
<com.cloudbees.opscenter.server.config.SharedConfiguration plugin="operations-center-server@2.235.0.2">
  <snippets/>
  <properties/>
</com.cloudbees.opscenter.server.config.SharedConfiguration>
--
<?xml version='1.1' encoding='UTF-8'?>
<com.cloudbees.opscenter.clouds.ec2.EC2Configuration plugin="operations-center-ec2-cloud@2.235.0.2">
  <actions/>
  <description></description>
  <snippets>
    <com.cloudbees.opscenter.clouds.ec2.EC2CloudConfigurationSnippet>
      <value>
        <string>&lt;hudson.plugins.ec2.EC2Cloud plugin=&quot;ec2@1.50.3&quot;&gt;
  &lt;name&gt;ec2-ec2-cloud-configuration-shared&lt;/name&gt;
  ...
</com.cloudbees.opscenter.clouds.ec2.EC2Configuration>

In the provided example, we observe that email was an empty Miscellaneous Configuration Container after removing it the issue was solved.

Once the Shared Configuration items have been fixed, restart the Operation Center. If the problematic Shared Cloud in the Operations Center still not be recognized by the connected Masters, take a BackUp of the Operation Center and then delete all the Shared Configuration items apart from the Shared Cloud you are troubleshooting. In case it fixes the NullPointerException, start adding one by one the required Shared Configuration items to understand which is the problematic one.

Tested products/plugins version

The latest update of this article was tested with:

Have more questions?

0 Comments

Please sign in to leave a comment.