Issue
- We detected that some build data is not reported to Operations Center Analytics
- We saw the following message in CJP-OC/CJE-OC logs
Mar 27, 2018 11:39:44 AM WARNING com.cloudbees.opscenter.analytics.reporter.items.ItemSubmissionQueue readChanges
The item does not exist anymore Update{type=update, item=null, locator= ... }
- We saw the following message in CJP-CM/CJE-MM logs
Apr 06, 2018 12:20:53 PM WARNING com.cloudbees.opscenter.analytics.reporter.items.ItemSubmissionQueue submitQueueItems
Failed to submit changes
java.io.IOException: Unexpected return code calling items : 500
at com.cloudbees.opscenter.analytics.reporter.async.AsyncReporter$1.onCompleted(AsyncReporter.java:78)
at com.cloudbees.opscenter.analytics.reporter.async.AsyncReporter$1.onCompleted(AsyncReporter.java:69)
at com.ning.http.client.AsyncCompletionHandler.onCompleted(AsyncCompletionHandler.java:63)
at com.ning.http.client.providers.netty.NettyResponseFuture.getContent(NettyResponseFuture.java:279)
at com.ning.http.client.providers.netty.NettyResponseFuture.done(NettyResponseFuture.java:307)
- We see this Administrative monitor active in CJP-OC/CJE-OC
- We see this Administrative monitor active in CJP-OC/CJE-OC/CJP-CM/CJE-MM
Environment
- CloudBees Jenkins Platform - Client Master (CJP-CM)
- CloudBees Jenkins Platform - Operations Center (CJP-OC)
- CloudBees Jenkins Enterprise - Managed Master (CJE-MM)
- CloudBees Jenkins Enterprise - Operations Center (CJE-OC)
Resolution
Potential causes and how to find each one:
Elasticsearch is offline
If you see the Administrative monitor like the two below in CJP-OC/CJE-OC, then your Elasticsearch is in red status or offline. Please follow our troubleshooting guide - Elasticsearch Cluster Health
Client masters unable to reach CJOC
If you see the following Administrative monitor active in CJP-CM/CJE-MM, you have to check your CJP-OC/CJE-OC for the Administrative monitor described before. If none are active in Operations Center, then you have to check the connectivity between CJP-CM/CJE-MM and CJP-OC/CJE-OC, see Operations Center-Client Master connectivity issues
Missing build data
If you are missing some build data or you saw the following message in your CJP-CM/CJE-MM logs:
Mar 27, 2018 11:39:44 AM WARNING com.cloudbees.opscenter.analytics.reporter.items.ItemSubmissionQueue readChanges
The item does not exist anymore Update{type=update, item=null, locator= ... }
You should check the build history of your jobs. Only the existing items are reported to Operations Center Analytics which means that when the analytics reporter queue sends the data to CJP-OC/CJE-OC the related item should exist. Because of that, it is recommended to keep builds for at least 24 hours or the maximum period that you would have an outage in your infrastructure. Thus if you keep builds for at least 24 hours and Elasticsearch or CJOC is offline for this period, you would not lose builds which are rotated during the outage.
Deleting Old Builds - Best Strategy for Cleanup and disk space management
Missing Nodes data
If you are missing some nodes data or you saw the following message in your CJP-CM/CJE-MM logs:
Mar 27, 2018 11:39:44 AM WARNING com.cloudbees.opscenter.analytics.reporter.items.ItemSubmissionQueue readChanges
The item does not exist anymore Update{type=update, item=null, locator= ... }
Some types of Agents are not permanent items in your Client/Managed Master, e.g., Share Agents, Shared Cloud, or One-Shot-executors. These Agents are used for a short period and then removed. If the time between transient Agent creation and deletion is shorter than one minute, then analytics reporter queue would not find the item and would not be able to report the data to Operations Center Analytics.
0 Comments