Issue

• The CloudBees Analytics administrative monitor is active and showing the following message:

  The CloudBees analytics system is having problems processing the generated data, this may affect the system performance.If the system is offline it is recommended to disable analytics. Otherwise please contact support for instructions on how to customize the CloudBees analytics queue size.
  

Environment

• CloudBees CI (CloudBees Core) on modern cloud platforms - Managed controller
• CloudBees CI (CloudBees Core) on modern cloud platforms - Operations Center
• CloudBees CI (CloudBees Core) on traditional platforms - Client controller
• CloudBees CI (CloudBees Core) on traditional platforms - Operations Center
• CloudBees Jenkins Platform - Client controller
• CloudBees Jenkins Platform - Operations Center
• [CloudBees Analytics plugin] > 1.38 and < 1.41

Related Issue(s)

• BEE-16827 : Analytics monitor reporting queue full is being displayed way too often

Explanation

Before version 1.38, the CloudBees Analytics plugin uses an unbounded queue to process the instance telemetry data. This could cause a memory leak in environments that cannot access the telemetry server api at api.segment.io (See BEE-2824 in CloudBees Analytics version 1.38).

Starting from version 1.38, the events queue is now limited, to avoid the memory leak while still dealing with backpressure. Additionally, an administrative monitor has been added to inform a Jenkins administrator when that queue is full, sign of pressure in the processing of data. HOWEVER, the default limit is small AND when a large amount of events needs to be enqueued rapidly, the queue is not flushed until it is full. This can cause the monitor to be shown, even though the overload is not real.
That being said, the monitor might also show up, as designed, because of an overload. In many cases due to a lack of connectivity to api.segment.io.

In version 1.41, the queue settings have been adjusted to ensure that the monitor reports a true overload.

Solution

Check connectivity to api.segment.io

If api.segment.io cannot be reached, task are just filling up the queue and the monitor will show up.

Ensure that api.segment.io can be reached from the instance environment including through a configured HTTP proxy. See Required URLs to allowlist. The URL https://api.segment.io/v1/p and https://api.segment.io/v1/t to test the connectivity.

Note: if for any reason, the instance cannot be allowed access to api.segment.io, disable the data collection for the CloudBees Analytics Plugin until it can

Upgrade CloudBees CI

The problem has been fixed in CloudBees Analytics version 1.41 that is available under CAP since CloudBees CI 2.332.3.2.

Upgrade CloudBees CI to version 2.332.3.2 or later.

Workaround

Assuming api.segment.io is reachable from the instance, a workaround could be to increase the queue size with the system property com.cloudbees.analytics.driver.segment.SegmentQueue.queue=<numberOfItems>, that would require a restart. See How to add Java arguments to Jenkins?.

CloudBees Analytics 1.41

The queue size default to 250. It may be increased. Though if this keeps happening despite increasing the queue size, please contact Support for guidance as it could well be due to performances problems on the controller.

CloudBees Analytics 1.38

The queue size default to 50. The monitor may still activate despite increasing the queue though because of the unreliable flushing mechanism fixed in 1.41.