- I am running the Datadog Tracing Agent and my Jenkins JVM heap utilization is unexpectedly climbing over time.
- CloudBees CI (CloudBees Core) on Traditional Platforms
- CloudBees Jenkins Distribution
- CloudBees Jenkins Enterprise (CJE)
- CloudBees Jenkins Enterprise - Managed controller (CJE-MM)
- CloudBees Jenkins Enterprise - Operations Center (CJE-OC)
- CloudBees Jenkins Team (CJT)
- CloudBees Jenkins Platform - Client controller (CJP-CM)
- CloudBees Jenkins Platform - Operations Center (CJP-OC)
- Jenkins LTS
This issue has since been resolved by Datadog as noted in this GitHub Commit
The fixed version of 0.20 was released December 13, 2018 per the Release Notes. It is recommended to upgrade to the latest version of the agent to ensure you are taking advantage of improvements and new features.
Upon initial observation of a client provided heap dump, a 5GB+ chunk of memory was observed as a potential leak suspect.
Further investigation proved that the Datadog Java Agent was leaking memory from the
WeakConcurrentMap to the tune of 5GB, causing an OOM event which ultimately brought down Jenkins.
One instance of "com.blogspot.mydailyjava.weaklockfree.WeakConcurrentMap" loaded by "datadog.trace.bootstrap.DatadogClassLoader @ 0x1c0667058" occupies 5,450,446,944 (22.28%) bytes. The memory is accumulated in one instance of "java.util.concurrent.ConcurrentHashMap$Node" loaded by "<system class loader>"
The Datadog Tracing Agent is invoked through JVM Arguments: