- I am running the DataDog Tracing Agent and my Jenkins JVM heap utilization is unexpectedly climbing over time.
- CloudBees Core on Traditional Platforms
- CloudBees Jenkins Distribution
- CloudBees Jenkins Enterprise (CJE)
- CloudBees Jenkins Enterprise - Managed Master (CJE-MM)
- CloudBees Jenkins Enterprise - Operations Center (CJE-OC)
- CloudBees Jenkins Team (CJT)
- CloudBees Jenkins Platform - Client Master (CJP-CM)
- CloudBees Jenkins Platform - Operations Center (CJP-OC)
- Jenkins LTS
This issue has since been acknowledged by DataDog as notated in this GitHub Commit
However it is unclear when the issue will be released at the time of writing this, therefore it is recommended to remove the agent until this commit is referenced in the release notes.
Upon initial observation of a client provided heap dump, a 5GB+ chunk of memory was observed as a potential leak suspect.
Further investigation proved that the DataDog Java Agent was leaking memory from the
WeakConcurrentMap to the tune of 5GB, causing an OOM event which ultimately brought down Jenkins.
One instance of "com.blogspot.mydailyjava.weaklockfree.WeakConcurrentMap" loaded by "datadog.trace.bootstrap.DatadogClassLoader @ 0x1c0667058" occupies 5,450,446,944 (22.28%) bytes. The memory is accumulated in one instance of "java.util.concurrent.ConcurrentHashMap$Node" loaded by "<system class loader>"
The DataDog Tracing Agent is invoked through JVM Arguments: