Memory Leak in DataDog Agent

Issue

  • I am running the DataDog Tracing Agent and my Jenkins JVM heap utilization is unexpectedly climbing over time.

indicative_of_memory_leak.png

Environment

Resolution

This issue has since been acknowledged by DataDog as notated in this GitHub Commit

However it is unclear when the issue will be released at the time of writing this, therefore it is recommended to remove the agent until this commit is referenced in the release notes.

Explanation

datadog_1.png

Upon initial observation of a client provided heap dump, a 5GB+ chunk of memory was observed as a potential leak suspect.

datadog_2.png

Further investigation proved that the DataDog Java Agent was leaking memory from the WeakConcurrentMap to the tune of 5GB, causing an OOM event which ultimately brought down Jenkins.

One instance of "com.blogspot.mydailyjava.weaklockfree.WeakConcurrentMap" loaded by "datadog.trace.bootstrap.DatadogClassLoader @ 0x1c0667058" occupies 5,450,446,944 (22.28%) bytes. The memory is accumulated in one instance of "java.util.concurrent.ConcurrentHashMap$Node[]" loaded by "<system class loader>"

The DataDog Tracing Agent is invoked through JVM Arguments: -javaagent:/etc/datadog-agent/dd-java-agent.jar

Have more questions? Submit a request

0 Comments

Please sign in to leave a comment.