Issue
- The master thread dump shows hundreds / thousands of
OkHttp ConnectionPool
threads similar to:
"OkHttp ConnectionPool" id=679 (0x2a7) state=TIMED_WAITING cpu=0%
- waiting on <0x09293902> (a okhttp3.ConnectionPool)
- locked <0x09293902> (a okhttp3.ConnectionPool)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:460)
at okhttp3.ConnectionPool$1.run(ConnectionPool.java:67)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Locked synchronizers: count = 1
- java.util.concurrent.ThreadPoolExecutor$Worker@2b7a6107
- The master Jenkins logs shows socket exception like the following:
Dec 12, 2018 1:22:08 AM io.fabric8.kubernetes.client.dsl.internal.ExecWebSocketListener onFailure
SEVERE: Exec Failure: HTTP:0. Message:No response
java.net.SocketException: Socket closed
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
at sun.security.ssl.InputRecord.read(InputRecord.java:503)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:983)
at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:940)
at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
at okio.Okio$2.read(Okio.java:139)
[...]
Environment
- CloudBees CI (CloudBees Core) on modern cloud platforms - Managed controller
- CloudBees CI (CloudBees Core) on modern cloud platforms - Operations Center
- CloudBees CI (CloudBees Core) on traditional platforms - Client controller
- CloudBees CI (CloudBees Core) on traditional platforms - Operations Center
- CloudBees Jenkins Team
- CloudBees Jenkins Platform - Client controller
- CloudBees Jenkins Platform - Operations Center
- Jenkins LTS
- Kubernetes Plugin < 1.13.6
Related Issue(s)
Resolution
This is caused by a thread leak in the Kubernetes Plugin. Every time plugin needs to connect to the kubernetes server, a new client object is created.
The issue is captured by JENKINS-54770 that has been fixed in version 1.13.6 of the Kubernetes Plugin.
Solution
The solution is to upgrade the Kubernetes Plugin to version 1.13.6 or later.
0 Comments