- Agent provisioning fails and Jenkins logs show a stacktrace similar to the following:
okhttp3.internal.http2.ConnectionShutdownException at okhttp3.internal.http2.Http2Connection.newStream(Http2Connection.java:219) at okhttp3.internal.http2.Http2Connection.newStream(Http2Connection.java:205) [...] at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:404)
Or the following:
okhttp3.internal.http2.StreamResetException: stream was reset: PROTOCOL_ERROR at okhttp3.internal.http2.Http2Stream.takeHeaders(Http2Stream.java:158) at okhttp3.internal.http2.Http2Codec.readResponseHeaders(Http2Codec.java:131) at okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.java:88) [...] at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:411)
- CloudBees CI (CloudBees Core)
- CloudBees CI (CloudBees Core) on traditional platforms - Client controller
- CloudBees CI (CloudBees Core) on traditional platforms - Operations Center
- CloudBees Jenkins Platform - Client controller
- CloudBees Jenkins Platform - Operations Center
- CloudBees Jenkins Distribution
- Jenkins LTS
- Kubernetes Client API Plugin < 4.9.2
- Kubernetes Plugin < 1.26.0
This is caused by Java - more precisely the okhttp library used by the kubernetes client - that chooses the wrong protocol HTTP/2 to communicate with the Kubernetes API Server although it does not support it. There a couple of changes meant to bring support for HTTP/2 to Java 8 and Java clients that could explain this problem. But also a change in Kubernetes:
- JDK 8u252 bring support for HTTP/2 with JEP-244, see Jetty, ALPN & Java 8u252. This may cause again
okhttpto use HTTP/2 in some circumstances.
- In Kubernetes 1.17 and later,
okhttpseems to wrongly choose the HTTP/2 protocol to communicate with Kubernetes but the
kube-apiserverdoes not support it. Our experience is that this happens also with Openshift 4.2 and later.
Some solutions emerged to workaround that problem:
- In kubernetes-client 4.4.0, there is a system property
http2.disablethat can be used to disable HTTP/2 and can workaround the problem.
- In CloudBees CI 22.214.171.124, kubernetes-client 4.4.0 is available and the system property
http2.disablecan be used
Kubernetes Client maintainers addressed the problem directly:
- In Kubernetes Client 4.9.2, kubernetes client forces http1.1 to avoid the problem caused by JDK 8u252 fabric8/kubernetes-client #2212
- In CloudBees CI 126.96.36.199, kubernetes-client 4.9.2 is available which should definitely fix those problems.
Note: From our experience, JDK version 8u272 and later do not cause this http/2 problem anymore
The recommended solution is to upgrade CloudBees CI to version 188.8.131.52 or later. That version guarantee that the kubernetes client uses http1.1 when communicating with Kubernetes and prevent this issue from happening.
If an upgrade is not possible, there are potential workarounds.
CloudBees CI >= 184.108.40.206
If impacted, add the System Property
http2.disable=true to the startup of the Controller. See How to add Java arguments to Jenkins? for details.
CloudBees CI < 220.127.116.11
If impacted, upgrade the JDK to a version greater than 8u272.