- Agent / Controller provisioning fails and Jenkins logs show a stacktrace similar to the following:
okhttp3.internal.http2.ConnectionShutdownException at okhttp3.internal.http2.Http2Connection.newStream(Http2Connection.java:219) at okhttp3.internal.http2.Http2Connection.newStream(Http2Connection.java:205) [...] at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:404)
Or the following:
okhttp3.internal.http2.StreamResetException: stream was reset: PROTOCOL_ERROR at okhttp3.internal.http2.Http2Stream.takeHeaders(Http2Stream.java:158) at okhttp3.internal.http2.Http2Codec.readResponseHeaders(Http2Codec.java:131) at okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.java:88) [...] at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:411)
- CloudBees CI (CloudBees Core)
- CloudBees CI (CloudBees Core) on modern cloud platforms - Managed Master
- CloudBees CI (CloudBees Core) on modern cloud platforms - Operations Center
- Kubernetes Client API Plugin < 4.9.2
- Kubernetes Plugin < 1.26.0
- Master Provisioning Plugin < 2.5.5
- CPLT2-5621: Fix “Is alpn-boot on the boot class path?”
- CPLT2-6044: revert CPLT2-5621
- fabric8/kubernetes-client #2212: kubernetes-client could not work with java 8u252
This is caused by Java - more precisely the okhttp library used by the kubernetes client - that chooses the wrong protocol HTTP/2 to communicate with the Kubernetes API Server although it does not support it. There a couple of changes meant to bring support for HTTP/2 to Java 8 and Java clients that could explain this problem. And also a change in Kubernetes:
- In CloudBees CI 188.8.131.52, the
alpn-boot.jarwas added to the classpath in CloudBees images to fix CPLT2-5621. That brings support for HTTP/2 and cause
okhttpto use HTTP/2 in some circumstances.
- In Kubernetes 1.17 and later,
okhttpseems to wrongly choose the HTTP/2 protocol to communicate with Kubernetes but the
kube-apiserverdoes not support it. Our experience is that this happens also with Openshift 4.2 and later.
Running version 184.108.40.206 of CloudBees CI in Kubernetes version older than 1.17 or Openshift version older than 4.2 is very risky. If Kubernetes/Openshift is upgraded, controller and agent provisioning would likely stop working and there is no workaround possible other than upgrading CloudBees CI. In public Cloud, clusters might be automatically upgraded by the Cloud operator (GKE, EKS, AKS, …).
Some solutions emerged to workaround that problem:
- In kubernetes-client 4.4.0, there is a system property
http2.disablethat can be used to disable HTTP/2 and can workaround the problem.
- In CloudBees CI 220.127.116.11, kubernetes-client 4.4.0 is available and the system property
http2.disablecan be used
- In CloudBees CI 18.104.22.168, the
alpn-boot.jarwas removed from the classpath in CloudBees images to fix the problem CPLT2-6044. The system property
http2.disableshould not be needed as this should prevent
okhttpfrom choosing HTTP/2 if it is not supported.
Kubernetes Client maintainers addressed the problem directly as it is discovered that the same behavior is caused by JDK 8u252 that brings support for HTTP/2 with JEP-244:
- In Kubernetes Client 4.9.2, kubernetes client forces http1.1 to avoid the problem caused by JDK 8u252 fabric8/kubernetes-client #2212
- In CloudBees CI 22.214.171.124, JDK 8u252 is installed but kubernetes-client 4.9.2 is available which should definitely fix those problems.
The recommended solution is to upgrade CloudBees CI to version 126.96.36.199 or later. That version guarantee that the kubernetes client uses http1.1 when communicating with Kubernetes and prevent that
CloudBees CI >= 188.8.131.52
If impacted, add the System Property
http2.disable=true to the startup of Operations Center and Managed Controllers. See How to add Java arguments to Jenkins on CloudBees CI Modern ? for details.
CloudBees CI <= 184.108.40.206
If impacted, there is no workaround. CloudBees CI must be upgraded to a version that has a workaround (220.127.116.11 or later) or a version that contains the solution (18.104.22.168 or later).