TLS termination for Jenkins is setup in Jetty / Winstone
Running 2.277.3 / After upgrading CloudBees CI to version 2.277.3 or later:
Many items and global configurations cannot be saved from the UI or the REST API
Plugin HPI and Files cannot be uploaded
External service fail to POST payload to Jenkins endpoints
In Chrome, attempts to save configuration in Jenkins results in
org.eclipse.jetty, the following exception can be seen while reproducing the problem:
javax.net.ssl.SSLHandshakeException: Encrypted buffer max length exceeded [...] at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:540) at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:395) at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:161) [...]
- CloudBees CI (CloudBees Core) >= 2.277.3 and < 2.289.2
- CloudBees CI (CloudBees Core) on traditional platforms - Client Master >= 2.277.3 and < 2.289.2
- CloudBees CI (CloudBees Core) on traditional platforms - Operations Center >= 2.277.3 and < 2.289.2
- CloudBees Jenkins Platform - Client Master >= 2.277.3 and < 2.289.2
- CloudBees Jenkins Platform - Operations Center >= 2.277.3 and < 2.289.2
- Jetty 9.4.39.v20210325 (with TLS)
This issue happens in Jenkins version 2.277.3 to 2.289.2 (excluded) when Jetty / Winstone (the embedded servlet container for Jenkins) is set up to terminate TLS (commonly using
--httpsCertificate to provide a certificate, see Configuring HTTP).
It is caused by a known issue Jetty Issue #6082 in the Jetty 9.4.39.v20210325 initially introduced while fixing a security vulnerability. When Jetty receives POST requests (when a user saves a configuration in Jenkins), various issues related to compaction and calculation of buffer length now may cause Jetty to abort connection unexpectedly.
A backport has also been provided for the CloudBees CI 2.277 release line in version 2.277.4.4, that is version 2.277.4.3 packaged with the fixed Jetty / Winstone version.
- JENKINS-65280 - Update Winstone 5.16 which includes Jetty 9.4.39.v20210325 (cause in Jenkins 2.277.3)
- JENKINS-65624 - Webhook failures after upgrading jetty to 9.4.39.v20210325 in 2.277.3 (fix in Jenkins 2.289.2)
- Jenkins Security Advisory 2021-04-20 (cause)
- Jetty Issue #6072 / Jetty PR #6073 (cause)
- Jetty Issue #6082 / Jetty PR #6083 (fix)
The solution is to upgrade CloudBees CI to version 2.289.2.2 or later.
If impacted, but the upgrade to 2.289.2 is not an option, the workaround is to upgrade to version 2.277.4.4.