Issue

The Sidecar Injector does not inject the configuration data and the Sidecar Injector pod container logs shows the following error:
```
TLS handshake error from <IP>:<PORT>: remote error: tls: bad certificate
```

Environment

CloudBees CI (CloudBees Core) on Modern Cloud Platforms
sidecar injector v1 (manual installation scripts < 2.164.2.1)
sidecar injector v2 < 2.1.2

Explanation

Sidecar injector (and Admission Controllers in general) needs a keypair in order to secure its access from the Kubernetes API Server using TLS:

TLS key and certificate used to set up TLS for the webhook server (i.e. the sidecar injector)
CA Certificate / Public Key used by the MutatingWebhookConfiguration to validate the webhook server (i.e. the sidecar injector) certificate

The TLS key and certificate used to set up TLS for the Sidecar Injector server are generated when deploying the Sidecar Injector (from helm chart or scripts). A corresponding TLS Secret is created from the key and certificate generated and this secret is mounted to the Sidecar Injector deployment:

In Sidecar Injector v1, this used to be done by the script webhook-create-signed-cert.sh and the TLS secret name is sidecar-injector-webhook-certs by default
In Sidecar Injector v2, this is done by a Kubernetes Job and the TLS secret name is cloudbees-sidecar-injector by default

The CA Certificate / Public Key used by the Kubernetes API Server to validate the Sidecar Injector server certificate is also automatically set when deploying the Sidecar Injector (from helm chart or scripts). It is added as the .webhooks[].clientConfig.caBundle in the MutatingWebhookConfiguration of the Sidecar Injector.

The TLS handshake error from <IP>:<PORT>: remote error: tls: bad certificate may happens in the following scenario:

If Sidecar Injector has been working for a while and this just started happening, most likely the TLS certificate of the Sidecar Injector server expired.
If this happened when deploying the Sidecar Injector, most likely the Sidecar Injector deployment produced a bad / wrong certificate or is not able to request one.

Note the following chronology:

Sidecar Injector until version 2.1.0 relies on kubernetes Certificate Signing Request (CSR) that can cause problems in environment where the Kubernetes API server CA differs from the Cluster Signing Certificate:
The Sidecar Injector TLS key and certificate are retrieved (rather issued) by Kubernetes through a Certificate Signing Request
The CA Certificate / Public Key used to validate the Sidecar Injector server certificate the Kubernetes API Server CA (mounted in all pods)
Sidecar Injector v1 scripts prior to version 2.164.2.1 have a known issue in the way the CA AuthoritycaBundle to use in the MutatingWebhookConfiguration is retrieved.
Sidecar Injector v2 starting from version 2.0.7 introduce a feature that automatically renew the Server TLS certificate before it expires.
Sidecar Injector v2 starting from version 2.1.0 does not rely on Certificate Signing Request anymore and creates its own self-signed certificate whose public key is injected in the MutatingWebhookConfiguration object so that the API server can contact it securely.
Sidecar Injector v2 starting version 2.1.2 guarantee that the Sidecar Injector server - i.e. the pod - is restarted upon certificate renewal (that should happen a month before its expiration).

Related Issues

CPLT2-5479: sidecar-injector setup doesn’t work if CA is not provided in configmap - fixed in 2.164.1.2 packaged scripts
CPLT2-6615: Sidecar should handle CSR renewal automatically - fixed in 2.0.5
CPLT2-6774: Compatibility fixes for Kubernetes 1.18+ / Remove CSR Handling - fixed in 2.1.0
CPLT2-6981: Certificate renewal is causing errors until sidecar injector is restarted - fixed in 2.1.2

Resolution

The recommended solution is to upgrade or reinstall Sidecar Injector deployment to version 2.1.2 or later:

If running Sidecar Injector version 1.x, delete the Sidecar Injector with kubectl delete -f sidecar-injector.yaml -n <namespace> and install Sidecar Injector 2.1.2 or later following Using self-signed certificates in CloudBees CI on Kubernetes.
If running Sidecar Injector version 2.x, upgrade the Sidecar Injector to version 2.1.2 or later

Note: If upgrading from version 2.0.7 or later, the CronJob that manages certificates will not be automatically started upon upgrade. In that case, start a new Job based on it with a command like kubectl create job --from=cronjob/cloudbees-sidecar-injector -n sidecar-injector.

Troubleshooting

To troubleshoot such issues, check the Sidecar Injector TLS certificate (for the TLS secret) and the .clientConfig.caBundle of the MutatingWebhookConfiguration.

Issue with Certificate Expiration

This issue may appear if the certificate used to set up TLS for the Sidecar Injector server have expired. To confirm this, check the expiry date of this certificate:

In Sidecar Injector v1: check the cert.pem in secret sidecar-injector-webhook-certs:

kubectl get secret sidecar-injector-webhook-certs -o "go-template={{ index .data \"cert.pem\" }}" | base64 --decode | openssl x509 -enddate -noout

In Sidecar Injector v2: check the tls.crt in secret cloudbees-sidecar-injector:

kubectl get secret cloudbees-sidecar-injector -o "go-template={{ index .data \"tls.crt\" }}" | base64 --decode | openssl x509 -enddate -noout

Issue with Deployment

This can happen if the Kubernetes API Server certificate is different from the Cluster Signing Certificate. To confirm this, check the issuer and CN of the Sidecar Injector TLS certificate and the MutatingWebhookConfiguration Public Key: