Helm deployment fails due to ingress security group conflicts

Issue

I am using AWS ALB for ingress in an AWS EKS environment. When I try to update the security groups annotation (alb.ingress.kubernetes.io/security-groups) with new security groups, the deployment fails with a message similar to the following:

Warning FailedBuildModel 12m (x68 over 13h) ingress Failed build model due to conflicting securityGroups: [sg-123 sg-abc sg-456] | [old-security-group]

Environment

Explanation

The alb.ingress.kubernetes.io/security-groups annotation has exclusive merge behavior, which is defined as follows:

Exclusive: such annotation should only be specified on a single Ingress within IngressGroup or specified with same value across all Ingresses within IngressGroup.

In CloudBees CI, ingress resources (CJOC and MM) are part of the same IngressGroup. Therefore, the security group annotation values must match for all ingress resources. When provisioning MMs, the default behavior is to propagate all CJOC ingress annotations to the ingress resources created for each MM. The issue is when updating the security groups via Helm (using OperationsCenter.Ingress.Annotations value) the new security groups are only applied to the CJOC ingress resource and not the MM ingress resources.

Resolution

The solution is to limit exclusive annotations to the CJOC ingress and apply non-exclusive annotations to each Managed Master.

Here is an example set of non-exclusive annotations that should be applied to each MM ingress resource:

alb.ingress.kubernetes.io/group.name: <groupName>
alb.ingress.kubernetes.io/listen-ports: '[{"HTTP":80}, {"HTTPS":443}]'
alb.ingress.kubernetes.io/target-type: ip
kubernetes.io/ingress.class: alb

Note: Any custom (non-exclusive) ingress annotations you’ve applied in your environment should be specified as well.

To prevent propagation of CJOC ingress annotations, specify the non-exclusive annotations under CJOC > Configure Master Provisioning > Ingress Annotations (replacing with the group name defined in your environment).

This will ensure all newly provisioned MMs will not include the security group annotation. Existing MMs need to be restarted for the annotations to be applied.

Tested product/plugin versions

References

Have more questions?

0 Comments

Please sign in to leave a comment.