EC2 agents failing to connect to Controller.
You observe below error when connecting EC2 instance to controller.
The instance EC2 (aws-xxx-xxx-xxxx) - docker-2020-09-18 (i-052d99921a8f6fc42) has a blank console. Maybe the console is yet not available. If enough time has passed, consider changing the key verification strategy or the AMI used by one printing out the host key in the instance console Failed to connect via ssh: There was a problem while connecting to X.X.X.X:22 Jun 19, 2020 10:54:07 AM hudson.plugins.ec2.EC2Cloud INFO: The instance console is blank. Cannot check the key. The connection to EC2 (Jenkins) - Default Slave (i-052d99921a8f6fc42) is not allowed Jun 19, 2020 10:54:07 AM hudson.plugins.ec2.EC2Cloud INFO: Failed to connect via ssh: There was a problem while connecting to X.X.X.X:22 Jun 19, 2020 10:54:07 AM hudson.plugins.ec2.EC2Cloud
- CloudBees CI (CloudBees Core)
- CloudBees CI (CloudBees Core) on modern cloud platforms - Managed Master
- CloudBees CI (CloudBees Core) on modern cloud platforms - Operations Center
- CloudBees CI (CloudBees Core) on traditional platforms - Client Master
- CloudBees CI (CloudBees Core) on traditional platforms - Operations Center
- CloudBees Jenkins Platform - Client Master
- CloudBees Jenkins Platform - Operations Center
- CloudBees Jenkins Distribution
- Jenkins LTS
The above error message is related to host key verification introduced in version
1.50.3 of the Amazon EC2 plugin.
When you set up a template for a Unix instance (Type AMI field), you can select a strategy to be used to guarantee the instance you’re connecting to is the expected one. There are
four options to select for this strategy under the Advanced… configuration, on the Host Key Verification Strategy field of every configured AMI.
- Check New Hard: Check the key presented by the instance against the instance console and stores it to check subsequent connections. If the key is not printed on the console, the connection is not trusted. This is the default behavior for new AMIs.
- Check New Soft: Check the key against the instance console and stores it to check subsequent connections. If the key is not printed on the console, the connection is trusted anyway. This is the default behavior for existing AMIs (upgrading from a previous plugin version). This avoids future attacks but cannot guarantee the instance is the right one if a man-in-the-middle attack has already been committed.
- Accept New: Accept the key on first connection and stores it to check subsequent connections. This doesn’t try to check the key against the console as the check-new-soft strategy does
- Off: Don’t check the host key on any connection
If the Connect by SSH Process field is checked, the equivalent host key verification options are:
- check-new-hard = yes
- check-new-soft = accept-new
- accept-new = accept-new
- off = no
This error usually occurs when using
Check New Hard or
Check New Soft options and if any one of the below requirement is not met.
- The IAM credentials configured for the plugin should have
- The AMI used should print the key used. It’s a common behaviour, for example the Amazon Linux 2 AMI prints it out. You can consult the AMI documentation to figure it out.
- The launch timeout should be long enough to allow the plugin to check the instance console. With this strategy, the plugin waits for the console to be available, which can take a few minutes. The Launch Timeout in seconds field should have a number to allow that, for example 600 (10 minutes). By default there is no timeout, so it’s safe.
The long term fix is to ensure your environment is setup as per above requirement.
Accept Newverification strategy where the first key is accepted blindly without verifying against the instance console output. This should only be used as a temporary fix, eventually you should setup your environment to use
Check New Hardwhich is considered to be the safest option.