EC2 agent unable to connect to Controller with host key verification error

Issue

EC2 agents failing to connect to Controller.
You observe below error when connecting EC2 instance to controller.

The instance EC2 (aws-xxx-xxx-xxxx) - docker-2020-09-18 (i-052d99921a8f6fc42) has a blank console. Maybe the console is yet not available. If enough time has passed, consider changing the key verification strategy or the AMI used by one printing out the host key in the instance console
Failed to connect via ssh: There was a problem while connecting to X.X.X.X:22
Jun 19, 2020 10:54:07 AM hudson.plugins.ec2.EC2Cloud
INFO: The instance console is blank. Cannot check the key. The connection to EC2 (Jenkins) - Default Slave (i-052d99921a8f6fc42) is not allowed
Jun 19, 2020 10:54:07 AM hudson.plugins.ec2.EC2Cloud
INFO: Failed to connect via ssh: There was a problem while connecting to X.X.X.X:22
Jun 19, 2020 10:54:07 AM hudson.plugins.ec2.EC2Cloud

Environment

Resolution

The above error message is related to host key verification introduced in version 1.50.3 of the Amazon EC2 plugin.
When you set up a template for a Unix instance (Type AMI field), you can select a strategy to be used to guarantee the instance you’re connecting to is the expected one. There are
four options to select for this strategy under the Advanced… configuration, on the Host Key Verification Strategy field of every configured AMI.

  • Check New Hard: Check the key presented by the instance against the instance console and stores it to check subsequent connections. If the key is not printed on the console, the connection is not trusted. This is the default behavior for new AMIs.
  • Check New Soft: Check the key against the instance console and stores it to check subsequent connections. If the key is not printed on the console, the connection is trusted anyway. This is the default behavior for existing AMIs (upgrading from a previous plugin version). This avoids future attacks but cannot guarantee the instance is the right one if a man-in-the-middle attack has already been committed.
  • Accept New: Accept the key on first connection and stores it to check subsequent connections. This doesn’t try to check the key against the console as the check-new-soft strategy does
  • Off: Don’t check the host key on any connection

If the Connect by SSH Process field is checked, the equivalent host key verification options are:

  • check-new-hard = yes
  • check-new-soft = accept-new
  • accept-new = accept-new
  • off = no

This error usually occurs when using Check New Hard or Check New Soft options and if any one of the below requirement is not met.

  • The IAM credentials configured for the plugin should have "ec2:GetConsoleOutput" permission allowed.
  • The AMI used should print the key used. It’s a common behaviour, for example the Amazon Linux 2 AMI prints it out. You can consult the AMI documentation to figure it out.
  • The launch timeout should be long enough to allow the plugin to check the instance console. With this strategy, the plugin waits for the console to be available, which can take a few minutes. The Launch Timeout in seconds field should have a number to allow that, for example 600 (10 minutes). By default there is no timeout, so it’s safe.

The long term fix is to ensure your environment is setup as per above requirement.

Workaround

  • Use Accept New verification strategy where the first key is accepted blindly without verifying against the instance console output. This should only be used as a temporary fix, eventually you should setup your environment to use Check New Hard which is considered to be the safest option.

References

JENKINS-62724
Amazon EC2 Plugin

Have more questions?

0 Comments

Please sign in to leave a comment.