How to Connect to Remote SSH Agents?

Issue

  • How to do the initial connection of SSH agents (formerly called slaves) to Jenkins, using SSH keys.

Environment

Resolution

Ideally, we recommend that the same user account jenkins is created on each Server that will be connected to your Jenkins Master. However, it is possible to run agents with a different user. Indeed, each agent acts as an SSH server and the Jenkins Master acts as the SSH Client.

Requirements

The master needs to be able to connect to the agent via ssh, with the Private key known by the Master, and the corresponding public key is put in the agent’s ~/.ssh/authorized_keys file.

So, a SSH server needs to be installed in the agent (For instance, on Ubuntu: openssh-server)

Before trying to connect an agent to a master. Make sure that when you login to the console of the Master, you can ssh to the agent:

  > ssh $AGENT_HOSTNAME

Expected output would be similar to:

ECDSA key fingerprint is SHA256:rX96v9WGIabHecNJusEYhXvENKyKe6GzQ/B/5ozRH9Q.
Are you sure you want to continue connecting (yes/no)?

Steps

In order to connect an SSH agent with the user $AGENT_USER, follow these steps:

On the agent machine:

1) Log in to the agent machine as $AGENT_USER:

  > sudo su $AGENT_USER

2) Create private and public SSH keys. The following command creates the private key jenkinsAgent_rsa and the public key jenkinsAgent_rsa.pub. It is recommended to store your keys under ~/.ssh/ so we move to that directory before creating the key pair.

  > cd ~/.ssh/; ssh-keygen -t rsa -C "Jenkins agent key" -f "jenkinsAgent_rsa"

Note: To increase security, a passphrase is advisable to associate it to your Private Key.

3) Add the public SSH key to the list of authorized keys on the agent machine

  > cat jenkinsAgent_rsa.pub >> ~/.ssh/authorized_keys

4) Copy the private SSH key (~/.ssh/jenkinsAgent_rsa) from the agent machine to your OS clipboard (eg: xclip, pbcopy, or ctrl-c). The SSH private key should be similiar to this:

-----BEGIN RSA PRIVATE KEY-----
...
-----END RSA PRIVATE KEY-----

In Jenkins:

5) Go to the Build Executor Status link on the Jenkins dashboard:

Build Executor Status

6) Choose New Node, and Permanent agent type:

New Node
Permanent Agent

7) Fill in the Remote root directory with a path the user on the agent is allowed to write to, set the Host value to the hostname of the agent, and press the Add button for Credentials:

ssh agent configuration

8) Choose SSH Username with private key, the Username is the user account on the agent machine (usually jenkins), and choose Private Key -> Enter directly and paste the key from your OS clipboard, and give a useful Description for this credential.

ssh agent credentials

Note: If you are setting up multiple agents, you can use the same public/private key for all agents (the public key would be put into ~/.ssh/authorized_keys on every agent, and the private key added as a credential on the Master can be re-used for every new agent)

9) Save the configuration and the agent connection log should be similar to:

Oct 2, 2018 3:13:43 PM [ssh] Opening connection to ssh-agent:22 as jenkins (agent ssh key)
Oct 2, 2018 3:13:52 PM [ssh] Authenticating...
Oct 2, 2018 3:13:56 PM [ssh] Verifying server host key...
Oct 2, 2018 3:13:56 PM [ssh] ECDSA key fingerprint is SHA256:...
Oct 2, 2018 3:13:56 PM [ssh] Server host key verified
Oct 2, 2018 3:14:00 PM [ssh] Authentication successful.
Oct 2, 2018 3:14:00 PM [ssh] Checking for header junk...
Oct 2, 2018 3:14:01 PM [ssh] No header junk found.
Oct 2, 2018 3:14:01 PM [ssh] Inspecting operating system...
Oct 2, 2018 3:14:02 PM [ssh] Operating system: UNKNOWN
Oct 2, 2018 3:14:02 PM [ssh] Remote SSH server: SSH-2.0-OpenSSH_7.4p1 Debian-10+deb9u3
Oct 2, 2018 3:14:02 PM [ssh] Verifying Java...
Oct 2, 2018 3:14:10 PM [ssh] Trying 'java' as a java command...
java -version returned 1.8.0.
Oct 2, 2018 3:14:11 PM [ssh] Verifying slave.jar...
Oct 2, 2018 3:14:12 PM [ssh] Copying latest slave.jar...
Oct 2, 2018 3:14:12 PM [ssh] Copied 32,768 bytes.
...
Oct 2, 2018 3:14:14 PM [ssh] $ cd '/home/jenkins/' && java  -jar slave.jar 
<===[JENKINS REMOTING CAPACITY]===>channel started
Remoting version: 3.25
This is a Unix agent
Evacuated stdout
Agent successfully connected and online

On the agent machine:

Under the path specified in Remote root directory, the slave.jar would have been copied, and builds for jobs that run on this agent will allocate their workspace under ${remoteRootDirectory}/workspace.

NOTE:
Depending on the permissions of the home directory for the agent machine’s user account, it may be necessary to set the jar cache directory - due to the issue JENKINS-18578
The default jar cache location is hardcoded to ~/.jenkins/cache/jars

To change the default setting:

In the agent configuration, under the Advanced method of Launch Option,
Place the following in the Suffix Start Agent Command -jar-cache <path to jar cache directory>

Make sure there’s a leading space so the parameters aren’t tacked directly onto the slave.jar itself.

References

Have more questions? Submit a request

8 Comments

  • 0
    Avatar
    João miguel Silva Ferreira

    This is the right way to connect to a slave. But there is a problem when you use this method for OSX slaves to run jobs that needs to use codesign. How can we have an OSX slave connecting via ssh keys and working properly with xcode build.

    Thank you 

  • 1
    Avatar
    Arnaud Heritier

    Hi João

    AFAIK for OSX to have a proper integration with all xcode tools you need to have a real user session and thus to use a JNLP slave. With SSH you can open a user session and thus the emulator and others tools like this aren't working

    BR

  • 0
    Avatar
    Milind Bangar

    Hi,

    Good one.. Helps...

    We are trying similar from a Jenkins Box (Linux) and trying to connect to a Windows box, wherein Windows box is ssh server.

    We are using Key based connectivity.

    The 2 servers are in different domain.

    Result : SSH connectivity cannot be established.

    Is there anything like above, is addressed.

    Regards

    Milind

  • 0
    Avatar
    Arnaud Heritier

    > The 2 servers are in different domain.

    What do you mean Milind ? On different windows domains ? But both servers can reach each others ? Did you try from the linux box command line using ssh to connect to the windows box ?

  • 0
    Avatar
    Milind Bangar

    Hi Arnaud,

    No, not different domains , but different  DMZs and traffic is routed via third party software.

    There is a trust between the domains (AD) to which the servers belong.

    We are able to RDP the server, but ssh does not work.

    Tried ssh from Linux box  as well, but did not work. The windows boxes in the same domain , able to ssh and run scripts remotely.

    Hope I was clear.

    Regards

    Milind

     

  • 0
    Avatar
    Arnaud Heritier

    Hi Milind

    > different  DMZs and traffic is routed via third party software

    Isn't it a firewall issue? Are you sure the SSH port is accessible ?

  • 0
    Avatar
    Pedro Cortes

    Hi,

     

      I have another trouble, I'm using Linux as master and AIX as slave, everything is set correctly, when I select the Launch agent in the slave, nothing happens, just an icon of a circle running is showed, if I go to the slave, the slave.jar file can be found, but in jenkins the next error is displayed: This node is offline because Jenkins failed to launch the slave agent on it. logs are empty too. Any idea what can be happening?

    Regards

     

     

  • 0
    Avatar
    Carlos leonardo camilo Vargas Huamán

    Thank you very much for the great information. 

    I just have a question. I currently have a macOS High Sierra(my localhost machine). It was really amazing to be able to connect from my Jenkins master node to my macOS slave node through the Launch Agent via Java Web Start launch method. But I actually wanted to connect to my Unix slaves via SSH but I didn't get any result. In fact I can say that is so difficult to do it when working with macOS.

    I've tried a lot of things:

    1. I've enabled the Preferences -> Sharing -> Remote Login -> Allow access for : All users. When I simply open my terminal: ssh username@private_ip

    I always get ssh_exchange_identification: Connection closed by remote host message. I've used the -v parameter the see the log and I always get errors like related to key_load_public. In conclusion, starting from this error all the others attempts I make have no results :(

    In a nutshell, the only way to connect o macOS slave is via Launch Agent via Java Web Start? 

Please sign in to leave a comment.