How to Connect to Remote SSH Slaves?

Issue

  • Connection of SSH slaves agents or nodes with different users.
  • Using Certificates (SSH Private Key) for connecting to slaves agents or nodes.

Environment

  • CloudBees Jenkins Enterprise
  • Dump slave/agents Launch method via SSH

Resolution

Ideally, we recommend that the same user jenkins is created on each Server where slaves agents or nodes will be deployed. However, it is possible to run slaves with a different user. Indeed, each slave acts as an SSH server and the Jenkins master acts as the SSH Client.

Requirements

That being said, here are the requirements to make it work:

  • The slave needs to authorize the master to access via SSH for a specific user.
  • The master needs a way to authenticate with a specific user to the slave via SSH.

So, a SSH server needs to be installed in the slaves agents or nodes (For instance, on Ubuntu openssh-server)

Before trying to connect a slave to a master. Make sure of the following:

a. From the Master, there is connectivity to the Slave

> ping $MASTER_IP

b. From the Slave, there is connectivity to the Master

> ping $SLAVE_IP

Expected output for “a” and “b” would be similar to:

PING 192.168.1.86 (192.168.1.86) 56(84) bytes of data.
64 bytes from 192.168.1.86: icmp_seq=1 ttl=64 time=0.417 ms
64 bytes from 192.168.1.86: icmp_seq=2 ttl=64 time=0.369 ms
64 bytes from 192.168.1.86: icmp_seq=3 ttl=64 time=0.419 ms
64 bytes from 192.168.1.86: icmp_seq=4 ttl=64 time=0.406 ms
^X64 bytes from 192.168.1.86: icmp_seq=5 ttl=64 time=0.587 ms
64 bytes from 192.168.1.86: icmp_seq=6 ttl=64 time=0.309 ms
64 bytes from 192.168.1.86: icmp_seq=7 ttl=64 time=0.278 ms
^C
--- 192.168.1.86 ping statistics ---
7 packets transmitted, 7 received, 0% packet loss, time 6000ms

c. From the Master, you can ssh to the Slave

> ssh $MASTER_IP

Expected output for “c” would be similar to:

ECDSA key fingerprint is SHA256:rX96v9WGIabHecNJusEYhXvENKyKe6GzQ/B/5ozRH9Q.
Are you sure you want to continue connecting (yes/no)?

Steps

In order to connect an SSH slave with the user $SLAVE_USER, follow these steps:

On the slave machine:

1) Log in as $SLAVE_USER. We create the SSH Keys with that user to ensure it has the required permissions later on when connecting to the slave from Jenkins.

> sudo su $SLAVE_USER

2) Create private and public SSH keys. The following command creates the private key $myslave_rsa and the public key $myslave_rsa.pub. It is recommended to store your keys under ~/.ssh/ so we move to that directory before creating the key pair.

> cd ~/.ssh/; ssh-keygen -t rsa -C "$myslave" -f "$myslave_rsa"

Note: To increase security, a passphrase is advisable to associate it to your Private Key.

3) Add the public SSH key to the list of authorized keys in the slave machine

> cat $myslave_rsa.pub >> ~/.ssh/authorized_keys

4) Copy the private SSH key from the Slave machine to your Master. Two options here:

  • 4.a Copy the content by using the OS clipboard (in linux, xclip). SSH private key should be similiar to this:
-----BEGIN RSA PRIVATE KEY-----
Proc-Type: 4,ENCRYPTED
DEK-Info: AES-128-CBC,DBA2145CD0590DE762D19A306B3AE233

7pzgoksx5ZivR/awfo2wX4zHROigUEzfFUl0jZryVK5Pzx3Ubr9gc7xLHQ8jo7p2
...
Ewuk0EAiN1qBZWutdBOxxUnopLsML7SFkzeSXuRjdHQJR76V5WBclnxkS2AK9y6q
-----END RSA PRIVATE KEY-----

  • 4.b Copy the full file. For instance, if you have installed scp on your Slave Server:
    > scp /home/$SLAVE_USER/$myslave_rsa $MASTER_USER@$MASTER_IP:/home/$MASTER_USER/.ssh

In Jenkins:

5) Go to the configuration of credentials under Manage Jenkins/Manage Credentials/Add Credentials ( JENKINS_URL/credential-store/) and choose SSH username with private key.

ScreenShot2014-07-15at11.23.28AM.png

According to the option selected on “step 4”, select now the proper Private Key type. So for “4.a” choose Enter Directly, whereas for “4.b” choose From a file on the Jenkins Master

6) For every slave that you want to connect using the user $SLAVE_USER, configure the slave under Manage Nodes/$NODE_NAME/configure ( $JENKINS_URL/computer/$NODE_NAME/configure) and use this user credential.

node-config

Note: Using that configuration, only one credential per user can be used. Several slaves can be launched for the same user on the same machine without any further configuration. To connect slaves hosted on a different machine using the same credential, the same SSH public Key needs to be added in the authorized_keys logged as $SLAVE_USER

7) On every slave you connected successfully, a similar output to this one is expected:

Evacuated stdout
Slave successfully connected and online
[07/15/16 17:49:14] [SSH] Opening SSH connection to 192.168.1.85:22.
[07/15/16 17:49:14] [SSH] Authentication successful.
[07/15/16 17:49:14] [SSH] The remote users environment is:
BASH=/bin/bash
BASHOPTS=cmdhist:complete_fullquote:extquote:force_fignore:hostcomplete:interactive_comments:progcomp:promptvars:sourcepath
BASH_ALIASES=()
BASH_ARGC=()
BASH_ARGV=()
BASH_CMDS=()
BASH_EXECUTION_STRING=set
BASH_LINENO=()
BASH_SOURCE=()
BASH_VERSINFO=([0]="4" [1]="3" [2]="42" [3]="1" [4]="release" [5]="x86_64-pc-linux-gnu")
BASH_VERSION='4.3.42(1)-release'
DIRSTACK=()
EUID=1000
GROUPS=()
HOME=/home/carlosrodlop
HOSTNAME=carlosrodlop-VirtualBox
HOSTTYPE=x86_64
IFS=$' \t\n'
LANG=es_ES.UTF-8
LOGNAME=carlosrodlop
MACHTYPE=x86_64-pc-linux-gnu
MAIL=/var/mail/carlosrodlop
OPTERR=1
OPTIND=1
OSTYPE=linux-gnu
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games
PIPESTATUS=([0]="0")
PPID=3777
PS4='+ '
PWD=/home/carlosrodlop
SHELL=/bin/bash
SHELLOPTS=braceexpand:hashall:interactive-comments
SHLVL=1
SSH_CLIENT='192.168.1.86 53746 22'
SSH_CONNECTION='192.168.1.86 53746 192.168.1.85 22'
TERM=dumb
UID=1000
USER=carlosrodlop
XDG_RUNTIME_DIR=/run/user/1000
XDG_SESSION_ID=3
_=']'
[07/15/16 17:49:14] [SSH] Checking java version of java
[07/15/16 17:49:14] [SSH] java -version returned 1.7.0_80.
[07/15/16 17:49:14] [SSH] Starting sftp client.
[07/15/16 17:49:14] [SSH] Copying latest slave.jar...
[07/15/16 17:49:14] [SSH] Copied 506,667 bytes.
Expanded the channel window size to 4MB
[07/15/16 17:49:14] [SSH] Starting slave process: cd "/opt/Labs/slaves/lab3/ssh3" && java  -jar slave.jar

On the slave machine:

On the hand, under the path specified in Remote root directory, slave.jar should has been copied. And those item/jobs which use this slave agent/node will allocate their workspace here after they are built.

NOTE:
Depending on the permissions of the node/slave machine’s user account, it may be necessary to set the jar cache directory - due to the issue JENKINS-18578
The default jar cache location is hardcoded to ~/.jenkins/cache/jars

To change the default setting:

In the slave configuration, under the Advanced method of Launch Option,
Place the following in the Suffix Start Slave Command -jar-cache <path to jar cache directory>

Make sure there’s a leading space so the parameters aren’t tacked directly onto the slave.jar itself.

References

Have more questions? Submit a request

7 Comments

  • 0
    Avatar
    João miguel Silva Ferreira

    This is the right way to connect to a slave. But there is a problem when you use this method for OSX slaves to run jobs that needs to use codesign. How can we have an OSX slave connecting via ssh keys and working properly with xcode build.

    Thank you 

  • 0
    Avatar
    Arnaud Heritier

    Hi João

    AFAIK for OSX to have a proper integration with all xcode tools you need to have a real user session and thus to use a JNLP slave. With SSH you can open a user session and thus the emulator and others tools like this aren't working

    BR

  • 0
    Avatar
    Milind Bangar

    Hi,

    Good one.. Helps...

    We are trying similar from a Jenkins Box (Linux) and trying to connect to a Windows box, wherein Windows box is ssh server.

    We are using Key based connectivity.

    The 2 servers are in different domain.

    Result : SSH connectivity cannot be established.

    Is there anything like above, is addressed.

    Regards

    Milind

  • 0
    Avatar
    Arnaud Heritier

    > The 2 servers are in different domain.

    What do you mean Milind ? On different windows domains ? But both servers can reach each others ? Did you try from the linux box command line using ssh to connect to the windows box ?

  • 0
    Avatar
    Milind Bangar

    Hi Arnaud,

    No, not different domains , but different  DMZs and traffic is routed via third party software.

    There is a trust between the domains (AD) to which the servers belong.

    We are able to RDP the server, but ssh does not work.

    Tried ssh from Linux box  as well, but did not work. The windows boxes in the same domain , able to ssh and run scripts remotely.

    Hope I was clear.

    Regards

    Milind

     

  • 0
    Avatar
    Arnaud Heritier

    Hi Milind

    > different  DMZs and traffic is routed via third party software

    Isn't it a firewall issue? Are you sure the SSH port is accessible ?

  • 0
    Avatar
    Pedro Cortes

    Hi,

     

      I have another trouble, I'm using Linux as master and AIX as slave, everything is set correctly, when I select the Launch agent in the slave, nothing happens, just an icon of a circle running is showed, if I go to the slave, the slave.jar file can be found, but in jenkins the next error is displayed: This node is offline because Jenkins failed to launch the slave agent on it. logs are empty too. Any idea what can be happening?

    Regards

     

     

Please sign in to leave a comment.