Any kind of communication issue between Jenkins master and a dedicated SSH agent.
This article describes how to collect the minimum required information for communication issues between a Jenkins master and a SSH agent so that it can be efficiently troubleshooted.
If the required data is bigger than 20 MB you will not be able to use ZenDesk to upload all the information. On this case we would like to encourage you to use our upload service in order to attach all the required information.
- CloudBees Jenkins Platform (CJP)
- Operations Center (OC)
- Client Master (CM)
- CloudBees Jenkins Enterprise (CJE)
- Time and Date the issue was exposed
- Support bundle
- Agent name
- Agent config.xml
- Agent logs
- Master logs
A support bundle from the Jenkins instance after the issue is exposed. Please, follow the KB below in case you don’t know how to generate a support bundle.
The name of the agent under Manage Jenkins -> Manage Nodes
It is possible to access to the agent config.xml from the UI. You just need to add at the end of the item URL
config.xml. i.e http://<JENKINS_URL>/computer/<AGENT_NAME>/config.xml
<slave> <name>unix-agent</name> <description/> <remoteFS>/home/vagrant/agent</remoteFS> <numExecutors>1</numExecutors> <mode>NORMAL</mode> <retentionStrategy class="hudson.slaves.RetentionStrategy$Always"/> <launcher class="hudson.plugins.sshslaves.SSHLauncher" plugin="firstname.lastname@example.org"> <host>127.0.0.1</host> <port>2222</port> <credentialsId>358aae70-309a-4e9e-a522-71a85b96920b</credentialsId> <jvmOptions>-Djava.util.logging.config.file=logging.properties</jvmOptions> <maxNumRetries>0</maxNumRetries> <retryWaitTime>0</retryWaitTime> </launcher> <label/> <nodeProperties> <org.jenkinsci.plugins.envinject.EnvInjectNodeProperty plugin="email@example.com"> <unsetSystemVariables>true</unsetSystemVariables> </org.jenkinsci.plugins.envinject.EnvInjectNodeProperty> </nodeProperties> </slave>
When writing this document (July, 2017) the remoting-3.8 which contains support for work directories was not still included in any Jenkins LTS release. When this enhancement is available in a LTS version, it will be possible to easily record the agent logs without almost any effort on the Jenkins admin side. Meanwhile this new feature is coming the easiest workaround is to use a Java logging to record the information in the agent file system.
In the Remote root directory of this agent create a file called
logging.properties with the following content.
# -Djava.util.logging.config.file=logging.properties .level = ALL handlers= java.util.logging.FileHandler java.util.logging.FileHandler.level = ALL java.util.logging.FileHandler.formatter = java.util.logging.SimpleFormatter java.util.logging.FileHandler.pattern=support/jenkins-ssh-agent-%u.log java.util.logging.FileHandler.limit = 10000000 java.util.logging.FileHandler.count = 10 javax.jms.connection.level = INFO hudson.level = INFO hudson.remoting.Channel.level = FINE hudson.remoting.FileSystemJarCache.level = INFO hudson.remoting.jnlp.level = FINE hudson.remoting.RemoteClassLoader.level = INFO jenkins.slaves.level = FINE hudson.slaves.level = FINE org.jenkinsci.remoting.engine.level = FINE jenkins.AgentProtocol.level = FINE
After this, in the agent configuration at Jenkins side under the Advanced section add the JVM Options
-Djava.util.logging.config.file=logging.properties. You will need to relaunch the agent to take the effects.
$REMOTE_ROOT_DIRECTORY/support* you will see files called
jenkins-ssh-agent-* which will contail the agent logs. This needs to be attached when the issue is exposed.
Remoting [work directory]https://github.com/jenkinsci/remoting/blob/7909e174bbe08eb9e750608225b35589d85a98b0/docs/workDir.md) is available starting from Remoting 3.8,which is available in Jenkins 2.68). Before this version there was no working directory concept in the library itself; all operations were managed by library users (e.g. Jenkins agent workspaces).
Attach the master logs around the time the issue is exposed that you can directly find in the UI under Manage Jenkins-> System Log -> All Jenkins Logs.
Notice that the idea is to correlate the master logs with the agent logs at the time the issue is exposed.