Issue
Any kind of communication issue between Jenkins master and a dedicated SSH agent.
Required Data SSH dedicated agents
This article describes how to collect the minimum required information for communication issues between a Jenkins master and a SSH agent so that it can be efficiently troubleshooted.
If the required data is bigger than 50 MB you will not be able to use ZenDesk to upload all the information. On this case we would like to encourage you to use our upload service in order to attach all the required information.
Environment
- CloudBees Jenkins Platform (CJP)
- Operations Center (OC)
- Client Master (CM)
- CloudBees Jenkins Enterprise (CJE)
Required Data check list
- Time and Date the issue was exposed
- Support bundle
- Agent name
- Agent config.xml
- Agent logs
- Master logs
Support bundle
A support bundle from the Jenkins instance after the issue is exposed. Please, follow the KB below in case you don’t know how to generate a support bundle.
Agent name
The name of the agent under Manage Jenkins -> Manage Nodes
Agent config.xml
It is possible to access to the agent config.xml from the UI. You just need to add at the end of the item URL config.xml. i.e http://<JENKINS_URL>/computer/<AGENT_NAME>/config.xml
<slave>
<name>unix-agent</name>
<description/>
<remoteFS>/home/vagrant/agent</remoteFS>
<numExecutors>1</numExecutors>
<mode>NORMAL</mode>
<retentionStrategy class="hudson.slaves.RetentionStrategy$Always"/>
<launcher class="hudson.plugins.sshslaves.SSHLauncher" plugin="ssh-slaves@1.11">
<host>127.0.0.1</host>
<port>2222</port>
<credentialsId>358aae70-309a-4e9e-a522-71a85b96920b</credentialsId>
<jvmOptions>-Djava.util.logging.config.file=logging.properties</jvmOptions>
<maxNumRetries>0</maxNumRetries>
<retryWaitTime>0</retryWaitTime>
</launcher>
<label/>
<nodeProperties>
<org.jenkinsci.plugins.envinject.EnvInjectNodeProperty plugin="envinject@1.93.1">
<unsetSystemVariables>true</unsetSystemVariables>
</org.jenkinsci.plugins.envinject.EnvInjectNodeProperty>
</nodeProperties>
</slave>
Agent logs
When writing this document (July, 2017) the remoting-3.8 which contains support for work directories was not still included in any Jenkins LTS release. When this enhancement is available in a LTS version, it will be possible to easily record the agent logs without almost any effort on the Jenkins admin side. Meanwhile this new feature is coming the easiest workaround is to use a Java logging to record the information in the agent file system.
In the Remote root directory of this agent create a file called logging.properties with the following content.
# -Djava.util.logging.config.file=logging.properties
.level = ALL
handlers= java.util.logging.FileHandler
java.util.logging.FileHandler.level = ALL
java.util.logging.FileHandler.formatter = java.util.logging.SimpleFormatter
java.util.logging.FileHandler.pattern=support/jenkins-ssh-agent-%u.log
java.util.logging.FileHandler.limit = 10000000
java.util.logging.FileHandler.count = 10
javax.jms.connection.level = INFO
hudson.level = INFO
hudson.remoting.Channel.level = FINE
hudson.remoting.FileSystemJarCache.level = INFO
hudson.remoting.jnlp.level = FINE
hudson.remoting.RemoteClassLoader.level = INFO
jenkins.slaves.level = FINE
hudson.slaves.level = FINE
org.jenkinsci.remoting.engine.level = FINE
jenkins.AgentProtocol.level = FINE
After this, in the agent configuration at Jenkins side under the Advanced section add the JVM Options -Djava.util.logging.config.file=logging.properties. You will need to relaunch the agent to take the effects.
Now, under $REMOTE_ROOT_DIRECTORY/support* you will see files called jenkins-ssh-agent-* which will contail the agent logs. This needs to be attached when the issue is exposed.
Remoting work directory
Remoting work directory is available starting from Remoting 3.8,which is available in Jenkins 2.68). Before this version there was no working directory concept in the library itself; all operations were managed by library users (e.g. Jenkins agent workspaces).
Master logs
Attach the master logs around the time the issue is exposed that you can directly find in the UI under Manage Jenkins-> System Log -> All Jenkins Logs.
Notice that the idea is to correlate the master logs with the agent logs at the time the issue is exposed.
1 Comments