Issue
- There is apparently a connectivity issue between two components (Be it CJOC, a Client Master, an agent, or anything requiring connectivity).Looking at the support bundle and informations provide can lead to think the issue could be coming from something in between those components.
Prerequisite
IMPORTANT: machines must be synchronized in time for us to be able to analyze them.
Required Data Network Dump
This article describes how the collect a Network Dump which helps to troubleshoot Connectivity issues between nodes in your Cloudbees Core cluster.
If the required data is bigger than 20 MB, you will not be able to use ZenDesk to upload all the information. On this case, we would like to encourage you to use our upload service in order to attach all the required information.
Environment
- CloudBees Core
- Jenkins LTS
- Linux
Required Data check list
- Network Dump
- Time around the issue is exposed
- A support bundle created when the issue is exposed
Network Dump
1.- Run the Network Dump
Network dumps need to be run in background mode from each of the nodes which are trying to connect. For example, if you have issues between a Operation Center, a Client Master and one agent, you want to run this 3 times for each node/host.
The following list of commands need to be run it before the issue occurs again.
Indeed: this is going to set up rotating dump files and dump the network traffic there. So, it will take up to ~1 GB of disk on each node (100 x ~10 MB), and not more.
$ export CASEID=CHANGE_IT_TO_THE_ZENDESK_ISSUE_ID
$ mkdir zd-$CASEID-tcpdump-$(hostname)
$ cd zd-$CASEID-tcpdump-$(hostname)
$ nohup sudo tcpdump -i eth0 -s 1522 -C 10000000 -w tcpdump.cap -W 100 -Z root &
2.- Collect the Network Dump
When the issue is exposed again, zip the directory you created above for each of the nodes.
$ zip -9 -r zd-$CASEID-tcpdump-$(hostname).zip zd-$CASEID-tcpdump-$(hostname)
Time around the issue is exposed
Make sure to tell us the exact time slot where you saw the issue occur again. This is required for us to be able to correlate the dates between network dumps, support bundles data and any information you may have provided us with
Support Bundle
A support bundle from the problematic instance ideally when the issue is happening, or in the worst case, right after the issue is exposed. Please, follow the KB below in case you don’t know how to generate a support bundle.
That may help us correlate network packets to potential error logs.
0 Comments