Selecting support-bundle options

Issue

  • What support-bundle options should I select for X?
  • I am trying to troubleshoot X and CloudBees Support is asking me for a support bundle, which options should I select?
  • What do the support-bundle options mean?

support-bundle-options

Environment

Resolution

Agents

  • Agent Command Statistics

    Used to diagnose issues with agent to controller communication.

    $ cat [support bundle]/nodes/slave/[agent]/command-stats.md
    
    # Totals
    * Writes: 913
      * sent 2.4Mb
    * Reads: 1044
      * received 1.4Mb
    * Responses: 92
      * waited 9.1 sec
    
    # Commands sent
    * `Close`: 1
      * sent 0.0Mb
    * `Pipe.Chunk`: 5
      * sent 0.0Mb
    * `Pipe.Flush`: 2
      * sent 0.0Mb
    ...
    
  • Agent Log Recorder

    Captures agent logs.

    $ ls [support bundle]/nodes/slave/[agent]/logs/*
    $ cat [support bundle]/nodes/slave/[agent]/jenkins.log
    2021-02-20 01:33:03.955+0000 [id=172]   WARNING o.j.r.u.AnonymousClassWarnings#warn: Attempt to (de-)serialize anonymous class org.jenkinsci.plugins.durabletask.FileMonitoringTask$FileMonitoringController$1; see: https://jenkins.io/redirect/serialization-of-anonymous-classes/
    2021-02-20 13:05:20.772+0000 [id=21622] INFO    h.r.RemoteInvocationHandler$Unexporter#reportStats: rate(1min) = {0,number,0.0}±{1,number,0.0}/sec; rate(5min) = {2,number,0.0}±{3,number,0.0}/sec; rate(15min) = {4,number,0.0}±{5,number,0.0}/sec; rate(total) = {6,number,0.0}±{7,number,0.0}/sec; N = {8,number}
    
  • Agents config file [Encrypted secrets are redacted]

    Agents configuration.

    $ cat [support bundle]/nodes/slave/[agent]/config.xml
    
    <slave>
      <name>agent-name</name>
      <remoteFS>/tmp/jenkins</remoteFS>
      <numExecutors>1</numExecutors>
      <mode>NORMAL</mode>
      <retentionStrategy class="hudson.slaves.RetentionStrategy$Always"/>
      <launcher class="com.cloudbees.jenkins.plugins.sshslaves.SSHLauncher" plugin="cloudbees-ssh-slaves@2.7">
        <host>agent-name.example.com</host>
        <connectionDetails>
    ...
    
  • Agent JVM process system metrics (Linux only)

    Agent JVM details.

    ls [support bundle]/nodes/slave/[agent]/proc/*
    cpuinfo.txt		mounts.txt		self			system-uptime.txt
    meminfo.txt		net			swaps.txt
    
  • Agent system configuration (Linux only)

    General agent system information.

    $ ls [support bundle]/nodes/slave/[agent]/sysctl.txt
    $ ls [support bundle]/nodes/slave/[agent]/dmesg.txt
    $ ls [support bundle]/nodes/slave/[agent]/dmi.txt
    $ ls [support bundle]/nodes/slave/[agent]/environment.txt
    $ ls [support bundle]/nodes/slave/[agent]/file-descriptors.txt
    $ ls [support bundle]/nodes/slave/[agent]/sysctl.txt
    $ ls [support bundle]/nodes/slave/[agent]/proc/*
    
  • Dump agent export tables (could reveal some memory leaks)
    Note: agent memory leaks

    Agent export tables.

    $ cat [support bundle]/nodes/slave/[agent]/exportTable.txt
    #1 (ref.1) : object=hudson.remoting.Channel@57234fcd:jnk2stl30 type=hudson.remoting.Channel interfaces=[hudson.remoting.IChannel]
      Created at Fri Feb 19 19:14:54 CST 2021
    	at hudson.remoting.ExportTable$Entry.<init>(ExportTable.java:114)
    	at hudson.remoting.ExportTable.export(ExportTable.java:365)
    	at hudson.remoting.Channel.internalExport(Channel.java:821)
    ...
    

Client

  • About user (basic authentication details only)

    User details.

    User
    ====
    
    Authentication
    --------------
    
      * Authenticated: true
      * Name: GROUP\UserID
      * Authorities
          - `synchronized`
          - `authenticated`
    ...
    

CloudBees

  • CloudBees Assurance Program

    Beekeeper and Beekeeper-related properties.

    $ cat [support bundle]/cap/beekeeper.md
    
    # Configuration
    
    WAR Envelope detected [cje-2.249.2.4]
    Installed Envelope detected [cje-2.249.2.4]
    CloudBees Assurance Program enabled: false
    Allow automatic upgrades of individual plugins on startup: false
    Allow automatic downgrades of individual plugins on startup: false
    Notify when security warnings affecting the core are detected: true
    Notify when security warnings affecting plugins are detected: true
    
  • Operations Center Connector Logs

    Logs recorded on controller for operation center connection.

    $ ls [support bundle]/operations-center-connector-logs/*
    connection-details.md			operations-center-connector.log.3
    operations-center-connector.log		operations-center-connector.log.4
    operations-center-connector.log.1	operations-center-connector.log.5
    operations-center-connector.log.2
    

Jenkins

  • About Jenkins

    Fundamental Jenkins details.

    $ cat [support bundle]/about.md
    
    Jenkins
    =======
    
    Version details
    ---------------
    
      * Version: `2.263.4.2`
      * Instance ID: `0123456789012345678901234`
      * Mode:    Webapp Directory
      * Url:     https://jenkins.example.com/cjoc/
    ...
    
  • All loggers currently enabled.

    Enabled loggers.

    $ cat [support bundle]/loggers.md
    Loggers currently enabled
    =========================
    org.apache.sshd - WARNING
    org.jenkinsci.plugins.gwt - ALL
    disabled - OFF
    com.splunk.splunkjenkins.HealthMonitor - WARNING
    winstone - INFO
     - INFO
    
  • Build queue

    $ cat [support bundle]/buildqueue.md
    Current build queue has 0 item(s).
    ---------------
    Is quieting down: false
    
  • Jenkins Global Configuration File (Encrypted secrets are redacted)

    Main configuration file ($JENKINS_HOME/config.xml)

    $ cat [support bundle]/jenkins-root-configuration-files/config.xml
    <hudson>
      <disabledAdministrativeMonitors>
        <string>hudson.diagnosis.TooManyJobsButNoView</string>
        <string>com.cloudbees.jenkins.plugins.assurance.SecurityWarningsWatch</string>
      </disabledAdministrativeMonitors>
      <version>2.235.1.2</version>
      <installStateName>RUNNING</installStateName>
      <numExecutors>0</numExecutors>
      <mode>NORMAL</mode>
      <useSecurity>true</useSecurity>
    ...
    
  • Items Content (Computationally expensive)

    Summary of jobs and types.

    Item statistics
    ===============
    
      * `hudson.model.FreeStyleProject`
        - Number of items: 35
        - Number of builds per job: 20.085714285714285 [n=35, s=40.0]
      * `org.jenkinsci.plugins.workflow.job.WorkflowJob`
        - Number of items: 20
        - Number of builds per job: 11.9 [n=20, s=20.0]
    
    Total job statistics
    ======================
    
      * Number of jobs: 55
      * Number of builds per job: 17.10909090909091 [n=55, s=40.0]
    ...
    
  • Master Log Recorders

    Custom log recorders (master/controller)

    $ ls [support bundle]/nodes/master/logs/custom/*
    
  • Master system configuration (Linux only)

    Specific controller configuration for your instance.

    $ ls [support bundle]/nodes/master/proc/*
    $ ls [support bundle]/nodes/master/systctl.txt
    $ ls [support bundle]/nodes/master/dmesg.txt
    $ ls [support bundle]/nodes/master/userid.txt
    $ ls [support bundle]/nodes/master/dmi.txt
    ...
    
  • Metrics

    Collection of metrics in json format.

    $ cat [support bundle]/nodes/master/metrics.json
    {
      "version" : "4.0.0",
      "gauges" : {
        "jenkins.executor.count.value" : {
          "value" : 41
        },
        "jenkins.executor.free.value" : {
          "value" : 36
        },
        "jenkins.executor.in-use.value" : {
          "value" : 5
    ...
    
  • Node monitors

    $ cat [support bundle]/node-monitors.md
    Node monitors
    =============
    Approved Folders
    ----
     - Is Ignored: false
     - Computers:
       * master: null
       * agent01: null
       * agent02: null
       * agent03: null
    ...
    
  • Other Jenkins Configuration File (Encrypted secrets are redacted)

    The rest of the configuration files in the $JENKINS_HOME directory.

    $ ls [support bundle]/jenkins-root-configuration-files/
    io.jenkins.plugins.casc.CasCGlobalConfig.xml
    audit-trail.xml
    esr-feeder.xml
    jenkins.metrics.api.MetricsAccessKey.xml
    ...
    
  • Administrative monitors

    Built-in monitors.

    $ cat [support bundle]/admin-monitors.md
    Monitors
    ========
    
    `AsyncResourceDisposer`
    --------------
    (active and enabled)
    
    `jenkins.diagnostics.URICheckEncodingMonitor`
    --------------
    (active and enabled)
    
  • Update Center

    Update Center details.

    === Sites ===
     - Url: file:/tmp/jenkins_home/WEB-INF/plugins/update-center.json
     - Connection Url: null
     - Implementation Type: hudson.model.UpdateSite
     - Url: https://jenkins-updates.cloudbees.com/update-center/envelope-cje/update-center.json
     - Connection Url: http://www.google.com/
     - Implementation Type: hudson.model.UpdateSite
    ======
    Last updated: N/A
    === Proxy ===
    

JVM/Server

  • Environment variables

    Environment for the JVM process of your instance.

    $ cat [support bundle]/nodes/master/environment.txt
    HOME=/var/lib/jenkins
    LANG=en_US.UTF-8
    LOGNAME=jenkins
    PATH=/sbin:/usr/sbin:/bin:/usr/bin
    PWD=/
    ...
    
  • File descriptors (Unix only)

    Open files of your instance.

    $ cat [support bundle]/nodes/master/file-descriptors.txt
    Jenkins
    ======
    
    core file size          (blocks, -c) 0
    data seg size           (kbytes, -d) unlimited
    scheduling priority             (-e) 0
    file size               (blocks, -f) unlimited
    pending signals                 (-i) 123756
    max locked memory       (kbytes, -l) 64
    max memory size         (kbytes, -m) unlimited
    open files                      (-n) 65534
    pipe size            (512 bytes, -p) 8
    ...
    
  • Master Heap Histogram

    A heap histogram if enabled.

    $ cat [support bundle]/nodes/master/heap-histogram.txt
    Histogram generation is disabled. If you want to enable it, do either:
    * Add the system property: -Dcom.cloudbees.jenkins.support.impl.HeapUsageHistogram.DISABLED=false
    * Run from Script Console the line: com.cloudbees.jenkins.support.impl.HeapUsageHistogram.DISABLED=false
    
  • Master JVM process system metrics (Linux only)

    $ ls [support bundle]/nodes/master/proc/meminfo.txt
    $ ls [support bundle]/nodes/master/proc/self/*
    cpuinfo.txt		mounts.txt		self			system-uptime.txt
    meminfo.txt		net			swaps.txt
    
  • Network Interface

    $ cat [support bundle]/nodes/master/networkInterface.md
    -----------
     * Name eth0
     ** Hardware Address - abba3f5d6003
     ** Index - 3
     ** InetAddress - /fe80:0:0:0:d0af:1fff:fe5d:6003%eth0
     ** InetAddress - /10.4.1.9
     ** MTU - 1500
     ** Is Up - true
     ** Is Virtual - false
     ** Is Loopback - false
     ** Is Point to Point - false
     ** Supports multicast - true
    
  • Reverse Proxy

    $ cat [support bundle]/reverse-proxy.md
    Reverse Proxy
    =============
     * Detected `X-Forwarded-For` header: TRUE
    
  • Root CAs

    $ cat [support bundle]/nodes/master/RootCA.txt
    ===== Trust Manager 0 =====
    It is an X.509 Trust Manager containing 204 certificates:
    CN=ITF MC Production Network Applications sub CA, OU=Global Information Security, O=MasterCard WorldWide, DC=mastercard, DC=com
    CN=SecureTrust CA, O=SecureTrust Corporation, C=US
    CN=Entrust Root Certification Authority - EC1, OU="(c) 2012 Entrust, Inc. - for authorized use only", OU=See www.entrust.net/legal-terms, O="Entrust, Inc.", C=US
    CN=DigiCert Global Root CA, OU=www.digicert.com, O=DigiCert Inc, C=US
    OU=Security Communication RootCA1, O=SECOM Trust.net, C=JP
    ...
    $ cat [support bundle]/slave/[agent]/RootCA.txt
    ===== Trust Manager 0 =====
    It is an X.509 Trust Manager containing 133 certificates:
    CN=Hongkong Post Root CA 1, O=Hongkong Post, C=HK
    CN=SecureTrust CA, O=SecureTrust Corporation, C=US
    CN=Entrust Root Certification Authority - EC1, OU="(c) 2012 Entrust, Inc. - for authorized use only", OU=See www.entrust.net/legal-terms, O="Entrust, Inc.", C=US
    ...
    
  • System properties

    JVM system properties (i.e. java -XshowSettings)

    $ cat [support bundle]/nodes/master/system.properties
    #Fri May 01 14:31:28 UTC 2020
    MASTER_DOMAIN=domain
    MASTER_ENDPOINT=https\://example.com/devops/
    MASTER_GRANT_ID=14a45fff-3985-eeee-977f-aa8f4e0e1425
    MASTER_INDEX=1
    ...
    $ cat [support bundle]/nodes/slave/[agent]/system.properties
    #Fri Aug 21 03:51:33 CST 2020
    awt.toolkit=sun.awt.X11.XToolkit
    file.encoding=UTF-8
    file.encoding.pkg=sun.io
    file.separator=/
    ...
    

Performance

  • Deadlock Records

    Detected deadlocks.

    $ ls [support bundle]/deadlocks/*
    DeadlockDetected-20200320-185437.txt
    
  • Garbage Collection Logs

    $ ls [support bundle]/nodes/master/logs/gc/*
    gc.log.1		gc.log.2.current
    
  • Load Statistics

    $ ls [support bundle]/load-stats/
    label		no-label	overall
    $ ls [support bundle]/load-stats/label/
    service-f8b099dd-7777-9999-98b0-3599e73a21bc
    java-8
    master
    $ ls [support bundle]/load-stats/label/master/
    gnuplot		hour.csv	min.csv		sec10.csv
    
  • Out of Memory Errors

    $ cat [support bundle]/nodes/master/oome.md
    OutOfMemoryError Statistics
    ===========================
    
      * Time of first OOME: N/A
    
  • Slow Requests Records

    URI requests which responded slowly.

    $ ls [support bundle]/slow-requests/
    20210225-023235.323.txt	20210225-095644.322.txt	20210225-155605.322.txt	20210225-164326.323.txt
    20210225-023235.324.txt	20210225-140117.322.txt	20210225-155929.322.txt	20210225-164929.322.txt
    20210225-064626.322.txt	20210225-141029.322.txt	20210225-160032.322.txt	20210225-165253.322.txt
    20210225-064629.322.txt	20210225-143444.322.txt	20210225-160111.322.txt	20210225-165332.322.txt
    ...
    
  • Thread dumps

    Snapshot of all Jenkins threads and their stacktraces
    (Note: critical performance data)

    $ cat [support bundle]/nodes/master/thread-dump.txt
    Master
    ======
    
    "Analytics" id=1540 (0x604) state=TIMED_WAITING cpu=17%
        - waiting on <0x58430729> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        - locked <0x58430729> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2083)
    ...
    
  • Thread dumps of running Pipeline builds

    Currently running pipelines and their thread dumps.

    $ cat [support bundle]/nodes/master/pipeline-thread-dump.txt
    Build: [Job] #1
    Thread #0
            at WorkflowScript.run(WorkflowScript:30)
    Approximate graph size: 4
    
    Build: [Job]/dynatrace mapping #3
    Thread #0
            at org.[organization].pipeline.workflow.PCFWorkflow.createJsonStageList(PCFWorkflow.groovy:82)
            at com.cloudbees.groovy.cps.CpsDefaultGroovyMethods.each(CpsDefaultGroovyMethods:2030)
            at com.cloudbees.groovy.cps.CpsDefaultGroovyMethods.each(CpsDefaultGroovyMethods:2015)
            at com.cloudbees.groovy.cps.CpsDefaultGroovyMethods.each(CpsDefaultGroovyMethods:2056)
            at org.[organization].pipeline.workflow.PCFWorkflow.createJsonStageList(PCFWorkflow.groovy:81)
            at my.call(my.groovy:67)
            at WorkflowScript.run(WorkflowScript:2)
    Approximate graph size: 21
    ...
    
  • Timing data about recently completed Pipeline builds

    Pipeline build timings.

    $ cat [support bundle]/nodes/master/pipeline-timings.txt
    No timings available for [Folder]/[Job]/dev #1: java.io.IOException: [Folder]/[Job]/dev #1 did not yet start
    
    No timings available for [Folder]/[Job]/master #1: java.io.IOException: [Folder]/[Job]/master #1 did not yet start
    ...
    

Plugins

  • About Jenkins

    Global information about the instance.

    $ cat [support bundle]/about.md
    
  • Plugins

    A list of active plugins (plugins/active.txt), disable plugins (plugins/disabled.txt), failed plugins (plugins/failed.txt and backed up plugins (plugins/backup.txt).

User Interface (UI)

  • About browser

    Your browser.

    $ cat [support bundle]/browser.md
    Browser
    =======
    
      * Screen size: 2048x1152
      * User Agent
          - Type:     Browser
          - Name:     Chrome
          - Family:   CHROME
          - Producer: Google Inc.
          - Version:  88.0.4324.96
          - Raw:      `Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.96 Safari/537.36`
      * Operating System
          - Name:     OS X
          - Family:   OS_X
          - Producer: Apple Computer, Inc.
          - Version:  10.13.6
    

References

Have more questions?

0 Comments

Please sign in to leave a comment.