Issue

Best Practice for monitoring Jenkins instance performance.

Environment

Resolution

CloudBees’ Products

CloudBees Jenkins monitoring capabilities rely on Jenkins Metrics plugin which explodes a set a metrics including: System and Java Virtual Machine metrics, Web UI metrics and Jenkins specific metrics. Other plugins can also expose different metrics like the Metrics Disk Usage plugin.

There are three complementary solutions you can use for monitoring:

1 CloudBees Jenkins Monitoring

This feature allows you to define different alerts based on metrics. Jenkins will send emails when the alerts start and finish.

It is not available for CloudBees Jenkins Distribution and Jenkins LTS.

2 Elastic Search Reporter

Elasticsearch Reporter is a set of Jenkins plugins that provides insight into your usage of Jenkins and the health of your Jenkins cluster by storing data into an ElasticSearch 5.x/6.x cluster that you must provide. Once you have installed and configured the Elasticsearch Reporter plugins, then each Managed Master will report events and metrics to CloudBees Jenkins Operations Center, and in turn to ElasticSearch. Then you can use Kibana and other tools that you devise to visualize and analyze that data.

IMPORTANT: Elasticsearch Reporter replaces CloudBees Jenkins Analytics which is not longer recommended and it should be uninstalled prior to installing Elasticsearch Reporter, see:

It is not available for CloudBees Jenkins Distribution and Jenkins LTS.

3 CloudBees DevOptics

It allows organizations to measure, analyze, and manage DevOps. CloudBees DevOptics provides a comprehensive view of the status of your CD platform as well as software value streams, correlated from data collected across software delivery pipelines. It acts as a single source of truth to analyze and continually optimize the flow of value from idea to production.

Built-in Value Streams capabilities make it easy to map then actively measure and manage your software delivery process and activities.
Industry standard DevOps metrics allow you to measure performance across teams to leverage best practices and enable continuous improvement.
CD platform monitoring (Run Insights) capabilities allow you to track availability, usage and team engagement of your CD platforms to ensure they are optimally supporting your software delivery activities.

There is a DevOptics free version (check this comparison table) for CloudBees Jenkins Distribution and Jenkins LTS.

Third-party alternatives

Enabling JMX

Java 8 Monitoring and Management Using JMX Technology

Cloudbees Core on Modern Platform with DataDog

See Monitoring with Datadog | CloudBees Docs.

Tier 3 plugins

As indicated in our support policies, we have limitations on the support we can provide for these plugins. Since these plugins are contributed by third parties, we cannot guarantee bug fixes or other resolutions. While we will not be able to escalate these issues to our engineering team, we will do our best to help you diagnose the problem and provide you with workarounds if possible.

Application Performance Monitoring (APM) Tools: External Java JVM Profiling Agents

From our support experience, we have observed in certain scenarios that some External Java JVM Profiling Agents (it might depend on their versions) could create undesired side effects like System and network overhead.

The data collected by the Java agent will be stored in the Application heap. It is because this kind of agents run as a part of the JVM.

While the actual overhead incurred depends on the specificity of the data gathered and the application itself, less than 5% performance overhead is an optimistic general estimate.

References:

Pushing Custom Metrics to your desired Store

Another option would be not using the Jenkins Metrics and create custom metric packages and push them to your desired store. As an example refer How to send build data to an Elasticsearch server.

Monitoring Best Practices