Metrics integrations: JMX

The Sysdig agent retrieves data from your Java virtual machines using the JMX protocol. The agent is configured to automatically discover active Java virtual machines and poll them for basic JVM metrics like Heap Memory and Garbage collector as well as application specific metrics for ActiveMQ, Cassandra, Elasticsearch, HBase, Kafka, Tomcat, and Zookeeper.

The agent can also be easily configured to extract custom JMX metrics coming from your own Java processes. Metrics extracted are shown in the pre-defined Application views or under the Metrics > JVM and JMX menus. 

The default JMX metrics configuration is found in the /opt/draios/etc/dragent.default.yaml file. When customizing existing entries, copy the complete application's bean listing from that defaults yaml file into the user settings file /opt/draios/etc/dragent.yaml. The Sysdig agent will merge configurations of both files.

Here is what your dragent.yaml file might look like for a customized entry for the Spark application:

customerid: 07c948-your-key-here-006f3b
tags: local:nyc,service:db3
jmx:
  per_process_beans:
    spark: 
      pattern: "spark"
      beans:
        - query: "metrics:name=Spark shell.BlockManager.disk.diskSpaceUsed_MB"
          attributes:
            - name: Value 
              alias: spark.metric
 

Include the jmx: and per_process_beans: section headers at the beginning of your application/bean list.  For more information on adding parameters to a container agent's configuration file, see the FAQ: How-can-I-edit-the-agent-s-configuration-file?

 

Bean Configuration

Basic JVM metrics are pre-defined inside the default_beans: section. This section is defined in the agent's default settings file and contains beans and attributes that are going to be polled for every Java process, like memory and garbage collector usage:

jmx:
default_beans:
- query: "java.lang:type=Memory" attributes: - HeapMemoryUsage - NonHeapMemoryUsage - query: "java.lang:type=GarbageCollector,*" attributes: - name: "CollectionCount" type: "counter" - name: "CollectionTime" type: "counter"

Metrics specific for each application are specified in sections named after the applications. For example, this is the Tomcat section:

  per_process_beans:
tomcat:
pattern: "catalina" beans: - query: "Catalina:type=Cache,*" attributes: - accessCount - cacheSize - hitsCount
- . . .

The key name, "tomcat" in this case, will be displayed as a process name in the Sysdig Monitor user interface instead of just "java". The pattern: parameter specifies a string that is used to match a java process name and arguments with this set of JMX metrics. If the process main class full name contains the given text, the process is tagged and the metrics specified in the section will be fetched. 

Note: The class names are matched against the process argument list.  if you implement JMX metrics in a custom manner that does not expose the class names on the command line, you will need to find a pattern which conveniently matches your java invocation command line.

The beans: section contains the list of beans to be queried, based on JMX patterns. JMX patterns are explained in details in the Oracle documentation, but in practice, the format of the query line is pretty simple: you can specify the full name of the bean like java.lang:type=Memory, or you can fetch multiple beans in a single line using the wildcard * as in: java.lang:type=GarbageCollector,*

To get the list of all the beans that your application exports, you can use JVisualVM, Jmxterm, JConsole or other similar tools.  Here is a screenshot from JConsole showing where to find the namespace, bean and attribute (metric) information (JConsole is available when you install the Java Development Kit):

 

 

For each query, you can specify the attributes that you want to retrieve. We support the following JMX attributes (For these attributes, all the subattributes will be retrieved):

 

Attributes may be absolute values or rates. For values, we need to calculate a per second rate before sending them. In this case, you can specify type: "counter", the default is "rate" which can be omitted, so usually you can simply write the attribute name.

 

Limits

The total number of JMX metrics polled per host is limited to 500.  The maximum number of beans queried per process is limited to 300. If more metrics are needed please contact your sales representative with your use case. 

In agents 0.46 and earlier, the limit was 100 beans for each process.

 

Aliases

JMX beans and attributes can have very long names. To avoid interface cluttering we added support for aliasing, you can specify an alias in the attribute configuration. For example:

  cassandra:
    pattern: "cassandra"
    beans:
      - query: "org.apache.cassandra.db:type=StorageProxy
attributes: - name: RecentWriteLatencyMicros alias: cassandra.write.latency - name: RecentReadLatencyMicros alias: cassandra.read.latency

In this way the alias will be used in Sysdig Monitor instead of the raw bean name. Aliases can be dynamic as well, getting data from the bean name - useful where you use pattern bean queries. For example:

      - query: "java.lang:type=GarbageCollector,*"
        attributes:
          - name: CollectionCount
            type: counter
            alias: jvm.gc.{name}.count
          - name: CollectionTime
            type: counter
            alias: jvm.gc.{name}.time

This query will match multiple beans (All Garbage collectors) and the metric name will reflect the name of the Garbage Collector. For example:jvm.gc.ConcurrentMarkSweep.count. General syntax is: {<bean_property_key>}, to get all beans properties you can use a JMX explorer like JVisualVM or Jmxterm.

 

Troubleshooting

The Sysdig agent normally auto-discovers Java processes running on your host and enables the JMX extensions for polling them.  If your java application is not discovered automatically by the agent, try adding the following parameter on your application's command line:

 -Dcom.sun.management.jmxremote

For more information, see Oracle's web page on monitoring using JMX technology

 

Disabling JMX Polling

If you do not need it or otherwise want to disable JMX metrics reporting, you can add the following two lines to the agent's user settings configuration file /opt/draios/etc/dragent.yaml

jmx:
 enabled: false

After editing the file, restart the native Linux agent via service dragent restart or restart the container agent to make the change take effect.

If using our containerized agent, instead of editing the dragent.yaml file, you can add this extra parameter in the 'docker run' command when starting the agent:

-e ADDITIONAL_CONF="jmx:\n  enabled: false\n"
Have more questions? Submit a request