InSpire Diagnostics and Resource Monitoring

Diagnostics & Monitoring

Creating Java Heap Dumps

Tool JMX Terminal

The "JMX Terminal" tool allows you to query and execute JMX Beans. The tool can be found under /opt/mindbreeze/tools/jmxterm-uber.jar or under C:\Program Files\Mindbreeze\Enterprise Search\Server\Tools\jmxterm-uber.jar. The tool contains an interactive shell with integrated help.

Starting the Tools:

cd /opt/mindbreeze/tools/

java -jar jmxterm-uber.jar

The interactive shell starts.

Note: on Windows, make sure that the Java JDK is used.

Creating Heap Dumps

The following information is necessary for a heap dump:

• PID (for Java sandboxes, such as Item Transformer, Query Transformer) or JMX URL (for Client Service or Query Service)
• Directory for storing the heap dumps

Note: the JMX URL is output in the log file of the corresponding service. For example:

TomcatLauncher INFO: Started JMX Server on address service:jmx:rmi://localhost/jndi/rmi://localhost:41518/jmxrmi

The dumps are created with the JMX terminal. Execute the following commands inside the Inspire container:

# starting the tool

cd /opt/mindbreeze/tools/

java -jar jmxterm-uber.jar

# Welcome to JMX terminal. Type "help" for available commands.

$> # connect to the service via PID

$>open 12345

$> # alternative: connect to the service via JMX URL

$>open service:jmx:rmi://localhost/jndi/rmi://localhost:41518/jmxrmi

# Connection to service:jmx:rmi://localhost/jndi/rmi://localhost:41518/jmxrmi is opened

$> # set the domain:

$>domain com.sun.management

$> # set the bean:

$>bean com.sun.management:type=HotSpotDiagnostic

$> # execute the dumpHeap operation to the path e.g. /data/clientservice.hprof

$>run dumpHeap /data/clientservice.hprof true
# calling operation dumpHeap of mbean com.sun.management:type=HotSpotDiagnostic with params [/data/clientservice.hprof, true]
# operation returns:
null

Then, you can find the heap dump under /data/clientservice.hprof

Monitoring JMX Values into a CSV

The JMX terminal can request values from JMX MBean attributes.

The following example shows the query of java.lang.Memory.HeapMemoryUsage:

# starting the tool

cd /opt/mindbreeze/tools/

java -jar jmxterm-uber.jar

Welcome to JMX terminal. Type "help" for available commands.

$> # connect to the service via PID

$>open 12345

$> # alternative: connect to the service via JMX URL

$>open service:jmx:rmi://localhost/jndi/rmi://localhost:41518/jmxrmi

#Connection to service:jmx:rmi://localhost/jndi/rmi://localhost:41518/jmxrmi is opened

$> # set the domain:

$>domain java.lang

$> # set the bean:

$>bean java.lang:type=Memory

$> # execute the dumpHeap operation to the path e.g. /data/clientservice.hprof

$>get HeapMemoryUsage

HeapMemoryUsage = {

  committed = 1558708224;

  init = 262144000;

  max = 3702521856;

  used = 208660496;

};

Collect performance data with the Collect script

/opt/mindbreeze/scripts/collect.sh

The Collect script collects the following information:

• all memory-usage.csv files from the current log directory
• Heap dumps of all mes-jvms (mesclientservice, mesindex, messdkjavaservice)
• Thread dumps and native thread dumps of these processes
• Perf report using perf record and perf script for 10 seconds
• Jvmtop Report with optional profiling

The collected information is stored in a tar.gz file in /data/collect

usage: collect.py [-h] [--output_dir OUTPUT_DIR] [--tarname TARNAME]

                  [--no-compress-tar] [--perf | --no-perf]

                  [--perf-time PERF_TIME] [--jvmtop | --no-jvmtop]

                  [--jvmtop-profile] [--jvmtop-iterations JVMTOP_ITERATIONS]

                  [--jvmtop-delay JVMTOP_DELAY]

Tool to collect resource usage information

optional arguments:

  -h, --help            show this help message and exit

  --output_dir OUTPUT_DIR, -o OUTPUT_DIR

                        base directory for collect reports (default:

                        /data/collect)

  --tarname TARNAME     filename of output file (default:

                        2019-05-22_10-02-27.tar)

  --no-compress-tar     do not compress the output tarfile

  --perf                collect 'perf' information (default)

  --no-perf             do not collect 'perf' information

  --perf-time PERF_TIME

                        'perf' collection time in seconds (default: 2)

  --jvmtop              collect 'jvmtop' information

  --no-jvmtop           do not collect 'jvmtop' information (default)

  --jvmtop-profile      enable profiling when collecting jvmtop information

  --jvmtop-iterations JVMTOP_ITERATIONS

                        number of iteration for 'jvmtop' (default: 5)

  --jvmtop-delay JVMTOP_DELAY

                        delay (in seconds) between iterations of 'jvmtop'

                        (default: 1)

Monitoring of Resources with CSV Files

Monitoring the Index

In the section „Memory Analysis Settings“ oft he Index, monitoring of resources can be configured.

The Index Service will generate a memory.csv file in the log directory of the Index Service.

Monitoring the Client Service

In the section „Memory Analysis Settings“ of the Client Service , monitoring of resources can be configured.

The Client Service will generate a memory.csv file in the log directory of the Client Service.

Setting the level for logging

Select the Log Level

The configuration in the log pool includes the configuration "Permanent Software Telemetry Log Level"

The different levels are Log, Standard, Detail and Debug, where “Log” is the lowest and “Debug” is the highest level. Depending on the selection, all or certain sub-areas are now logged.