How to Monitor Processes With SNMP

Monitoring processes that don't provide network services is a default use case in network monitoring. Because they aren't providing network services, black box testing won't work- you need an agent on your system providing an inside view of your operating system. SNMP monitoring can help. The Net-SNMP agent is easy to install and configure on Linux or Unix. It's compatible with any monitoring solution that supports SNMP, such as OpenNMS.

By default, there are basically two methods utilizing Net-SNMP:

1. Using the HOST-RESOURCES-MIB
2. Using the UCD-SNMP-MIB. Both are supported by the Net-SNMP agent.

Both methods produce the same result, but each will impact your monitoring system configuration and maintenance differently.

To help you decide which method to use, this article will discuss the costs and benefits for each. In this example, we'll discuss different monitoring solutions for two single processes as well as a multiprocess application.

When you're using the Host Resources MIB, the key information is in two tabular objects: the hrSWRunName and the according hrSWRunStatus.

In OpenNMS, you can use the HostResourceSwRunMonitor.

The key parameters are:

• service-name: The name of the process you want to monitor from the hrSWRunName object
• match-all: In case there are multiple processes running, all processes need to be in the named run-level state
• run-level: The status of the process status, by default it is set to 2 and does not need to be changed

A typical SNMP output:

snmpwalk -v 2c -c useVersion3 localhost .1.3.6.1.2.1.25.4.2.1.2

.1.3.6.1.2.1.25.4.2.1.2.895 = STRING: "dockerd"

.1.3.6.1.2.1.25.4.2.1.2.938 = STRING: "tincd"

.1.3.6.1.2.1.25.4.2.1.2.1416 = STRING: "docker-proxy"

.1.3.6.1.2.1.25.4.2.1.2.1428 = STRING: "docker-proxy"

.1.3.6.1.2.1.25.4.2.1.2.1635 = STRING: "docker-proxy"

.1.3.6.1.2.1.25.4.2.1.2.1761 = STRING: "docker-proxy"

.1.3.6.1.2.1.25.4.2.1.2.1880 = STRING: "docker-proxy"

.1.3.6.1.2.1.25.4.2.1.2.1902 = STRING: "docker-proxy"

.1.3.6.1.2.1.25.4.2.1.2.1916 = STRING: "docker-proxy"

.1.3.6.1.2.1.25.4.2.1.2.1928 = STRING: "docker-proxy"

.1.3.6.1.2.1.25.4.2.1.2.1941 = STRING: "docker-proxy"

.1.3.6.1.2.1.25.4.2.1.2.1954 = STRING: "docker-proxy"

.1.3.6.1.2.1.25.4.2.1.2.1972 = STRING: "docker-proxy"

snmpwalk -v 2c -c useVersion3 localhost .1.3.6.1.2.1.25.4.2.1.7

.1.3.6.1.2.1.25.4.2.1.7.895 = INTEGER: runnable(2)

.1.3.6.1.2.1.25.4.2.1.7.938 = INTEGER: runnable(2)

.1.3.6.1.2.1.25.4.2.1.7.1416 = INTEGER: runnable(2)

.1.3.6.1.2.1.25.4.2.1.7.1428 = INTEGER: runnable(2)

.1.3.6.1.2.1.25.4.2.1.7.1635 = INTEGER: runnable(2)

.1.3.6.1.2.1.25.4.2.1.7.1761 = INTEGER: runnable(2)

.1.3.6.1.2.1.25.4.2.1.7.1880 = INTEGER: runnable(2)

.1.3.6.1.2.1.25.4.2.1.7.1902 = INTEGER: runnable(2)

.1.3.6.1.2.1.25.4.2.1.7.1916 = INTEGER: runnable(2)

.1.3.6.1.2.1.25.4.2.1.7.1928 = INTEGER: runnable(2)

.1.3.6.1.2.1.25.4.2.1.7.1941 = INTEGER: runnable(2)

.1.3.6.1.2.1.25.4.2.1.7.1954 = INTEGER: runnable(2)

.1.3.6.1.2.1.25.4.2.1.7.1972 = INTEGER: runnable(2)

For each process you want to monitor, you have to create a service monitor named something like Process-docker-proxy, Process-tincd and Process-dockerd.

Pros: It allows you to set up granular monitoring for every process. The service monitors can be used in "Service Level Management" on the start page for availability calculation and notifications for each process.

Cons: You have to configure and maintain a service monitor for each process you want to monitor in OpenNMS. It is not possible to configure something like: "At least a number x of y processes need to be running to have the service to be ok."

snmpwalk -v 2c -c useVersion3 localhost .1.3.6.1.4.1.2021.2.1.2
iso.3.6.1.4.1.2021.2.1.2.1 = STRING: "tincd"
iso.3.6.1.4.1.2021.2.1.2.2 = STRING: "dockerd"
iso.3.6.1.4.1.2021.2.1.2.3 = STRING: "docker-proxy"

snmpwalk -v 2c -c useVersion3 localhost .1.3.6.1.4.1.2021.2.1.5
iso.3.6.1.4.1.2021.2.1.5.1 = INTEGER: 3
iso.3.6.1.4.1.2021.2.1.5.2 = INTEGER: 1
iso.3.6.1.4.1.2021.2.1.5.3 = INTEGER: 13

snmpwalk -v 2c -c useVersion3 localhost .1.3.6.1.4.1.2021.2.1.101
iso.3.6.1.4.1.2021.2.1.101.1 = ""
iso.3.6.1.4.1.2021.2.1.101.2 = ""
iso.3.6.1.4.1.2021.2.1.101.3 = ""

snmpwalk -v 2c -c useVersion3 localhost .1.3.6.1.4.1.2021.2.1.100
iso.3.6.1.4.1.2021.2.1.100.1 = INTEGER: 0
iso.3.6.1.4.1.2021.2.1.100.2 = INTEGER: 0
iso.3.6.1.4.1.2021.2.1.100.3 = INTEGER: 0

The PrTableMonitor uses the tables above to monitor the status of running processes. There are no specific configuration parameters necessary, because the configuration of how and which processes are monitored is located in the Net-SNMP configuration.

In OpenNMS, configure just one monitor named something like Process-Table. As soon as the Net-SNMP agent identifies a process is not running in the specified boundaries, the Error Flag table is updated and is changed from 0 to 1. The OpenNMS monitor will populate a list of names with the processes that are not in a desired state.

Pros: You have only to configure one PrTable Monitor in OpenNMS, regardless of how many processes you want to monitor. The detailed configuration of which process needs to be monitored is configured on the monitored system itself; the SNMP agent needs to be configured anyway, and configuration management tools are in place in larger environments. It is possible to configure in detail how many processes and which processes need to be monitored. Only one service goes down if multiple processes fail.

Cons: There is only one service in OpenNMS for all services, it is not possible to notify different people for specific processes. In case you have one process that fails and the monitor goes down and a second process fails, the event reason in OpenNMS documents only the event reason for the initial process failure.