OpenNMS Meridian Security Reference Architecture

The security implications of a network monitoring solution extend beyond the installation of the platform itself. This reference architecture presents the structural components of the OpenNMS monitoring solution and provides guidelines and best practices for using them.

This reference architecture describes only "out-of-the-box" components and typical use patterns for OpenNMS software. However, as an open source project, OpenNMS is frequently customized by the community (sometimes collaborating with OpenNMS staff) for unique integration patterns and use cases that are beyond the scope of this document.

This document's intent is to help you get started with a more secure deployment of OpenNMS software.

Note: This is a companion to product documentation, blog posts, community discussions, and source code. This document does not explain how to install or customize OpenNMS software. Refer to OpenNMS documentation to learn more.

OpenNMS is available in two different distributions that share the same basic architecture:

• OpenNMS Horizon contains the latest developed features and has packages available for anyone to download and install. Horizon is available with community-level support only.
• OpenNMS Meridian contains the latest stable version of Horizon components, includes additional security penetration testing, and offers long-term paid support. Meridian is recommended for production use in enterprise networks.

Regardless of which version you choose, OpenNMS is a robust monitoring platform that scales from small to large enterprise networks—and integrates easily with other systems. In addition, being open-source encourages community contributions to extend the base OpenNMS functionality.

It's possible to install the OpenNMS Meridian / Horizon Core—and the associated PostgreSQL database—on a single system to create a full-featured monitoring system for use within a small network. This is represented below, in Figure 1.

The Meridian / Horizon Core contains all of the functionality necessary to run the OpenNMS Java application, collect data from the local network, handle events and alarms, manage message queues, host a web server, and display the web console. It also provides APIs for integration.

Figure 2 illustrates a typical OpenNMS deployment within an enterprise environment. (In this case, with a data center and some remote offices.) This figure describes the components that distribute processing away from the Core to meet high-performance requirements or to provide additional functionality.

Minions can feed high volumes of monitoring data to the OpenNMS Core via dedicated messaging systems (Kafka or ActiveMQ), as shown below in Figure 3. If the connection between Minion and the Core fails, the message broker can buffer the data and deliver once the connection resumes.

Minions help the OpenNMS Core easily understand the topology of networks that have duplicate overlapping address ranges (for example, multiple remote offices that all have 192.168.x.x addresses).

Minions initiate communication to the OpenNMS Core and act as a proxy for OpenNMS communications across remote networks. These features are especially useful in environments where the OpenNMS Core cannot initiate connections to remote locations.

Minions are available as installable applications or as a pre-packaged OpenNMS Appliance (virtual or physical). OpenNMS Appliances are centrally managed through the OpenNMS Portal, which automatically handles Minion software upgrades, patches, and maintenance and is especially useful for fleets of Minions.

Recommendations:

• Minions need a username configured within OpenNMS Core that has the ROLE_MINION security permission. Do not use the Admin user.
• For high-security environments, use a unique username for each Minion. In this deployment model, the unique username for a security-compromised Minion can be quickly disabled without affecting other Minions.
• To simplify Minion deployments, all Minions can share the same username. In this deployment model, if a single Minion is compromised, then all Minions would be affected. All Minions would also be affected in the case where the shared username is disabled or reset.

Figure 4 Illustrates a logical network view of a more complex OpenNMS deployment. In this example, the deployment makes use of a reverse proxy and load balancers in front of OpenNMS Minions.

Figure 5 illustrates the OpenNMS Core Java runtime environment. Minion and Sentinel both use similar Java environments, but with superfluous functions removed, to streamline their functional focus.

Recommendations:

• Each OpenNMS Core, Sentinel, and Minion has a unique Karaf instance that installs with a default admin/admin username/password. You should change the default password to secure your installation. The Core's admin account can be managed via the Users section of the admin web UI. Sentinel and Minion documentation provides instructions to update default passwords (see Install Sentinel Runtime docs and Install Minion docs).
• Before changing the password, enable secure SHA-256, SHA-384, or SHA-512 password storage to prevent Karaf from storing clear-text passwords. In addition, enable salted password storage to help prevent password cracking by someone who obtains a copy of the Karaf configuration file.
• For high-security environments, use a unique admin password for each Karaf instance—each OpenNMS Core, Minion, and Sentinel. In this deployment model, the unique password for a compromised Karaf instance can be quickly disabled without affecting other Karaf instances.
• To simplify administration, you can use the same password for all Karaf instances. However, in this model, if a single Minion, Sentinel, or Core is compromised, then all other Karaf instances are vulnerable.
• Enable TLS encryption to protect communications to the web application and REST APIs.

Unlike some security systems, OpenNMS does not assign permissions to groups, nor are they used as a security feature. Permissions can be assigned only to users. Users can have more than one permission.

OpenNMS uses groups to manage alert notifications (for example, the Evening-Shift group) and for limiting which nodes trigger notifications (for example, the Routers-Switches group). Group notification controls are ignored for users that have the ROLE_ADMIN permission.

OpenNMS installs an Admin group. However, that group has no special permissions (users in the Admin group do not automatically inherit the ROLE_ADMIN permission).

The Jetty and Spring modules enforce authentication and authorization to the web user interface and REST APIs. Unless integrated with SSO, OpenNMS authenticates users against the $OPENNMS_HOME/etc/users.xml file.

The users.xml file contains the username, password (typically stored as salted hashes), security permissions, group membership, and other basic information. Because it's critical to OpenNMS security, you must protect access to the $OPENNMS_HOME/etc folder—and especially the users.xml and users.properties files—against tampering, copying, or inspection outside of OpenNMS.

Recommendations:

• Use the UNIX/Linux "chmod 600 …" to set the appropriate access control on the users.xml and users.properties files within the $OPENNMS_HOME/etc folder.

The Web UI requires a valid session cookie to access OpenNMS pages. If a user attempts to access any page within the web UI without a valid session cookie, they'll be redirected to the login page to validate their username and password. A successful login creates a session cookie.

For SSO implementations, the login mechanism is externalized from OpenNMS. However, the external login still results in an OpenNMS session cookie (see "SSO Integration" section below).

Most non-administrative users have the ROLE_USER permission which is granted by default when users don't have an explicit role/permission assignment. Users with ROLE_USER have access to most of the OpenNMS user interface and can respond to alarms and notifications. The ROLE_READONLY permission also gives access to the entire user interface, except that the user cannot respond to alarms and notifications.

Users with the ROLE_ADMIN permission can access all user interface pages, including the administrator functions to manage users, roles, and groups.

Note: Placing users into the Admin group does not give them the ROLE_ADMIN permissions.

The OpenNMS REST APIs usually require proactive HTTP basic authentication, where the caller provides a valid username and password with each call. The APIs will not redirect or challenge the caller for credentials. The rules are slightly different for SSO (see "SSO Integration" section below).

Third-party applications that call OpenNMS APIs should typically have a dedicated user ID / password assigned with the ROLE_REST permission for full access to all API functions. Specialized ROLE_MINION, ROLE_MOBILE, and ROLE_REPORT_DESIGNER permissions exist for unique API integrations.

Note: ROLE_ADMIN users can also access the full API. However, use of this permission is not recommended for REST API integrations.

Recommendations:

• Assign unique usernames within OpenNMS for each API-integrated application or feature. Do not use the Admin user for API calls.
• Assign prescribed roles (permissions) for API calls. Specifically, most API usernames should be assigned the ROLE_REST permission. Do not assign the ROLE_ADMIN to API usernames.

Single Sign On (SSO) integration externalizes authentication for the Jetty web application server running within the OpenNMS Core JVM. With SSO enabled, all Core web UI and REST API calls are authenticated via SSO. After initial authentication, web UI users receive a session cookie which avoids re-authentication for each web page.

SSO integration can be configured directly within the Jetty server (using Spring Security) or through a reverse proxy in front of Jetty.

• See External Authentication documentation for specific instructions on how to integrate SSO directly into Jetty using LDAP directories (for example, Microsoft Active Directory).

With direct SSO integration, Jetty performs LDAP (or similar) calls to authenticate each user or REST API call.

• See Web UI Pre-Authentication in the OpenNMS product documentation for instructions on how to configure web UI pre-authentication in Jetty and Spring to support SSO via a reverse proxy. When using pre-authentication, the reverse proxy passes the username and role(s)/permissions via HTTP headers to the OpenNMS Jetty web server. This integration means you must restrict access to the Jetty web UI and REST API by firewall rules or other means to prevent malicious users from bypassing authentication. Typically, this involves limiting access to the Jetty IP address/ports to only the reverse proxy's address or configuring Jetty to listen only on localhost and running the reverse proxy on the same server as OpenNMS.

Search the community Discourse pages for more discussions about various SSO integration patterns using NGINX, Jetty, LDAP, Kerberos, etc.

Recommendations:

• When using pre-authentication SSO integration, restrict access to the Jetty web UI by using either a firewall with rules to only allow the SSO front-end, or by running the reverse proxy on the same server as the OpenNMS Core and binding the Jetty web UI to the local loopback interface (127.0.0.1).

All networks are subject to confidentiality risks if an attacker gains physical or virtual access to network infrastructure located along communication paths, particularly if the communications are not secured with encryption.

If an attacker inserts themself into a communication path—either by physical means (rewire some cables), or virtually (for example, spoof a Wi-Fi access point, gratuitous ARP spoofing, DNS spoofing, etc)—then the attacker could intercept and modify data in motion, thereby reducing the integrity of the communications.

Note: Encryption does not always protect integrity (for example, the attacker could use man-in-the-middle encryption).

Popular networking and monitoring protocols such as SNMP, ICMP, Syslog, Telemetry, DHCP, ARP, NTP, and OSPF were designed to run within protected networks and are unsuitable for transmissions over unprotected networks (the Internet).

Zero-Trust Architecture dictates the following principles for secure communications:

• Assume the network between client and servers is unsafe.
• Use end-to-end encryption to protect confidentiality and data integrity.
• Authenticate the identity of clients connecting to servers.
• Authenticate the identity of all servers that clients connect to (typically done with certificates).

OpenNMS Minions can collect monitoring data at remote protected networks—even when those networks are not Zero-Trust compliant—and then repackage and transmit that data to the OpenNMS Core using Zero-Trust principles by enabling TLS for communications to the OpenNMS Core and Kafka or ActiveMQ.

Recommendations:

• Use Zero-Trust principles (for example, encrypt using HTTPS protocols with TLS, authenticate server certificates, authenticate API callers, authenticate users) wherever possible within OpenNMS deployments.

Due to security concerns, default installations of the OpenNMS Core restrict access to the RMI interface to the local loopback address (127.0.0.1). Despite this restriction, a netstat command will show RMI listening for all interfaces (0.0.0.0). However, the internal RMI service will reject access attempts unless reconfigured to allow them. If RMI is set to allow other addresses, then you should use TLS to secure communications because OpenNMS admin credentials are transmitted during RMI calls.

See Enable RMI in the OpenNMS product documentation for details about configuring RMI with TLS (HTTPS).

Recommendations:

• Do not enable the OpenNMS Core RMI interface beyond local loopback (127.0.0.1) unless necessary.
• If the RMI interface is enabled for remote access, configure it to use HTTP/TLS encryption and authentication and restrict access via firewalls.

OpenNMS strongly recommends using TLS (HTTPS) for the OpenNMS web UI, REST API services, and RMI (if enabled). You can enable HTTPS/TLS either on the Jetty web service running inside of the OpenNMS JVM, or by running OpenNMS behind a web proxy (for example, Apache or NGINX), and letting the proxy handle HTTP/TLS, or both.

See HTTPS/SSL and Web UI Pre-Authentication in the OpenNMS documentation for specific instructions.

The messaging services (Kafka or ActiveMQ) and database service (PostgreSQL) OpenNMS uses should also be configured to use TLS to protect data integrity and confidentiality (particularly the connection usernames/passwords). Refer to the product-specific documentation for enabling HTTPS/TLS.

• Kafka docs
• ActiveMQ "Classic" docs
• PostgresSQL docs

Similarly, if Cassandra or ScyllaDB are used to store time series network data, you should also encrypt communications to them.

• Cassandra docs
• ScyllaDB docs

Note: Default installations of OpenNMS do not use these databases.

OpenNMS and many of its dependent services (ActiveMQ, Kafka) are built on Java, which uses a Java KeyStore to securely store the associated certificates and private keys needed for HTTPS/TLS servers.

When OpenNMS Core, Sentinel, and Minion documentation specifies URLs as HTTP within configuration screens or files, HTTPS URLs are also acceptable if the server supports HTTPS, and the associated certificates are trusted by OpenNMS. Like most Java applications, OpenNMS uses the Java TrustStore to list the certificates it trusts.

See OpenNMS documentation for instructions about managing the OpenNMS Java Keystore and TrustStore:

• Secure Jetty with HTTPS docs
• Enable Request Logs docs
• SSL Client Connections docs

Recommendations:

• Enable TLS (HTTPS) on all services OpenNMS provides or uses. Specifically, the OpenNMS Core web UI, messaging subsystem (Kafka or ActiveMQ), PostgreSQL database, Cassandra (if used), ScyllaDB (if used), and RMI interface (if enabled) should all be configured for HTTPS/TLS.
• Set access controls on Java KeyStore and TrustStore to prevent unauthorized users from modifying or obtaining a copy of these files.

OpenNMS Core allows an event to be captured via simple TCP or UDP sockets (default ports 5817 for both TCP and UDP) used by the send-event.pl script. The send-event ports are not secure because they require no authentication and use no encryption. Because of these security concerns, default OpenNMS installations listen only with the send-event ports on the local loopback interface (127.0.0.1). If eventd is reconfigured (in eventd-configuration.xml) to listen on alternate addresses, you should use firewalls to restrict access to the opened send-event ports.

Recommendations:

• Do not bind the send-event ports (TCP and UDP 5817) to other than the local loopback interface (127.0.0.1) on the OpenNMS Core installation.
• If the send-event ports are bound to alternate interfaces, then use firewalls or similar technologies to limit access to those ports to explicit users or applications.
• Consider using the /events REST API endpoint instead of the XML-over-port-5817 mechanism to create event records from external code.

Many OpenNMS configuration files and database tables contain sensitive information such as

• OpenNMS users
• Passwords
• Configurations
• IP addresses
• Network device passwords
• Security keys
• SNMP community names

Some of the data is encrypted or hashed (for example, passwords in the users.xml file). However, other sensitive data is not.

In addition to OpenNMS data, security-related data about systems and services that OpenNMS can monitor (for example, LDAP, Active Directory, web login pages) or interface with (for example, Jira, Remedy) are also not encrypted within OpenNMS configuration files. For these reasons, access to all the data contained in $OPENNMS_HOME and its subfolders must be restricted to the operating system account used to run OpenNMS. In addition, backup copies of these folders and databases must also be protected for similar reasons.

The Java/Jetty/Karaf environment used within OpenNMS was designed to be extensible and flexible for system integrators. Integrators can place Java modules into the $OPENNMS_HOME/jetty-webapps, $OPENNMS_HOME/lib, and $OPENNMS_HOME/deploy folders, which OpenNMS could install and start when OpenNMS starts. Malicious actors with access to these folders could also subvert OpenNMS for nefarious reasons using the same techniques.

Recommendations:

• Restrict access to the $OPENNMS_HOME folder and all subfolders to only the operating system username used to run OpenNMS components (for example, the "opennms" user). All other operating system usernames should have no access to these folders.
• All folders and files within $OPENNMS_HOME should also be owned by the OpenNMS system account. Running the /usr/share/opennms/bin/fix-permissions script will reset folder and file ownership within the $OPENNMS_HOME folder structure.

Note: The script will not use chmod commands or reset access controls, it will only use chown commands to set ownership.

• Place similar restrictions on folder structures associated with other subsystems associated with OpenNMS (for example, PostgreSQL server, Kafka, and Elasticsearch).

For example, protect the following PostgreSQL folders with ownership by the "postgres" username:

• /etc/postgresql/<version>/main
• /var/lib/pgsql/data
• Limit operating system-level access (for example, console login, ssh, RDP) to operating system administrators and OpenNMS administrators for use only during system installation and software maintenance. Do not allow other users, including other OpenNMS users, to directly log-on to the operating system of OpenNMS servers.
• Starting with Horizon version 29 and Meridian version 2022, the system account used to run OpenNMS (JVM, etc) has no special operating system privileges. The OpenNMS system account should be non-root, non-admin, and should not allow interactive logon.

Note: Some versions of Linux require a configuration change to allow OpenNMS components to use the ICMP protocol from a non-root account.

• Related community article: H29+ won’t start with permission error to open ICMP socket

OpenNMS Core components were designed to run within a protected data center, where most people would not have physical access to the associated servers.

Recommendations:

• Run OpenNMS Core and Sentinels only within protected data centers, isolated from the public internet or other unprotected networks. Use firewalls or similar techniques to prevent internet access to the Core web user interface and REST APIs.
• Use Minions to collect data from distributed networks and provide them access to the OpenNMS Core via firewalls restricted REST API and message broker (for example, Kafka or ActiveMQ) target addresses.