WinRM in OpenNMS

WSMAN support has been present in OpenNMS for a while. The main motivator for adding support of this was Dell iDRAC, but it is Microsoft Windows infrastructure the one that can have more benefits from it, as an alternative of SNMP to gather statistics from Windows.

Through WinRM, everything exported by WMI is accessible (if the user has the right permissions), which makes it a more attractive solution that using WMI directly.

The challenge is always, how to easily configure the Windows Infrastructure in terms of user authentication. Fortunately, thanks to Active Directory we have GPOs (Group Policy Objects), which can be used to configure a set of Windows machine from a central place.

Traditionally, WinRM (which stands for Windows Remote Management) is just a tool used by PowerShell to have remote access to a Windows machine to perform administrative tasks. For this reason, the protocol and the access to WMI statistics were meant for administrators.

That said, when speaking about monitoring Windows infrastructure, there is no need to have a privileged account, as the purpose of the monitoring tool will be only read statistics, and never perform changes.

Using an administrator account makes monitoring easy, but it can be seen as a security breach. This has been the reason why the WMI Tools in OpenNMS are rarely used, as the credentials have to be stored in plain text inside a configuration file.

Fortunately, to use WinRM, Kerberos has to be the algorithm to use when using Domain Accounts, as basic authentication is only allowed for local administrators. If AD and GPOs are going to be used to facilitate deployment, for sure a Domain Account will be used for WinRM, so Kerberos is going to be mandatory.

With Kerberos, OpenNMS and the WSMAN library will use GSS to negotiate authentication, and the only requirement is having an initialized KeyTab. That means, there is no need to store the credentials in plain text on a configuration file. Of course, the KeyTab file should be protected to make sure it won’t be misused to compromise the Windows infrastructure.

Using privileged Domain Account (or accounts with administrator access) can make the Windows Administrators nervous, so it is reasonable to consider using a non-domain account.

In terms of WinRM access, this is entirely possible, but the problem is going to be having access to WMI.

The changes required to allow a non-admin account to perform WMI queries cannot be done through a GPO directly, meaning that this GPO would have to distribute a PowerShell script that will be executed through the Task Manager, to perform all the changes required locally to allow non-users to have access to WMI, as explained in this article.

This article is not about explaining how to configure the Domain Account, is about configuring WinRM with Kerberos on Linux to monitor Windows Server using OpenNMS, so those details are not going to be explained here, and all the upcoming context will assume that there is an account with WinRM and WMI access (regardless if it is an administrative account or not).

DNS

To use Kerberos, DNS should be perfectly working on the Linux machine where OpenNMS is running; otherwise, it will be impossible to test connectivity, verify authentication and use WSMAN in general.

Usually, the same DNS used by the Domain Controller should be used on the OpenNMS server, and it should be able to perform forward and reserve lookups against any Windows Server that will be monitored by OpenNMS.

This is very important for the following reason:

Kerberos only work with FQDNs, not IP addresses, while OpenNMS only work with IP addresses not FQDNs

This might sound contradictory, but fortunately, that is not the case, and DNS plays an important role here.

So, as FQDNs will be specified when performing any Kerberos related operation, the DNS should be able to properly translate FQDNs to IP addresses the same way the Domain Controller would do it.

When configuring OpenNMS, usually through requisitions, the operator declares nodes and set at least one IP Address to the node. Usually, this IP is marked as primary if SNMP is going to be used, but with WSMAN, it is crucial that a reverse lookup against that IP resolves to the same FQDN used by the Domain Controller to recognize the Windows machine in question; otherwise Kerberos authentication is going to fail.

In other words, the WSMan library within OpenNMS will translate that IP into an FQDN, and if that operation succeeds, it will use that Hostname to perform the WinRM request, and Kerberos will work if the FQDN is valid from Active Directory’s perspective.

Here is a simple example:

Let’s say that the domain is called mynet.local, and the server that it is required to access is called winsrv01, its FQDN would be winsrv01.mynet.local.

This is how to validate if DNS works on the OpenNMS server:

As shown in the above output, both, a forward and a reverse lookup are working. That means, the requisitioned node should use that IP to detect and collect data through WSMAN using WinRM, for example:

NOTE: The node should only have the interface marked as primary if SNMP is intended to be used. Also, there is no need to force the WSMAN service. It was done to demonstrate how to define the node properly.

Verify connectivity using the wsman CLI

For simplicity, I’m going to assume that the Operating System where OpenNMS is running is CentOS/RHEL 7.

To install the wsman CLI:

yum install wsmancli

The command should be part of the core packages of the OS.

Then, we also need the Kerberos utilities:

yum install krb5-workstation

The command should be part of the core packages of the OS.

Now that we have DNS correctly configured, the required tools are installed, and there is a domain account with WinRM/WMI access, we can verify the installation.

Of course, before use Kerberos, the Kerberos Realm must be configured.

This can be done by overriding the main configuration file (i.e. etc/krb5.conf, or by extending it by adding the following to /etc/krb5.conf.d/winrm.conf

The configuration requires knowing the domain (i.e. mynet.local), the realm that Kerberos will be used (i.e. MYNET.LOCAL), and the FQDN of the Domain Controller (i.e. dc1.mynet.local).

Now, we’re ready to initialize the Ticket Cache.

We should use the username in Kerberos format, which is USERNAME@KERNEROS_REALM, let’s assume that the username we got from the Windows Team is operator, so the username will be [email protected].

To initialize the cache:

kinit [email protected]

The above will ask for the user’s password. If everything went well, it should show something like this:

Now that we got the TGT (Ticket Granting Ticket), represented by the krbtgt service, we can use the wsman command: Here is an explanation about the options:

a) -d 6

Enable DEBUG.

b) -h winsrv01.mynet.local

The FQDN of the target machine from which it is desired to get statistics. Keep in mind that this FQDN should be the same used when the server joined the domain in Active Directory.

c) -P 5985

The WinRM Port. For now, HTTP will be used. HTTPS is recommended, but that increases the level of complexity of the solution, which is why it is going to be omitted.

d) --username [email protected]

This is an optional parameter with the username in Kerberos format as mentioned before. If omitted, the tool will prompt for it.

f) --password "password_here"

This is an optional parameter with the user’s password. If omitted, the tool will prompt for it.

g) -y gss

To enable GSS/Kerberos authentication, or instruct the command to initiate GSS/Kerberos negotiation.

f) enumerate http://schemas.microsoft.com/wbem/wsman/1/wmi/root/cimv2/Win32_OperatingSystem

To execute a WinRM enumerate command against the provided WMI resource.

If the above command works, a bunch of information about the server will be displayed. The output itself can be huge, so the following includes a partial content as an example:

The good news is that if the above output is displayed, that means that WinRM, as well as WMI, are properly configured. If not, the Windows Team should verify the permissions, depending on the error reported by the tool.

Remember, for a non-admin account, having access to WinRM doesn’t guarantee access to WMI. This is only guaranteed when using privileged users (or administrator accounts).

NOTE: Unfortunately, the command requires the username and the password even if the authentication is going to be GSS. Fortunately, this is just for testing.

Verify connectivity using the WSMAN Java Library

The idea here is to use the WSMAN library that OpenNMS uses directly from the command line to verify that the JDK can communicate with the server using WSMAN and Kerberos.

Most of the configuration files required for this CLI are going to be the same that OpenNMS will be using, so the location of these files will reflect this fact.

Previously, we’ve used the OS Kerberos Cache. Even if it is possible to do this with Java, it is advised to use a Kerberos KeyTab file instead.

To generate and initialize the KeyTab the credentials of the WinRM/WMI user are going to be required.

NOTE: If an administrator account is going to be used, and the Windows Team doesn’t want to share the credentials, they should be able to generate the KeyTab file and initialize it for you on any Windows server. That way, they can provide the KeyTab to you (without the credentials). Unfortunately, that also means that the wsman command won’t be used to verify the communication and the operator must rely on the Java tool for this purpose.

To generate the KeyTab file, use the following command:

kutil

After running the command, a special prompt will be displayed. When this happens, introduce the following command:

addent -password -p --username [email protected] -k 1 -e RC4-HMAC

Note that the username must be provided in Kerberos format. After hitting enter at the end, the tool will ask for the password.

Save the Keytab File by entering the following command and press enter:

wkt /opt/opennms/etc/winrm.keytab

Finally, exit the KUtil CLI by entering q and press enter.

Now that we have a KeyTab, we should initialize it to store the TGT on it:

kinit [email protected] -kt /opt/opennms/etc/winrm.keytab

The tool will request the user’s password.

Now that the KeyTab exist, we need to provide the Kerberos configuration. The following is similar to the initial one but includes additional information and should be stored at /opt/opennms/etc/krb5.conf (which is the equivalent of overriding /etc/krb5.conf with everything requried):

Finally, we need to instruct the WSMAN Client how to authenticate the requests. For this purpose, create a file called /opt/opennms/etc/winrm-login.conf with the following content: Note that it has a reference to the principal (i.e. the username in Kerberos format), and more importantly, the location of the KeyTab created earlier.

debug=true is optional and can be used to understand what’s going on during the Kerberos negotiation and authentication phase (it helps for troubleshooting).

At this point, we have everything required to use the Java Client.

It is recommended to download the latest version of the WSMAN CLI for Java from the following link:

https://github.com/OpenNMS/wsman/releases/download/v1.2.3/org.opennms.core.wsman.cli-1.2.3.jar

Here is the syntax for the Java client to get the same content as the wsman CLI we’ve used before:

IMPORTANT: By the time this document was written, the above tool only works with Java 8. Java 11 is not supported due to JAXB issues, as reported here. That means, until that problem is fixed, WinRM/WSMAN can only be used with OpenNMS when using either OpenJDK 8 or Oracle JDK 8, even if Horizon 24+ or Meridian 2019 are being used.

Note that multiple JVM level settings have to be added when using Kerberos. The following explains each of them:

a) -Dsun.security.krb5.debug=true

This is for enabling DEBUG, for troubleshooting purposes.

b) -Djava.security.krb5.conf=/opt/opennms/etc/krb5.conf

This is to tell Java where is the Kerberos Configuration (with the domain and realm information).

If not specified, it defaults to /etc/krb5.conf

c) -Djava.security.krb5.realm=MYNET.LOCAL

To specify the Kerberos realm to be used.

If not specified, it defaults to the one declared on the kerberos configuration. If it is used, java.security.krb5.kdc must be specified as well.

d) -Djava.security.krb5.kdc=dc1.mynet.local

To specify the FQDN of the KDC or the Domain Controller.

If not specified, it defaults to the one declared on the kerberos configuration. If it is used, java.security.krb5.realm must be specified as well.

e) -Djava.security.auth.login.config=/opt/opennms/etc/winrm-login.conf

The location of the authentication handler we’ve created earlier.

For more information, follow this link.

The rest are arguments of the Java CLI

a) -jar /root/org.opennms.core.wsman.cli-1.2.3.jar

This is to specify the JAR that contains the CLI implementation.

b) -gssAuth

To enable GSS/Kerberos authentication via KeyTab

c) -r http://winsrv01.catsnet.local:5985/wsman

The Full URL of the WinRM location for the target host in question. Once again, note that HTTP is being used. For this reason, the port is the default for HTTP 5985, and the path has to be /wsman. Note that the FQDN of the target host is used (i.e. winsrv01.mynet.local).

d) -resourceUri http://schemas.microsoft.com/wbem/wsman/1/wmi/root/cimv2/Win32_OperatingSystem

Similar to the wsman CLI, that’s the URL of the WMI information of interest.

e) -o enum

To specify to the Java CLI that we’re going to run a WSMAN enumeration

f) -w WSMAN_1_0

The version of WSMAN. This is very important as it seems that Microsoft only supports version 1.0, which is not the default of the Java CLI.

g) -vvv

To increase verbosity for troubleshooting purposes.

When executing the command, it shows a lot of information, even more than the wsman command (especially for the Kerberos authentication). In the end, the output is the same. It shows the SOAP response, but this client also enumerates it in plain text to facilitate reading.

Configure OpenNMS

Most of the required configuration files should be present:

• /opt/opennms/etc/krb5.conf
• /opt/opennms/etc/winrm-login.conf
• /opt/opennms/etc/winrm.keytab

The only additional file would be /opt/opennms/etc/opennms.conf, and this file should include all the JVM level properties used on the Java CLI, for example:

As mentioned before, this link provides more information about the Kerberos settings at JVM level.

If all the tests were successful, and the inventory of the Windows machines have been added to be consistent with the DNS resolution, OpenNMS is now ready to collect WMI statistics through WinRM using Kerberos authentication against a Domain Account.

The default configuration contains basic metrics, but there are lots of room for improvements and use specialized metrics for Windows applications like Microsoft Exchange, Active Directory, etc.