“We saw a common need among equipment manufacturers and other companies for a powerful platform that they could use to showcase the capabilities that they had built into their products”

Junos Space is a unique and disruptive platform that unites the management of disparate devices running the Junos operating system. Think of it as a universal remote for network devices – on steroids. Not only can it control all of those devices through the Network and Security Manager (NSM), it can allow applications to be built to fully automate the management process across devices to reduce the time and cost of administering a network to meet real business needs.

The “Powered by OpenNMS” product allows third parties to license the award winning OpenNMS network management application platform for integration with their systems, allowing them to focus on differentiating their products without having to also develop broad, feature rich management platform. OpenNMS brings highly scalable provisioning, event handling, and fault and performance management features across a wide range of devices, from routers and switches, through servers, virtual machines and applications.

“We saw a common need among equipment manufacturers and other companies for a powerful platform that they could use to showcase the capabilities that they had built into their products,” states Tarus Balog, CEO of The OpenNMS Group. “Building an enterprise scale management system is hard, and by leveraging OpenNMS, companies can focus on providing the most features for their clients and their products, while leaving the rest to us.”

Usama Anqud, Senior Director of Engineering at Juniper Networks, states “After evaluating several products, we found that OpenNMS was best-in-class, complementing Junos Space by providing a highly scalable, full featured and customizable fault and performance management capabilities.”

OpenNMS is available to everyone under an open source license (the GNU GPL, version 3), but for those companies that can’t use that license without exposing either proprietary features or trade secrets to competitors, purchasing the “Powered by OpenNMS” program will allow them greater licensing options.

For more information on “Powered by OpenNMS” contact sales@opennms.com, or call +1 919 533 0160.

ABOUT OPENNMS

OpenNMS (www.opennms.org) is the world’s first enterprise-grade network management application platform developed under the open-source model. It is a 100% free software alternative to commercial products such as HP Operations Manager, IBM Netcool and Tivoli, and CA Unicenter.

ABOUT THE OPENNMS GROUP

The OpenNMS Project is maintained by The OpenNMS Group (www.opennms.com), who also provide commercial support, services and training for the OpenNMS platform.

ABOUT JUNIPER NETWORKS

Juniper Networks is in the business of network innovation. From devices to data centers, from consumers to cloud providers, Juniper Networks delivers the software, silicon and systems that transform the experience and economics of networking. Additional information can be found at Juniper Networks (www.juniper.net).

Speaking of triaging the JIRA database, I’d like to re-post an article I wrote on my personal blog about reporting issues. The TL;DR version is: don’t be afraid to open an issue! Here it is in it’s entirety:

Schrödinger’s Bugs

Working on an open-source project teaches you a few things about dealing with software developers, and reporting bugs. I’ve been in the open-source world for a long time, and I remember when I first started out as a user of software, I felt glad to even have access to these tools at all, and I felt a reluctance to “bother” the developers with issues if I wasn’t sure it was only me.

The problem is, issues are a bit like Schrödinger’s cat: they don’t exist until the developer knows about them.

Since I’ve become a developer of open-source software and seen things from the other side, I have one request: err on the side of opening an issue. There’s nothing I love more than having an issue opened, and being able to fix it, and tell the user their problem is solved. It’s that kind of feedback loop that is one of the best parts of developing software without a marketing and sales department sitting between you and your users.

So without further ado, I’d like to point out a few comments in this vein. Note that when I say “issue,” it could be anything: a showstopper bug, an annoyance, or just a new feature you wish the software had.

Always Open an Issue

Sure, sometimes it’s a pain to figure out where the issue reporter link is, and create an account, and validate your email, and figure out what component it goes into… but don’t worry about it. If you get it in the wrong place, they’ll know where it belongs and (hopefully) triage it. But if you don’t open that issue, they may never know it’s a problem.

Don’t Worry If It’s a Duplicate

Of course, you should always try searching for your issue first, maybe someone else has reported it. Make a comment if someone has. But if you can’t find it, don’t worry that it might be a duplicate, go ahead and open that issue. As a developer, I’d rather close a million duplicates than to never know about the issue in the first place.

Don’t Just Describe the Issue, Describe What You’re Trying to Do

It may be that the issue you’re trying to solve is meant to work a different way, or is part of another feature you haven’t used yet, or has a workaround. Make sure when you describe the problem you’re having, also describe what you want to accomplish.

A Closed Issue is Not an Ultimatum

This is a corollary to “describe what you’re trying to do.” Just because an issue is closed does not mean it is closed for discussion. Sometimes the developer doesn’t realize what you’re trying to actually do, or the original issue was described in a way that doesn’t make it clear that the real issue is elsewhere.

For example, OpenNMS supports creating a “path outage,” which describes how particular nodes are related. There was an issue opened that said if you created a path outage, it would be wiped out when using Provisiond. It was closed, saying that you create path outage relationships with the “parent-id” tag in the provisioning group file. What the issue did not say is that these manually-created path outages were created through the UI. So the real issue is that the web UI path outage editor is not Provisiond-aware, and the issue should be reopened.

It’s Better to Be Too Verbose than Not Enough

Configuration files, logs, output from `dmesg` or similar, anything you can add that makes it easier to diagnose the problem. It’s a lot harder to fix a problem with a one-line error message than with 200 lines of context telling you what the software was doing just before the error. The more information you give, the more likely it is the developer will be able to figure out what’s going on when the issue happened.

Does this mean your issue will be resolved quickly? Not necessarily. Everyone has their own set of priorities, and their own time set aside for working on issues. I can say that a good issue report, with a lot of detail and a good description of what you’re trying to accomplish, will get a lot more traction than a 1-line report saying “it doesn’t work,” and it will get a heck of a lot more traction than no report at all. To paraphrase Wayne Gretzky, you miss fixing 100% of the issues you never report.

Project Updates

• 1.8: Current Release is 1.8.14 (Red-breasted Nuthatch)
  1.8.14 is the current stable release, tagged 13 Sep, 2011. For a complete list of changes and updates, see the “New and Noteworthy” page on the OpenNMS wiki. As always, it is recommended that you back up your database before upgrading.
• 1.9: Current release is 1.9.91 (Crumhorn)
  1.9.91 is the current unstable release, tagged 14 Sep, 2011. This release is not yet recommended for production use, but for developers and users who want to try out the very latest features, or for test environments to evaluate upgrades to 1.10 when it is released. For a list of changes and updates, see the “New and Noteworthy” page on the OpenNMS wiki.

Upcoming Events

OpenNMS training will be held in Fulda, Germany, September 26th through 30th, 2011. It is done in conjunction with our partner, NETHINKS. You can register on their site, here.

Issues Resolved

Since the last TWiO, the following issues were resolved as “fixed”:

• [NMS-1322] – CAPSD loops infinitely on misbehaving ipAddrTable
• [NMS-2052] – small patch to add regex filtering of eventparms
• [NMS-2374] – When running the database checker rethrow any exceptions with details (e.g.: database URL)
• [NMS-2375] – Make the severity element in event configuration an enumeration and fix our default config files
• [NMS-2738] – Include a failure reason in data collection failure events
• [NMS-2812] – normalize radius code
• [NMS-2822] – PSQLException when discovering interface/address that has not yet been deleted from database
• [NMS-2889] – Move org.opennms.netmgt.ping.Ping and its dependencies into jicmp package
• [NMS-2907] – More reduction keys for APC events
• [NMS-2977] – Deleted node appear in KSC report creation
• [NMS-3005] – ONMS Version not display correct – Windows install
• [NMS-3066] – JavaMailNotificationStrategy fails when passing -nm
• [NMS-3073] – Weird redirect on login
• [NMS-3143] – GUI error if we remove Switches from Surveillance
• [NMS-3194] – Reporting is truncated by URL length limitation
• [NMS-3319] – HTTPS don’t support SSLv3-only servers
• [NMS-3515] – Broken paged grid in Select SNMP Interfaces
• [NMS-3540] – General node information not displayed using Internet Explorer 8 in IE7 Compatability view or IE7
• [NMS-3594] – provisiond does not create nodelabelchanged event
• [NMS-3606] – WMI/WQL Poller – Wrong text in event
• [NMS-3622] – Allow HttpCollector and PageSequenceMonitor to accept all SSL certificates
• [NMS-3651] – Create detectors for all protocol plugins
• [NMS-3785] – Provisiond crashes against Procurve 1810G
• [NMS-3823] – Clicking a Vlan interface (SVI) on a Cisco router throws Jasper Exception
• [NMS-3848] – Reasons Missing From nodeLostService events
• [NMS-3863] – provisiond has issues with two IPs on the same ifIndex – I think
• [NMS-3887] – webUI shows 100% availability although service is not monitored
• [NMS-3912] – Alarm Description in Dashboard not formatted
• [NMS-3959] – eventd logging missing file name
• [NMS-3979] – Notification Dashlet doesn’t format HTML message text
• [NMS-4020] – Too many CLOSE_WAIT on port 5817
• [NMS-4029] – dns provisiond – fqdn trailing dot
• [NMS-4040] – SNMPPoller is the cause of loss of snmpinterfaces during the re-import / synchronization Provision Groups.
• [NMS-4049] – Can’t provision a node with one IP address and a policy to avoid all IP address
• [NMS-4091] – Page Sequence Monitor and invalid SSL certificates
• [NMS-4157] – org.apache.mina.util.DefaultExceptionMonitor: Unexpected exception
• [NMS-4232] – Configure scheduling outages via RESTful Web Service
• [NMS-4390] – Nearly impossible to make head or tail of why Maven build is failing.
• [NMS-4475] – Provisiond fails with a Unable to return specified BeanFactory instance exception at startup
• [NMS-4502] – [patch] Show correct values in net-snmp CPU Usage graph
• [NMS-4526] – Remote Poller implodes with lack of WMI classes
• [NMS-4544] – Provisiond HOST-RESOURCES process detector
• [NMS-4593] – Report Issue – Surveillance Category Not Correctly Chosen
• [NMS-4673] – Discovery fails with JNA pinger
• [NMS-4675] – Resource Graph Resources – limited to 55 Resources or less
• [NMS-4683] – SNMP interface poller filtering not allowing data collection after provisioning group sync
• [NMS-4704] – Service monitored with 100% availability instead of Not Monitored
• [NMS-4749] – Requisition REST Service allows duplicate nodes
• [NMS-4754] – Event analysis report, missing legend in pie graph
• [NMS-4799] – Events generated from trapd are not associated with any node
• [NMS-4800] – Node.jsp – double clicking physical interfaces goes to interface.jsp instead of snmpinterface.jsp
• [NMS-4815] – Make Jetty headerBufferSize property configurable
• [NMS-4817] – Null (\0) characters in logmsg field of events causes org.postgresql.util.PSQLException: ERROR: invalid byte sequence for encoding “UTF8″: 0×00
• [NMS-4831] – notifd.log – Info if message was send
• [NMS-4833] – Poorly used INFO log message
• [NMS-4834] – Remote Poller state changes from Paused to Running
• [NMS-4838] – jmx collector does direct db lookup of nodeid
• [NMS-4839] – Check if a node is currently covered by a scheduled outage using Rest
• [NMS-4841] – OpenNMS Version not displayed correctly
• [NMS-4845] – RrdUtils.createRRD log message is unclear
• [NMS-4851] – Notification not being sent event if status=”on”, looks like notifd is not using the status in the config file properly
• [NMS-4853] – unit tests on windows creates directories outside of temp directory
• [NMS-4857] – StorageStrategy documentation does not match API in code
• [NMS-4861] – Runaway threads consuming CPU when rendering certain graphs
• [NMS-4871] – With Jetty + HTTPS, certain Web UI actions prompt browser to “Save As” JSP and HTML files
• [NMS-4872] – Show all nodes with asset info not working
• [NMS-4880] – java.lang.ClassCastException when building an event notification with a category filter
• [NMS-4882] – IP address formatting does not match
• [NMS-4886] – HttpCollector ignores “port” parameter from Collectd config
• [NMS-4895] – 1.9.90 newer graphics display inconsistency – node.jsp
• [NMS-4899] – notifd DEBUG message “supress” mispelling
• [NMS-4903] – base-url not used when viewing resource graphs
• [NMS-4913] – Change StorageStrategy to throw an IllegalArgumentException when the arguments (or parameters) are not properly configured on datacollection-config.xml
• [NMS-4914] – wrong implementation of org.opennms.netmgt.correlation.Correlator$EngineAdapter.getName() ?
• [NMS-4917] – update ksc page
• [NMS-4921] – after upgrade to 1.8.13 RadiusAuthMonitor gives exception “The RADIUS Server returned the wrong Identifier”
• [NMS-4923] – Add PostgreSQL 9.1 support
• [NMS-4926] – OpenNMS 1.8.14-1 fails to install on Ubuntu Server 10.04.3 LTS , reason opennms-db depends on iplike-pgsql91 which is not available at debian.opennms.org
• [NMS-4931] – Availability Miscalculated in NodeAvailabilityReport.jrxml (and others)

Until Next Week…

As always, if there’s anything you’d like me to talk about in a future TWiO, or you just have a comment or criticism that you’d like to share, don’t hesitate to say hi.

We are an ISP. Our network is what we actually sell, and that’s why we must have top performance from our network management system. We didn’t get top performance with our old system, but we do with OpenNMS.

- Michael Micheletti, Manager of Engineering, Towerstream

Towerstream is a leading wireless service provider utilizing 4G technology to deliver advanced, high-speed Internet access to businesses in 12 markets including New York City, Boston, Los Angeles, Chicago, the San Francisco Bay Area, Miami, Seattle, Dallas/Fort Worth, Philadelphia, Nashville, Las Vegas/Reno, and the greater Providence area where the company is based.

As a fast-growing ISP that guarantees 99.99% uptime, Towerstream places great demands on its network. In mid-2010, Towerstream saw that its legacy NMS was more burden than help for Towerstream’s multiple, expanding networks. After intensive experimentation with dozens of alternatives, they switched to OpenNMS in November 2010.

“An NMS That Works For Us, Instead Of The Other Way Around”

Chris Manigan, Systems Engineer at Towerstream, doesn’t mince words when he talks about the old system:

We had to restart processes multiple times a day and kick it in the butt just to keep it working. It was the equivalent of a full time job for one person, and it didn’t even do all the things we wanted. We were monitoring ~5,000 nodes and growing every day. We needed a better solution, and we got it with OpenNMS.

OpenNMS never needs kicking. If we leave it alone, it just keeps doing what it’s supposed to do. Maybe this shouldn’t be exceptional, but we find that it is. The only time we have to restart it is when we make configuration changes. I still spend a lot of time working on our NMS, but the difference is that I’m using that time to get OpenNMS to do more and more useful things – not to keep it from dying, like I had to with the old system. It’s great to have an NMS that works for us instead of the other way around.

“We Get A Lot For What We’re Paying”

Michael Micheletti, Towerstream’s Manager of Engineering, is too professional to call out their old system by name, but he tactfully refers to it as “another widely used open source NMS from a well-known provider.”

Beyond the issue of just keeping it alive, we weren’t happy with the options they gave to address our unique needs. When we’d ask them about possible improvements, the answer usually sounded like, ‘Sure, we can probably do that for you but you have to pay us lots of money first and then more money when you get in the door – but no guarantees.’

By contrast, the OpenNMS platform does nearly all we need, out of the box with out a costly commercial version upgrade. For anything else, we can make a solution on our own or get fast help from the OpenNMS Group. The OpenNMS support contract has been very cost-effective: we’re saving time on labor, getting things done sooner, and spending less money on hardware. Just looking at these benefits, we get a lot for what we’re paying.

Bonus: Customer Service And Finance Also Get A Boost

Chris sounds more like a brand evangelist than an engineer when he talks about the extra ways they use OpenNMS:

OpenNMS shows us much more than the health of our network – it actually helps us see the health of our company. In a few months, it’s pushed different departments together: to validate data, like our inventory; automate customer service processes and streamline things that we weren’t doing as efficiently; and even identify internal policies or procedures that needed changing. Yes it’s created some more work for a few people but it’s all great work and nobody’s complaining. It’s really helped us shape up at our customer care, engineering and finance groups.

Asset Management – Not Just Network Management

As Chris describes it:

It doesn’t matter how talented you are, it’s very hard to keep track of thousands and thousands of physical assets that are constantly getting installed, removed, repaired or replaced by both your staff and your clients. Our old process required many manual steps and a lot of repetition that added up to more labor and more errors. But the SNMP asset data collection in OpenNMS pulls a serial number or other identifier off every device automatically. That gives us a fast and highly accurate inventory that’s automatically adjusted every time someone connects or disconnects a piece of hardware. This feature really set OpenNMS apart from its major competitors, none of which came with a comparable option. With OpenNMS, we can even integrate the asset data with our NetSuite® CRM for faster and more accurate customer management.

Huge Cost Savings on Installs

Michael explains the redundant, manual process they had before the switch to OpenNMS.

For every customer install, we had to update three systems: our CRM system, our inventory system and our monitoring system. We had to do the same three steps for everything from a new install, to the addition or replacement of a single device. But now with OpenNMS, we can update one system and have everything added to monitoring automatically, which saves us 10 or 15 minutes per install. When you’re doing hundreds of installs, that’s a huge cost savings right there.

Chris continues the story:

Engineering, customer service, and finance use NetSuite in different ways, but in the end they’re all using the same pool of data. When we integrated NetSuite with OpenNMS, we made a lot of processes go faster, more accurately and with less work. It was an incredible bonus that we didn’t expect when we decided to switch our network management system to OpenNMS just a few months earlier. And yet, it works so smoothly that we almost feel it should have been obvious from the start.

It’s fair to say that we wouldn’t have gotten this win without the Greenlight support contract, which brought Dave Hustace out from OpenNMS to help us ramp up the new system. Dave got us to think about our existing resources and whether we could benefit by tying any of them in with OpenNMS. Dave didn’t know we used NetSuite coming through the door – and he wasn’t at all an expert with NetSuite. But when we combined his knowledge about business processes and OpenNMS with our knowledge about NetSuite and Towerstream, the pieces just started snapping together. ‘Oh wow,’ we said, ‘look how easy it will be to pull our nodes in and keep everything up to date, without manual updates on multiple systems. The systems are actually going to update each other!’ Without Dave’s help, we would never have figured out the way we get requisitions from NetSuite.

Acquiring Another Company’s Networks – No-Brainer

Michael shares the details on Towerstream’s acquisition of another ISP.

We recently acquired another company that also managed its networks with OpenNMS, but a different installation that wasn’t the same as ours. We found that adding all the new nodes into OpenNMS was like a total no-brainer. Their CRM was significantly different from our own, so it took some work to get those records into our system, but as soon as that was done, OpenNMS was happy to just accept more nodes. It did its thing and there were no complaints.

Really, it was genius the way it was set up. We didn’t have to do anything in Engineering – we just had to wait for the IT guys to get the new networks’ CRM data into NetSuite. Once that got done with the right customer status and IP addresses, monitoring started just about automatically, with no manual configuring of anything. There was no intensive discovery effort, because OpenNMS asset management pulled all the equipment data, and automatically passed the data to the NetSuite CRM and inventory databases. In the past, someone would collect the same serial number data and MAC addresses three times: when’d we install it, we’d write it down; when operations would sign off for the customer, he’d validate the data; then when our asset manager put it into NetSuite, he’d validate the data again. But now it’s one shot. Genius.

Customer Care — Faster, Cheaper and Better

Michael shares:

Proactive customer care got better when we switched to OpenNMS because it monitors not just ICMP, but SNMP and HTTP as well. For example: in the past, if a radio’s web GUI went down and our customer care staff couldn’t log in, they’d think the radio might be down and would dispatch a tech to diagnose and perhaps repair. Now, with the SNMP and HTTP data, we have more information that lets us know if a faster and cheaper solution – like restarting a simple web interface – is all we need.

With the way we’ve configured OpenNMS, we can identify issues much faster and resolve problems before the customers even notice anything was wrong. Customers really value us when we call to warn them about a link problem before they’ve noticed it. We get them back on track sooner with less wasted effort because they’re burning time and energy to figure things out from their end. The system is very accurate and has helped us get our SLA metrics even tighter. Everybody’s winning.

Share Network Control with Finance and Customer Service?

At Towerstream, the engineering group trusts its colleagues, its controls and its network data enough to let a decision in the finance department flip a big switch on the NMS. As Chris describes,

Like every business, we have the occasional delinquent customer. If their bill gets too far behind, finance just makes a ‘didn’t pay bill’ record in NetSuite and the information automatically feeds into OpenNMS, which automatically quits monitoring every device in their network because it knows why we’ve taken their link down. It’s a smooth, automatic chain of events and we trust the OpenNMS and NetSuite integration enough to hand over the controls.

Advice to the CTO

For the CTO or CIO considering OpenNMS, Michael Micheletti has this advice:

Choose OpenNMS First for What it Does, Not for What it Costs

I’ve used everything from WhatsUp Gold to Netcool/OMNIbus, and many systems in between, like SolarWinds and Zenoss. The OpenNMS platform incorporates the best of them all. Like premium systems such as Netcool, it has all the events so you can see everything in real time. And like basic systems such as WhatsUp Gold, it’s easy to configure. You can just slap a new node on it.

When we were looking for a new NMS, our VP of Engineering and Network Operations asked us to take a wide look at options. We tested dozens of commercial and open source systems, from the well known to the barely known. In the end, OpenNMS looked like it would do 99% of what we wanted, and would do it well with comparatively modest hardware requirements. It looked like a good bet, and we found out quickly that it was.

Very Reasonable TCO Plus Strong Performance Equals “Everybody here loves OpenNMS”

The total cost of ownership of OpenNMS is very reasonable compared to other systems. The software is free, and we paid only $30k to have the OpenNMS Group do a Greenlight Plus project to help us with configuration and getting it running. In comparison, we would have to pay $175k for a commercial version upgrade to the system we had been using, and yet more money for startup help, with no guarantee we’d get rid of the slowness and unreliable performance we were getting with their free version. It’s no wonder that everybody here loves OpenNMS.

Turned Off The Old System In Less Than One Month

Getting started with OpenNMS took no time at all: we set up requisitions with our NetSuite® CRM, sucked all the nodes in and were running. We turned off the old system a few weeks later – as soon as we’d configured our data collection and display graphs the way we wanted.

Our support contract with OpenNMS was well worth the price, especially at startup. We have smart staff and can figure out almost anything we want to do with OpenNMS platform, given enough time. But taking months to do it all on our own wouldn’t be smart business.

The OpenNMS Group was a tremendous help from day one. In fact, they were supporting us even before we bought our support contract. One of the first things they did was to help us identify the right hardware. Their president, Dave Hustace, walked us through details like, ‘this is the kind of RAID array that you should use because it’s been proven to work with OpenNMS and a network with the nodes and activity plan to have over the next few years.’ We’d show him our plans and he’d let us know which parts looked good and which parts weren’t going to serve our expansion plans. Now we know we won’t need to buy new hardware next year because we outgrew ourselves.

Smart Code Under The Hood

The more we get under the hood, the smarter it looks. For example, the code saves resources by reducing the number of writes to the RRD files. OpenNMS collects data every 5 minutes but might only write every 20. That really helps performance because by storing more data in memory between writes, the system needs less time for disk access. Our old system did a write every 5 minutes for every one of our ~5,000 nodes, which crippled its performance compared to OpenNMS. The RRD file queue is just one example of how OpenNMS is both well thought out and clever.

No System is Perfect – But OpenNMS Group Quickly Gets You Closer

When we asked the Group to create several custom surveillance views, the first iteration ran very slowly – sometimes taking tens of minutes to load. They understood immediately that this was a big problem for us, so Dave and his team worked through the weekend on new code. Now it’s lightning fast, and we are convinced of both their technical savvy and their commitment to taking care of Towerstream. We’re also glad to have helped Dave spot a problem that would almost certainly affect other OpenNMS users – it’s a win for everyone when the platform gets better.

Chris Manigan adds his advice to the CTO:

OpenNMS is Not for Everyone. The Platform Helps Smart Users. The Group Helps Smart Businesses with Large Networks.

Success with the OpenNMS platform depends on the user’s level of expertise. We’ve got a good team at Towerstream and guys at the OpenNMS Group don’t treat me like I need my hand held; they know I can do pretty much anything on my own unless it’s something brand new. If you run a small network like a handful of servers in a closet, it will probably do what you need to out of the box, and you won’t need to buy the OpenNMS Group’s support. But if you have a larger network like ours that spans 11 markets across the country, their support becomes essential. If we had all the time in the world, we might be able to take care of everything, but don’t have all the time in the world. It’s great to have the OpenNMS Group’s experience and staff to get the job done faster, and smart from the beginning.

Responsive Experts And No Runaround

All of the OpenNMS Group guys are experts, so ready to help, and very responsive. They also treat me like I’m a professional, and that goes a long way. It’s a lot different from when you talk to something like a PC support line. With OpenNMS, you don’t get passed around from a Level 1 guy to a Level 2 guy to a Level 3 guy until you get where you need to be.

Everybody at OpenNMS seems to keep an eye on each others’ open cases. When I’ve opened a case, Dave might grab it to start, then Jeff will jump in with comments because he’s worked on something similar. They all work together and are way, way friendly.

“More Service Than We Pay For”

During the first week of our Greenlight startup, Dave came to our site and worked all day, then he went back to his hotel and worked all night. He worked through the weekend because he was concerned that we should get that we needed. The same thing happened when he came back for the second week of our Greenlight support. That’s way more service than we pay for.

The Bottom Line

Michael sums up the before and after with OpenNMS:

The old system was really a joke around here because it was so unreliable. At any given moment, it was either not working or it was working slowly. And don’t even think about asking the old system to do something like integrate with NetSuite. For starters, we’d have to pay $175k for a commercial version upgrade, and there’d still be no guarantee that it would work. It’s no wonder that everybody here loves OpenNMS – it’s saving us time and money in every department from engineering to customer service to finance. I have never heard a negative comment toward OpenNMS since it was installed, and Chris actually enjoys all the time he spends modifying the platform, because it keeps doing more things that he wants.

We are extremely happy with OpenNMS – both the platform, which we got for free; and the support and consulting, which we get for a very reasonable price. We are an ISP. Our network is what we actually sell, and that’s why we must have top performance from our network management system. We didn’t get top performance with our old system, but we do with OpenNMS.

New Releases

It’s been a bit since we last did a release, we pretty much put all releases on hold until we could put a 1.10 release candidate out. That said, we found a couple of reasonably important bugs (in both 1.8.x and 1.9.x) and decided it was time to take the plunge. 1.9.90 is the first release candidate for the 1.10 series, meaning that we are feature-complete and plan only on fixing important bugs before 1.10.0 gets released. That said, Linkd should be still considered beta for this release. It’s gone through a significant refactor, but still has not reached parity with 1.6 (or even 1.8) Linkd. We will not release 1.10.0 until we have that parity, but for now, if you’re relying on Linkd, stick with 1.8. If you have the ability to help us test the new Linkd code against your devices, bug reports would be helpful, however.

Podcast

Jeff has been planning on making an OpenNMS podcast for a while now, but Dev-Jam was the perfect opportunity to get a few folks on the mic and talk about the state of OpenNMS and what happened during Dev-Jam. If you subscribe to the Planet OpenNMS RSS feed, then you’ve already seen this, but if not, please check it out!

• OpenNMS Audiocast (MP3)
• OpenNMS Audiocast (Ogg Vorbis)

Project Updates

• 1.8: Current Release is 1.8.13 (Northern Lapwing)
  1.8.13 is the current stable release, tagged 10 Aug, 2011. For a complete list of changes and updates, see the “New and Noteworthy” page on the OpenNMS wiki. As always, it is recommended that you back up your database before upgrading.
• 1.9: Current release is 1.9.90 (Balafon)
  1.9.90 is the current unstable release, tagged 10 Aug, 2011. This release is not recommended for production use, but for developers and users who want to try out the very latest features, or for test environments to evaluate upgrades to 1.10 when it is released. For a list of changes and updates, see the “New and Noteworthy” page on the OpenNMS wiki.

Issues Resolved

• [JICMP-4] – jicmp IcmpSocket.h byte swapping macros don’t work on OpenIndiana
• [JRD-7] – JRRD Report runs fine in iReport but has errors when running via Reportd
• [NMS-696] – Availability report time periods
• [NMS-1769] – columnName argument to AssetModel.searchAssets allows SQL injection
• [NMS-1802] – HTTP monitor nits
• [NMS-1865] – The installer should catch systems where TCP connections cannot be made to localhost
• [NMS-2079] – Opennms not honoring changes in opennms.conf
• [NMS-2893] – Node when deleted remains in performance report list
• [NMS-2995] – Trapd is not able to process SNMPv3 traps
• [NMS-3651] – Create detectors for all protocol plugins
• [NMS-3771] – reportd missing ability to select mailer from javamail-configuration.xml
• [NMS-3905] – reload of Threshold configuration does not work – only after restart OpenNMS new threshold are applied
• [NMS-4038] – translated events are displayed like the original event
• [NMS-4350] – spring/beanfactory issue Java 1.7.0 – pointcut issues
• [NMS-4668] – Create a poller monitor to “proxy” pings via the CISCO-PING-MIB
• [NMS-4725] – LDAP authorization fails – group to role mapping does not work
• [NMS-4745] – java.net.SocketException: Too many open files
• [NMS-4817] – Null (\0) characters in logmsg field of events causes org.postgresql.util.PSQLException: ERROR: invalid byte sequence for encoding “UTF8″: 0×00
• [NMS-4862] – Add command option to NRPE in provisiond
• [NMS-4873] – Upgrade bug when Linkd tables contain data
• [NMS-4875] – Older builds ov OpenNMS are not available for download
• [NMS-4877] – Typo in datacollection-config.xml

Until Next Week…

As always, if there’s anything you’d like me to talk about in a future TWiO, or you just have a comment or criticism that you’d like to share, don’t hesitate to say hi.

IPv6 Testing

I spent the first half of the week testing IPv6, making sure that the packaging is ready and that you can have a system with only an IPv4 or IPv6 stack (but not both) and have everything still come up correctly. Also, if you need to enforce the availability (or not) of either stack, some options have been added to $OPENNMS_HOME/etc/opennms.properties to take care of that.

# By default, OpenNMS will start up if either ICMPv4 *or* ICMPv6 are
# available and initialize properly. If you wish to force IPv4 or IPv6
# explicitly, set one or both of these properties.
#
#org.opennms.netmgt.icmp.requireV4=true
#org.opennms.netmgt.icmp.requireV6=true

Smoke Tests

We’ve had pretty good unit test coverage in general, but we’ve been lacking a few things, namely, UI testing, and a “smoke test” which makes sure that OpenNMS snapshots come up properly, don’t have nasty exceptions, and other similar things.

Our intern Philip was back this summer helping us out with a few things, and he’s been working on the smoke test infrastructure for the last few weeks. I spent a bit of time putting together scripts on the server side to let us bring up a snapshot and run the smoke tests against it, from Bamboo. As of this week, the smoke tests are not only running, but green. (Yay!) Tomorrow is Philip’s last day, and we’ve heard he’s going to try again to beat the Andy’s Big-A Challenge again this year. Last year, it didn’t work out so well. I swear, this is not a hazing thing, he volunteered. Anyways, wish him luck.

Other Code Cleanups

Alejandro is wrapping up making sure that there is a Provisiond detector for everything that has a Capsd plugin. Seth and I have both been working on cleaning up miscellaneous bugs and other stuff in preparation for 1.10. Donald has finished moving all of our Ext-JS code to use GWT instead, which has given us a number of spiffy controls.

I also spent some time profiling OpenNMS, and cleaning up some easy-to-fix-stuff, including working on moving the data collection configuration parsing to JAXB and changing that code to optimize usage of string data some. Data collection should use less memory now, and be a little faster.

Project Updates

• 1.8: Current Release is 1.8.12 (Black-capped Petrel)
  1.8.12 is the current stable release, tagged 10 May, 2011. For a complete list of changes and updates, see the “New and Noteworthy” page on the OpenNMS wiki. As always, it is recommended that you back up your database before upgrading.
• 1.9: Current release is 1.9.8 (Pulalu)
  1.9.8 is the current unstable release, tagged 10 May, 2011. This release is not recommended for production use, but for developers and users who want to try out the very latest features. For a list of changes and updates, see the “New and Noteworthy” page on the OpenNMS wiki.

Issues Resolved

• [NMS-1121] – threshd.log shows wrong nodeId in certain circumstances
• [NMS-1174] – SNMP collection on NAT’d interfaces produces ifIndex of -100
• [NMS-1316] – service not being marked up after interfaceUp event.
• [NMS-4038] – translated events are displayed like the original event
• [NMS-4856] – MSExchangeDetectorClient is too verbose on exceptions
• [NMS-4859] – OpenNMS doesn’t start

Until Next Week…

As always, if there’s anything you’d like me to talk about in a future TWiO, or you just have a comment or criticism that you’d like to share, don’t hesitate to say hi.

Please send feedback to audiocast AT opennms DOT org.

0:00:25 Introduction

0:01:10 Alex Finger’s thoughts on Dev-Jam 2011

* The German OpenNMS book was written by Alex along with Ronny and Klaus
* Modbus

0:03:30 Ben Reed’s reflections

* If you’re really pedantic and need help getting to sleep, have a look at the massive git commitdiff for Ben’s GPLv3 changes
* Ben’s call on mailing list for MIB dumps

0:06:48 Jeff Gehlbach’s ramblings

* DJ Gregor could not make it to Dev-Jam this year, but he very thoughtfully sent ice cream from Jeni’s
* ExpectJ

0:10:00 Markus Neumann’s take on his first Dev-Jam

* The assets page GWT refresh and related report work will be included in the OpenNMS 1.10 release
* Matt Raykowski’s Invd work is unfolding in a branch called “feature-invd”

0:14:35 Discussion of exactly what OpenNMS 1.10 is and is not

* Today’s 1.10 snapshots are not production quality but are pretty solid.  RPM and Debian packages are available in the “unstable” repo.
* Getting the syslog message parser to work right in 1.8 and prior releases often required a black-belt in regular expressions.  The 1.10 message parsing engine uses pluggable parsers that are much easier to get going.
* IPv6 support is the hallmark feature of OpenNMS 1.10.
* Since Dev-Jam, we’ve been taking stock of the week’s accomplishments to decide which bits are suitable for inclusion in 1.10.
* Post-1.10, the original plan was to tackle a redesign of the notification subsystem, but that is not set in stone.

0:19:20 Tips for aspiring OpenNMS developers to avoid being overwhelmed by the code’s sheer size

* Ben mentions Eclipse instructions on wiki
* For most developers, the minimum set of Maven projects to import is opennms-model, opennms-dao, and probably opennms-services.

Thanks for listening to the OpenNMS audiocast.  The show is produced entirely with free software including Audacity.  You can find more information, subscribe to the show’s RSS feed, and submit your own ideas for future episodes at http://opennms.org/audiocast.  This episode and the accompanying show notes are Copyright 2011 by The OpenNMS Group, Inc. and are licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported license.

This episode’s music is from the track “Deep Sea Diver” by Heifervescent.  It’s a Creative Commons Attribution-ShareAlike-NonCommercial track.  You can find it and support the artist on Jamendo or on the artist’s own site at http://heifervescent.com.  The full-length track is included between the outro and outtakes.

The artwork is made from a Creative Commons-Attribution-2.0 licensed image from ErnestDuffoo on Flickr.

Please send feedback to audiocast AT opennms DOT org.

0:00:25 Introduction

0:01:10 Alex Finger’s thoughts on Dev-Jam 2011

* The German OpenNMS book was written by Alex along with Ronny and Klaus
* Modbus

0:03:30 Ben Reed’s reflections

* If you’re really pedantic and need help getting to sleep, have a look at the massive git commitdiff for Ben’s GPLv3 changes
* Ben’s call on mailing list for MIB dumps

0:06:48 Jeff Gehlbach’s ramblings

* DJ Gregor could not make it to Dev-Jam this year, but he very thoughtfully sent ice cream from Jeni’s
* ExpectJ

0:10:00 Markus Neumann’s take on his first Dev-Jam

* The assets page GWT refresh and related report work will be included in the OpenNMS 1.10 release
* Matt Raykowski’s Invd work is unfolding in a branch called “feature-invd”

0:14:35 Discussion of exactly what OpenNMS 1.10 is and is not

* Today’s 1.10 snapshots are not production quality but are pretty solid. RPM and Debian packages are available in the “unstable” repo.
* Getting the syslog message parser to work right in 1.8 and prior releases often required a black-belt in regular expressions. The 1.10 message parsing engine uses pluggable parsers that are much easier to get going.
* IPv6 support is the hallmark feature of OpenNMS 1.10.
* Since Dev-Jam, we’ve been taking stock of the week’s accomplishments to decide which bits are suitable for inclusion in 1.10.
* Post-1.10, the original plan was to tackle a redesign of the notification subsystem, but that is not set in stone.

0:19:20 Tips for aspiring OpenNMS developers to avoid being overwhelmed by the code’s sheer size

* Ben mentions Eclipse instructions on wiki
* For most developers, the minimum set of Maven projects to import is opennms-model, opennms-dao, and probably opennms-services.

Thanks for listening to the OpenNMS audiocast. The show is produced entirely with free software including Audacity. You can find more information, subscribe to the show’s RSS feed, and submit your own ideas for future episodes at http://opennms.org/audiocast. This episode and the accompanying show notes are Copyright 2011 by The OpenNMS Group, Inc. and are licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported license.

This episode’s music is from the track “Deep Sea Diver” by Heifervescent. It’s a Creative Commons Attribution-ShareAlike-NonCommercial track. You can find it and support the artist on Jamendo or on the artist’s own site at http://heifervescent.com. The full-length track is included between the outro and outtakes.

The artwork is made from a Creative Commons-Attribution-2.0 licensed image from ErnestDuffoo on Flickr.

Aaaaanyways, without further ado, here’s what we’ve done since, uh, well, May 16th. =)

Dev-Jam 2011

We had a total blast at Dev-Jam 2011. If you’re interested in catching up on what went on, check out Tarus’s Dev-Jam blog posts.

• Day 1
• Day 2
• Day 3
• Day 4
• Day 5

OpenNMS 1.10

We’ve been slowly working our way towards a 1.10 release candidate. The only things left are to go over the remainder of the IPv6 infrastructure (which I am spending this week on), and to then spend some time giving some attention to pending stuff in JIRA.

At the high level, there is a lot of exciting stuff coming in 1.10:

IPv6 Support

The obvious one: support for IPv6. We have done a ton of work to add support for passing around IPv6 addresses in the code at all levels, as well as adding specific support for ICMPv6.

We’ve actually got 2 different strategies for IPv6 ping:

• JNA – Uses embedded native code which is distributable. This means that ping will be supported from remote pollers!
• JICMP6 – A port of the our internal JNI ICMP code to support IPv6. This will be the default in 1.10 for polling ICMP, as it is faster than the JNA implementation.

Reporting

The new JasperReports-based reports introduced later in the 1.8 series has gone through a ton of improvements in 1.10, including a number of new reports, support for passing arguments to reports (date ranges, etc.), and other nifty stuff. Many other general updates to the reporting infrastructure have been added as well.

Installer Updates

The Liquibase database upgrade tools introduced in 1.8 have been updated to the latest (2.0.1) version, which is considerably faster than the 1.9.x version we used previously.

New XML Processing Infrastructure

While we haven’t finished converting all of our code to use it, we have started moving our Castor-based XML processing code to use JAXB instead. This provides a speed improvement when processing XML (including events posted to Eventd). This includes support for validation of inputs in the ReST interface.

Drools Support is Back

OpenNMS used to have support for using Drools for doing complex correlation of events. Due to dependency conflicts with other things we used in OpenNMS, it has been non-functional for quite some time. As of 1.10, this facility is working again, and will let you create code or scripts to do event correlation. Examples of a simple event correlation can be found in the $OPENNMS_HOME/etc/examples/ directory.

UI Updates

Many new UI changes have been made which give searchable controls for reports, nodes, etc., thanks to Donald’s new code for embedding GWT widgets in the OpenNMS UI.

Instrumentation Log Reader

Philip’s Instrumentation Log Reader tool has been updated with a number of new features, including sorting. It’s a great way to investigate data collection issues.

Syslogd Improvements

The Syslog infrastructure has been retooled to support a wider variety of input styles, as well as providing more optimized parsers for the most common Syslog formats. Using one of the specific parsers (Syslog-NG or RFC 5424) should provide performance improvements for handling large amounts of Syslog messages.

Data Collection and Thresholding Updates

Many improvements to data collection and thresholding have been made, including support for using regular expressions against column names when determining data to persist. Also, data collection configuration files have now been split like events are.

Many New Detectors

We’re adding detectors for everything that has a Capsd plugin, so you should be able to completely migrate to Provisiond.

Provisiond Updates

The provisioner has had a number of updates, including:

• UI changes
• Support for enabling or disabling polling and data collection through policies
• Support for scanning the IP-MIB ipAddressTable, which detects IPv6 interfaces on an SNMP-enabled device
• Updates to the DNS requisition import including IPv6 support, support for specifying the default services to apply to DNS-imported nodes, and the ability to determine how to compute the foreign ID (“ipAddress” or “nodeLabel”).

This is not an exhaustive list, but just what I came across doing some ‘git diff’ magic between the 1.8 and 1.10 trees. For a giant list of all of the issues closed against 1.10, see this link.

Project Updates

• 1.8: Current Release is 1.8.12 (Black-capped Petrel)
  1.8.12 is the current stable release, tagged 10 May, 2011. For a complete list of changes and updates, see the “New and Noteworthy” page on the OpenNMS wiki. As always, it is recommended that you back up your database before upgrading.
• 1.9: Current release is 1.9.8 (Pulalu)
  1.9.8 is the current unstable release, tagged 10 May, 2011. This release is not recommended for production use, but for developers and users who want to try out the very latest features. For a list of changes and updates, see the “New and Noteworthy” page on the OpenNMS wiki.

Issues Resolved

• [JRB-12] – Vertical labels disappear on a certain graphs
• [NMS-1275] – eventd truncates the length of trap messages to 256 bytes
• [NMS-2834] – Enhancement – Assets with clean date input
• [NMS-2895] – The method sanitizeString(String) is undefined for the type WebSecurityUtils
• [NMS-3164] – Implement PersistenceSelectorStrategy framework
• [NMS-3322] – XMPPNotificationStrategy (or JavaMailNotificationStrategy) does not utilise the “Numeric Message” field -nm
• [NMS-3516] – auto-clean=”true” suddenly is not working
• [NMS-3722] – Link with parentifindex -1 are not displayed in jsp pages
• [NMS-3956] – allow trapd to bind to specific address
• [NMS-3982] – Interface Deleted with SNMP supported and no ipAddrTable
• [NMS-3997] – Provisiond Deleting IpAddr On Nodes After Rescan
• [NMS-4034] – SNAPSHOT installer scripts are faulty
• [NMS-4039] – provisioning node with NodeCategorySettingPolicy policy in foreign source does not work if node has no SNMP available
• [NMS-4047] – Win32ServiceDetector fails to detect services
• [NMS-4106] – We need a WmiDetector
• [NMS-4114] – Support relativetime in graph URL
• [NMS-4208] – Memcached graph definitions left out of default configuration
• [NMS-4244] – threshd process wrong counter-type SNMP data after SNMP data collection failed or restored
• [NMS-4261] – In-line thresholder ignores scheduled outages
• [NMS-4295] – JRobin VDEFs sometimes do not work
• [NMS-4339] – Update Copyright Notice to include 2011
• [NMS-4375] – jetty allows directory listings
• [NMS-4376] – Problems adding nodes during discovery
• [NMS-4423] – Provisiond NodeCategorySettingPolicy. Nodes lose categorization in subsequent Synchronizations
• [NMS-4463] – SnmpAsset Adapter has dependency on Trapd
• [NMS-4483] – Services drop down list not alphabatized
• [NMS-4494] – SiblingIndexStorageStrategy does SNMP Queries and makes collection VERY slow
• [NMS-4500] – Collectd’s ServiceCollector class was erroneously changed to take Map<String, String>
• [NMS-4530] – IPAddress class overrides equals but not hashCode
• [NMS-4545] – Importer doesn’t log to correct log4j category
• [NMS-4546] – Provisiond silently fails to import an invalid model importer file but reports importSuccessful anyway.
• [NMS-4568] – Policy Rule using ipAddress with Match Snmp Policy does not work
• [NMS-4569] – ipinterface.jsp and snmpinterface.jsp does not show snmp poller data
• [NMS-4570] – link.jsp does not show interface details for some entries
• [NMS-4572] – Let the Snmp Poller send down event the first time an interface is found “Down”
• [NMS-4585] – hardware asset fields need more space and one more field
• [NMS-4590] – Node label changes ourside requisition editor on nodes with a foreign-source ID
• [NMS-4623] – Provision Groups do not synchronize on startup
• [NMS-4633] – Add an optional services field to the query string of the DNS URL
• [NMS-4655] – Returns directory of “opennms/admin/notification/” when required fields are not provided for notification destination paths.
• [NMS-4657] – Outstanding notice page not aligning correctly
• [NMS-4663] – Capsd may reparent duplicate interfaces from requisitioned nodes
• [NMS-4682] – Permissions on multiple files/directories are poor (allow world-write, have setuid)
• [NMS-4691] – OpenNMS GoogleMaps geo-enocder no longer functioning and creates a 503 error in UI
• [NMS-4692] – Email Notifications are not properly encoded when the message contains non us-ascii characters.
• [NMS-4694] – Better provisiond debugging
• [NMS-4695] – Provisiond allows duplicate nodes
• [NMS-4697] – Path not filtered correctly during build, etc/response-graph.properties
• [NMS-4698] – Surveilance part of WEB GUI crashes after all default categories were removed and custom ones were created
• [NMS-4700] – snmpStorageFlag=”all” is being ignored by the threshold procesing
• [NMS-4701] – Unable to add IPv6 address for discovery via web UI
• [NMS-4703] – Data Collection Broken for some nodes in testing
• [NMS-4705] – Add time it takes to persist the data to the ILR
• [NMS-4706] – Enable support for filtering the displayed data
• [NMS-4707] – Add ability for HttpMonitor to use node label as virtual host for HTTP polls
• [NMS-4708] – MicroBlog doesn’t support -nm
• [NMS-4710] – Erroneous text on a node resource graph selection page
• [NMS-4712] – Add the possibility to modify eventparms (as text) from Vacuumd
• [NMS-4717] – race condition in Provisiond IPv6 scanning
• [NMS-4719] – Persist and Error Messages do not Take Date into Account
• [NMS-4721] – DbHelper class should use dao’s
• [NMS-4722] – Add new opennms mib events definition
• [NMS-4728] – Sort by Stat
• [NMS-4734] – When using the GoogleMaps remote poller interface, unchecked markers are visible on initialization
• [NMS-4742] – Add Label to thresholds for display on “Edit Group” page
• [NMS-4752] – Event Analysis Report should be usable on postgres older than 8.4
• [NMS-4753] – Event Analysis report is missing in default configuration
• [NMS-4770] – Traffic Report for last month returns data from the current month
• [NMS-4772] – Add support for matching syslog messages by process name, severity, facility in ueiMatch
• [NMS-4776] – Alphabetize group names in the Users/Groups list
• [NMS-4777] – nodeList page fails to pass the foreignSource when “show interfaces” is selected
• [NMS-4778] – Time zone WEST is not able to cast in a postgres date
• [NMS-4779] – Split syslogd-configuration.xml
• [NMS-4783] – DNS provisioning expression matching matches hostname but not record data
• [NMS-4784] – Warning and Critical thresholds in InterfaceAvailabilityReport and SnmpInterfaceOperAvailabilityReport
• [NMS-4785] – Standardize the time zone format reports
• [NMS-4801] – DNS provisioning needs to allow foreign ID to be a hash of IP address instead of nodeLabel if administrator so chooses
• [NMS-4808] – ICMP concurrency issue in JnaPinger, JniPinger, Jni6Pinger
• [NMS-4809] – collectd log entries could be enhanced
• [NMS-4819] – SELECT tag not closed in asset/modify.jsp
• [NMS-4820] – Reportd JavaMailDeliveryService always copies address in sendmail-message “to” attribute or root@localhost
• [NMS-4822] – Trapd node-matching should prefer SNMP primary ifaces
• [NMS-4832] – AttributeGroup and SiblingColumnStorageStrategy are producing a StackOverflowError
• [NMS-4837] – Add a BSF (bean scripting framework) notification strategy
• [NMS-4846] – Provisiond leaks file handles, eventually causing “Too many open files” crashes
• [NMS-4847] – Upgraded to 1.8.12 and I started to recieve an error when I try to see node information
• [NMS-4850] – Convert Linkd to use Hibernate
• [RANCID-4] – RWS does not support inventory for ios-xr devices

Until Next Week…

As always, if there’s anything you’d like me to talk about in a future TWiO, or you just have a comment or criticism that you’d like to share, don’t hesitate to say hi.