We are an ISP. Our network is what we actually sell, and that’s why we must have top performance from our network management system. We didn’t get top performance with our old system, but we do with OpenNMS.

- Michael Micheletti, Manager of Engineering, Towerstream

Towerstream is a leading wireless service provider utilizing 4G technology to deliver advanced, high-speed Internet access to businesses in 12 markets including New York City, Boston, Los Angeles, Chicago, the San Francisco Bay Area, Miami, Seattle, Dallas/Fort Worth, Philadelphia, Nashville, Las Vegas/Reno, and the greater Providence area where the company is based.

As a fast-growing ISP that guarantees 99.99% uptime, Towerstream places great demands on its network. In mid-2010, Towerstream saw that its legacy NMS was more burden than help for Towerstream’s multiple, expanding networks. After intensive experimentation with dozens of alternatives, they switched to OpenNMS in November 2010.

“An NMS That Works For Us, Instead Of The Other Way Around”

Chris Manigan, Systems Engineer at Towerstream, doesn’t mince words when he talks about the old system:

We had to restart processes multiple times a day and kick it in the butt just to keep it working. It was the equivalent of a full time job for one person, and it didn’t even do all the things we wanted. We were monitoring ~5,000 nodes and growing every day. We needed a better solution, and we got it with OpenNMS.

OpenNMS never needs kicking. If we leave it alone, it just keeps doing what it’s supposed to do. Maybe this shouldn’t be exceptional, but we find that it is. The only time we have to restart it is when we make configuration changes. I still spend a lot of time working on our NMS, but the difference is that I’m using that time to get OpenNMS to do more and more useful things – not to keep it from dying, like I had to with the old system. It’s great to have an NMS that works for us instead of the other way around.

“We Get A Lot For What We’re Paying”

Michael Micheletti, Towerstream’s Manager of Engineering, is too professional to call out their old system by name, but he tactfully refers to it as “another widely used open source NMS from a well-known provider.”

Beyond the issue of just keeping it alive, we weren’t happy with the options they gave to address our unique needs. When we’d ask them about possible improvements, the answer usually sounded like, ‘Sure, we can probably do that for you but you have to pay us lots of money first and then more money when you get in the door – but no guarantees.’

By contrast, the OpenNMS platform does nearly all we need, out of the box with out a costly commercial version upgrade. For anything else, we can make a solution on our own or get fast help from the OpenNMS Group. The OpenNMS support contract has been very cost-effective: we’re saving time on labor, getting things done sooner, and spending less money on hardware. Just looking at these benefits, we get a lot for what we’re paying.

Bonus: Customer Service And Finance Also Get A Boost

Chris sounds more like a brand evangelist than an engineer when he talks about the extra ways they use OpenNMS:

OpenNMS shows us much more than the health of our network – it actually helps us see the health of our company. In a few months, it’s pushed different departments together: to validate data, like our inventory; automate customer service processes and streamline things that we weren’t doing as efficiently; and even identify internal policies or procedures that needed changing. Yes it’s created some more work for a few people but it’s all great work and nobody’s complaining. It’s really helped us shape up at our customer care, engineering and finance groups.

Asset Management – Not Just Network Management

As Chris describes it:

It doesn’t matter how talented you are, it’s very hard to keep track of thousands and thousands of physical assets that are constantly getting installed, removed, repaired or replaced by both your staff and your clients. Our old process required many manual steps and a lot of repetition that added up to more labor and more errors. But the SNMP asset data collection in OpenNMS pulls a serial number or other identifier off every device automatically. That gives us a fast and highly accurate inventory that’s automatically adjusted every time someone connects or disconnects a piece of hardware. This feature really set OpenNMS apart from its major competitors, none of which came with a comparable option. With OpenNMS, we can even integrate the asset data with our NetSuite® CRM for faster and more accurate customer management.

Huge Cost Savings on Installs

Michael explains the redundant, manual process they had before the switch to OpenNMS.

For every customer install, we had to update three systems: our CRM system, our inventory system and our monitoring system. We had to do the same three steps for everything from a new install, to the addition or replacement of a single device. But now with OpenNMS, we can update one system and have everything added to monitoring automatically, which saves us 10 or 15 minutes per install. When you’re doing hundreds of installs, that’s a huge cost savings right there.

Chris continues the story:

Engineering, customer service, and finance use NetSuite in different ways, but in the end they’re all using the same pool of data. When we integrated NetSuite with OpenNMS, we made a lot of processes go faster, more accurately and with less work. It was an incredible bonus that we didn’t expect when we decided to switch our network management system to OpenNMS just a few months earlier. And yet, it works so smoothly that we almost feel it should have been obvious from the start.

It’s fair to say that we wouldn’t have gotten this win without the Greenlight support contract, which brought Dave Hustace out from OpenNMS to help us ramp up the new system. Dave got us to think about our existing resources and whether we could benefit by tying any of them in with OpenNMS. Dave didn’t know we used NetSuite coming through the door – and he wasn’t at all an expert with NetSuite. But when we combined his knowledge about business processes and OpenNMS with our knowledge about NetSuite and Towerstream, the pieces just started snapping together. ‘Oh wow,’ we said, ‘look how easy it will be to pull our nodes in and keep everything up to date, without manual updates on multiple systems. The systems are actually going to update each other!’ Without Dave’s help, we would never have figured out the way we get requisitions from NetSuite.

Acquiring Another Company’s Networks – No-Brainer

Michael shares the details on Towerstream’s acquisition of another ISP.

We recently acquired another company that also managed its networks with OpenNMS, but a different installation that wasn’t the same as ours. We found that adding all the new nodes into OpenNMS was like a total no-brainer. Their CRM was significantly different from our own, so it took some work to get those records into our system, but as soon as that was done, OpenNMS was happy to just accept more nodes. It did its thing and there were no complaints.

Really, it was genius the way it was set up. We didn’t have to do anything in Engineering – we just had to wait for the IT guys to get the new networks’ CRM data into NetSuite. Once that got done with the right customer status and IP addresses, monitoring started just about automatically, with no manual configuring of anything. There was no intensive discovery effort, because OpenNMS asset management pulled all the equipment data, and automatically passed the data to the NetSuite CRM and inventory databases. In the past, someone would collect the same serial number data and MAC addresses three times: when’d we install it, we’d write it down; when operations would sign off for the customer, he’d validate the data; then when our asset manager put it into NetSuite, he’d validate the data again. But now it’s one shot. Genius.

Customer Care — Faster, Cheaper and Better

Michael shares:

Proactive customer care got better when we switched to OpenNMS because it monitors not just ICMP, but SNMP and HTTP as well. For example: in the past, if a radio’s web GUI went down and our customer care staff couldn’t log in, they’d think the radio might be down and would dispatch a tech to diagnose and perhaps repair. Now, with the SNMP and HTTP data, we have more information that lets us know if a faster and cheaper solution – like restarting a simple web interface – is all we need.

With the way we’ve configured OpenNMS, we can identify issues much faster and resolve problems before the customers even notice anything was wrong. Customers really value us when we call to warn them about a link problem before they’ve noticed it. We get them back on track sooner with less wasted effort because they’re burning time and energy to figure things out from their end. The system is very accurate and has helped us get our SLA metrics even tighter. Everybody’s winning.

Share Network Control with Finance and Customer Service?

At Towerstream, the engineering group trusts its colleagues, its controls and its network data enough to let a decision in the finance department flip a big switch on the NMS. As Chris describes,

Like every business, we have the occasional delinquent customer. If their bill gets too far behind, finance just makes a ‘didn’t pay bill’ record in NetSuite and the information automatically feeds into OpenNMS, which automatically quits monitoring every device in their network because it knows why we’ve taken their link down. It’s a smooth, automatic chain of events and we trust the OpenNMS and NetSuite integration enough to hand over the controls.

Advice to the CTO

For the CTO or CIO considering OpenNMS, Michael Micheletti has this advice:

Choose OpenNMS First for What it Does, Not for What it Costs

I’ve used everything from WhatsUp Gold to Netcool/OMNIbus, and many systems in between, like SolarWinds and Zenoss. The OpenNMS platform incorporates the best of them all. Like premium systems such as Netcool, it has all the events so you can see everything in real time. And like basic systems such as WhatsUp Gold, it’s easy to configure. You can just slap a new node on it.

When we were looking for a new NMS, our VP of Engineering and Network Operations asked us to take a wide look at options. We tested dozens of commercial and open source systems, from the well known to the barely known. In the end, OpenNMS looked like it would do 99% of what we wanted, and would do it well with comparatively modest hardware requirements. It looked like a good bet, and we found out quickly that it was.

Very Reasonable TCO Plus Strong Performance Equals “Everybody here loves OpenNMS”

The total cost of ownership of OpenNMS is very reasonable compared to other systems. The software is free, and we paid only $30k to have the OpenNMS Group do a Greenlight Plus project to help us with configuration and getting it running. In comparison, we would have to pay $175k for a commercial version upgrade to the system we had been using, and yet more money for startup help, with no guarantee we’d get rid of the slowness and unreliable performance we were getting with their free version. It’s no wonder that everybody here loves OpenNMS.

Turned Off The Old System In Less Than One Month

Getting started with OpenNMS took no time at all: we set up requisitions with our NetSuite® CRM, sucked all the nodes in and were running. We turned off the old system a few weeks later – as soon as we’d configured our data collection and display graphs the way we wanted.

Our support contract with OpenNMS was well worth the price, especially at startup. We have smart staff and can figure out almost anything we want to do with OpenNMS platform, given enough time. But taking months to do it all on our own wouldn’t be smart business.

The OpenNMS Group was a tremendous help from day one. In fact, they were supporting us even before we bought our support contract. One of the first things they did was to help us identify the right hardware. Their president, Dave Hustace, walked us through details like, ‘this is the kind of RAID array that you should use because it’s been proven to work with OpenNMS and a network with the nodes and activity plan to have over the next few years.’ We’d show him our plans and he’d let us know which parts looked good and which parts weren’t going to serve our expansion plans. Now we know we won’t need to buy new hardware next year because we outgrew ourselves.

Smart Code Under The Hood

The more we get under the hood, the smarter it looks. For example, the code saves resources by reducing the number of writes to the RRD files. OpenNMS collects data every 5 minutes but might only write every 20. That really helps performance because by storing more data in memory between writes, the system needs less time for disk access. Our old system did a write every 5 minutes for every one of our ~5,000 nodes, which crippled its performance compared to OpenNMS. The RRD file queue is just one example of how OpenNMS is both well thought out and clever.

No System is Perfect – But OpenNMS Group Quickly Gets You Closer

When we asked the Group to create several custom surveillance views, the first iteration ran very slowly – sometimes taking tens of minutes to load. They understood immediately that this was a big problem for us, so Dave and his team worked through the weekend on new code. Now it’s lightning fast, and we are convinced of both their technical savvy and their commitment to taking care of Towerstream. We’re also glad to have helped Dave spot a problem that would almost certainly affect other OpenNMS users – it’s a win for everyone when the platform gets better.

Chris Manigan adds his advice to the CTO:

OpenNMS is Not for Everyone. The Platform Helps Smart Users. The Group Helps Smart Businesses with Large Networks.

Success with the OpenNMS platform depends on the user’s level of expertise. We’ve got a good team at Towerstream and guys at the OpenNMS Group don’t treat me like I need my hand held; they know I can do pretty much anything on my own unless it’s something brand new. If you run a small network like a handful of servers in a closet, it will probably do what you need to out of the box, and you won’t need to buy the OpenNMS Group’s support. But if you have a larger network like ours that spans 11 markets across the country, their support becomes essential. If we had all the time in the world, we might be able to take care of everything, but don’t have all the time in the world. It’s great to have the OpenNMS Group’s experience and staff to get the job done faster, and smart from the beginning.

Responsive Experts And No Runaround

All of the OpenNMS Group guys are experts, so ready to help, and very responsive. They also treat me like I’m a professional, and that goes a long way. It’s a lot different from when you talk to something like a PC support line. With OpenNMS, you don’t get passed around from a Level 1 guy to a Level 2 guy to a Level 3 guy until you get where you need to be.

Everybody at OpenNMS seems to keep an eye on each others’ open cases. When I’ve opened a case, Dave might grab it to start, then Jeff will jump in with comments because he’s worked on something similar. They all work together and are way, way friendly.

“More Service Than We Pay For”

During the first week of our Greenlight startup, Dave came to our site and worked all day, then he went back to his hotel and worked all night. He worked through the weekend because he was concerned that we should get that we needed. The same thing happened when he came back for the second week of our Greenlight support. That’s way more service than we pay for.

The Bottom Line

Michael sums up the before and after with OpenNMS:

The old system was really a joke around here because it was so unreliable. At any given moment, it was either not working or it was working slowly. And don’t even think about asking the old system to do something like integrate with NetSuite. For starters, we’d have to pay $175k for a commercial version upgrade, and there’d still be no guarantee that it would work. It’s no wonder that everybody here loves OpenNMS – it’s saving us time and money in every department from engineering to customer service to finance. I have never heard a negative comment toward OpenNMS since it was installed, and Chris actually enjoys all the time he spends modifying the platform, because it keeps doing more things that he wants.

We are extremely happy with OpenNMS – both the platform, which we got for free; and the support and consulting, which we get for a very reasonable price. We are an ISP. Our network is what we actually sell, and that’s why we must have top performance from our network management system. We didn’t get top performance with our old system, but we do with OpenNMS.

In six hours, we had OpenNMS running with all the functionality that it had taken us three years to develop with Openview NNM. This was a rude but happy awakening.

- Dale Meyerhoffer, Senior Network Analyst, Hershey Medical Center

The Pennsylvania State Milton S. Hershey Medical Center

The Penn State University Milton S. Hershey Medical Center uses OpenNMS to manage networks at its medical center, medical school, children’s hospital and numerous clinics. They switched to OpenNMS from HP Openview Network Node Manager (NNM) in 2008 and have realized significant gains in network operations while also reducing costs in both time and direct expenses. The following are comments by Dale Meyerhoffer, Senior Network Analyst, on his experiences with OpenNMS.

The Switch from Openview NNM to OpenNMS

Before OpenNMS, the Hershey Medical Center used HP Openview NNM for its network management. By 2008, we had reached its optimal capability but were still missing many things we needed such as graphs, notifications, and other tools to easily alert us to issues. We were frustrated by the lack of these things, as well as several simple reports that we thought should be available.

We made a list of all the things we wanted from an NMS that might replace Openview NNM. First, we looked at commercial software, then we looked at Zenoss and other open source solutions. OpenNMS (v 1.3.9 at the time) seemed to fit most of what we wanted to do. We didn’t yet know how sophisticated the programming was beneath the surface, but the default screens and menus gave us information on the complicated things that we needed, and we were impressed by the simplicity of its design, so we decided to give it a try.

After nearly a year of testing OpenNMS on the side, we reached a decision point when our Openview NNM installation began calling for a $75k upgrade. That potential expense tipped the scales in favor of OpenNMS, and we committed to the conversion. In six hours, we had OpenNMS running with all the functionality that it had taken us three years to develop with Openview NNM. This was a rude but happy awakening.

How do the finances add up? Here’s the comparison we made: $40 to $50k to install Openview NNM plus consulting time covered us for the first three years. To get the next update, we would have to spend $75k (including a new server). In contrast, the OpenNMS software was free, and we added a $5k “getting started” consult to make sure the process went smoothly. The OpenNMS support contract price was comparable to that of Openview NNM. From a cash standpoint, OpenNMS was the clear winner. But more importantly, OpenNMS does a better job of helping us manage our network, and requires less of our time to do it. Openview NNM took up 90% of my time; OpenNMS requires 10-15 hours, combined, from me and one colleague. Lower investment plus higher return is a great solution.

Comparing Openview NNM and OpenNMS – Functionality

With Openview NNM, it was very difficult to produce graphs, and there were limits on how many we could do. To add insult to injury, every time we did an upgrade, the graphics broke and it took us two months to recover. Graphs are easy in OpenNMS.

Metrics with Openview NNM could take 12 or 16 hours to produce. With OpenNMS, it takes only a couple of clicks and two minutes. Clearly, we gained a lot of time there.

With Openview NNM, it was difficult to customize alerts. With OpenNMS, we customize alerts all the time. We save time and headaches by customizing whether an alert goes to email vs. pager, based on severity or any other criterion. That saves us a lot of time and headache. Of course, we also benefit from OpenNMS’s constant improvement and evolution. OpenNMS is adding new functionality all the time, and we just have to tweak it for our own use. In one instance, we had plans to manage investigative events in a particular way, but Jeff Gelbach at OpenNMS convinced us that alarming and notifications would take away our problem and some of our administrative time – a double win.

OpenNMS Platform – Integration and Customization

With the switch to OpenNMS, we’ve reduced operational costs. We’ve started to find ways to save money by running scripts in the background that are tied to the database, which is something we could never do with Openview NNM. We’ve come up with solutions so that OpenNMS now tells not only that a switch is down, but what rooms are down, and what departments are affected. We’re tying this to a department phone directory so that we know who to call before they call us.

We also use OpenNMS to manage UPS’s – a simple but important task. We’re using OpenNMS to see which UPS’s are too hot and won’t last the five years that would otherwise be expected. We’re being proactive.

Working on the OpenNMS platform is just great. Most of the time, we can make changes all within the XML. It’s very easy to copy a section, paste it elsewhere and make the adjustments we need. We could never even think about doing that before.

Support from the OpenNMS Group

Support from the OpenNMS Group has been outstanding and they consistently give us better support than we’re paying for. Sometimes they giving us fast responses to the support tickets we file outside our service hours at 4 in the morning or 9 at night. Other times, they give us quick but very effective consulting advice even though we only pay for support and not consulting. During a recent challenge, Tarus Balog worked with me until 11 at night, saying, “I’m not giving up on you and I’m not getting off the phone until you tell me this is done right.” As it turns out, the problem was caused by an error on my part, not a problem with OpenNMS, but the point is that the OpenNMS team is committed to making sure we’re taken care of.

In related instance, Jeff Gelbach inferred from our logs that we had a problem with the database, not with the network. He walked our team through the solution and the network has been working like a dream since then. The level at which they understand our system is not typical for most companies. They are small enough to be intimate, and intelligent enough to tie things together where other companies would be blind. I have every confidence this will remain true even if the OpenNMS Group grows. That’s just they way they run the business.

What the CTO Should Know about OpenNMS

Who is a good client for OpenNMS? Anybody who has services running that they want running all the time with software that’s 100 percent free. That is phenomenal in and of itself.

Big commercial providers do not give you more than open source! That’s a myth that needs to go away. OpenNMS performance – both the platform and the Group – is the standard that our medical center is beginning to expect from all of our vendors. The OpenNMS platform is powerful, flexible and easy to use. We can do things with OpenNMS that we could never have done with Openview NNM, or could never have done so quickly. Support with OpenNMS is excellent, while support with Openview NNM was bad in the beginning and then got worse, to the point that we were depending on third party vendors who knew Openview NNM. The solution was far from optimal.

The learning curve for OpenNMS is very inexpensive. It will cost you nothing but a little bit of hardware to try it, and the installation process is simple. The key ingredient to OpenNMS’s success is the quality of the thinking that goes into it. The way they’ve designed it is very intelligent, and that’s why you can start to navigate with any application with a few minutes. You won’t find that in many of the big commercial packages that require much more configuration and effort.

Before we fully committed to OpenNMS, I was naturally worried that they didn’t have enough people to provide good support. David Hustace eliminated most of my worries when he asked, “in the year you’ve been running OpenNMS on the side, how many problems have you had?” To my surprise, I realized that the answer was “none.” David pointed out that 80% of support for bad packages is needed because the product isn’t designed well. With OpenNMS, you only need support when you change something, and you can plan for that. Since then, we’ve observed that the OpenNMS Group provides exceptionally fast and thorough support on the rare occasions that we need it. The support comes from engineers who developed the platform and know what they are doing, and that makes the support immediately effective. We are very satisfied, and happy we’re not getting multiple layers of ineffective support that is typical of large commercial vendors.

If you have a large enterprise and you’re dealing with many different types of equipment and things that aren’t the easiest to build around, it’s great to have that support. I can say to OpenNMS, “I’ve got a brand new device that you’ve never seen before – can you help me with the midbuild?” In forty-five minutes, they’ll send something back and say “here you go”.

We really win with the ability to customize OpenNMS. Any large enterprise is going to want custom things that no commercial or open source vendor will give you out of the box. OpenNMS in its current release covers 80% of what we’re hoping for in the forseeable future. We’re going to build that last 20% to do exactly what we want – for example, perhaps tying the network database with another database so that we can make a visual picture of every closet, what’s in it, and where it’s located in our buildings. When we’re done, I’ll present our change to OpenNMS and say “we don’t know if any of your other users need this, but here you go.” That will advance the project, and OpenNMS will keep getting better at no charge to everyone who uses it. That’s the open source advantage. And unlike the commercial source we mentioned earlier, access to the upgrade won’t cost $75k.

I was the only IT person at the office, but with OpenNMS, it was easy to keep track of everything on the network, including our snow and ice-covered traffic light system. If there had been a problem, OpenNMS would have let me know.

- John Jennery, City of Grapevine, Texas

City of Grapevine and OpenNMS – Case Study and Customer Story

Traffic matters in Grapevine, Texas. The Dallas-Ft. Worth airport takes up the southern third of the city, which is part of one of the country’s largest metro areas; and Grapevine has a bustling, historic downtown district of its own. To keep the cars moving, Grapevine uses one of the nation’s first traffic light systems that automatically adjusts its timing in response to real-time traffic. A Motorola Canopy® radio network connects the pole-mounted cameras and controls that the City installed in late 2010.

In the first week of February 2011, Grapevine had a unique opportunity to test the traffic system under stress, when Superbowl XLV brought an extra 150,000 people to the area just days after a rare snowstorm laid down 3 inches of snow and ice. Would the cameras, controls and radios hold up under extreme weather, or would the system break down just in time for the Superbowl traffic?

Fortunately for Grapevine, the traffic system did just fine. And fortunately for the City’s IT Manager, John Jennery, OpenNMS made it easy to be sure the system never went down. As Jennery tells the story:

Right after the storm, the roads in Grapevine and Dallas-Ft.Worth weren’t in great shape, so we told most of our staff to stay home. I was the only IT person at the office, but with OpenNMS, it was easy to keep track of everything on the network, including our snow and ice-covered traffic light system. If there had been a problem, OpenNMS would have let me know.

In support of the traffic light system, our main job in IT is to know if a camera is working and if a unit is online. What makes our traffic light management process different from regular network management is that we’re working with radio, and the nature of radio is that the signal goes up and down all the time. A standard protocol could give us false positives for down equipment, so we had to create a special protocol with a properly timed series of pings that would give us an accurate status.

When the City decided to install the traffic system, we knew that OpenNMS– which we’d been using since 2004 – had all the flexibility we needed. In our first few months with the traffic light system, we’ve had no trouble adjusting the alarms, the monitoring logic and the XML to make it work for us. We also make heavy use of the OpenNMS iPhone app to monitor immediate conditions and the last few days of data.

The Traffic Department is one of many whose functions are supported by Grapevine IT network. As the City’s IT Manager since 1991, Jennery knows what network management was like before and after OpenNMS, and he’s quick to point out the benefits:

OpenNMS Works Better and Does More than Industry’s Leading “Solutions”

Before moving to OpenNMS, we spent two years working with one of the leading proprietary solutions for network management. Our initial software setup was successful, but then we lost access to our first support engineer. The next support person we got was totally useless. He gave up after failing to solve several of our challenges and then he just disappeared. At this point, we still couldn’t get alerts to work right and had to use a back door to manage many of our basic tasks. The software required agents that conflicted with nearly everything on our network and the best we could do was a ping test. It was horrible.

After our first year, we still had so many problems. They suggested we get onto the latest version of their software for another $38k plus $7k or $8k per year in support. No thank you! By that time, it was clear to us that the proprietary software could never pull off several functions that were very important to us what we later discovered OpenNMS does out of the box.

OpenNMS Proved Itself Quickly

In just three months with OpenNMS, we threw out our old system. That was all the time we needed to get familiar with OpenNMS and make enough modifications to run our network about as well as it had been. OpenNMS replaced the proprietary system that we were using to monitor our physical infrastructure, along with another well-known open source system that we were using for service monitoring. While no network management system can be perfect for every user, we have found that OpenNMS comes much closer than any other.

Total Solutions with the OpenNMS Group and Platform

Nearly always, we can do our own fine-tuning on OpenNMS to do things like change a variable or make the alerts more useful to us. But when we need to, we can ask the OpenNMS Group for help. For example, Grapevine has an atypical approach to scheduling our IT staff, so we needed to customize OpenNMS to coordinate high and low-priority alert messages to fit our staff availability. OpenNMS couldn’t handle these needs out of the box, but the OpenNMS Group made a few quick changes to our config and we were set. After they showed me what they’d done, I could clone the process and take care of everything from there.

The OpenNMS Group has been absolutely great – no matter which support engineer we find ourselves talking to. I highly recommend the model of using the free OpenNMS platform and paying the Group for occasional support. With the previous software vendor, I remember that when we’d call the support engineers, they often couldn’t help me do what I needed. But when we consult with the OpenNMS Group, they often give us not only the answer to the question we asked, but also give us tips for an even better approach.

Support at OpenNMS is very fast and very strong. We treat each other with professional respect, which means that I can tell them whether my request is an “I need this now” or “I need this next week” so that they can schedule their help exactly as I need it. I don’t feel like I have to play games to get timely support.

“With the OpenNMS Solution – We Have No Reason to Look Elsewhere”

There’s no tradeoff between effectiveness and cost with the OpenNMS solution. Virtually all of our special needs are covered with OpenNMS – either with solutions that come out of the box, or with modifications that we can do on our own or with help from the OpenNMS Group’s. Meanwhile, we are saving tens of thousands of dollars in both cash (for expensive proprietary software and support) and staff time. With the OpenNMS Solution we have no reason to look elsewhere.

Our busiest dataflow for online ordering is dinner and lunch on Friday, Saturday and Sunday. Despite this, we only need one staffer on the weekend – for all of Papa John’s 3,400 locations.

- Chris Rodman, Papa John’s International

Papa John’s and OpenNMS – Case Study and Testimonial

With more than 3,400 stores, Papa John’s is the world’s third-largest pizza company. Papa John’s is renowned for its commitment to breaking technology barriers to provide an unprecedented customer experience.

Great customer experience demands the best network technology and business service management partner. Since 2004, Papa John’s has successfully used the OpenNMS platform and the OpenNMS Group as a powerful, flexible and extremely cost-effective network management solution.

The Customer Challenge

Papa John’s was the first pizza take-out company to institute text and online ordering at 100% of its domestic locations. The company also uses a wide array of leading edge technologies to connect more deeply with its customers, including social media, augmented reality and video.

In 2004, IT managers at Papa John’s International had come to the conclusion that their existing network management system (NMS) was not serving them as well as it should. The system was labor-intensive and lacked both the reliability and flexibility that Papa John’s needed to manage its ever-growing network.

In particular, Papa John’s observed that:

• The existing NMS (CA Unicenter) was inflexible, high-cost and labor-intensive.
• Administrators and technicians were unhappy with the agent-based NMS, which often hampered network performance instead of improving it.
• NMS software costs were rising rapidly, into the multiple hundreds of thousands of dollars each year.

The OpenNMS Solution

Papa John’s installed open source-based OpenNMS in late 2004, and has since obtained a continuing series of successes with better network management and lower operating costs. Among the results:

• Papa John’s reduced its network operating costs drastically.
• Network performance is measurably better, resulting in more satisfied internal and external customers.
• Network management tools are more secure, flexible, scalable and reliable.
• The Network Operating Center is now working faster, better and on a much larger scale.

In the five years since Papa John’s first installed OpenNMS, the pizza company and its network have continued to grow. As a network manager, Chris Rodman describes their system:

Right now in Louisville we’re monitoring around 816 nodes, which correspond to approximately 1,412 actual interfaces and 5,747 services. In 2010, things will ramp up very quickly when we add OpenNMS to the stores. We plan to start monitoring approximately 3,000 nodes and then we plan to monitor other devices in the stores, which would take us up as high as 30,000. Like I’ve said before – OpenNMS is scalable.

Remarkably, just one operator can manage the entire network using OpenNMS.

According to Chris:

Our busiest dataflow for online ordering is dinner and lunch on Friday, Saturday and Sunday. Despite this, we could operate throughout the weekend with one staffer – for all of Papa John’s 3,400 locations – if needed.

Papa John’s customers have noticed. Papa John’s was honored by Restaurants & Institutions Magazine (R&I) with the 2009 Gold Award for Consumers’ Choice in Chains in the pizza segment, ranked first among pizza companies in the 2008 Brand Keys Customer Loyalty Engagement Index, and was named 2007 Pizza Today Chain of the Year.

Why Use the OpenNMS Group?

The OpenNMS technology platform is backed by people who share Papa John’s commitment to customer service, the OpenNMS Group.

The small staff at Papa John’s frequently customizes OpenNMS – something they can do because OpenNMS is open source. But they also have a support contract with the OpenNMS Group, and sometimes hire the Group to speed the process.

As Chris puts it:

Our business requires a great deal of customized monitoring. Our brand was built on quality and that extends throughout the system, even to the configuration of our monitoring solution. That’s where the team at OpenNMS has assisted us tremendously. They’ve always helped us find the best solution.

The OpenNMS Group is truly customer driven. They sell you what you need to get the job done. Other NMS providers are money driven with twice the sales staff. But OpenNMS people don’t beat around the bush. They’re not selling vapor. They tell me like it is; when I should expect to see what I want, or what I honestly won’t see. The OpenNMS Group is a valued partner.

In the last five years we’ve gone from 10,000 interfaces to 160,000 interfaces. It’s always going up and now there are more than a million RRD files on our Net App. And yet, our internal support costs have dropped dramatically as we got things more stable and acquired better management tools.

- Scott McLaughlin, Director Infrastructure Operations    

New Edge Networks and OpenNMS – Case Study and Testimonial

New Edge Networks provides managed IP-based networks through a mix of private and public connections to deliver secure, managed network solutions. Connecting business sites at virtually any address in the United States, New Edge allows businesses to choose any blend of available access technologies – from DSL and T1 lines to fiber-optic connections – for building private and secure MPLS networks with performance guarantees managed over an award-winning portal.

OpenNMS manages the entire New Edge client network which currently includes more than 160,000 interfaces and a million RRD files. New Edge evaluated several network management platforms and chose the OpenNMS solution after judging OpenNMS to be superior on performance, flexibility and total investment costs.

New Edge Networks and the OpenNMS Solution

New Edge Networks began using the OpenNMS platform in 2004 when New Edge concluded that a legacy system that came with one of its acquired businesses was no longer sufficient for its expanding needs. In the five years of rapid growth that have followed, New Edge has used the OpenNMS platform and the OpenNMS Group to gain:

• Exceptional network stability in a Sun/Solaris environment.
• Easy NMS scalability to more than 160,000 interfaces.
• Custom monitoring applications and other applications built quickly to solve critical New Edge needs.
• Incorporation of New Edge-required functionality into new releases of the OpenNMS Platform.
• High ROI from a very cost-effective NMS that allows a small staff to manage a large network.

Five Years on the OpenNMS Platform at New Edge Networks

Five years of OpenNMS is five years of performance.

Scott McLaughlin, Director Infrastructure Operations at New Edge Networks, summarizes his experience with the OpenNMS platform and the OpenNMS Group:

In 2004, we received some new client requests that our legacy network monitoring system (NMS) could not support, so we started looking around at other options. At the time, the decision really came down to what we thought we could do for the money, which gave OpenNMS an advantage that helped us make our choice. But as we’ve grown, we’ve realized that we made a good choice for far more reasons than acquisition cost. The big thing was finding something we could grow with really fast. OpenNMS has proven to be the most scalable option

Expenses fall while the network grows.

It’s reasonable to expect that NMS costs would increase steadily as the size of the networks grows, but the New Edge experience has been just the opposite.

In the last five years we’ve gone from 10,000 interfaces to 160,000 interfaces. It’s always going up and now there are more than a million RRD files on our Net App. And yet, our internal support costs have dropped dramatically as we got things more stable and acquired better management tools. Today we have just one person who takes care of it all and it’s not even a half-time job.

OpenNMS Group works directly with New Edge to ensure that OpenNMS meets the customer’s needs.

One key to our ongoing success with OpenNMS is that it keeps evolving to match our needs. The OpenNMS product that’s out today has a lot of pieces that were driven by New Edge requirements several years ago. We don’t need new customizations with every OpenNMS upgrade because they’ve built our previous customizations into the platform.

This never would have happened if the platform wasn’t open source and if we didn’t have the OpenNMS Group. Personally, I’m very glad that the source code for OpenNMS now reflects our needs and innovations, and that other companies are using the technology that we requested for New Edge. It’s really cool, and you would never see that happen with any black box application.

More Reasons to Use the OpenNMS Group?

Fast, effective support means fewer, shorter network outages. The OpenNMS Group provides the highest level of expertise because they’re the same people who wrote the existing code and are writing new code every day. They’re present all the time.

Support speed is one of the things we’ve really appreciated with the OpenNMS Group, and an important reason we’ve been able to justify buying our 24/7 contract. The OpenNMS Group response time is amazing, and we need that because our customers expect New Edge to be working 24/7 for them.

For example, somebody from OpenNMS is logged into their IRC server almost all the time. The support teams here stay there all day – that’s how they communicate. There have been times when we’ve had problems and we’ve just set up a chat on the IRC channel, “Hey Tarus” or “Hey Ben, this is what’s going on” and they say “OK we’ll look at it.” They’re just right there. That’s not something that’s in the contract. That’s just something they do because of the geek in them. Those are the kinds of things we love getting from them that just don’t exist anywhere else.

When we’ve had an issue and asked “what is going on here?” we’ve usually received an answer from one of the people who wrote the piece of code or is an expert in PostgreSQL. They’ve responded, “Oh yeah, I know what that is” and it’s that quick. We’ve never had a major outage that’s lasted long. Once a problem is in their hands, it’s never very long before it’s solved. We’ve never really had a multi-day or multi-hour outage that I can remember.

Advice for the CTO considering OpenNMS

McLaughlin has some detailed observations to share with the CTO responsible for large networks.

First, they need to understand that a smart solution can combine software that’s free plus support that you pay for. People didn’t understand that model in the old days, but it’s getting easier now.

It’s hard to describe the biggest selling point for the OpenNMS solution. The platform is great, of course. You can take all the arguments for open source and apply them to the OpenNMS platform. Beyond that, though, it’s really about the OpenNMS Group and the way they support us. They really care about our environment. They’re going to do their best to understand it and to make sure it stays up.

If I didn’t have the OpenNMS Group, I’d have to hire more people . I’m getting a lot more value for what I pay OpenNMS because I get a Postgres guy, I get the folks who wrote the code, I get Tarus, I get SNMP experts – I get all the stuff that I need for the NMS and save $250,000 a year or more in salaries. It’s like outsourcing – it’s one of those times where it makes a lot more sense to buy that contract than to staff it on your own.