Customer Case Study: Towerstream

Towerstream is a leading wireless service provider utilizing 4G technology to deliver advanced, high-speed Internet access to businesses in 12 markets including New York City, Boston, Los Angeles, Chicago, the San Francisco Bay Area, Miami, Seattle, Dallas/Fort Worth, Philadelphia, Nashville, Las Vegas/Reno, and the greater Providence area where the company is based.

As a fast-growing ISP that guarantees 99.99% uptime, Towerstream places great demands on its network. In mid-2010, Towerstream saw that its legacy NMS was more burden than help for Towerstream’s multiple, expanding networks. After intensive experimentation with dozens of alternatives, they switched to OpenNMS in November 2010.

“An NMS That Works For Us, Instead Of The Other Way Around”

We were monitoring ~5,000 nodes and growing every day. We needed a better solution, and we got it with OpenNMS. ... It’s great to have an NMS that works for us instead of the other way around.

Chris Manigan, Systems Engineer at Towerstream, doesn’t mince words when he talks about the old system:

We had to restart processes multiple times a day and kick it in the butt just to keep it working. It was the equivalent of a full time job for one person, and it didn’t even do all the things we wanted. We were monitoring ~5,000 nodes and growing every day. We needed a better solution, and we got it with OpenNMS.

OpenNMS never needs kicking. If we leave it alone, it just keeps doing what it’s supposed to do. Maybe this shouldn’t be exceptional, but we find that it is. The only time we have to restart it is when we make configuration changes. I still spend a lot of time working on our NMS, but the difference is that I’m using that time to get OpenNMS to do more and more useful things – not to keep it from dying, like I had to with the old system. It’s great to have an NMS that works for us instead of the other way around.

Michael Micheletti, Towerstream’s Manager of Engineering, is too professional to call out their old system by name, but he tactfully refers to it as “another widely used open source NMS from a well-known provider.”

Beyond the issue of just keeping it alive, we weren’t happy with the options they gave to address our unique needs. When we’d ask them about possible improvements, the answer usually sounded like, ‘Sure, we can probably do that for you but you have to pay us lots of money first and then more money when you get in the door – but no guarantees.’

...we’re saving time on labor, getting things done sooner, and spending less money on hardware.

By contrast, the OpenNMS platform does nearly all we need, out of the box with out a costly commercial version upgrade. For anything else, we can make a solution on our own or get fast help from the OpenNMS Group. The OpenNMS support contract has been very cost-effective: we’re saving time on labor, getting things done sooner, and spending less money on hardware. Just looking at these benefits, we get a lot for what we’re paying.

Asset Management – Not Just Network Management

...the SNMP asset data collection in OpenNMS pulls a serial number or other identifier off every device automatically. That gives us a fast and highly accurate inventory that’s automatically adjusted every time someone connects or disconnects a piece of hardware.

As Chris describes it:

It doesn’t matter how talented you are, it’s very hard to keep track of thousands and thousands of physical assets that are constantly getting installed, removed, repaired or replaced by both your staff and your clients. Our old process required many manual steps and a lot of repetition that added up to more labor and more errors. But the SNMP asset data collection in OpenNMS pulls a serial number or other identifier off every device automatically. That gives us a fast and highly accurate inventory that’s automatically adjusted every time someone connects or disconnects a piece of hardware. This feature really set OpenNMS apart from its major competitors, none of which came with a comparable option. With OpenNMS, we can even integrate the asset data with our NetSuite® CRM for faster and more accurate customer management.

Huge Cost Savings on Installs

Michael explains the redundant, manual process they had before the switch to OpenNMS.

For every customer install, we had to update three systems: our CRM system, our inventory system and our monitoring system. We had to do the same three steps for everything from a new install, to the addition or replacement of a single device. But now with OpenNMS, we can update one system and have everything added to monitoring automatically, which saves us 10 or 15 minutes per install. When you’re doing hundreds of installs, that’s a huge cost savings right there.

It’s fair to say that we wouldn’t have gotten this win without the Greenlight support contract, which brought Dave Hustace out from OpenNMS to help us ramp up the new system.

Chris continues the story:

Engineering, customer service, and finance use NetSuite in different ways, but in the end they’re all using the same pool of data. When we integrated NetSuite with OpenNMS, we made a lot of processes go faster, more accurately and with less work. It was an incredible bonus that we didn’t expect when we decided to switch our network management system to OpenNMS just a few months earlier. And yet, it works so smoothly that we almost feel it should have been obvious from the start.

It’s fair to say that we wouldn’t have gotten this win without the Greenlight support contract, which brought Dave Hustace out from OpenNMS to help us ramp up the new system. Dave got us to think about our existing resources and whether we could benefit by tying any of them in with OpenNMS. Dave didn’t know we used NetSuite coming through the door – and he wasn’t at all an expert with NetSuite. But when we combined his knowledge about business processes and OpenNMS with our knowledge about NetSuite and Towerstream, the pieces just started snapping together. ‘Oh wow,’ we said, ‘look how easy it will be to pull our nodes in and keep everything up to date, without manual updates on multiple systems. The systems are actually going to update each other!’ Without Dave’s help, we would never have figured out the way we get requisitions from NetSuite.

Acquiring Another Company’s Networks

There was no intensive discovery effort, because OpenNMS asset management pulled all the equipment data, and automatically passed the data to the NetSuite CRM and inventory databases.

Michael shares the details on Towerstream’s acquisition of another ISP:

We recently acquired another company that also managed its networks with OpenNMS, but a different installation that wasn’t the same as ours. We found that adding all the new nodes into OpenNMS was like a total no-brainer. Their CRM was significantly different from our own, so it took some work to get those records into our system, but as soon as that was done, OpenNMS was happy to just accept more nodes. It did its thing and there were no complaints.

Really, it was genius the way it was set up. We didn’t have to do anything in Engineering – we just had to wait for the IT guys to get the new networks’ CRM data into NetSuite. Once that got done with the right customer status and IP addresses, monitoring started just about automatically, with no manual configuring of anything. There was no intensive discovery effort, because OpenNMS asset management pulled all the equipment data, and automatically passed the data to the NetSuite CRM and inventory databases. In the past, someone would collect the same serial number data and MAC addresses three times: when’d we install it, we’d write it down; when operations would sign off for the customer, he’d validate the data; then when our asset manager put it into NetSuite, he’d validate the data again. But now it’s one shot. Genius.

Customer Care — Faster, Cheaper and Better

With the way we’ve configured OpenNMS, we can identify issues much faster and resolve problems before the customers even notice anything was wrong.

Michael shares:

Proactive customer care got better when we switched to OpenNMS because it monitors not just ICMP, but SNMP and HTTP as well. For example: in the past, if a radio’s web GUI went down and our customer care staff couldn’t log in, they’d think the radio might be down and would dispatch a tech to diagnose and perhaps repair. Now, with the SNMP and HTTP data, we have more information that lets us know if a faster and cheaper solution – like restarting a simple web interface – is all we need.

With the way we’ve configured OpenNMS, we can identify issues much faster and resolve problems before the customers even notice anything was wrong. Customers really value us when we call to warn them about a link problem before they’ve noticed it. We get them back on track sooner with less wasted effort because they’re burning time and energy to figure things out from their end. The system is very accurate and has helped us get our SLA metrics even tighter. Everybody’s winning.

Share Network Control with Finance and Customer Service?

At Towerstream, the engineering group trusts its colleagues, its controls and its network data enough to let a decision in the finance department flip a big switch on the NMS. As Chris describes:

Like every business, we have the occasional delinquent customer. If their bill gets too far behind, finance just makes a ‘didn’t pay bill’ record in NetSuite and the information automatically feeds into OpenNMS, which automatically quits monitoring every device in their network because it knows why we’ve taken their link down. It’s a smooth, automatic chain of events and we trust the OpenNMS and NetSuite integration enough to hand over the controls.

Advice to the CTO

For the CTO or CIO considering OpenNMS, Michael Micheletti has this advice:

I’ve used everything from WhatsUp Gold to Netcool/OMNIbus, and many systems in between, like SolarWinds and Zenoss. The OpenNMS platform incorporates the best of them all. Like premium systems such as Netcool, it has all the events so you can see everything in real time. And like basic systems such as WhatsUp Gold, it’s easy to configure. You can just slap a new node on it.

It’s no wonder that everybody here loves OpenNMS.

When we were looking for a new NMS, our VP of Engineering and Network Operations asked us to take a wide look at options. We tested dozens of commercial and open source systems, from the well known to the barely known. In the end, OpenNMS looked like it would do 99% of what we wanted, and would do it well with comparatively modest hardware requirements. It looked like a good bet, and we found out quickly that it was.

The total cost of ownership of OpenNMS is very reasonable compared to other systems. The software is free, and we paid only $30k to have the OpenNMS Group do a Greenlight Plus project to help us with configuration and getting it running. In comparison, we would have to pay $175k for a commercial version upgrade to the system we had been using, and yet more money for startup help, with no guarantee we’d get rid of the slowness and unreliable performance we were getting with their free version. It’s no wonder that everybody here loves OpenNMS.

It’s great to have the OpenNMS Group’s experience and staff to get the job done faster, and smart from the beginning.

Chris Manigan adds his advice to the CTO:

Success with the OpenNMS platform depends on the user’s level of expertise. We’ve got a good team at Towerstream and guys at the OpenNMS Group don’t treat me like I need my hand held; they know I can do pretty much anything on my own unless it’s something brand new. If you run a small network like a handful of servers in a closet, it will probably do what you need to out of the box, and you won’t need to buy the OpenNMS Group’s support. But if you have a larger network like ours that spans 11 markets across the country, their support becomes essential. If we had all the time in the world, we might be able to take care of everything, but don’t have all the time in the world. It’s great to have the OpenNMS Group’s experience and staff to get the job done faster, and smart from the beginning.

Turned Off The Old System In Less Than One Month

Getting started with OpenNMS took no time at all: we set up requisitions with our NetSuite® CRM, sucked all the nodes in and were running. We turned off the old system a few weeks later – as soon as we’d configured our data collection and display graphs the way we wanted.

We have smart staff and can figure out almost anything we want to do with OpenNMS platform, given enough time. But taking months to do it all on our own wouldn’t be smart business.

Our support contract with OpenNMS was well worth the price, especially at startup. We have smart staff and can figure out almost anything we want to do with OpenNMS platform, given enough time. But taking months to do it all on our own wouldn’t be smart business.

The OpenNMS Group was a tremendous help from day one. In fact, they were supporting us even before we bought our support contract. One of the first things they did was to help us identify the right hardware. Their president, Dave Hustace, walked us through details like, ‘this is the kind of RAID array that you should use because it’s been proven to work with OpenNMS and a network with the nodes and activity plan to have over the next few years.’ We’d show him our plans and he’d let us know which parts looked good and which parts weren’t going to serve our expansion plans. Now we know we won’t need to buy new hardware next year because we outgrew ourselves.

Smart Code Under The Hood

The more we get under the hood, the smarter it looks. For example, the code saves resources by reducing the number of writes to the RRD files. OpenNMS collects data every 5 minutes but might only write every 20. That really helps performance because by storing more data in memory between writes, the system needs less time for disk access. Our old system did a write every 5 minutes for every one of our ~5,000 nodes, which crippled its performance compared to OpenNMS. The RRD file queue is just one example of how OpenNMS is both well thought out and clever.

...we are convinced of both their technical savvy and their commitment to taking care of Towerstream. ... They all work together and are way, way, friendly.

When we asked the Group to create several custom surveillance views, the first iteration ran very slowly – sometimes taking tens of minutes to load. They understood immediately that this was a big problem for us, so Dave and his team worked through the weekend on new code. Now it’s lightning fast, and we are convinced of both their technical savvy and their commitment to taking care of Towerstream. We’re also glad to have helped Dave spot a problem that would almost certainly affect other OpenNMS users – it’s a win for everyone when the platform gets better.

All of the OpenNMS Group guys are experts, so ready to help, and very responsive. They also treat me like I’m a professional, and that goes a long way. It’s a lot different from when you talk to something like a PC support line. With OpenNMS, you don’t get passed around from a Level 1 guy to a Level 2 guy to a Level 3 guy until you get where you need to be.

Everybody at OpenNMS seems to keep an eye on each others’ open cases. When I’ve opened a case, Dave might grab it to start, then Jeff will jump in with comments because he’s worked on something similar. They all work together and are way, way friendly.

That’s way more service than we pay for.

During the first week of our Greenlight startup, Dave came to our site and worked all day, then he went back to his hotel and worked all night. He worked through the weekend because he was concerned that we should get that we needed. The same thing happened when he came back for the second week of our Greenlight support. That’s way more service than we pay for.

The Bottom Line

Our network is what we actually sell, and that’s why we must have top performance from our network management system. We didn’t get top performance with our old system, but we do with OpenNMS.

Michael sums up the before and after with OpenNMS:

The old system was really a joke around here because it was so unreliable. At any given moment, it was either not working or it was working slowly. And don’t even think about asking the old system to do something like integrate with NetSuite. For starters, we’d have to pay $175k for a commercial version upgrade, and there’d still be no guarantee that it would work. It’s no wonder that everybody here loves OpenNMS – it’s saving us time and money in every department from engineering to customer service to finance. I have never heard a negative comment toward OpenNMS since it was installed, and Chris actually enjoys all the time he spends modifying the platform, because it keeps doing more things that he wants.

We are extremely happy with OpenNMS – both the platform, which we got for free; and the support and consulting, which we get for a very reasonable price. We are an ISP. Our network is what we actually sell, and that’s why we must have top performance from our network management system. We didn’t get top performance with our old system, but we do with OpenNMS.

Bonus: Customer Service And Finance Also Get A Boost

Chris sounds more like a brand evangelist than an engineer when he talks about the extra ways they use OpenNMS:

OpenNMS shows us much more than the health of our network – it actually helps us see the health of our company. In a few months, it’s pushed different departments together: to validate data, like our inventory; automate customer service processes and streamline things that we weren’t doing as efficiently; and even identify internal policies or procedures that needed changing. Yes it’s created some more work for a few people but it’s all great work and nobody’s complaining. It’s really helped us shape up at our customer care, engineering and finance groups.