IT Management and Cloud Blog

« | Home | »

Possible Cloud Monitoring Tools

By John | September 16, 2008

I have been wanting to start a thread on what open source tools are best for monitoring in the cloud. At this point this is just a list with some of my initial thoughts. I hope to gather some more data in future posts. Here are some of my initial thoughts on monitoring in a cloud provider like Amazon’s EC2, Flexiscale, Gogrid, and 3Tera.

  1. The tool probably has to be lite weight.
  2. Performance of the monitoring tool will be key.
  3. Should be easily customizable to add or remove plugins. Not all metric’s will need to be monitored in the cloud.
  4. Heavy network based products might not be a good fit for the cloud.
  5. The tool should be low cost most likely open source based.

Please feel free to add any comments or experiences of anyone using monitoring tools in the cloud.

Nagios
Most prevalent open source monitoring product.
Hyperic
At this point they are leading the pack with the most work done with (Amazon, Rackspace, and Mosso).
Zenoss
Not many cloud stories yet. Very sound monitoring tool.
Zabbix
I have used this product before and it is a solid tool. I am not sure if it’s a good play in the clouds. However it is written in C and might be better suited for performance (ala clouds).
OpenNMS
I need to research.
Ganglia
Very prevalent in the grid space. Might make a great candidate for cloud monitoring.
Collectd
Rightscale uses it as their monitoring tool. They seem to be making all the right moves, who am I to argue.
Ntop
Written in C. Might be a good tool for the cloud.
Monit
Needs research
Gkrellm
Needs research

Topics: cloud, monitoring | 22 Comments »

22 Responses to “Possible Cloud Monitoring Tools”

  1. Matt Raykowski Says:
    September 16th, 2008 at 12:33 pm

    John,

    I wouldn’t say that I represent OpenNMS in any way other than that I am a user of OpenNMS. I’m not entirely clear on what you meant by “not a cloud play.” Maybe I’m just not down with the lingo…

    OpenNMS is a fairly powerful, extremely scalable NMS platform and I’m certain that if you had a description of the types of things you would like to monitor, trend and collect I could tell you what it does and does not do, or how you could jury-rig suitable workarounds. Feel free to drop me an email.

  2. John Says:
    September 16th, 2008 at 12:39 pm

    Matt,

    I didn’t intend that to be a dig. I will update that line. However, there are two reasons why I said that.

    1) OpenNMS is a network management centric tool and is mostly based on SNMP. I am not sure those are two strong qualities for monitoring a cloud. However, I could be wrong.

    2) In previous discussions with Tarus, he seemed to have no interest in positioning OpenNMS towards the clouds (e.g., sort of like what Hyperic is doing).

    John

  3. Cote' Says:
    September 16th, 2008 at 12:49 pm

    Citrix and VMWare both announced cloud initiatives this week. I’m guessing at this point that it’s more management – provisioning, updating, and capacity management in real-time – than monitoring, but they’re both worth paying attention to and checking out on the “IT management for the cloud” front.

  4. John Says:
    September 16th, 2008 at 12:53 pm

    It’s gonna get real interesting as the private cloud vendors do battle with the public cloud vendors. The Cloud Holy Wars!

  5. Robert Says:
    September 16th, 2008 at 1:45 pm

    Hi John,

    Could you elaborate on what makes a tool “Cloud-Ready” as opposed to another?

  6. John Says:
    September 16th, 2008 at 1:51 pm

    Robert,

    Fair question. I was planning a more detailed post about what monitoring means in the cloud. A short list is, IMO, a monitoring tool that can be easily modified or customizable to only include what is needed in a cloud instance. For example, if you look at collectd that Rightscale uses. It seems light-weight, C based for performance, and very plugable. I think update this blog to include .. Is anyone using any of these tools …

    Thanks
    John

  7. Matt Raykowski Says:
    September 16th, 2008 at 3:34 pm

    John,

    I didn’t mean to take it as a dig. I just didn’t know what you meant by it. (=

    1) While you are correct that OpenNMS is largely SNMP-centric that isn’t its only tool and trick. It has a veritable swiss-army knife of tools and plugins to accomplish a variety of tasks. We talked about something as simple as the HttpMonitor or PageSequenceMonitor being used to query cloud availability as well as the HttpCollector for collecting a variety of availability and performance statistics.

    2) I know I’ve heard Tarus say he dislikes the term “cloud” but I think that it’s not so much an opposition to the technology as it is to the over-hype and buzz over the *word.*

    See his podcast where he says “they” said the same thing about virtualization. But they’re not saying “virtualization” anymore, they’re saying “cloud.” But real innovation is truly happening in virtualization today. I tend to agree with that assessment – that “clouds” aren’t the only exciting thing evolving right now.

    But I’ll take your response to Robert and elaborate a little bit:

    Out of the box OpenNMS provides a wealth of pollers (availability/response time checks) and collectors (performance collection and graphing.) Beyond what they already have there is a pretty simple Java API for adding new capabilities.

    Lets say, though, that you have a Perl script or something similar that already accomplishes what you require – it has a “general purpose” poller that runs a shell on the local NMS box.

    If you want to talk about running it within a cloud… I can honestly say I’m not aware whether that has been attempted.

  8. Benjamin Reed Says:
    September 16th, 2008 at 3:46 pm

    From what I understand, it’s not really a dislike of cloud computing, just that from a network management perspective, it isn’t really any different from… “computing.”

    If you remove the word “cloud”, all of the problems you have to solve in monitoring cloud services are the same problems you have to solve in monitoring traditional servers.

    The only thing to add is that a lot of the cloud-computing solutions have lightweight (RESTful or otherwise) APIs for querying some metrics on your usage of the cloud; a monitor to query those specific things in OpenNMS would probably be a matter of subclassing our HTTP monitor and adding 50 lines of code.

    The question really, is, “what makes monitoring cloud computing any different from monitoring anything else?” and the answer is, “not much.”

    Tools saying they monitor the cloud are putting a little icing on existing network monitoring practices and now they can mark off the “cloud” hype check-box. ;)

  9. People Over Process » Links for September 15th through September 16th Says:
    September 16th, 2008 at 10:59 pm

    [...] Possible Cloud Monitoring Tools [...]

  10. William Louth Says:
    September 17th, 2008 at 9:13 am

    JXInsight Probes – A Unified Approach to Performance Management and Cost Management for Cloud Computing
    http://www.jinspired.com/products/jxinsight/meteringthecloud.html

    The last time I looked none of the products listed above are activity based which seems to be a key requirement for monitoring cloud computing and providing fine grain chargeback mechanisms.

    William

  11. William Louth Says:
    September 17th, 2008 at 9:20 am

    Hi Benjamin,

    I have to disagree with the “icing” and “check box” statements unless of course you are referring to monitoring solutions which in general are devoid of execution context (user, workflow, request, message id,…) in their reporting.

    If one is going to do basic system level (health) monitoring then yes there is nothing different once the “cloud” term is removed. But if a cloud computing is to be managed as an utility then metering, costing, and billing have to be integrated into a model – a model that is not just system based but activity based.

    William

  12. Jamie Says:
    September 18th, 2008 at 5:04 am

    You’ve got Monit already, might I suggest you add Munin to the list.

    It is standard on a debian distro, and you can easily control which metrics it reports on.

    And you can use it as the client side plugins for Nagios to poll.

    Only downside is that you need some command line skills to configure it. It’s not really web2.0 plug n play.

    Groundworks would also seem like a good option. Afaik, it’s basically Nagios wrapped up to be very easy to install and configure and use.

  13. System Monitoring » Blog Archive » Smörgåsbord of links Says:
    September 18th, 2008 at 12:23 pm

    [...] That came via John M Willis  who also has an interesting post about what makes for a good monitoring tool “in the cloud” [...]

  14. Benjamin Reed Says:
    September 27th, 2008 at 4:48 pm

    William,

    We are in violent agreement!

    That is still just domain-specific application-level monitoring, to which the “cloud” is largely irrelevant. You can track anything by determining metrics and monitoring them, that doesn’t make monitoring the cloud anything new. It also doesn’t make them pointless, that monitoring is important.

    It’s just a matter of nomenclature; the cloud is just another word for distributed resources that we’ve had in one way or another for many years. Now that it’s a service that you lease from another company over the Internet, it doesn’t deserve it’s own category, in my opinion. The issues involved in monitoring the cloud have always existed since we’ve had networks.

    “Cloud Monitoring” looks a whole lot like, for example, running BMC Patrol and the BMC agent on your distributed servers in an intranet doing application-level performance monitoring (which I did 10 years ago at State Farm).

    Calling it something new only helps marketing folks associate themselves with trends, and blogs to get hits. And here I am, helping validate that for John. *waves* (grin)

    I really don’t mind, perhaps it’s useful to group a set of common use cases together as a “cloud,” but it’s not like it’s some kind of (cough) paradigm shift. Let’s just call a spade a spade.

  15. September Roundup! | IT Management and Cloud Blog Says:
    October 3rd, 2008 at 9:53 am

    [...] Possible Cloud Monitoring Tools [...]

  16. William Louth Says:
    October 3rd, 2008 at 2:46 pm

    Hi Benjamin,

    I think what is different with “cloud monitoring” as an approach rather than a technology is that the cost model (how one is charged for usage) does not necessarily reflect the application resource model (i.e. cpu, memory) typical of traditional system management models. The management model can consist of multi meters some used in the calculation of the monthly billing (vendor rate plans) and others used for standard service level management.

    What I really like about cloud computing is that there is a much stronger correlation of cost with resource usage and more than likely performance. Performance is not just some timing statistic used in service level management reporting but an actual cost that appears on a balance sheet. We now just need to start placing a value on activities (fine grain and not just a large service label) and calculating the real/virtual profits of such activity.

    William

  17. William Louth Says:
    October 3rd, 2008 at 2:48 pm

    What I really like about cloud computing is that there is a much stronger correlation of cost with *** service activity *** and more than likely performance.

  18. Nicholas Whitehead Says:
    February 20th, 2009 at 1:32 pm

    Hello John;

    I have been compiling a list of open source monitoring systems, tools and supporting what-nots here: http://www.heliosdev.org/open-source

    Cheers.

  19. onthecloud Says:
    April 23rd, 2009 at 12:19 pm

    Although it’s not open source, we have heard great things from our clients about Techout, which offers cloud monitoring with a “software as a service” approach. Outsourcing web monitoring is another option that’s available; it might be a viable one for some businesses.

  20. Hovhannes Avoyan Says:
    July 18th, 2009 at 3:45 am

    Take a look at http://www.monitorcloud.com/monitorcloud

  21. NETwork Monitoring DIVA Says:
    August 10th, 2009 at 1:34 pm

    Take a look at GWOS (GroundWork) with the Tap-In plugin. It’s a pretty sweet package for cloud management. http://www.groundworkopensource.com/exchange/cloud_management

  22. Derek Haynes (Scout) Says:
    September 29th, 2009 at 1:37 pm

    As the founder of Scout, a server monitoring solution with cloud support, I believe there are distinct needs when monitoring a cloud enviroment. John hit on most of them in his list.

    The problem with many existing tools is that monitoring solutions that are saved with backup images or configuration scripts are difficult to update and test. You don’t really want to mess with your deployment process and verifying it’s working after a change is also a pain.

    Our approach with Scout: a single crontab entry that loads a monitoring profile from our web interface at scoutapp.com. This profile can be changed at any time and won’t break a deployment. It’s super light-weight (just 1 line of code) and you can tailor monitoring without fear.

    More info on our approach: http://bit.ly/PJW4o

Comments