IT Management and Cloud Blog

« | Home | »

Why the Enterprise will use Open Source in the Cloud

By John | December 5, 2008

The other day I attended an Amazon Web Service (Cloud) bootcamp. There were about twenty-five people attending the workshop and by my estimation at least a third of them were enterprise customers. During the break I was able to talk to a few of these “Brand” name enterprises about their plans for using the cloud. It was obvious to me that they were all working with the “Do now and ask for forgiveness later” kind of mindset. Most of them explained the headaches they have to go through to requisition equipment from their in house IT to be able to do quick hit campaigns. One of the “enterprise” customers told me that over a year ago they had to run a special media event where they took over all of the screens on Times Square for a day. They needed over a three-month lead time with their IT department and it was an absolute administrative nightmare process. In the end it was nowhere’s as automated as it could have been if they could of controlled the process themselves, he told me. Over a year and half later they still have the twenty high-powered blade servers, they had to purchase for the event, sitting around doing nothing. Another student in the bootcamp me assured me that this was not going to happen again in his enterprise, now that he knows he can use Amazon’s cloud. There were other students in the class with similar stories. One of the students in the class runs short marketing campaigns and sees AWS as a fabulous vehicle for quick response time and logistical control of his projects. The idea that these individuals can pull out their corporate credit card and solve their problems without getting bogged down in the IT Meat Cloud has overwhelming appeal to them. In fact most department managers have around 10k signing approval and most short term AWS project would fall well below those numbers. I also talked to the class instructor and he assured me that enterprise customers are coming to the class and, they are indeed using the cloud.

So what does this have to do with open source? In a word “everything.” For the same reason theses customer’s will be pulling out their credit cards for servers and services, they will also not be going to their corporate IT department to use their “Enterprise” IT management approved tools. For one, most of those products today do not even have ELA/licensing provisions for use in the cloud. The appeal of AWS to these customers is the time to deliver a solution. They can make snap marketing decisions on demand and deliver solutions with out inter-department dependencies. This is probably the strongest reason for them using open source. They will not want to be bound to IT procedures and process to respond to market demands. It is also highly unlikely that a nimble media or marketing group in an enterprise would start a long and arduous RFP process with a large enterprise vendor for software to manage their cloud solution.

A lot of people often ask me if you need ESM in the cloud. Then I remind them that the “E” in EC2 is not elastic. Someone is going to have to provide automation, configuration, and provisioning in order to make their solution work. AWS doesn’t just start your servers because you want it to. Most people who are using elastic cloud solutions are using tools like Puppet and RightAWS for configuration and automatic provisioning. They are also using open source monitoring tools like Collectd or Zenoss for making autonomic configuration and provisioning decisions. Even the PaaS on the IaaS vendors like RightScale and Elastra are using open source tools and not proprietary solutions to solve the elasticity problems.

I can mainly speak for the IBM Tivoli products on this subject, and is my opinion that the Tivoli provisioning and monitoring product would be near impossible for the small “Do now and ask for forgiveness later project”. I have seen large enterprises struggle to implement these products with armies of veteran experts to get them to work in the enterprise and I just can’t see how they could be used in the aforementioned scenarios. If you have a different opinion I would love to hear your thoughts.

Topics: cloud | 14 Comments »

14 Responses to “Why the Enterprise will use Open Source in the Cloud”

  1. Lyle Says:
    December 5th, 2008 at 4:39 pm

    “Then I remind them that the “E” in EC2 is not elastic.” ???
    “Amazon ELASTIC Compute Cloud”

    Perhaps you meant, “the ‘E’ in ESM is not elastic.”

  2. John Says:
    December 5th, 2008 at 4:54 pm

    There is nothing elastic about EC2 by itself. Someone has to provide scripts to start and configure other servers. That is why companies like Rightscale and Elastra exist. EC2 will not automatically start other servers when you need more. There is nothing, IMO, elastic about starting one or more servers on EC2. What is elastic is when you use AWS API’s along with scripts to handle a slashdot or the digg effect. Servers can be automatically start based on tools that I call ESM (monitoring, configuration, and provisioning). Someone has to provide these tools like (e.g., Puppet or RightAWS). EC2 does not do that for you.

  3. William Louth Says:
    December 6th, 2008 at 3:50 am

    ” In fact most department managers have around 10k signing approval and most short term AWS project would fall well below those numbers.”

    If it is so short term would you even think about ESM for such an application even if it was open source (which means free to most) or it? Why would not the cloud platform provider not already offer this service as a value add and already integrated into their provision and billing solution. The only hurdle to this is how does the customer expose his business services and activities to the cloud monitoring solution in such a way that allows him to track cost for each business service and activity and to get monitoring out-of-the box that is meaningful – business monitoring model combined with resource consumption model.

    If open source ever fills this gap then it will not last long because there is no business model (subscription/support) for a short term project with a budget of 10,000. The only one directly benefiting would be the platform vendor how might consider patching a solution together on the cheap but in the end this will more than likely be replaced because it would represent a core value added service to the business which to an internal engineering team represents a justification for reinventing the wheel.


  4. Jayadeep Purushothaman Says:
    December 7th, 2008 at 9:27 pm

    Interesting way to “leak” IT resources to get things done. In one of the companies I worked earlier, the control regime is so bad that a laptop battery takes 3 weeks to get approved which could have been bought off-the-shelf. They would have escalated to the IT VP if someone was doing this without their approval.

    But ultimately, the story is that cloud services will enable agility for the business that these enterprise IT silos have never been able to provide even with those expensive adaptive enterprise tools that costs millions. Cloud sure is a disruption for the IT honchos.

  5. Zenoss in the Blogosphere | Zenoss Blog Says:
    December 8th, 2008 at 10:56 am

    [...] Why the Enterprise will use Open Source in the Cloud [...]

  6. William Vambenepe Says:
    December 8th, 2008 at 1:58 pm


    What percentage of enterprise IT spending corresponds to one-off projects with short lifetime and no (or very limited) backend integration such as the examples you describe?

    From you discussions with these “enterprise” people, did you hear about other use cases for EC2?

    The “Animoto is on fire this week” and “I rented half of Times Square for a day” scenarios make really good success story situations for EC2. But how representative are they?

    Once people get a taste from these scenarios, do they come up with compelling ways to use EC2 for more conventional IT tasks? Maybe you can ask during the next class? :-)

  7. John Says:
    December 8th, 2008 at 2:39 pm


    It’s funny, all the cloud vendors I talk to assure me that they have enterprise customer’s using the cloud. However, they can not disclose. I am constantly talking to my enterprise customers and the only thing “cloudy” going on are utility virtualized provisioning. Until last week I could not connect the dots. What I found out last week is that the BU’s are subversively moving apps to the cloud (1-off’s) “Do now ask for 4givness later”. Groups like marketing/media departments that have thier own IT liaisons. Those were the type of people in the cloud class.

    The bottom line is that I have been looking in the wrong places. The IT infra guys are not really moving to the cloud (from my small view of the world). I talk a little bit about this in my last two CloudDroplets podcasts.

    I think that as more of these subversive projects keep occurring the “glass” house will have to come up with comparable offerings.

    Sorry to say, enterprise use of the cloud still seems a little cloudy to me…


  8. Stu Says:
    December 9th, 2008 at 11:25 am

    I think elasticity and auto-scaling is mostly a “checkbox” feature that looks good on paper but works only in limited circumstances. In most enterprise systems, adding the wrong sort of resources elastically will make your app slower!

    I also don’t foresee a wholesale rewrite of enterprise apps to use partitioned MySql databases; the jury is also out as to how applicable that architecture is to enterprise workloads (I remember the shared nothing debate raging back in the 90s… Guess who won? Oracle, the shared disk vendor.)

    So I tend to agree with William that elasticity is a tricky case in the enterprise, given how apps work today. If anything, I think the benefit is more about finally having a way to truly impove utilization thru virtualization AND cloud elasticity.

    I think the real value of the enterprise cloud is split between (as you point out) first, a retail outsourcing channel, whether it’s rogue or not, has a very low barrier to entry (though shared hosting providers have helped rogue marketing and biz depts this way for a long while), and second but more importantly, reduced lead times to enact change at some level of your IT infrastructure. The latter comes down to putting software and APIs in front of what used to be a ticket, or form, or whole project..

  9. William Louth Says:
    December 9th, 2008 at 2:15 pm

    “Groups like marketing/media departments that have thier own IT liaisons. Those were the type of people in the cloud class.”

    John these sound like the type of individuals who would not necessarily be interested in monitoring solutions – open source (i.e. free) or not.

    It is hard enough getting core IT management teams putting in place proper SLM and CM processes. What hope it there with marketing/media departments with a small development team.

    The market is not there at least not the way you see it at this moment in time.


  10. William Louth Says:
    December 9th, 2008 at 2:23 pm

    The real benefit to the enterprise is that awareness and expectations will be higher for internally managed environments. CC will also inspire innovations that will eventually become part and parcel of day to day IT management.

    Before you can achieve a goal you must first believe it is possible even though the reality might indicate otherwise – fake it until you make it which I am sure will happen.

    What I like about CC is that the bar is set so high that even if we get half way there in the near term we have already overcome many obstacles that have not been challenged today.


  11. John Says:
    December 9th, 2008 at 2:24 pm


    Actually most of the auto scaling solutions have built in monitoring. Most are using things like Collectd, Monit, God, etc …. Rightscale is using CollectD under the covers. I agree that most of these groups will not go to monitoring directly. However, for example, if they have a Ruby based application they might decide to swap out something like Monit for GOD. Or some similar hack for other languages.

    IMO, the important factor is that Autonomics needs provisioning, configuration, and monitoring to cooperate and they are not likely to go to Big 4 for those solutions.


  12. John Says:
    December 9th, 2008 at 2:30 pm

    Totally agree.

  13. William Louth Says:
    December 9th, 2008 at 4:34 pm

    I agree on the big 4 but I also think it is applicable to even open source vendors of management solutions. The vendor is unlikely to give back to the community as it is a competitive edge.

    Open source will more than likely be used as a stop gap whilst the CC vendors build out their business and expertise. This is more so when one considers the degree of integration required across management domains (provisioning, config, monitoring,…). Open source projects typically target a particular domain or aspect (if narrower in scope) and are not interested in the bigger picture which is left to customers.

    Eventually the CC vendor will replace the OS stack with their own specialized solution(s) tailored to their own needs, business and strategic vision. I believe this has already been shown to be the case in one of your previous blog entries (Rightscale? Puppet?).
    anyone other than the CC vendor.


  14. You don’t have to pay £$000’s for business software… here’s some alternatives | The 845 Club Says:
    December 22nd, 2008 at 10:14 am

    [...] Why the Enterprise will use Open Source in the Cloud [...]