A Path to Private Clouds for the Enterprise
Bob Lozano
* Good to hear from you Paul.to which Ross Cooney replied:
Below are a few thoughts (keep in mind it's 2:00 AM where I am now)...
Everything we need to know about cloud computing can be discovered by
looking at the behavior of real clouds and weather.
There will be high clouds, low clouds, and in the short term for some no
clouds at all (a sunny day). When the clouds are visible they will be
of varying shapes, sizes, and will change as often as the wind blows.
Clouds will come and clouds will go. At the end of the day, the only
thing that will matter is if the clouds efficiently did their job while
they were around.
So here's what I predict... We'll soon see all the cloud vendors talk
about delivering their Cloud Foundations into Fortune 500 data centers.
The cloud services will be delivered as appliances (likely virtual) and
will offer integration and simple low cost of change transition to
remote hardware in the cloud vendors virtualized everything energy
efficient data centers. Grid, fabric, and managed services we've built
are a lot like "cloud services" so I would anticipate the leading cloud
vendors to begin picking up grid / fabric software providers, so they
can buy teams and show revenue while they mature their hosted versions.
Now the only question left is... who will build the magic carpet to
transcend all the clouds?
Regards,
Jake
Jacob,Now my thoughts ...Interesting analogy.
> Everything we need to know about cloud computing can be discovered by
> looking at the behavior of real clouds and weather.
There is a vast eco-system of consultants and applications building up
> Now the only question left is... who will build the magic carpet to
> transcend all the clouds?
large skill sets to help companies use the various cloud providers. I
am thinking of companies such as Rightscale and Arjuna here. While I
dont know of a product or service that allows you to swap and change
between them I am sure that one will appear soon.
I wrote a blog post close to this topic a few weeks ago:
http://www.spoutingshite.com/2008/06/30/the-aws-ecosystembarrier-to-innovation/
Ross Cooney
www.emailcloud.com
No doubt said magic carpet will be virtual ...
On a serious note, I do think the the step towards the management of private clouds (within an enterprise) is definitely an evolutionary one, particularly for those organizations that have already been aggressive about virtualization and perhaps, to some degree, a sense of commoditization.
In those cases the top initiatives to become more cloud-like internally will probably involve
- standardization on a much smaller number of vm types
- more aggressive move towards commoditization of HW - both computation and storage
- Programmatic & web interfaces for automatically provisioning and releasing infrastructure
- VM Orchestration / Placement Engine - to dynamically allocate work either in a private or public cloud, as needed.
- cloud-enablement
frameworks that provide the bridge between the "blank pool of
resources" provided by the earlier stuff, and the more app-oriented
layers below. This will make it easier to write cloud-native facilities
and apps that intrinsically are enterprise-grade (i.e. reliable, scale
arbitrarily, managable, secure, etc.)
- standard cloud-native application frameworks that use above
and scale as needed. This would include stuff like cloud-native web
tiers; Hadoop, GridBatch, and any other framework that enables a class
of applications to maintain data orthogonality as they scale; and so
forth. These are an extension of the generic cloud-enablement
frameworks (above) into specific app domains.
- actual apps that have been cloud-enabled, either using one of the previous frameworks or something similar, or (as will always be the case in anything this new) in a more hand-crafted manner.
- enterprise-wide storage services, provisionable as above that
include reliable, raw objects (S3 like, perhaps with an optional file system interface), BigTable / SimpleDB - like
facility, and instantiatable SQL instances of your choosing.
Remember I'm proposing a path towards creation of private clouds, and in particular cloud-enabled apps that can make use of either public or private clouds. I think if an org continues down the evolutionary path (top bullets), and begins to cloud-enable apps (middle bullets) using either public clouds or their current commodity-ish infrastructure (if they have any), they'll be moving down the right path.
Utpal Datta
who is more RESTful than others :-)
Initially, I was very interested in that thread but became somewhat
disappointed pretty quickly.
I am looking at this from the point of view of a uniform management
paradigm across what enterprises has on the ground (i.e. their
physical and virtual (may be even grid) infrastructure in their data
centers, parts of which they are planning to migrate to the cloud.
I think it will be a quite a while (may be forever) that the
enterprises will operate across the "ground" and the "cloud".
As such the enterprises are dealing with a multitude of tools to
manage their environment on the "ground", they will feel a lot better
and will be more receptive to moving to the "cloud" if the same tools
worked (at least at an overall level) for the "cloud" too.
So, what are we dealing with?
1) We have SNMP management tools primarily for servers and network
devices (SNMP protocol and MIB for modeling)
2) We have CIM-XML, WS-MAN tools for storage devices and storage
inter-connects (CIM-XML using CIMOM and CIM for modeling) through SMIS
3) We are moving towards (not there yet) for a similar (almost
identical) (as storage management) management paradigm for Physical
and Virtual Servers through SMASH (and WS-MAN is there too)
And now we will have to find a new management paradigm to the virtual
computing and storage resources in the cloud.
But from an business applications' point of view (an that is all the
enterprise business people care and need to care about) they are all
nothing but computing and storage resources. They really do not have
the time to care about (nor should they have to) whether each of these
components are different and have the extra headache to have to manage
them separately.
We need a management paradigm that, at least at a broad level, works
across all these different types of computing and storage resources.
I think CIM as a modeling paradigm works better than what else we have
(definitely better than SNMP V1, V2) and that fact that SNMP V1, V2
has no security, my vote will be use CIM as the base, especially when
SMIS and SMASH has pushed it reasonable far along the adoption curve.
I think WS-MAN is cumbersome and slow, but atleast it is there working
across all the resources (that needs management).
Will this uniform management paradigm flatten the playing field for
all the "Cloud Computing Middleware" companies? I really do not think
so. I am pretty sure all of them have a lot more to offer to
differentiate themselves. But this common gound will definitely help
the users and help them along the adoption of "cloud computing".
If I remember correctly, I saw a posting from Reuven a while ago that
Enomaly is doing work in the similar line (i.e. providing a uniform
platform for managing across the "on the ground" and "on the cloud"
resources), I would like to hear what others think, especially if
there are open source (or ther) efforts going on in the same
direction.
--utpal
> this new) in a more hand-crafted manner.bonus round:
Sam Johnston
I think it will be a quite a while (may be forever) that the
enterprises will operate across the "ground" and the "cloud".
Despite my tirades against 'private clouds' (eg The case against 'private clouds' - a [counter]example.) I agree with you on this point - local infrastructure as a component of a cloud architecture makes more sense than as an alternative to it.
<snip what="details" />
If I remember correctly, I saw a posting from Reuven a while ago that
Enomaly is doing work in the similar line (i.e. providing a uniform
platform for managing across the "on the ground" and "on the cloud"
resources), I would like to hear what others think, especially if
there are open source (or ther) efforts going on in the same
direction.
There is a very interesting looking screenshot for 2.0 (ok, a mockup) where you can see local clusters (Xen, MySQL) in the same list as cloud providers (Amazon). The feature list sounds cool too. These guys are obviously doing some very interesting things and I've been keeping a close eye on them for some time now.
I'm happy enough to let them deal with how to make a common (graphical and programmatic) interface for the various underlying components (be it Amazon's API, WS-MAN, SOAP, REST, etc.) - it's a good value add for them and it gives the others room to move (standardising too soon can stifle innovation). Eventually someone will do something noticably more sensible than the others (probably with a very low barrier to entry and relying heavily on existing protcols like HTTP/TLS/etc. - I'd expect it should pass the cURL test) and then this will be adopted. Bear in mind that the underlying providers can expose multiple APIs during any transition period too.
Sam
Paul Renaud
4) We are also seeing the emergence of client system management thru DASH
(which leverages WS-Man)
5) Windows server architecture management is also evolving along the WBEM
route from WMI to WS-Man.
It's not immediately obvious that we need to move to an entirely new
management paradigm. There are probably two distinct areas of management to
consider:
a) the public-facing management interface that customers use to obtain
information on and to manage the use of the SaaS service residing in the
cloud. This interface needs to be highly standardized to assure successful
transition from what we have today to what we need tomorrow. That is why
I've been so vocal in promoting WS-Man for this interface.
b) the cloud-facing management interface used to manage internal resources
within the cloud. This is likely to be an autonomic interface that may or
may not be decentralized. It will ultimately also need to be standardized
so that higher levels of function can be built upon it to provide richer
cloud functionality.
Using current management paradigm, the "provider" that is used to implement
the public-facing management interface can aggregate as required from as
many internal management sources as necessary. Viewed from the outside,
there is no reason to adopt an new management paradigm.
Randy Bias
Now the only question left is... who will build the magic carpet to
transcend all the clouds?
There is a vast eco-system of consultants and applications building up
large skill sets to help companies use the various cloud providers. I
am thinking of companies such as Rightscale and Arjuna here. While I
dont know of a product or service that allows you to swap and change
between them I am sure that one will appear soon.
- standardization on a much smaller number of vm types
- more aggressive move towards commoditization of HW - both computation and storage
- Programmatic & web interfaces for automatically provisioning and releasing infrastructure
- VM Orchestration / Placement Engine - to dynamically allocate work either in a private or public cloud, as needed.
These first initiatives are a natural evolution. Now comes the parts that I think may be a bigger step for many organizations:
- cloud-enablement frameworks that provide the bridge between the "blank pool of resources" provided by the earlier stuff, and the more app-oriented layers below. This will make it easier to write cloud-native facilities and apps that intrinsically are enterprise-grade (i.e. reliable, scale arbitrarily, managable, secure, etc.)
bonus round:
- enterprise-wide storage services, provisionable as above that include reliable, raw objects (S3 like, perhaps with an optional file system interface), BigTable / SimpleDB - like facility, and instantiatable SQL instances of your choosing.
All but the "cloud storage services" are, in the end, perhaps the most crucial to gain what's actually possible. When done well, they will enable the apps to comfortably execute in a private cloud, a public cloud, or whatever mix makes sense at that moment. I think the "cloud storage services" will be a later step for many orgs, but I could easily be wrong on this.
Remember I'm proposing a path towards creation of private clouds, and in particular cloud-enabled apps that can make use of either public or private clouds. I think if an org continues down the evolutionary path (top bullets), and begins to cloud-enable apps (middle bullets) using either public clouds or their current commodity-ish infrastructure (if they have any), they'll be moving down the right path
--Randy
ZULKARNAIN KAGALWALLA
Nice Q&A With Nicholas Carr on CC:http://www.cioinsight.com/c/a/Expert-Voices/Nicholas-Carr-Why-IT-Will-Change/?kc=CIOMINUTE07282008CIO2 |
Rob Blake
There is a vast eco-system of consultants and applications building up
large skill sets to help companies use the various cloud providers. I
am thinking of companies such as Rightscale and Arjuna here. While I
dont know of a product or service that allows you to swap and change
between them I am sure that one will appear soon.A lot of the current solutions are sadly cloud dependent. Something like AWS is largely it's own ecosystem and most, if not all, of the work done to support it to date is completely non-portable as a solution.Only something like EUCALYPTUS has promise to bring an internal AWS compatible tool for building internal clouds. But AWS (and EUCALYPTUS???) only support para-virtualized Xen instances at the moment, which means restricting any internal cloud deployment that uses them to Linux. The enterprise needs Windows.
At Arjuna we agree that cloud dependent solutions are a barrier to the adoption of both internal and external clouds. We do have our own provisioning mechanism but we also attempt to abstract away from the physical provisioning through the use of service agreements. If a service agreement is formed between consumer and provider, then exactly how that agreement is fulfilled is down to the provider. As Sam Johnston said earlier in the week (http://samj.net/2008/07/future-of-cloud-computing-army-of.html), there could be an army of monkeys behind the scenes, but as long as the provider does what they said they'd do, we're happy. This gives us the benefit of being able to bridge between internal and external clouds pretty seamlessly as providers are simply represented by available service agreements. We also provide extra benefit by actively monitoring the terms of available service agreements, integrating this with user provided policy to decide which service agreement should be used to best fulfill their application needs.
Alternatively, moving to a non-VM based mechanism for building VMs. In other words, using Puppet, rPath, or FastScale-like technologies to model and build VMs dynamically. The notion of continuing to manage large numbers of opaque VM images is inherently broken.
- standardization on a much smaller number of vm types
Yes!! Including built-in versioning, roll back, live reconfiguration, and just-in-time environments (e.g. 'staging on-demand'). Most of which is hard to impossible using VM images as a configuration management system.
- more aggressive move towards commoditization of HW - both computation and storage
- Programmatic & web interfaces for automatically provisioning and releasing infrastructure
I'm uncertain of whether you are talking about the grid/VM workload component here or the application management piece here.
- VM Orchestration / Placement Engine - to dynamically allocate work either in a private or public cloud, as needed.
- cloud-enablement frameworks that provide the bridge between the "blank pool of resources" provided by the earlier stuff, and the more app-oriented layers below. This will make it easier to write cloud-native facilities and apps that intrinsically are enterprise-grade (i.e. reliable, scale arbitrarily, managable, secure, etc.)
I think Randy hits the nail on the head . In my opinion we should be focussing on the application itself and this requires us to have a far greater control over the granularity of management. If you want to provision everything from a virtual machine image, then carry on and go for it. But surely both providers and consumers will come to want finer control over the provisioning process? As a provider I want to define policy that guides how consumers can use my infrastructure and exactly the type of applications I'm willing to host. The opaqueness of vm's removes this from me. As a consumer, I want to provide you with a recipe for building my application and let you do it for me because that's what I'm paying you for. I don't want to (and shouldn't have to) provide a virtual machine image with everything pre-configured. I'm responsible for telling you how the application fits together, you're responsible for realising my requirements. I'm also likely to want to wrap policy around that recipe that says exactly what application requirements a service provider should be able to fulfil before I allow them to provision my application for me.
Rob Blake
Arjuna Technologies
www.arjuna.com
http://blog.arjuna.com
Stuart Charlton
On Aug 6, 2008, at 10:50 AM, Utpal Datta wrote:
>
> But from an business applications' point of view (an that is all the
> enterprise business people care and need to care about) they are all
> nothing but computing and storage resources. They really do not have
> the time to care about (nor should they have to) whether each of these
> components are different and have the extra headache to have to manage
> them separately.
>
> We need a management paradigm that, at least at a broad level, works
> across all these different types of computing and storage resources.
>
... And works with higher level concepts of software configurations,
dependencies, and into business applications, policies and IT service
dependencies.
A central problem in IT manageability is this divorce between hardware
devices and resources and _software_ resources leading to semantic
divergence.
CIM is attempting to bridge that in some cases, but still stays
perilously close to the metal -- its ventures into specifying policy
or application models feel as if there's a "chasm" being crossed.
MIB, of course, feels hopelessly close to the wire, even though there
are dozens of proprietary MIBs out there for software monitoring &
management.
But then there's ITIL, which provides a sketch of a metamodel for
configuration items, incidents, problems, IT services, etc. that
should be captured in a CMDB. Some CMDB's are using CIM, some are
not. Some go straight to OMG's MOF. Some use XSD, even though it
is primarily a syntactic model. Some use RDF/OWL (remember the Data
Center Markup Language?). There's the CMDBf workgroup doing their
thing with the DMTF (mostly based on WSDL, XSD and XPath). There's
also the Microsoft's SDM , in Operations Center, which begat the W3C
SML (which adds logical constraints to XSD via Schematron, though this
also may be going nowhere fast). Let's also not forget JMX (which is
often RMI serialized objects under the covers),
In the applications space, there's MIME types, including XML (which
could be specified in XSD or RelaxNG), JSON, RSS, OpenSearch, Atom &
AtomPub, etc.
Then there's SOA Governance products, which have their own semantics,
usually specified, again, in a syntactic model (XSD).
In the end, "it's all just data & metadata". Aren't these all
just applications of document transfer? Why do we need a management
standard that's oriented specifically to lower levels of the
technology stack or is solely focused on manageability?
Certainly there are different performance requirements and patterns
when dealing with a vast number of hardware resources (particularly
when dealing with sensor networks and/or disbursed telecom switches),
but the limiting factor with these standards is that they keep coming
out of (and smelling like) the management & monitoring industry, and
while they keep trying to be a general purpose modeling & data
transfer framework, they don't quite break out of their sphere of
influence. They become a semantic island that must be
(haphazardly) bridged.
> Will this uniform management paradigm flatten the playing field for
> all the "Cloud Computing Middleware" companies? I really do not think
> so. I am pretty sure all of them have a lot more to offer to
> differentiate themselves. But this common gound will definitely help
> the users and help them along the adoption of "cloud computing".
I think the main problem is to tackle semantic interoperability and
traceability between hardware, the cloud architecture being used, and
the end business application. Otherwise the cloud will just building
silos again.
Cheers
Stu