What is a "Sponge" ?

[Miha Ahronovitz]

Librato has a very interesting new product, called Load Manager. For a
complete technical description see
http://www.librato.com/assets/media/Librato_Load_Manager_twp.pdf

A typical user case is when one buys CPU instances from Amazon or from
any other pay per use IaaS. There are two questions we rarely know the
answer:

1. How many CPU instances I should pay for to run optimally my
application?
2. How do I know the CPU instances are fully utilized?

Here is the what a "sponge" is (quote):

"Load Manager guarantees a business critical application all the
system resources it requires, while allowing background applications
to run on the same server (or CPU instance) but only take advantage
of unused
system resources. We refer to all the background tasks as “sponge”
applications because they safely “absorb” all unclaimed system
resources and allow companies to “squeeze” out the maximum possible
value from their server infrastructure . Load Manager dynamically
reallocates resources from a low priority “sponge” application to the
high priority, foreground application the instant they are needed."

Load Manager potentially can lower the bills on a pay per use public
cloud or dramatically improve the throughput. It works on Linux and
Windows

They say the devil is in the details. Librato keeps the devil away and
makes the cloud even more attractive.

Cheers,

Miha

[Rao Dronamraju]

Miha,

My first impressions, this product looks like just another "me too" cloud
product.

It is basically a scheduler with a policy engine and scheduling management
infrsatructure.

It can only optimize resources within a VM. It is at the mercy of the cloud
scheduler.

I am not sure how much more/better schduing it can do than OS level
schduler. How come their document does not have any comparison data between
OS level schduling of various applications and their application level
scheduling?...how much better they are?....

BTW, their "sponge" technique, wasn't it called cycle stealing decades
back:-)...

"Load Manager potentially can lower the bills on a pay per use public
cloud or dramatically improve the throughput. It works on Linux and
Windows"

They could have run it on some EC2 instances with and without their
scheduler and shown some data about lowering the bills....is it not?...

Cheers,

Miha

--
~~~~~
Register Today for Cloud Slam 2010 at official website -
http://cloudslam10.com
Posting guidelines:
http://groups.google.ca/group/cloud-computing/web/frequently-asked-questions
Follow us on Twitter http://twitter.com/cloudcomp_group or @cloudcomp_group
Post Job/Resume at http://cloudjobs.net
Buy 88 conference sessions and panels on cloud computing on DVD at
http://www.amazon.com/gp/product/B002H07SEC,
http://www.amazon.com/gp/product/B002H0IW1U or get instant access to
downloadable versions at http://cloudslam09.com/content/registration-5.html

~~~~~
You received this message because you are subscribed to the Google Groups
"Cloud Computing" group.
To post to this group, send email to cloud-c...@googlegroups.com
To unsubscribe from this group, send email to
cloud-computi...@googlegroups.com

[Fred van den Bosch]

Rao,

Load Manager is an application-aware scheduler that runs in user space; it
does a much better job than the OS scheduler. Take a look at the example
here: http://www.librato.com/products/silverline_example. In a few weeks EC2
users will be able to try it for themselves.

Best,

Fred van den Bosch

[mij...@sbcglobal.net]

The LM does for Linux and Windows a similar functionality as SRM - Solaris Resource Manager.

It seems it has been designed to create containers on top of instances of CPUs, not only physical servers.

Librato's engineers on this group can clarify further.

Miha
Sent from my Verizon Wireless BlackBerry

[Rao Dronamraju]

Fred,

My apologies. I was not trying to shoot down or put down your
product/technology.

I was bring critically analytical.

My sincere best wishes for your success!.

Rao

[Brian Olson]

Hi Rao,

As I understand it, Librato Load Manager runs completely in user-space
and provides transparent "virtual containers� for fine-grained
application-level scheduling control. It can be used to consolidate
applications or ensure SLAs--in both virtualized environments as well as
on 'bare-metal" servers. The idea is that it can significantly lighten
the load on the operating system's or hypervisor's scheduler, allowing
it to run more efficiently, while also providing application-centric
visibility, control and SLAs which more accurately reflect the
organization's business priorities.

As far as "me too", I'm not aware of another product that uses this
exact technical approach...

Brian
br...@brian-olson.net

[Rao Dronamraju]

Brian,

I understand that it runs in user space intercepting systems calls and
trying to optimize process/application scheduling. But (all kinds of)
schedulers have been around for a long time espcially in the OS and Grid
areas. About it making OS and hypervisor run more efficeintly, how can it
make them more efficiently unless there is some communication and
intelligence that they share and communciate with each other. I do not think
it is doing that. Infact you will not be able to do it in the public clouds
because the cloud scheduler has to work on a multi-tenancy basis. They
cannot do anything that is specific to a single tenent. The cloud scheduler
will schedule according to the cloud scheduling policies and when a VM gets
scheduled, then Librato will function as a scheduler within that VM time
slot. It will probably make that time slot usage as optimal as possible
tending towards 100% efficiency. But don't OS schedulars also try to make
the best use of this time slot as long as there is enough work to be done.
Why would a OS scheduler leave a resource unscheduled when there is enough
work to be done and it is still within the VM timeslot. My reference to "me
too" is with respect to a lot of former grid companies has rebranded
themselves as cloud companies because of the cloud hype.

Rao

-----Original Message-----
From: cloud-c...@googlegroups.com
[mailto:cloud-c...@googlegroups.com] On Behalf Of Brian Olson
Sent: Sunday, February 21, 2010 8:19 PM
To: cloud-c...@googlegroups.com
Subject: Re: [ Cloud Computing ] What is a "Sponge" ?

Hi Rao,

As I understand it, Librato Load Manager runs completely in user-space

and provides transparent "virtual containers" for fine-grained

Brian
br...@brian-olson.net

--

[gopu.r...@wipro.com]

Miha,

How different is this with VMware Resource pool ? There are other
orchestration products in market that identify the resource of
availability and machine is build on that resource pool.

Thanks and Warm Regards

R.Gopu

[Rollie Schmidt]

> I'm not aware of another product that uses this exact technical
approach...

Which is precisely the focal point of the IP and associated patents. Prior
work, whether classic application workload management found on mainframes,
or logical partitions, on AIX, Solaris Zones/Containers, and other Unix
implementations, all require hooks in apps and/or OS/kernel itself. If the
later, then context switching is also very expensive (slow/drag on
performance) and enterprises hate non-standard Linux kernels, not to mention
even obtain, run, or support a modified Windows anything.

> how can it make them more efficiently unless there is some communication

and intelligence >that they share and communicate with each other?

By making sure things are done/managed purely from a point of application
centricity. That's what people care about, as the apps are what are
delivering the business service(s) from an EU's perspective, whether in the
data center, cloud or anywhere.

And since OS schedulers lack application context, and deal with threads and
processes mostly in a vacuum and w/o attention to priority for the most
part, they can't really understand SLAs from the underlying resource
consumption standpoint. By capturing all application context, including any
child processes and associated threads, and keeping them in a the context of
a single "virtual container", it makes the work of any underlying scheduler
much more efficient/effective and adds application context transparently
above the OS without the need for any detailed communication or knowledge
the Load Manager even is present.

This approach works well whether there need to be several/dozens of apps on
a machine, each in a container, or groups of them in a container, or where
one is deploying a large single-instance, multi-tenant app in EC2, where
each "tenant" runs within a container. Now all manner of detail and
resources consumption can be monitored and controlled at 10ms precision:
pay-per-use billing, near real-time elasticity (scale up/down, out/in) and
any # of use cases enabled.

For

> Why would a OS scheduler leave a resource unscheduled when there is enough
work to be done >and it is still within the VM timeslot

And,

> are other orchestration products in market that identify the resource of
availability and >machine is build on that resource pool.

Mostly as per above, even if one can identify unused, underutilized
resources, by what policy/rules are the additional resources allocated?
Manually? Automatically? How dynamically? Again, which App(s) have priority?
Or should they all equal/fair sharing the excess? All these decisions none
of these other things can/will make, because they have no concept of
application context/importance of which I am aware. How then could
decisions/allocations be made in terms of app SLAs?

As an ex-colleague at StoredIQ used to frequently say, "context is worth a
hundred IQ points."

Rollie

-----Original Message-----
From: cloud-c...@googlegroups.com
[mailto:cloud-c...@googlegroups.com] On Behalf Of Brian Olson
Sent: Sunday, February 21, 2010 6:19 PM
To: cloud-c...@googlegroups.com

Subject: Re: [ Cloud Computing ] What is a "Sponge" ?

Hi Rao,

As I understand it, Librato Load Manager runs completely in user-space

and provides transparent "virtual containers" for fine-grained

Brian
br...@brian-olson.net

--

[miha ahronovitz]

Ropu,

I am not in position to compare technically VMware Resource pool with Librato's Load Manager.
Rao, Brian Olsen and Fred van der Bosch have more  insights.

But as Fred promised, in a few weeks everyone can try. The greatest ideas are so simple, that one wonders why we did not do this before. A couple of EC2 runs with and without LM,  will show when it LM works best and when it doesn't.

But right now, I challenge anyone to tell us how many EC2 processor  instances shall I buy and for how long (1 hour, 1 day, 1 year, etc.) to run my application with a defined SLA. We go by rule of thumb and past experience. And after we select a number, say 100, how do we know  the business critical application uses the processor as close to 100% as possible?  We don't.

If we do,  either (1) the unused capacity is sponged for background applications, which  increased  throughput, or (2) we buy less EC2 processors and we save money.

Any sufficiently advanced technology is indistinguishable from magic,  Arthur C. Clarke wrote. LM new functionality will be no exception, if it works. We will know in a few weeks.

2cents,

Miha

[Rao Dronamraju]

So how is this any different than what was existing for quite a few years
like gWLM (the pdf covers how it manages SAP for example)

http://docs.hp.com/en/13387/Using.gWLM.with.SAP.pdf

-----Original Message-----
From: cloud-c...@googlegroups.com
[mailto:cloud-c...@googlegroups.com] On Behalf Of Rollie Schmidt
Sent: Sunday, February 21, 2010 10:59 PM
To: cloud-c...@googlegroups.com

[Rollie Schmidt]

> So how is this any different than what was existing...

Conceptually similar, as I mentioned Load Manager comparisons to other App
WLMs from mainframe era on down through Unix too. But essentially it's the
implementation and the completely non-invasive, user-space intercept
technology/IP:

1. Works w/Windows (WS 2003, 2008, XP, Vista etc.) and Linux (RHEL, Centos,
SLES, Debian)
2. Runs on virtual AND physical machines
3. Supported on any/open x86_32 & x86_64 systems
4. Installs in minutes into existing infrastructure w/o disruption
5. Independent of vendor-specific requirements(HP Systems Insight Manager)

As opposed to:

Hardware Requirements
gWLM will operate on HP 9000 Servers, HP Integrity Servers and HP OpenVMS
AlphaServers.

Software Requirements
gWLM will operate on HP-UX 11i Version 1 and HP-UX 11i Version 2

gWLM will operate on OpenVMS Integrity 8.2-1 and above

gWLM will operate on OpenVMS AlphaServer 7.3-2 and above

-----Original Message-----
From: cloud-c...@googlegroups.com
[mailto:cloud-c...@googlegroups.com] On Behalf Of Rao Dronamraju

[Miha Ahronovitz]

Raom,

Very interesting. Did you use the SAP workload manager? First, itappears
it is only for SAP loads. Second, reading the paper I realize why SAP
is a nightmare to use.

How do you think can someone use Global Workload Manager in an EC2
environment, outside SAP framework to make sure all node are used fully?

Miha

[Ramki Balasubramanian]

From a look at the document pointed out by Fred, this solutions seems capable of keeping the primary application pool fully fed while redistributing the demand of other applications to work with the remaining capacity as much as possible.

This is a different perspective, in that it seems focused on maintaining turn-around time of the primary application pool rather than taking resources away whenever possible.

Curious about the name. Is the primary application the 'sponge' in the infrastructure, and all else is the water being absorbed? :-)

-
Ramki

[Rao Dronamraju]

Miha,

No I did not use or work on gWLM especially with SAP.

gWLM is a generic WLM solution that is primarily available in Integrity HW.

Although I must add that it is also available for Linux and Windows VMs running on the Integrity Systems/VSE environment.

"How do you think can someone use Global Workload Manager in an EC2 environment, outside SAP framework to make sure all node are used fully?"

About using Global Workload Manager on EC2, I think it depends on what the cloud provider allows you (tenent) to do or not do. If the GWLM does not try to do anything that is in conflict with the hypervisors management of the underlying HW then you should be able to use it. Please note that this means all the operations initiated by the GWLM will be executed within the contxt of the VM/OS or traps to the hypervisor and hypervisor does it on behalf of the VM/OS. If the hypervisor allocates the VM a timeslot and lets it manage the resources (only the ones that the VM sees) as the VM sees fit then in this case the GWLM should be able to manage the resources as it did before. But I am not sure what the cloud providers are allowing or not allowing in this respect. For instance Amazon, as I understand uses a modified/propietory Xen and so even if GWLM works on Xen outiside a cloud, it may or may not work in a modified environment.

Now there could be other problems that are independent of resource management that one might run into in a cloud. For instance, since VMs are dynamically moved around by the cloud scheduling and management solutions, GWLM nees to be maintain the connectivity with migrated VMs in order to continue to monitor and manage the workloads within a VM. This ofcourse will be a problem to all solutions that need to manage VMs in some form or the other including virtual data center management solutions of a tenent.

--

[Fred van den Bosch]

We usually refer to the “background” application as the sponge, since
it soaks up spare resources not need by the primary application; the
idea for using this term actually came from one of our customers.

Load Manager actually makes it possible to create an arbitrary number
of workload “containers” and assign resource quota to each of them.
The “sponge” use case is just a special case in which the “background
container” has resource quota of "zero", so only gets resources not
needed by the primary application; it doesn’t even require the primary
application to be run in a container. This is the first use case that
will be supported for EC2 users (http://www.librato.com/products/
silverline), because the payback is easy to see and it’s simple to
deploy without touching the primary applications.

We don’t claim to have invented application workload management (just
like for server virtualization that honor goes to IBM, close to 40
years ago), just to have built a robust user-space version that works
for Linux and Windows. We’re taking it to the cloud because if one
wants to improve utilization of cloud server instances, server
virtualization is not an option (one cannot virtualize a virtual
server) and application workload management (in our opinion) a natural
choice.

Fred

On Feb 22, 1:04 pm, Ramki Balasubramanian <ramk...@acm.org> wrote:
> From a look at the document pointed out by Fred, this solutions seems
> capable of keeping the primary application pool fully fed while
> redistributing the demand of other applications to work with the remaining
> capacity as much as possible.
>
> This is a different perspective, in that it seems focused on maintaining
> turn-around time of the primary application pool rather than taking
> resources away whenever possible.
>
> Curious about the name. Is the primary application the 'sponge' in the
> infrastructure, and all else is the water being absorbed? :-)
>
> -
> Ramki
>
> On Mon, Feb 22, 2010 at 8:37 AM, Rao Dronamraju <
>

> >http://groups.google.ca/group/cloud-computing/web/frequently-asked-qu...
> > Follow us on Twitterhttp://twitter.com/cloudcomp_groupor
> > @cloudcomp_group
> > Post Job/Resume athttp://cloudjobs.net

> > Buy 88 conference sessions and panels on cloud computing on DVD at
> >http://www.amazon.com/gp/product/B002H07SEC,

> >http://www.amazon.com/gp/product/B002H0IW1Uor get instant access to

> > downloadable versions at
> >http://cloudslam09.com/content/registration-5.html
>
> > ~~~~~
> > You received this message because you are subscribed to the Google Groups
> > "Cloud Computing" group.
> > To post to this group, send email to cloud-c...@googlegroups.com
> > To unsubscribe from this group, send email to
> > cloud-computi...@googlegroups.com
>
> > --
> > ~~~~~
> > Register Today for Cloud Slam 2010 at official website -
> >http://cloudslam10.com
> > Posting guidelines:
>

> >http://groups.google.ca/group/cloud-computing/web/frequently-asked-qu...
> > Follow us on Twitterhttp://twitter.com/cloudcomp_groupor
> > @cloudcomp_group
> > Post Job/Resume athttp://cloudjobs.net

> > Buy 88 conference sessions and panels on cloud computing on DVD at
> >http://www.amazon.com/gp/product/B002H07SEC,

> >http://www.amazon.com/gp/product/B002H0IW1Uor get instant access to

> > downloadable versions at
> >http://cloudslam09.com/content/registration-5.html
>
> > ~~~~~
> > You received this message because you are subscribed to the Google Groups
> > "Cloud Computing" group.
> > To post to this group, send email to cloud-c...@googlegroups.com
> > To unsubscribe from this group, send email to
> > cloud-computi...@googlegroups.com
>
> > --
> > ~~~~~
> > Register Today for Cloud Slam 2010 at official website -
> >http://cloudslam10.com
> > Posting guidelines:

> >http://groups.google.ca/group/cloud-computing/web/frequently-asked-qu...
> > Follow us
>
> ...
>
> read more »

[Rao Dronamraju]

 

“this solutions seems capable of keeping the primary application pool fully fed while redistributing the demand of other applications to work with the remaining capacity as much as possible.”
“This is a different perspective, in that it seems focused on maintaining turn-around time of the primary application pool rather than taking resources away whenever possible.”

 

gWLM also does the same….

 

“Using gWLM, you will be able to move CPU resources to or from the workload as needed to maintain acceptable performance, while also using less net CPU resources over time than an unmanaged workload would. This is a net win for you as the unused CPU resources can then be used for other computing tasks when the SAP workloads do not require them.”

 

---

[Miha Ahronovitz]

Rao,

> gWLM also does the same..
.
If one would tel us here in this forum what this new start up called
Google is a web search engine, many would have had said: But this what
Altavista does. Altavista was launched by Compaq on 15 December 1995
at altavista.digital.com It was DEC baby and now they are owned by
yahoo.com

Actually you can try altavista.com and "google yourself". Pardon me;
"altavista yourself" Google who appeared in 1998, when started in a
garage (actually Palo Alto has so few garages, every garage is
precious) after depositing Andy Bechtolsheim $100,000 check,
In December 1998, "PC Magazine" reports that Google "has an uncanny
knack for returning extremely relevant results" and recognizes us as
the search engine of choice in the Top 100 Web Sites for 1998.

and this was the end of Altavista. This is life.

Miha

On Feb 22, 2:43 pm, "Rao Dronamraju" <rao.dronamr...@sbcglobal.net>
wrote:

> "this solutions seems capable of keeping the primary application pool fully
> fed while redistributing the demand of other applications to work with the
> remaining capacity as much as possible."
> "This is a different perspective, in that it seems focused on maintaining
> turn-around time of the primary application pool rather than taking
> resources away whenever possible."
>

> gWLM also does the same..

>
> "Using gWLM, you will be able to move CPU resources to or from the workload
> as needed to maintain acceptable performance, while also using less net CPU
> resources over time than an unmanaged workload would. This is a net win for
> you as the unused CPU resources can then be used for other computing tasks
> when the SAP workloads do not require them."
>

>   _____  

>
> From: cloud-c...@googlegroups.com
> [mailto:cloud-c...@googlegroups.com] On Behalf Of Ramki Balasubramanian
> Sent: Monday, February 22, 2010 3:05 PM
> To: cloud-c...@googlegroups.com
> Subject: Re: [ Cloud Computing ] What is a "Sponge" ?
>
> From a look at the document pointed out by Fred, this solutions seems
> capable of keeping the primary application pool fully fed while
> redistributing the demand of other applications to work with the remaining
> capacity as much as possible.
>
> This is a different perspective, in that it seems focused on maintaining
> turn-around time of the primary application pool rather than taking
> resources away whenever possible.
>
> Curious about the name. Is the primary application the 'sponge' in the
> infrastructure, and all else is the water being absorbed? :-)
>
> -
> Ramki
>
> On Mon, Feb 22, 2010 at 8:37 AM, Rao Dronamraju
>

> Register Today for Cloud Slam 2010 at official website -http://cloudslam10.com

> Posting guidelines:http://groups.google.ca/group/cloud-computing/web/frequently-asked-qu...
> Follow us on Twitterhttp://twitter.com/cloudcomp_groupor @cloudcomp_group

> Post Job/Resume athttp://cloudjobs.net
> Buy 88 conference sessions and panels on cloud computing on DVD athttp://www.amazon.com/gp/product/B002H07SEC,http://www.amazon.com/gp/product/B002H0IW1Uor get instant access to
> downloadable versions athttp://cloudslam09.com/content/registration-5.html

>
> ~~~~~
> You received this message because you are subscribed to the Google Groups
> "Cloud Computing" group.
> To post to this group, send email to cloud-c...@googlegroups.com
> To unsubscribe from this group, send email to
> cloud-computi...@googlegroups.com
>
> --
> ~~~~~

> Register Today for Cloud Slam 2010 at official website -http://cloudslam10.com

> Posting guidelines:http://groups.google.ca/group/cloud-computing/web/frequently-asked-qu...
> Follow us on Twitterhttp://twitter.com/cloudcomp_groupor @cloudcomp_group

> Post Job/Resume athttp://cloudjobs.net

> Buy 88 conference sessions and panels on cloud computing on DVD athttp://www.amazon.com/gp/product/B002H07SEC,
>

> ...
>
> read more »

[Rao Dronamraju]

Miha,

We all know that even Yahoo had as much if not better limelight/success than
Google. Yahoo was a success even before Google. Google came to the search
scene later. Google, in the last 10 years, pulled away from the competition,
even though it needs to still be on the lookout for Microsoft.

Siimilarly Apple was ahead of MS in the PC business in early 80s. Apple
missed the (PC) bus very badly (btw, both Steve Jobs and Steven Wozniak were
at the helm) so did IBM and see where MS is now!.

So the point is why things (good and bad) happen to which companies how and
when, nobody knows. That is why life both that of an individual and a
company is LOT of LUCK!!!.

See where Sun is today, despite its mega Java driven boom in the dotcome
days, and was a major enterprie palyer for couple decades, had to beg around
to be bought!.

So even google is a mega case of LUCK...IMO...The doctcom luck ran out for
some companies like Yahoo and it is going on for some companies like Google.

Rao,

Miha

--
~~~~~

Register Today for Cloud Slam 2010 at official website -
http://cloudslam10.com
Posting guidelines:
http://groups.google.ca/group/cloud-computing/web/frequently-asked-questions
Follow us on Twitter http://twitter.com/cloudcomp_group or @cloudcomp_group
Post Job/Resume at http://cloudjobs.net
Buy 88 conference sessions and panels on cloud computing on DVD at
http://www.amazon.com/gp/product/B002H07SEC,

http://www.amazon.com/gp/product/B002H0IW1U or get instant access to
downloadable versions at http://cloudslam09.com/content/registration-5.html

[Miha Ahronovitz]

Rao, you spoke wisely my friend. I agree with every word below.
So any me-too, has a different angle, more luck, a better product
management team or is God Choosen as you rightly say.
We all have chances, if we believe in "yes we can"

Miha

>> read more �
>>
>
>

[Mohamed El-Refaey]

Hello Fred,
Interesting product actually. currently I am working in a research project named "CloudGauge" for workload characterization in virtual and cloud-based environments, part of this project output is to feed some load manger (one like yours) with the info needed to take decision for balancing the load in a virtualized cluster of VMs or so. If you are interested we can discuss this off the list, I think we can do a kind of integration between the two projects.

Regards,
Mohamed

[Jan Klincewicz]

La chance ne sourit qu'aux esprits bien préparés -  Louis Pasteur

Chance favors only the prepared mind.  The Google folks seem to have had the right service at the right time.  Just when the Internet was really making headway wit the masses, they earned a reputation as the "best search engine."  They exploited their goodwill (that is not a pejorative term) nevigated the rapid growth successfully, expanding their offerings and hiring smart people.

Their choice of platform on which to build their search engine probably had more to do with the ubiquity of x86 than anything else.  Google COULD have been built on expensive Alphas too and still made money.

--
Cheers,
Jan

[Jim Starkey]

Uh, Alta Vista was written by people at DEC's Palo Alto Research Center run by Bob Taylor.  Compaq didn't buy DEC until 1999.  Maybe the guys were Compaq moles, but they didn't look to me like Texans -- no hats, string ties, or oversize belt buckles.

-- 
Jim Starkey
Founder, NimbusDB, Inc.
978 526-1376

[Ken North]

Fred van den Bosch wrote
>> Load Manager is an application-aware scheduler ...

Is Load Manager a location-aware scheduler?

For a job involving analytics over distributed data sets, can it
schedule the processing with instances that are geographically closest
to the data in order to reduce network latency?

Does it "understand" EC2 Availability Zones?

[Fred van den Bosch]

That would be a good capability, but at this time Load Manager manages
resources on a per-server-instance basis and is not location aware. It's not
a job scheduler in the traditional sense.

Fred

-----Original Message-----
From: cloud-c...@googlegroups.com
[mailto:cloud-c...@googlegroups.com] On Behalf Of Ken North
Sent: Wednesday, February 24, 2010 1:25 PM
To: cloud-c...@googlegroups.com
Subject: [ Cloud Computing ] What is a "Sponge" ?

--
~~~~~

Register Today for Cloud Slam 2010 at official website -
http://cloudslam10.com
Posting guidelines:
http://groups.google.ca/group/cloud-computing/web/frequently-asked-questions
Follow us on Twitter http://twitter.com/cloudcomp_group or @cloudcomp_group
Post Job/Resume at http://cloudjobs.net
Buy 88 conference sessions and panels on cloud computing on DVD at
http://www.amazon.com/gp/product/B002H07SEC,

http://www.amazon.com/gp/product/B002H0IW1U or get instant access to
downloadable versions at http://cloudslam09.com/content/registration-5.html

[Miha Ahronovitz]

Fred,

Can Load Manager use an integration with Sun Grid Engine, open source
version?. There is a the concept of service containers, see the blog
from Zmanda, an Amazon AWS partner:

http://www.jcmartin.org/service-containers-in-sun-grid-engine/

On Solaris, Grid Engine is container-aware and in theory, it could bring
data locality to the Load Manager's containers on Linux and Windows.

Regarding the wish

> "schedule the processing with instances that are geographically closest
> to the data in order to reduce network latency"

Note that in the latest version 6.2 Update 5, SGE is integrated with Hadoop Apache, that can solve many problems by processing where the data is located rather bringing the data to the server. Not sure this Map/Reduce implementation can work geographically dispersed, meaning servers scattered in Japan, Europe and US. I am not sure why someone wants to do that anyway ,

See http://www.sun.com/software/products/container_mgr/

Other detail, the open source license for Grid Engine is SISSL
http://en.wikipedia.org/wiki/Sun_Industry_Standards_Source_License

This was the most liberal open source license ever, that , although is retired, still applies to the Grid Engine. Oracle may change that status, bur for now it still applies.

Other simple solution is to reach an OEM agreement with Oracle for SGE and the synergy Oracle Librato may work win-win wonders.

My 2 cents,

Miha

PS: Sun is now merged with Oracle, I am not an Oracle employee and the opinions above are solely mine

[Michael Brown]

Who provides location-aware schedulers?

--
~~~~~
Register Today for Cloud Slam 2010 at official website - http://cloudslam10.com
Posting guidelines: http://groups.google.ca/group/cloud-computing/web/frequently-asked-questions
Follow us on Twitter http://twitter.com/cloudcomp_group or @cloudcomp_group
Post Job/Resume at http://cloudjobs.net
Buy 88 conference sessions and panels on cloud computing on DVD at http://www.amazon.com/gp/product/B002H07SEC, http://www.amazon.com/gp/product/B002H0IW1U or get instant access to downloadable versions at http://cloudslam09.com/content/registration-5.html

~~~~~
You received this message because you are subscribed to the Google Groups "Cloud Computing" group.
To post to this group, send email to cloud-c...@googlegroups.com
To unsubscribe from this group, send email to cloud-computi...@googlegroups.com

--
Michael Brown
www.mobixy.com
www.visageapp.com
Email:  mbr...@mobiapi.com
Work: +1-(201) 677-8263
Fax:  +1-877-723-8789

_________________________________________
The information contained in this communication may be confidential, is intended only for the use of the recipient named above, and may be legally privileged.  If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication, or any of its contents, is strictly prohibited. If you have received this communication in error, please re-send this communication to the sender and delete the original message and any copy of it from your computer system.

Thank you.
_________________________________________

[Fred van den Bosch]

Load Manager can integrate and has been “integrated” (or better, “used in combination with”) with Job Schedulers, but mainly to allow background jobs to be scheduled for execution on servers that are also running other (interactive) jobs. Load Manager itself works on a per-server level and therefore has no placement engine or location awareness. As Michael points out, job schedulers do not have location awareness either, and I’m inclined to think that it will be challenging for job schedulers to figure out what the location of the data for a job is without being given this information.

 

Fred

[Frank Greco]

First... Hi Fred!  Haven't talked to you in a while.

The way I handled a variant of this requirement while toiling away at Lehman (somewhere out there Hank Paulson has a sly grin on his face) was to use a coarse grain scheduler like Autosys (don't laugh... it was forced on us) to schedule large master Jobs (not Steve) for grid subsystems that had the notion of affinity.  Affinity is a first-class citizen in Gigaspaces/Javaspaces and later companies like DataSynapse (now Tibco) added 'discriminators' which were less-elegant but usable.  These large jobs then did the scatter-gather, master-workers, (putting on my trendy glasses and giving my best "I'm hip" look) *hadoop* thing which we designed with data-affinity (and processor-affinity) in mind.   "Schedule this job where customer ABC's portfolio resides AND the box is a quad-core x86 with 16GB of RAM"

Side note:  it was hard to convince wall-street tech management that its way better to send the processing to the data as opposed to doing what they have been doing for 20 years, ie, sending the data to the engines via (Tibco RV) messaging.  It felt like I was describing fire to Frankenstein... "Fire... bad!!!"

Frank G.
 - Hey Hawaii... wish you the best for that upcoming tsunami later today... Stay safe.

[Fred van den Bosch]

Hi Frank! Good to hear from you. Couldn’t agree more on the need to bring the processing to the data; it’s often neglected when talking about cloud bursting as well.

 

Interesting solution for dealing with data location affinity. We have no plans to do something in this area in the near future; as a start-up we have a big vision and big ambitions, but need to start with doing something really simple (and hopefully really valuable) really well.

 

Best,

 

Fred