MATLAB computations in Cloud

[Khazret Sapenov]

Has anyone used cloud to do financial computations (risk modeling, portfolio optimization, pricing/structure, tech trading/charting) or any other complex mathematical models?

Last year I tried MATLAB/LSF to run some basic matrix operations (like parfor, D=distributed(A)) and Black-Sholes model on EC2 and it worked just fine and was dirt cheap. I'd be interested to create an adapter to allow scientists and analysts execute distributed computation in cloud, thus reducing costs, improving performance and decision making. 

Perhaps same approach might be used with similar products like Maple etc.

This Monday I was at last MathWorks conference in Toronto, but all they had there was plain old GridServer from DataSynapse, which apparently runs in their facility, so I don't know how their clients resolve issue with exposing data to third party. 

It would be good to see cloud used in this area too. Write me, if you are interested in collaboration on the topic.

regards,

Khazret Sapenov

E-mail: sap...@gmail.com

[Vishal Mehra]

Yes, I have used octave (free version of MATLAB) to compute Cloud Computing economics particularly using Black-Sholes. If you are interested in knowing more, email me privately

Vishal
http://www.linkedin.com/in/mehravishal

[Paco NATHAN]

Hi Khaz,

We have a project to run R atop a Hadoop cluster. The objective is to
run large R stats jobs on EC2.

R is open source, has many many packages written for it, but is
limited by the amount of data that can be read into memory.

This is at a very early stage, but results look good so far.

Paco
(apologies to people who've emailed / we've been buried under a large rock)

[Paco NATHAN]

There is also the Hama project at Apache Incubator
http://wiki.apache.org/incubator/HamaProposal

...for linear algebra based on Hadoop

[Guy Tel-Zur]

I think it will be interesting to see if Star-P by Interactive
Supercomputing can be used in a cloud in a way that the back-end
server (cluster) will be hosted in a cloud. Once the data is
transferred from the front-end client to the cloud the computation may
be accelerated at the back-end. Of course, this model will not be
efficient for data intensive computations due to the network
bottleneck.
Start-P currently supports Matlab, R and Python.

Regards,

Guy Tel-Zur
Email: tel...@computer.org

> E-mail: sape...@gmail.com

[Peter Denyer]

Certainly some of the big financial organizations use their own internal grids to run MATLAB and use LSF as the mechanism to distribute the parallel pieces of the computation over that grid. Hundreds of CPUs/cores is not uncommon. That's a well known environment for both Platform Computing (LSF) and for The MathWorks (MATLAB).  However, independently licensing these apps in a cloud environment seems a little problematic to me. The license managers in both these products (Flex/LM or FlexNet is a typical license manager) rely on specific attributes of the host machine to do their work. Is there a guarantee that this will be the case - especially for the license manager itself.

So, it would seem to me that these companies would have to have done a prior deal with the cloud operating company (as some commercial apps companies seem to have done) to get their infrastructure set up properly in the environment. In this case, a user could then use the cloud infrastructure purely on a pay as you go basis - but who gets paid? the operator - who is billing out use of his virtual machines at so much per hour, the application provider(s) who need to recover revenue for the use of their tools? both? all?

When I was associated with Sun's efforts on the Network.com infrastructure, these issues precluded commercial apps from the environment, and raised the value of open source apps as the target for the compute utility. Many vendors of commercial software that I spoke with about using such an environment as Sun proposed basically said "over my dead body" (or at least their sales VPs did) as the infrastructure posited (call it cloud computing ) had, or could have, such an impact on their current revenue models, and collecting the revenue was a function they didn't want to address themselves. That was left from the cloud operator to address (gee, sounds like the old timesharing stuff I used to do with Control Data's CYBERNET - back in the day!)

I think it will take some time before this kind of computing model makes it big in the commercial application space. 

Thanks,

Peter Denyer

pde...@pacbell.net

Peter Denyer

pde...@pacbell.net

Cell: 510.378.0788

Phone: 510.656.8715

FAX: 510.656.1059

[Khazret Sapenov]

On Thu, Oct 2, 2008 at 5:10 PM, Peter Denyer <pde...@pacbell.net> wrote:

Certainly some of the big financial organizations use their own internal grids to run MATLAB and use LSF as the mechanism to distribute the parallel pieces of the computation over that grid. Hundreds of CPUs/cores is not uncommon. That's a well known environment for both Platform Computing (LSF) and for The MathWorks (MATLAB).  However, independently licensing these apps in a cloud environment seems a little problematic to me. The license managers in both these products (Flex/LM or FlexNet is a typical license manager) rely on specific attributes of the host machine to do their work. Is there a guarantee that this will be the case - especially for the license manager itself.

If we use their Compiler toolbox and MCR (as they said it is free runtime), it is also possible to share algoritms as Java classes or .NET assemblies and run it with open source scheduler. Also, I wouldn't lock myself to only Mathworks/LSF.

 

So, it would seem to me that these companies would have to have done a prior deal with the cloud operating company (as some commercial apps companies seem to have done) to get their infrastructure set up properly in the environment. In this case, a user could then use the cloud infrastructure purely on a pay as you go basis - but who gets paid? the operator - who is billing out use of his virtual machines at so much per hour, the application provider(s) who need to recover revenue for the use of their tools? both? all?

Amazon EC2 provides some of such features.

 

When I was associated with Sun's efforts on the Network.com infrastructure, these issues precluded commercial apps from the environment, and raised the value of open source apps as the target for the compute utility. Many vendors of commercial software that I spoke with about using such an environment as Sun proposed basically said "over my dead body" (or at least their sales VPs did) as the infrastructure posited (call it cloud computing ) had, or could have, such an impact on their current revenue models, and collecting the revenue was a function they didn't want to address themselves. That was left from the cloud operator to address (gee, sounds like the old timesharing stuff I used to do with Control Data's CYBERNET - back in the day!)

I think it will take some time before this kind of computing model makes it big in the commercial application space. 

I see obstacles to wider adoption of running computations in cloud, but they are in other domain(legal and technical, rarther than psychological/physiological). Perhaps recent changes in financial sphere, would make organizations put some thought into how cloud computing might reduce operating expenditures for some LOBs (like algorithmic trading etc), involved with mentioned modeling activity.

regards,

Khazret Sapenov

[Peter Denyer]

I think we're both on the same wavelength here. MATLAB aside, there are a lot of in-house developed applications in the financial community and elsewhere that can benefit from cloud computing, given that the security of data issues are adequately addressed. Cloud Computing can certainly help address the problem of "no more electricity to run your datacenter in the heart of New York's financial district".

Commercial, licensed applications are another thing entirely and perhaps best addressed with a SaaS model that more clearly delineates how the user get billed, who gets to collects the revenues and how these are, or are not, distributed to all participants. There has to be a win-win for the software vendor and the user. 

According to what I read on Oracle's approach to cloud computing and software billing, t certainly appears that they are innovating to get to that win-win place - and more credit to them for doing so.

this whole notion of inexpensive cloud computing reminds me of a conversation I participated in with Dr. Steven Wolfram (Wolfram Research - the makers of Mathematica) wherein he posed the question (and I paraphrase) "what new things can we learn when we have what amounts to unlimited computational power at everyone's fingertips". Previously intractable problems become tractable with sufficient, and inexpensive, compute horsepower. Cloud Computing opens up new opportunities for a new level of innovation.

Regards

Peter

[Ian Foster]

Hi Khazret:

We are doing a lot of work with Matlab applications. We run them,
mostly, via Swift.
(E.g., see https://twiki.grid.iu.edu/pub/Education/MWGS2008Syllabus/8_SwiftWorkflow.pdf.)

The target platform can then be a local cluster, one or more remote
sites (e.g., Open Science Grid), or alternatively, via Nimbus, our
local virtual machine system or EC2.

Ian.

On Wed, Oct 1, 2008 at 4:08 PM, Khazret Sapenov <sap...@gmail.com> wrote:

[Rayson]

I am involved with the Grid Engine opensource project: http://gridengine.sunsource.net/

(Grid Engine is very similar to LSF, and in fact used by many LSF
customers as LSF complement or replacement.)

Chris from the BioTeam recently added the Grid Engine/EC2 integration:
http://gridengine.info/2008/09/10/webinar-and-q-a-on-univa-ud-unicluster-amazon-ec2

Rayson

> E-mail: sape...@gmail.com

[Khazret Sapenov]

I am happy to announce, that users of MATLAB can use Amazon EC2 for parallel computations now. Read more details at  http://www.mathworks.com/programs/techkits/ec2_paper.html

[Santhosh Mogili]

Not sure if this is a perfect match for what you are looking for,,however Star-P offers something called Star-P on demand http://www.interactivesupercomputing.com/starpondemand/

--SM

[Khazret Sapenov]

Mathematica followed this path as well:

CHAMPAIGN, Ill., Nov. 5 -- Wolfram Research announced an initiative today to develop a cloud computing service for users of their flagship technical computing software Mathematica. This project is a collaborative effort by Wolfram Research, Nimbis Services, Inc., a clearing-house for accessing third party compute resources and commercial software, and R Systems NA, Inc., a provider of computing resources to the commercial and academic research community.

According to Deborah Wince-Smith, president of the Council on Competitiveness, "High-performance computing systems (HPC) remain a largely underutilized competitiveness asset in the United States for the majority of companies. Opening access to HPC represents a huge productivity opportunity for the nation and a competitiveness transformation challenge." The collaboration of Wolfram Research, Nimbis Services, and R Systems will make the transition from desktop to HPC systems easier for Mathematica users by providing efficiently structured access to larger, more powerful computing systems.

Nimbis Services will enable the Mathematica cloud service to access many diverse HPC systems, including TOP500 supercomputers and the Amazon Elastic Compute Cloud. Nimbis Services, Inc. President and CEO Robert Graybill echoes the council's views on HPC systems and explains that the foundational principle of Nimbis Services is to focus on "ease of use" by providing experimental and periodic business users the choice of large-scale computing service alternatives, all in one "instant" computing storefront.

"Our partnership with Wolfram Research immensely benefits software users attempting to increase efficiency and capacity," says R Systems founder Brian Kucic. "As Mathematica users seek to extend resource capacity, the exceptionally large memory of our multicore HPC resources and the double-data rate and quad-data rate InfiniBand network will increase performance." HPC resources such as the R Smarr cluster by R Systems, Inc., which was recently named the 44th fastest system on the TOP500 list for supercomputing pioneers, are responsible for bringing HPC technology to the forefront.

The Mathematica cloud computing service will provide flexible and scalable access to HPC from within Mathematica, simplifying the transition from desktop technical computing to HPC. "The two largest challenges in using HPC are programming the HPC application itself and ensuring that you can get enough computing power to do the job," says Tom Wickham-Jones, Wolfram Research executive director of kernel technology. "Mathematica answers the programming challenge by providing an integrated technical computing platform, enabling computation, visualization, and data access. Cloud computing offers consistent access to large-scale computing capabilities. We are excited to be working with Nimbis and R Systems to offer HPC access to our customers."

original: http://www.hpcwire.com/offthewire/Wolfram_to_Provide_Cloud_Computing_for_HPC_Tasks.html