numpy and scipy question

[Thanikai Sokka]

I have a customer asking whether numpy and scipy functions will work in GAE. I realize there is a feature request for it and it isn't built in today, but is there any workaround or way for someone to use numpy and scipy in GAE?

Thanks,

Thani

[Anand Mistry]

NumPy is available on the Python 2.7 runtime, but SciPy is not.

[Hans Then]

Hi Anand,

Are there plans to also support SciPy in the future?

Hans Then

Op woensdag 25 januari 2012 00:34:53 UTC+1 schreef Anand Mistry het volgende:

[Drake]

SciPy is mostly in Fortran. So I don’t give you good odds.

 

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/-WRRRE6BhD8J.
To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

[Hans Then]

NumPy is also Fortran. 

But my confusion is that if SciPy will not be supported, then why support NumPy? NumPy and SciPy are hardly ever used separately.

Hans

Op zondag 22 juli 2012 09:11:00 UTC+2 schreef Brandon Wirtz het volgende:

SciPy is mostly in Fortran. So I don’t give you good odds.

 

From: google-appengine@googlegroups.com [mailto:google-appengine@googlegroups.com] On Behalf Of Hans Then
Sent: Sunday, July 22, 2012 12:05 AM
To: google-appengine@googlegroups.com
Subject: [google-appengine] Re: numpy and scipy question

 

Hi Anand,

 

Are there plans to also support SciPy in the future?

 

Hans Then

Op woensdag 25 januari 2012 00:34:53 UTC+1 schreef Anand Mistry het volgende:

NumPy is available on the Python 2.7 runtime, but SciPy is not.

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/-WRRRE6BhD8J.

To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to google-appengine+unsubscribe@googlegroups.com.

[Drake]

I don’t know then. I thought NumPy was all C…

Any how I use Numpy but not SciPy. Apparently Google caters to my every whim and so they gave me what I needed.  J

Depending what you need Matrix.py, and Euclid.py get you many of the most common things you would use Numpy and Scipy for.

 

 

From: google-a...@googlegroups.com [mailto:google-a...@googlegroups.com] On Behalf Of Hans Then
Sent: Sunday, July 22, 2012 1:34 AM
To: google-a...@googlegroups.com
Subject: Re: [google-appengine] Re: numpy and scipy question

 

NumPy is also Fortran. 

 

But my confusion is that if SciPy will not be supported, then why support NumPy? NumPy and SciPy are hardly ever used separately.

 

Hans

Op zondag 22 juli 2012 09:11:00 UTC+2 schreef Brandon Wirtz het volgende:

SciPy is mostly in Fortran. So I don’t give you good odds.

 

From: google-a...@googlegroups.com [mailto:google-a...@googlegroups.com] On Behalf Of Hans Then
Sent: Sunday, July 22, 2012 12:05 AM
To: google-a...@googlegroups.com
Subject: [google-appengine] Re: numpy and scipy question

 

Hi Anand,

 

Are there plans to also support SciPy in the future?

 

Hans Then

Op woensdag 25 januari 2012 00:34:53 UTC+1 schreef Anand Mistry het volgende:

NumPy is available on the Python 2.7 runtime, but SciPy is not.

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/-WRRRE6BhD8J.

To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

--

You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/iM54xnkAilIJ.
To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.

[Hans Then]

The application I develop uses libraries such as gensim and NLTK, which depend on SciPy. Currently my plan is to run it on Heroku, but I would much prefer to use appengine.

Hans

[Drake]

I am building an API that does much of what NLTK does.  Which parts are you looking to use? Likely making API calls to my optimized version would cost less than actually running the full version, (which includes a lot of things that won’t work on GAE  (like the cool drawing of the sentence diagram)

If the parts you need don’t take too much effort to add, I’ll put them on my teams task list for the API release in 2 weeks.

 

-Brandon

[Hans Then]

I am mainly using the classify and the cluster packages.

Thanks for considering this to be included.

Hans 

Op zondag 22 juli 2012 19:18:25 UTC+2 schreef Brandon Wirtz het volgende:

Hans

 

NumPy is also Fortran. 

 

To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to google-appengine+unsubscribe@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/iM54xnkAilIJ.

To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to google-appengine+unsubscribe@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to google-appengine+unsubscribe@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to google-appengine+unsubscribe@googlegroups.com.

[Drake]

I think we have everything you need.  We are working on some speed improvements because unfortunately the 60s limit doesn’t let you work with very big documents.

Do you know how many words at a time you need to put through?

 

 

From: google-a...@googlegroups.com [mailto:google-a...@googlegroups.com] On Behalf Of Hans Then
Sent: Sunday, July 22, 2012 3:51 PM
To: google-a...@googlegroups.com
Subject: Re: [google-appengine] Re: numpy and scipy question

 

I am mainly using the classify and the cluster packages.

 

Thanks for considering this to be included.

 

Hans 

Op zondag 22 juli 2012 19:18:25 UTC+2 schreef Brandon Wirtz het volgende:

I am building an API that does much of what NLTK does.  Which parts are you looking to use? Likely making API calls to my optimized version would cost less than actually running the full version, (which includes a lot of things that won’t work on GAE  (like the cool drawing of the sentence diagram)

If the parts you need don’t take too much effort to add, I’ll put them on my teams task list for the API release in 2 weeks.

 

-Brandon

 

From: google-a...@googlegroups.com [mailto:google-a...@googlegroups.com] On Behalf Of Hans Then
Sent: Sunday, July 22, 2012 7:04 AM
To: google-a...@googlegroups.com
Subject: Re: [google-appengine] Re: numpy and scipy question

 

The application I develop uses libraries such as gensim and NLTK, which depend on SciPy. Currently my plan is to run it on Heroku, but I would much prefer to use appengine.

 

Hans

On Sun, Jul 22, 2012 at 11:24 AM, Drake <dra...@digerat.com> wrote:

I don’t know then. I thought NumPy was all C…

Any how I use Numpy but not SciPy. Apparently Google caters to my every whim and so they gave me what I needed.  J

Depending what you need Matrix.py, and Euclid.py get you many of the most common things you would use Numpy and Scipy for.

 

 

From: google-a...@googlegroups.com [mailto:google-a...@googlegroups.com] On Behalf Of Hans Then
Sent: Sunday, July 22, 2012 1:34 AM
To: google-a...@googlegroups.com
Subject: Re: [google-appengine] Re: numpy and scipy question

 

NumPy is also Fortran. 

 

But my confusion is that if SciPy will not be supported, then why support NumPy? NumPy and SciPy are hardly ever used separately.

 

Hans

Op zondag 22 juli 2012 09:11:00 UTC+2 schreef Brandon Wirtz het volgende:

SciPy is mostly in Fortran. So I don’t give you good odds.

 

From: google-a...@googlegroups.com [mailto:google-a...@googlegroups.com] On Behalf Of Hans Then
Sent: Sunday, July 22, 2012 12:05 AM
To: google-a...@googlegroups.com
Subject: [google-appengine] Re: numpy and scipy question

 

Hi Anand,

 

Are there plans to also support SciPy in the future?

 

Hans Then

Op woensdag 25 januari 2012 00:34:53 UTC+1 schreef Anand Mistry het volgende:

NumPy is available on the Python 2.7 runtime, but SciPy is not.

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/-WRRRE6BhD8J.

To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/iM54xnkAilIJ.

To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/ESn4rOHx3a8J.
To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.

[Hans Then]

Ah. That is very good news. We have no real timing requirements. I think we will setup a backend instance for long running requests, to get around the 60s limit.

[Anand Mistry]

If the processing is considered "offline" and not latency sensitive, then you can put that work onto a task queue and take up to 10 minutes. No need for a backend.

As for SciPy, Brandon is exactly right. SciPy is written entirely in Fortran whereas the other third-party libraries we have (NumPy, PIL, lxml) are written in C. For NLTK, do you really need SciPy? According to nltk.org, only NumPy is a dependency (optional).

On Monday, 23 July 2012 09:33:35 UTC+10, Hans Then wrote:

Ah. That is very good news. We have no real timing requirements. I think we will setup a backend instance for long running requests, to get around the 60s limit.

On Mon, Jul 23, 2012 at 1:24 AM, Drake <dra...@digerat.com> wrote:

I think we have everything you need.  We are working on some speed improvements because unfortunately the 60s limit doesn’t let you work with very big documents.

Do you know how many words at a time you need to put through?

 

 

From: google-appengine@googlegroups.com [mailto:google-appengine@googlegroups.com] On Behalf Of Hans Then
Sent: Sunday, July 22, 2012 3:51 PM

To: google-appengine@googlegroups.com

Hans

 

NumPy is also Fortran. 

 

To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to google-appengine+unsubscribe@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/iM54xnkAilIJ.

To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to google-appengine+unsubscribe@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to google-appengine+unsubscribe@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to google-appengine+unsubscribe@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/ESn4rOHx3a8J.

To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to google-appengine+unsubscribe@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to google-appengine+unsubscribe@googlegroups.com.

[Hans Then]

Hi Anand,

Processing is offline, but the order of magnitude will be more like days than 10 minutes, so I guess backends are the way to go. Basically the application will perform statistical analysis on large datasets of news items and media data. Right now, we run this analysis on hardware we own, but we'd like to make the basic toolset available to the scientific community. 

As for SciPy being in Fortran, does this mean that there are no plans at all to make SciPy available on Appengine? Or only that it will take some more time to do so?

Regards,

Hans Then 

Op maandag 23 juli 2012 14:47:19 UTC+2 schreef Anand Mistry het volgende: