[Web-SIG] Shortening execution time of Python script

[David Shi]

I have a Python script that automatically downloads zip files containing large datasets from another server and then unzips the files to further process the data.   It has been used as a geoprocessor of ArcGIS Server.   The script works fine when two datasets each has several kilobytes size, but the script stops half way when datasets were about 11,000KBytes.   I think that the execution time is too long and ArcGIS Server just simply killed the process.   What actions can I try to reduce the execution time?   ArcGIS Server only works on the basis of 32 bits and I was told that the maximum memory it can utilise is 4 MBytes.   I should be grateful if someone can make suggestions/recommendations.   Sincerely,   David

[Dirkjan Ochtman]

On Wed, Sep 9, 2009 at 12:52, David Shi<davi...@yahoo.co.uk> wrote:
> What actions can I try to reduce the execution time?

This has absolutely nothing to do with the interests of the Web-SIG.

Please send more general Python questions to
http://mail.python.org/mailman/listinfo/python-list.

Cheers,

Dirkjan
_______________________________________________
Web-SIG mailing list
Web...@python.org
Web SIG: http://www.python.org/sigs/web-sig
Unsubscribe: http://mail.python.org/mailman/options/web-sig/python-web-sig-garchive-9074%40googlegroups.com

[Ionel Maries Cristian]

You're a bit contradicting yourself - what's the actual problem, process memory size or execution time ?

If it's the process memory size you could trick ArcGIS by using a subprocess that does the actual work (and eats into the memory).

-- ionel

_______________________________________________
Web-SIG mailing list
Web...@python.org
Web SIG: http://www.python.org/sigs/web-sig

Unsubscribe: http://mail.python.org/mailman/options/web-sig/ionel.mc%40gmail.com

[Paweł Stradomski]

W liście David Shi z dnia środa 09 września 2009:

> ArcGIS Server only works on the basis of 32 bits and I was told that the
> maximum memory it can utilise is 4 MBytes.

32bits give 4GB address space, not 4 MB

> I should be grateful if someone can make suggestions/recommendations.
>
> Sincerely,
>
> David

--
Paweł Stradomski

_______________________________________________
Web-SIG mailing list
Web...@python.org
Web SIG: http://www.python.org/sigs/web-sig

Unsubscribe: http://mail.python.org/mailman/options/web-sig/python-web-sig-garchive-9074%40googlegroups.com

[Chris Withers]

David Shi wrote:
> I have a Python script that automatically downloads zip files containing
> large datasets from another server and then unzips the files to further
> process the data.

This smells more like it belongs on comp.lang.python that wb-sig, but
here goes...

> The script works fine when two datasets each has several kilobytes size,
> but the script stops half way when datasets were about 11,000KBytes.
>
> I think that the execution time is too long and ArcGIS Server just
> simply killed the process.

How does ArcGIS execute this script?

> What actions can I try to reduce the execution time?
>
> ArcGIS Server only works on the basis of 32 bits and I was told that the
> maximum memory it can utilise is 4 MBytes.

For speed analysis, run the script through cProfile:

http://docs.python.org/library/profile.html

For analysis of how much memory your script is using, use heapy:

http://guppy-pe.sourceforge.net/heapy_tutorial.html

However, you mention downloading large files. Are you using httplib,
urllib or urllib2 to do this? If so, you could be suffering from this bug:

http://bugs.python.org/issue6838

cheers,

Chris

--
Simplistix - Content Management, Batch Processing & Python Consulting
- http://www.simplistix.co.uk

[Sean Gillies]

Chris,

Many Europeans call geographic information systems (GIS) "SIG". Web-
SIG, to someone who doesn't read the charter, looks like "Web GIS".
Hence the random ArcGIS question, I think.

Cheers,

--
Sean