Who is using simplekml

[Jaco plakkies]

Hello there.

I have just started dabling with module. I would love to see some
projects using the module, or just stories of what worked, and what
did not.

Thank you
Jaco

[souper]

I've been using it since v0.1 and it's been fantastic. Most of my
files are relatively simple (only placemarks, no lines/polys) and I
have never used TimePrimitive, Model, gx:Track, and gx:MultiTrack.
Worked through any issues that did crop up and as far I know they have
all been fixed in the current release (I'm still using 0.7.3 with some
personally implemented patches).

The only issue I now have is that the kml object grows too large and
memory errors occur during saving in the minidom module with 40,000 +
placemarks, all with many attributes and long descriptions.
Workaround could be to generate html descriptions and iframe to it
from the kml description. Looking into it more before I report
anything. Largest successful run was ~15000 placemarks which I then
piped to Google's regionator for performance. I've been looking at
building regionator support into simplekml but that's a 'maybe later'
kind of thing.

All of my data is sourced from CSV files which I have been iterating
through and inserting into the kml object directly. I am in the
process of developing some classes to make this more modular and
extensible for other projects I will have in the future.

I work for an airborne carcinogen research project. Here are some
samples of what I have made:
http://dl.dropbox.com/u/13303135/airports_Benzene.kmz
http://dl.dropbox.com/u/13303135/incinerators_Benzene.kmz
http://dl.dropbox.com/u/13303135/naps_benz.kmz
http://dl.dropbox.com/u/13303135/npri_air_Benzene.kmz
I'll leave these up for a couple weeks at least.

Again, all very basic, but I have found simplekml to be the best kml
solution for Python for my purposes. Personal opinion: FAR better and
more customiziable than any Excel based solution, better than Arc ->
ExportKML, more reliable than Arc2Earth, and MUCH more simple that
other xml based generation modules.

Cheers,
Basil

[kyle.lan]

Basil

I found your post very interesting. I have not tried to do any bench-
marking with simplekml, until I read your post. I tried to see how
many very simple points I could generate. By simple I mean just a
coordinate, no name, description, nothing. I did a couple of runs with
this code:

for i in range(100): # Vary value with each run
print i
for j in range(5000):
kml.newpoint(coords=[(i, j)])
kml.save("tests.kml",format=False)

but I was to impatient because each run takes about 5 minutes to
complete, so I estimate I can generate between 500,000 and 800,000
points (definitely 500,000 minimum). What you might find interesting
is the line kml.save("tests2.kml",format=False). By setting the format
to False I can get many more points because this bypasses the minidom
module. With format=True I could not even achieve 500,000 points. So,
if you do not care what the kml looks like when saved, just set format
to False.

Anyway, I would welcome any suggestions on how to improve the memory
management. I would also love to know what those "personally
implemented patches" are :)

Finally, I am very impressed with the samples you included. They may
be "very basic" points, but the balloons are very attractive, and I
loved the dots with the lines extended to the ground (guessing the
height of the line represents the amount of emissions?).

Regards
Kyle

On Feb 20, 9:34 pm, souper <basilveer...@gmail.com> wrote:
> I've been using it since v0.1 and it's been fantastic.  Most of my
> files are relatively simple (only placemarks, no lines/polys) and I
> have never used TimePrimitive, Model, gx:Track, and gx:MultiTrack.
> Worked through any issues that did crop up and as far I know they have
> all been fixed in the current release (I'm still using 0.7.3 with some
> personally implemented patches).
>
> The only issue I now have is that the kml object grows too large and
> memory errors occur during saving in the minidom module with 40,000 +
> placemarks, all with many attributes and long descriptions.
> Workaround could be to generate html descriptions and iframe to it
> from the kml description.  Looking into it more before I report
> anything.  Largest successful run was ~15000 placemarks which I then
> piped to Google's regionator for performance.  I've been looking at
> building regionator support into simplekml but that's a 'maybe later'
> kind of thing.
>
> All of my data is sourced from CSV files which I have been iterating
> through and inserting into the kml object directly. I am in the
> process of developing some classes to make this more modular and
> extensible for other projects I will have in the future.
>
> I work for an airborne carcinogen research project.  Here are some

> samples of what I have made:http://dl.dropbox.com/u/13303135/airports_Benzene.kmzhttp://dl.dropbox.com/u/13303135/incinerators_Benzene.kmzhttp://dl.dropbox.com/u/13303135/naps_benz.kmzhttp://dl.dropbox.com/u/13303135/npri_air_Benzene.kmz

[Basil Veerman]

Complete thread hijack.. sorry!

Works like a charm :).  Relevatant erformance analysis here.  The Jenks function is some classification preprocessing and unrelated to simplekml.  You can see 2 calls to  featgeom.py:353(__str__), one for kml and one for kmz (I know, unnecessary, but this product is still in dev) taking about 14 minutes each.   This is with 37999 placemarks processed with python 2.7 32bit on Win 7 x64, i7 2600k, 8G.  Maxes at around 800M memory usage.  My descriptions/attributes must drastically increase the size of the object and slow down processing.  

You will also see that I also call Regionator and in 10 seconds ( :-O ) it reads, parses, spatially processes (might be more like 'spatial guessing'), and re-exports the entire file.  I thought about looking into how that happens, but have no time.  

My other idea was to, if specified, create a kml that, for the description, uses an iframe of dimensions x,y and sources a relatively referenced html file (probably just in current dir, or perhaps option to specify http hard link).  The HTML file would simply contain what the description would have.  I have had to do this before when my kml got too big for GE to handle while maintaining acceptable performance (http://dl.dropbox.com/u/13303135/CAREX_Watershed_Rankings.kmz - kml generated with Arc2Earth, html generated with Python).  The result is that GE can process the kml just fine, but it may have a little delay while it fetches/displays the description.

As for modifications, they aren't much really:

1) manually patched the unicode errors on saving simply because I haven't upgraded yet

2) added a one liner count_features function in kml.py: return len(self._feature._features).  I currently don't use it for anything essential, but the idea is that based on the number of features I can specify regionation which I currently control with a boolean value.

3) basic integration of Google regionator.py, but my  implementation is bad and not worth of sharing.  Regionator needs a kml string/file as input and saves this to a base kml and the separate files in a subdir.  This messes up all relative links for images/etc.  I get around this by creating kml with simplekml I know will not work if viewed, but match with what regionator outputs so upon zipping to kmz after regionation all is well.  It makes me cringe but it works.  When/if time permits, I was going to code it based on your savekmz function to handle relative links.

Thank you for your compliments on the maps.  Seeing as I am basically the sole creator and designer, I usually only see ways that I could have made them better :-P.

Cheers!

Basil

[em.c...@gmail.com]

To continue on the hijacked thread, several years later..

I've been using simple kml for reasearch projects, trying to display many polygons (40 000 * 10 points), and found the limits of simple KML. I just found your post, explicitely saying that setting the keword format=False would allow many more points and would be much faster faster... and indeed it is.

So I am sending this message to ask you if the documentation of simplekml could be updated in order to reflect this feature.. it just changed my life afeter almost one year struggling with memory issues.

regards

[Kyle Lancaster]

Hi,

I just checked and the format option is documented, but I see it does not make any mention of the performance gain when using format=false.

I will take your suggention and update the documentation shortly (hopefully this weekend).

Thanks for the feedback!

Regards,

Kyle

--

---
You received this message because you are subscribed to the Google Groups "simplekml" group.
To unsubscribe from this group and stop receiving emails from it, send an email to simplekml+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Kyle Lancaster]

Hi,

Just to let you know, I have updated the documentation.

Thanks again.

Regards,

Kyle Lancaster

[charles...@gmail.com]

Hi,

I just tried format=False and that shaved over 0.5GB when testing with the sample code below:

import simplekml
import resource

kml = simplekml.Kml()

for i in xrange(10):
    for j in xrange(5000):
        kml.newpoint(coords=[(i, j)])

print 'Memory -- create kml: %s (kb)' % resource.getrusage(resource.RUSAGE_SELF).ru_maxrss

kml.savekmz("tests.kml")

print 'Memory -- save kml: %s (kb)' % resource.getrusage(resource.RUSAGE_SELF).ru_maxrss

Test:

$ python test.py 
Memory -- create kml: 466776 (kb)
Memory -- save kml: 513396 (kb)

Are there any technique(s) for adding lots of points while using limited amount of memory?

Thank you.

Charles

[Kyle Lancaster]

Hi,

Sorry for taking so long to reply. 

There is no way of reducing the memory that the kml object takes up, but I will need to look at implementing different methods for saving the kml to reduce memory consumption during the save operation.

Currently it build a string in memory and then writes it to file. I am looking at possibly implementing a way for it to build the string in a file instead of memory, thus reducing memory usage.

I am uncertain if it will work, but if it does I will include the option in a future release.

Regards,

Kyle Lancaster

--

[charles...@gmail.com]

Thanks Kyle. 

Looking forward to trying out the optimization once it becomes available.