I have written a Python script to analyze Minecraft levels and render a graph. Then I did the same with Clojure. It takes Python 10 seconds to analyze a map, while it takes Clojure over a minute.
After having tried different options without any significant improvement, I am lost as to why there is such a huge difference. I wouldn't mind an extra pair of eyes/brains to look at this.
On Thu, 4 Nov 2010 22:28:12 +0100 Pepijn de Vos <pepijnde...@gmail.com> wrote:
> Hi all,
> I have written a Python script to analyze Minecraft levels and render a graph. Then I did the same with Clojure. It takes Python 10 seconds to analyze a map, while it takes Clojure over a minute.
> After having tried different options without any significant improvement, I am lost as to why there is such a huge difference. I wouldn't mind an extra pair of eyes/brains to look at this.
I once ran into an issue where Python was running rings around an Eiffel version (compiled down to native code - no VM need apply). This looks similar to what you have, in that I built a large data structure, and then started groveling over it. Turned out that Eiffel was doing a mark-and-sweep GC, which was spending all of it's time marking and sweeping the large static data structure, whereas python doing a reference count GC didn't. Given that I know nothing about Java GCs, this is just a WAG.
Come to think of it, how about trying to run the program Jython? That should have the same GC issues. If it's some similar environmental problem, that would show up there as well.
<mwm-keyword-googlegroups.620...@mired.org> wrote: > On Thu, 4 Nov 2010 22:28:12 +0100 > Pepijn de Vos <pepijnde...@gmail.com> wrote:
>> Hi all,
>> I have written a Python script to analyze Minecraft levels and render a graph. Then I did the same with Clojure. It takes Python 10 seconds to analyze a map, while it takes Clojure over a minute.
>> After having tried different options without any significant improvement, I am lost as to why there is such a huge difference. I wouldn't mind an extra pair of eyes/brains to look at this.
> Can you check GC activity in the clojure version?
> I once ran into an issue where Python was running rings around an > Eiffel version (compiled down to native code - no VM need apply). This > looks similar to what you have, in that I built a large data > structure, and then started groveling over it. Turned out that Eiffel > was doing a mark-and-sweep GC, which was spending all of it's time > marking and sweeping the large static data structure, whereas python > doing a reference count GC didn't. Given that I know nothing about > Java GCs, this is just a WAG.
> Come to think of it, how about trying to run the program Jython? That > should have the same GC issues. If it's some similar environmental > problem, that would show up there as well.
There are many different collectors for the JVMs, too numerous to list here, all tunable.
I don't know how to check the GC activity on my project, but I did run
Mian on Jython. It performs much like my initial Clojure version. It
consumes absurd amounts of memory and never finishes.
So I think we can safely say that Java's GC or the way it stores data
is less efficient on this type of problem than Python.
On Nov 4, 10:43 pm, Mike Meyer <mwm-keyword-googlegroups.
620...@mired.org> wrote:
> On Thu, 4 Nov 2010 22:28:12 +0100
> Pepijn de Vos <pepijnde...@gmail.com> wrote:
> > Hi all,
> > I have written a Python script to analyze Minecraft levels and render a graph. Then I did the same with Clojure. It takes Python 10 seconds to analyze a map, while it takes Clojure over a minute.
> > After having tried different options without any significant improvement, I am lost as to why there is such a huge difference. I wouldn't mind an extra pair of eyes/brains to look at this.
> Can you check GC activity in the clojure version?
> I once ran into an issue where Python was running rings around an
> Eiffel version (compiled down to native code - no VM need apply). This
> looks similar to what you have, in that I built a large data
> structure, and then started groveling over it. Turned out that Eiffel
> was doing a mark-and-sweep GC, which was spending all of it's time
> marking and sweeping the large static data structure, whereas python
> doing a reference count GC didn't. Given that I know nothing about
> Java GCs, this is just a WAG.
> Come to think of it, how about trying to run the program Jython? That
> should have the same GC issues. If it's some similar environmental
> problem, that would show up there as well.
> <mwm-keyword-googlegroups.620...@mired.org> wrote:
> > On Thu, 4 Nov 2010 22:28:12 +0100
> > Pepijn de Vos <pepijnde...@gmail.com> wrote:
> >> Hi all,
> >> I have written a Python script to analyze Minecraft levels and render a graph. Then I did the same with Clojure. It takes Python 10 seconds to analyze a map, while it takes Clojure over a minute.
> >> After having tried different options without any significant improvement, I am lost as to why there is such a huge difference. I wouldn't mind an extra pair of eyes/brains to look at this.
> > Can you check GC activity in the clojure version?
> > I once ran into an issue where Python was running rings around an
> > Eiffel version (compiled down to native code - no VM need apply). This
> > looks similar to what you have, in that I built a large data
> > structure, and then started groveling over it. Turned out that Eiffel
> > was doing a mark-and-sweep GC, which was spending all of it's time
> > marking and sweeping the large static data structure, whereas python
> > doing a reference count GC didn't. Given that I know nothing about
> > Java GCs, this is just a WAG.
> > Come to think of it, how about trying to run the program Jython? That
> > should have the same GC issues. If it's some similar environmental
> > problem, that would show up there as well.
> There are many different collectors for the JVMs, too numerous to list
> here, all tunable.
On Fri, Nov 5, 2010 at 10:41 AM, pepijn (aka fliebel) <pepijnde...@gmail.com
> wrote: > I don't know how to check the GC activity on my project, but I did run > Mian on Jython. It performs much like my initial Clojure version. It > consumes absurd amounts of memory and never finishes.
> So I think we can safely say that Java's GC or the way it stores data > is less efficient on this type of problem than Python.
It's common that iteration heavy, mutation heavy code which is idiomatic in Python poses some challenges when translating to Clojure. Making this run faster than Python should be possible, and I would be surprised if it wasn't quite a bit faster. You should search the Google Group for the various threads on optimizing slow Clojure code.
I note that the repo does not contain the data file which your code runs against?
Meanwhile I wrote a function that is already twice as fast as I had,
no memory problems, no threads. One tinny problem: it doesn't produce
the same result.
The other day I found out that this kind of logic will actually refer
to the same same transient. This eliminates the remainder and associng
in the areduce fn second in the list, but I'm not sure this is
reliable, and it might be the reason why some results get lost.
On Nov 5, 4:24 pm, David Nolen <dnolen.li...@gmail.com> wrote:
> On Fri, Nov 5, 2010 at 10:41 AM, pepijn (aka fliebel) <pepijnde...@gmail.com
> > wrote:
> > I don't know how to check the GC activity on my project, but I did run
> > Mian on Jython. It performs much like my initial Clojure version. It
> > consumes absurd amounts of memory and never finishes.
> > So I think we can safely say that Java's GC or the way it stores data
> > is less efficient on this type of problem than Python.
> It's common that iteration heavy, mutation heavy code which is idiomatic in
> Python poses some challenges when translating to Clojure. Making this run
> faster than Python should be possible, and I would be surprised if it wasn't
> quite a bit faster. You should search the Google Group for the various
> threads on optimizing slow Clojure code.
> I note that the repo does not contain the data file which your code runs
> against?
> Meanwhile I wrote a function that is already twice as fast as I had, > no memory problems, no threads. One tinny problem: it doesn't produce > the same result.
> The other day I found out that this kind of logic will actually refer > to the same same transient. This eliminates the remainder and associng > in the areduce fn second in the list, but I'm not sure this is > reliable, and it might be the reason why some results get lost.
I'm not familiar with incanter, which defines update!, but the update! call makes me suspicious. Transients are not designed to be banged on in place. That would explain your losing results.
I'm very curios about this situation, please let us know if you manage to write a version that's faster than the python one (as David claims is possible). I would attempt it myself but I've only just recently had the time to dive back into Clojure. :-\
- Greg
On Nov 5, 2010, at 9:38 AM, pepijn (aka fliebel) wrote:
> Meanwhile I wrote a function that is already twice as fast as I had, > no memory problems, no threads. One tinny problem: it doesn't produce > the same result.
> The other day I found out that this kind of logic will actually refer > to the same same transient. This eliminates the remainder and associng > in the areduce fn second in the list, but I'm not sure this is > reliable, and it might be the reason why some results get lost.
> On Nov 5, 4:24 pm, David Nolen <dnolen.li...@gmail.com> wrote: >> On Fri, Nov 5, 2010 at 10:41 AM, pepijn (aka fliebel) <pepijnde...@gmail.com
>>> wrote: >>> I don't know how to check the GC activity on my project, but I did run >>> Mian on Jython. It performs much like my initial Clojure version. It >>> consumes absurd amounts of memory and never finishes.
>>> So I think we can safely say that Java's GC or the way it stores data >>> is less efficient on this type of problem than Python.
>> It's common that iteration heavy, mutation heavy code which is idiomatic in >> Python poses some challenges when translating to Clojure. Making this run >> faster than Python should be possible, and I would be surprised if it wasn't >> quite a bit faster. You should search the Google Group for the various >> threads on optimizing slow Clojure code.
>> I note that the repo does not contain the data file which your code runs >> against?
>> David
> -- > You received this message because you are subscribed to the Google > Groups "Clojure" group. > To post to this group, send email to clojure@googlegroups.com > Note that posts from new members are moderated - please be patient with your first post. > To unsubscribe from this group, send email to > clojure+unsubscribe@googlegroups.com > For more options, visit this group at > http://groups.google.com/group/clojure?hl=en
On Fri, Nov 5, 2010 at 2:31 PM, Greg <g...@kinostudios.com> wrote: > I'm very curios about this situation, please let us know if you manage to > write a version that's faster than the python one (as David claims is > possible). I would attempt it myself but I've only just recently had the > time to dive back into Clojure. :-\
> - Greg
I'm almost 100% certain it's possible. But having already optimized other people's code several times on this list as a learning experiment, I've become bored with optimizing other people's code ;)
> > Meanwhile I wrote a function that is already twice as fast as I had,
> > no memory problems, no threads. One tinny problem: it doesn't produce
> > the same result.
> > The other day I found out that this kind of logic will actually refer
> > to the same same transient. This eliminates the remainder and associng
> > in the areduce fn second in the list, but I'm not sure this is
> > reliable, and it might be the reason why some results get lost.
> I'm not familiar with incanter, which defines update!, but the update!
> call makes me suspicious. Transients are not designed to be banged on
> in place. That would explain your losing results.
Could you refer me to some of those relevant to my problem? I tried
searching for them, and most stuff I found is about killing
reflection, using buffered IO and other basics I've already covered.
On Nov 5, 7:37 pm, David Nolen <dnolen.li...@gmail.com> wrote:
> On Fri, Nov 5, 2010 at 2:31 PM, Greg <g...@kinostudios.com> wrote:
> > I'm very curios about this situation, please let us know if you manage to
> > write a version that's faster than the python one (as David claims is
> > possible). I would attempt it myself but I've only just recently had the
> > time to dive back into Clojure. :-\
> > - Greg
> I'm almost 100% certain it's possible. But having already optimized other
> people's code several times on this list as a learning experiment, I've
> become bored with optimizing other people's code ;)
You can use Visual VM (https://visualvm.dev.java.net/) to see how the
VM is using memory. I don't think it specifically show a log of GC
activity, but it is pretty clear from the graphs.
mch
On Nov 5, 8:41 am, "pepijn (aka fliebel)" <pepijnde...@gmail.com>
wrote:
> I don't know how to check the GC activity on my project, but I did run
> Mian on Jython. It performs much like my initial Clojure version. It
> consumes absurd amounts of memory and never finishes.
> So I think we can safely say that Java's GC or the way it stores data
> is less efficient on this type of problem than Python.
> On Nov 4, 10:43 pm, Mike Meyer <mwm-keyword-googlegroups.
> 620...@mired.org> wrote:
> > On Thu, 4 Nov 2010 22:28:12 +0100
> > Pepijn de Vos <pepijnde...@gmail.com> wrote:
> > > Hi all,
> > > I have written a Python script to analyze Minecraft levels and render a graph. Then I did the same with Clojure. It takes Python 10 seconds to analyze a map, while it takes Clojure over a minute.
> > > After having tried different options without any significant improvement, I am lost as to why there is such a huge difference. I wouldn't mind an extra pair of eyes/brains to look at this.
> > Can you check GC activity in the clojure version?
> > I once ran into an issue where Python was running rings around an
> > Eiffel version (compiled down to native code - no VM need apply). This
> > looks similar to what you have, in that I built a large data
> > structure, and then started groveling over it. Turned out that Eiffel
> > was doing a mark-and-sweep GC, which was spending all of it's time
> > marking and sweeping the large static data structure, whereas python
> > doing a reference count GC didn't. Given that I know nothing about
> > Java GCs, this is just a WAG.
> > Come to think of it, how about trying to run the program Jython? That
> > should have the same GC issues. If it's some similar environmental
> > problem, that would show up there as well.
> > <mike
> > --
> > Mike Meyer <m...@mired.org> http://www.mired.org/consulting.html > > Independent Network/Unix/Perforce consultant, email for more information.
I think you missed his point. (assoc! m k v) is *allowed* to modify m,
not *guaranteed*. It returns a pointer to a transient map, which may
be m, or may be a totally distinct map, or may be a new map that
shares some pointers with m. So your (do (update! blah foo
bar) ...more stuff) is potentially (and unpredictably) throwing away
the results of the update. You need to save the return value.
On Nov 5, 11:48 am, "pepijn (aka fliebel)" <pepijnde...@gmail.com>
wrote:
> > > Meanwhile I wrote a function that is already twice as fast as I had,
> > > no memory problems, no threads. One tinny problem: it doesn't produce
> > > the same result.
> > > The other day I found out that this kind of logic will actually refer
> > > to the same same transient. This eliminates the remainder and associng
> > > in the areduce fn second in the list, but I'm not sure this is
> > > reliable, and it might be the reason why some results get lost.
> > I'm not familiar with incanter, which defines update!, but the update!
> > call makes me suspicious. Transients are not designed to be banged on
> > in place. That would explain your losing results.
Here's what I have so far. The code splits blocks into 128 smaller
sub-arrays, each representing a level, then calls a modified version
of frequencies (using areduce instead of reduce) on each level. On my
machine, with server mode on, it takes about 20 seconds to compute the
frequencies for an array of 99844096 blocks. I haven't tested it on
your level, because I'm too lazy to get JNBT set up, but I'm curious
to see if it returns the correct result for you.
> I have written a Python script to analyze Minecraft levels and render a graph. Then I did the same with Clojure. It takes Python 10 seconds to analyze a map, while it takes Clojure over a minute.
> After having tried different options without any significant improvement, I am lost as to why there is such a huge difference. I wouldn't mind an extra pair of eyes/brains to look at this.
> Here's what I have so far. The code splits blocks into 128 smaller
> sub-arrays, each representing a level, then calls a modified version
> of frequencies (using areduce instead of reduce) on each level. On my
> machine, with server mode on, it takes about 20 seconds to compute the
> frequencies for an array of 99844096 blocks. I haven't tested it on
> your level, because I'm too lazy to get JNBT set up, but I'm curious
> to see if it returns the correct result for you.
> On Nov 4, 3:28 pm, Pepijn de Vos <pepijnde...@gmail.com> wrote:
> > Hi all,
> > I have written a Python script to analyze Minecraft levels and render a graph. Then I did the same with Clojure. It takes Python 10 seconds to analyze a map, while it takes Clojure over a minute.
> > After having tried different options without any significant improvement, I am lost as to why there is such a huge difference. I wouldn't mind an extra pair of eyes/brains to look at this.
Awesome. You managed to reproduce my initial solution, but working
with arrays all the way. It is already quite a bit faster than what I
have, but still nowhere near Python.
I'll put it on the list for things to look at.
On Nov 6, 1:27 am, Benny Tsai <benny.t...@gmail.com> wrote:
> On Nov 5, 4:57 pm, Benny Tsai <benny.t...@gmail.com> wrote:
> > Here's what I have so far. The code splits blocks into 128 smaller
> > sub-arrays, each representing a level, then calls a modified version
> > of frequencies (using areduce instead of reduce) on each level. On my
> > machine, with server mode on, it takes about 20 seconds to compute the
> > frequencies for an array of 99844096 blocks. I haven't tested it on
> > your level, because I'm too lazy to get JNBT set up, but I'm curious
> > to see if it returns the correct result for you.
> > On Nov 4, 3:28 pm, Pepijn de Vos <pepijnde...@gmail.com> wrote:
> > > Hi all,
> > > I have written a Python script to analyze Minecraft levels and render a graph. Then I did the same with Clojure. It takes Python 10 seconds to analyze a map, while it takes Clojure over a minute.
> > > After having tried different options without any significant improvement, I am lost as to why there is such a huge difference. I wouldn't mind an extra pair of eyes/brains to look at this.
> You can use Visual VM (https://visualvm.dev.java.net/) to see how the > VM is using memory. I don't think it specifically show a log of GC > activity, but it is pretty clear from the graphs.
Or just use -XX:+PrintGC and maybe -XX:+PrintGCDetails and -XX:+PrintGCTimeStamps.
I haven't checked what the code is doing, but if you suspect extremely poor performance due to GC it may be because your application happens to require some amount of memory that is below but fairly close to the default maximum heap size. That may easily cause very frequent GC:s and show up as poor performance. If this is the case, doubling the heap size should fix it (-Xmx...).
(The JVM does throw OutOfMemoryExceptions when it decides there is cause too, but it is a difficult heuristic to decide when that is actually the right thing to do. So it's very possible to be in situations that are not quite so bad in terms of time spent doing GC that the JVM throws an exception, yet bad enough to cause very frequent full GC:s at considerable cost in CPU time.)
For 30 second worth of calculations, this doesn't look to bad to me.
I increased the heap space a lot, but I'm just bordering on the edge
of my real memory, so it's not helping much. Enabling the -server
thing seemed to help a tinny bit though.
On Nov 6, 12:32 pm, Peter Schuller <peter.schul...@infidyne.com>
wrote:
> > You can use Visual VM (https://visualvm.dev.java.net/) to see how the
> > VM is using memory. I don't think it specifically show a log of GC
> > activity, but it is pretty clear from the graphs.
> Or just use -XX:+PrintGC and maybe -XX:+PrintGCDetails and
> -XX:+PrintGCTimeStamps.
> I haven't checked what the code is doing, but if you suspect extremely
> poor performance due to GC it may be because your application happens
> to require some amount of memory that is below but fairly close to the
> default maximum heap size. That may easily cause very frequent GC:s
> and show up as poor performance. If this is the case, doubling the
> heap size should fix it (-Xmx...).
> (The JVM does throw OutOfMemoryExceptions when it decides there is
> cause too, but it is a difficult heuristic to decide when that is
> actually the right thing to do. So it's very possible to be in
> situations that are not quite so bad in terms of time spent doing GC
> that the JVM throws an exception, yet bad enough to cause very
> frequent full GC:s at considerable cost in CPU time.)
On 2010-11-06, at 9:08 AM, pepijn (aka fliebel) wrote:
> I increased the heap space a lot, but I'm just bordering on the edge > of my real memory, so it's not helping much.
Did you try pushing the minimum heap space up. I'm usually lazy and set them to the same. I've had serious trouble caused by the way the JVM increases the heap space. Setting min to max (and max big) pretty much took care of that issue.
>> I increased the heap space a lot, but I'm just bordering on the edge >> of my real memory, so it's not helping much.
> Did you try pushing the minimum heap space up. I'm usually lazy and set them to the same. I've had serious trouble caused by the way the JVM increases the heap space. Setting min to max (and max big) pretty much took care of that issue.
People keep making claims like this in various situations but I don't tend to hear details. Exactly what problems are you having that would plausibly apply in this situation?
Not that there is no reason to set ms=mx (there are reasons), but the need to do so tends to be over-stated in my opinion. But if I'm missing something I'd like to know about it :)
>>> I increased the heap space a lot, but I'm just bordering on the edge >>> of my real memory, so it's not helping much.
>> Did you try pushing the minimum heap space up. I'm usually lazy and set them to the same. I've had serious trouble caused by the way the JVM increases the heap space. Setting min to max (and max big) pretty much took care of that issue.
> People keep making claims like this in various situations but I don't > tend to hear details. Exactly what problems are you having that would > plausibly apply in this situation?
> Not that there is no reason to set ms=mx (there are reasons), but the > need to do so tends to be over-stated in my opinion. But if I'm > missing something I'd like to know about it :)
I understand your scepticism but, even applaud it, but, in my case, it comes from actually trying it and measuring the difference (again in my case you didn't need anything fancy it was huge and highly visible). It happened often enough on different projects that I just do it routinely now. Anyway, if I *ever* see something that might be a GC-like problem I first eliminate heap growth from the picture (and, this is all I was suggesting here). Perhaps a hold over from earlier versions of the JVM but I don't personally care that much with my servers -- I have a machine dedicated to the application, it's got a lot of memory, use it. Might have a different attitude for a desktop app :-)
Cheers, Bob
> -- > / Peter Schuller
> -- > You received this message because you are subscribed to the Google > Groups "Clojure" group. > To post to this group, send email to clojure@googlegroups.com > Note that posts from new members are moderated - please be patient with your first post. > To unsubscribe from this group, send email to > clojure+unsubscribe@googlegroups.com > For more options, visit this group at > http://groups.google.com/group/clojure?hl=en
While grocery shopping this morning, it occurred to me that it would
be even faster to do a single pass over the blocks array, and update
the count in one of 128 maps depending on the current index. Of
course, when I got home and took another look at your gist page, it
turns out that's you've already done in the second most recent
iteration :) So the following is almost the same as that iteration,
except:
1. I call assoc! directly on the transient maps instead of writing my
own update-in!.
2. I use unchecked-remainder instead of rem to calculate the index of
the map to update; this shaved off 3-4 seconds.
3. The transient maps are stored in a plain old vector. Using a
transient vector didn't make things much faster for me, so I dropped
it to keep the code a bit simpler.
On my home machine, this computes the frequencies over 99844096 blocks
in about 11 seconds. My home machine is slower than the machine I
used earlier, so this version should be about twice as fast as my
earlier code.
On Nov 6, 4:23 am, "pepijn (aka fliebel)" <pepijnde...@gmail.com>
wrote:
> Awesome. You managed to reproduce my initial solution, but working
> with arrays all the way. It is already quite a bit faster than what I
> have, but still nowhere near Python.
> I'll put it on the list for things to look at.
> On Nov 6, 1:27 am, Benny Tsai <benny.t...@gmail.com> wrote:
> > Oops, sorry, got my terminology wrong. The sub-arrays represent
> > *layers*, not levels. So the code should actually read as follows:
> > On Nov 5, 4:57 pm, Benny Tsai <benny.t...@gmail.com> wrote:
> > > Here's what I have so far. The code splits blocks into 128 smaller
> > > sub-arrays, each representing a level, then calls a modified version
> > > of frequencies (using areduce instead of reduce) on each level. On my
> > > machine, with server mode on, it takes about 20 seconds to compute the
> > > frequencies for an array of 99844096 blocks. I haven't tested it on
> > > your level, because I'm too lazy to get JNBT set up, but I'm curious
> > > to see if it returns the correct result for you.
> > > On Nov 4, 3:28 pm, Pepijn de Vos <pepijnde...@gmail.com> wrote:
> > > > Hi all,
> > > > I have written a Python script to analyze Minecraft levels and render a graph. Then I did the same with Clojure. It takes Python 10 seconds to analyze a map, while it takes Clojure over a minute.
> > > > After having tried different options without any significant improvement, I am lost as to why there is such a huge difference. I wouldn't mind an extra pair of eyes/brains to look at this.
> I have written a Python script to analyze Minecraft levels and render a graph. Then I did the same with Clojure. It takes Python 10 seconds to analyze a map, while it takes Clojure over a minute.
> After having tried different options without any significant improvement, I am lost as to why there is such a huge difference. I wouldn't mind an extra pair of eyes/brains to look at this.
