Pynobody filt.Spheres are very slow all of a sudden?

[Nolan McPhaul]

Hello,

My name is Nolan McPhaul and I am currently using pynbody to look at the dark matter and gas particles in the simulation I have running on my school's supercomputer.

The main portion of my code uses pynbody filter spheres to find the nearest gas particle to each dark matter particle for my analysis. I have been using this code now for a couple of months and running through an entire halo of 150,000 dark matter particles plus other analysis in a little under 10 minutes.

However, all of a sudden I ran the same code again and noticed it was taking a lot longer to compute. I ran a couple of tests and found out that the creation of the spheres was causing it to take considerably longer. I used the following code to test the sphere time rates:

s = pynbody.load("/data1/romulus/cosmo25/cosmo25p.768sg1bwK1BHe75.007779")

h = s.halos()

h5 = h.load_copy(5)

h5.physical_units()

i = 0

while i < len(h5.dm):

    ...:     radius = "0.3 kpc"

    ...:     center = np.reshape(h5.dm[i]['pos'], 3)

    ...:     sphere = h5[pynbody.filt.Sphere(radius, center)]

    ...:     i = i + 1

This is only a minor portion of the code I'm using from my analysis that collectively used to only take 10 minutes. Running this code took too long, so I found that it's roughly computing results for 25 particles per second which gives a time of around 1.7 hours for the full halo. This is incredibly longer than what it previously was (and I've left out even more analysis which would make it longer).

I commented out the sphere portion from the previous code and it then only took around 2 minutes for all 150,000 dm particles. So simply adding in the spheres makes it take around 40 times as long computationally.

Also I tested the computation speed of other pynbody functions (particle data, computations, etc.) and they all seem to be working fine and at proper speed for me which makes me wonder if this problem is on my end.

If anyone has any advice (or even just if this is my fault or something with pynbody itself) that'd be great!

Thanks,

Nolan McPhaul

[Andrew Pontzen]

Hi Nolan,

I can't really say why it would have changed speed, sad to say, but the bottom line is what you're doing is expected to be slow. Partly that's because you're actually constructing effectively a new snapshot every time you do h5.dm[i]['pos']. You'll probably find it faster if you do h5.dm['pos'][i]. But perhaps more to the point, looping over all particles within a python loop will always be slow, whatever way you try to do it. I can't quite figure out what you're trying to do, but my guess would be there's almost certainly a much more efficient way to do it avoiding this loop. If you explain what your goal is, I might be able to suggest something.

Best wishes

Andrew Pontzen

--
You received this message because you are subscribed to the Google Groups "pynbody-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pynbody-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pynbody-users/93e9c4c5-bf8f-42f9-9c46-f85f44bd94a0n%40googlegroups.com.