Enumerating cells with new detections
Branimir Sesar
I'm adding new detections to my object catalog and LSD is doing
"Enumerating cells with new detections" part. The problem is, this part
is taking *forever* and only one python process is running. This does
not seem right. Is this a bug or a feature?
B
Mario Juric
IIRC, the PanSTARRS loader remembers which detections the object matcher
has seen, and uses that information to significantly speed up subsequent
loads (orders of magnitude difference).
We can meet on Skype over the weekend and I'll show you how to implement
something similar for PTF.
Cheers,
--
Mario Juric,
Data Mgmt. Project Scientist, Large Synoptic Survey Telescope
Web : http://www.cfa.harvard.edu/~mjuric/
Phone : +1 617 744 9003 PGP: ~mjuric/crypto/public.key
Branimir Sesar
> On 3/12/12 22:30 , Branimir Sesar wrote:
>> Hi all,
>>
>> I'm adding new detections to my object catalog and LSD is doing
>> "Enumerating cells with new detections" part. The problem is, this part
>> is taking *forever* and only one python process is running. This does
>> not seem right. Is this a bug or a feature?
Ok, I've pinned down the problem to the following "if" statement in
smf.py, routine 'get_cells_with_dets_from_exps"
try:
bb[tt] |= fov
except KeyError:
bb[tt] = fov
This block keeps killing python processes and destroys the scalable
nature of LSD that we all love :) Mario, any idea on how to replace this
block with something better?
B
Branimir Sesar
Replacing the above block with
if bb.has_key(tt):
bb[tt] |= fov
else:
bb[tt] = fov
helps to speed up things a bit, but the CPUs still work at only ~2%
(except for one which runs at 100%). Is it possible to somehow
parallelize the "bb[tt] |= fov" line?
B
Mario Juric
>
> Replacing the above block with
>
> if bb.has_key(tt):
> bb[tt] |= fov
> else:
> bb[tt] = fov
>
Branimir,
What's fovradius that gets passed to that function?
PS: Let's take the debugging offline.