Increasing the temporal pixel size from 1 day

[Branimir Sesar]

Dear all,

The large number of small files in my LSD tables is severely impacting
the performance. I've tried increasing the size of spatial pixels, but
that did not help much. Has anyone tried to increase the size of
temporal pixels? Can LSD work with temporal pixels greater than 1 day?

Cheers,

Brani

[Bertrand Goldman]

Hi Brani,

  you probably know better, but I thought that if you do not provide a
Temporal key
in your yaml file, there will be only one file per sky cell?

  Cheers,
    Bertrand.

PS: how do you actually change the size of the Healpix cells (parameter k in the wiki help)?

[Branimir Sesar]

On 11/11/2013 02:07 PM, Bertrand Goldman wrote:
> Hi Brani,
>
> you probably know better, but I thought that if you do not provide a
> Temporal key
> in your yaml file, there will be only one file per sky cell?

Hi Bertrand,

Well, a static catalog is the other extreme from the one I have right
now :) I want to tune the temporal pixel size so the average size of my
HDF5 files is ~100 MB. Right now, my files are *waaay* too small :)

> PS: how do you actually change the size of the Healpix cells (parameter

> k in the wiki help <http://mwscience.net/trac/wiki/LargeSurveyDatabase>)?

The size of Healpix pixels is preset depending on the level of
pixelization. The level of pixelization can be changed with the PIXLEVEL
environment variable. For example,

export PIXLEVEL=6

will use pixels of ~13 deg^2. Of course, this needs to be done when the
table is created.

Cheers,

Brani

[Mario Juric]

On 11/11/13 16:16 , Branimir Sesar wrote:
> On 11/11/2013 02:07 PM, Bertrand Goldman wrote:
>> Hi Brani,
>>
>> you probably know better, but I thought that if you do not provide a
>> Temporal key
>> in your yaml file, there will be only one file per sky cell?
>
> Hi Bertrand,
>
> Well, a static catalog is the other extreme from the one I have right
> now :) I want to tune the temporal pixel size so the average size of my
> HDF5 files is ~100 MB. Right now, my files are *waaay* too small :)
>

Hi Brani,
You'd have to hack the code a bit. Specifically, take a look at line
1067 in table.py:

https://github.com/mjuric/lsd/blob/master/src/lsd/table.py#L1067

where the Pixelization object gets created. You'd want to change t0 (MJD
origin) and dt (granularity in days), probably passed via an environment
variable as well.

These used to be user settable, but trying to join two databases with
different (t0, dt) leads to (silently) incorrect query results, so I
disabled it at some point. But in your case, this hack may be justified
(just don't try to join that database to others with different (t0, dt)).

>> PS: how do you actually change the size of the Healpix cells (parameter
>> k in the wiki help <http://mwscience.net/trac/wiki/LargeSurveyDatabase>)?
>
> The size of Healpix pixels is preset depending on the level of
> pixelization. The level of pixelization can be changed with the PIXLEVEL
> environment variable. For example,
>
> export PIXLEVEL=6
>
> will use pixels of ~13 deg^2. Of course, this needs to be done when the
> table is created.
>

Yup, with the same caveat that joining databases/tables with different
PIXLEVELs will not work.


Going back to your original question, though, would it be reasonable to
simply _not_ have a temporal key at all, as Bertrand suggested? In
retrospect, I think that temporal partitioning was a bad idea: most use
cases I've seen so far usually request the full light curve for every
object, rather than observations in a given time interval (which is what
temporal partitioning optimizes for).

Cheers,
--
Mario Juric,
Data Mgmt. Project Scientist, Large Synoptic Survey Telescope
Web : http://www.cfa.harvard.edu/~mjuric/
Phone : +1 617 744 9003 PGP: ~mjuric/crypto/public.key

[Branimir Sesar]

On 11/11/2013 02:35 PM, Mario Juric wrote:

> Going back to your original question, though, would it be reasonable to
> simply _not_ have a temporal key at all, as Bertrand suggested? In
> retrospect, I think that temporal partitioning was a bad idea: most use
> cases I've seen so far usually request the full light curve for every
> object, rather than observations in a given time interval (which is what
> temporal partitioning optimizes for).

I can give it a shot, but how do I disable the temporal key? I've
modified your smf.py file to ingest my data. Where do I disable the
temporal key in that file? If I do that, can I still have the *_obj table?

Cheers,

Brani

[Mario Juric]

Taking a quick look, you'd drop line 51 in your equivalent of smf.py:

https://github.com/mjuric/lsd/blob/master/src/lsd/smf.py#L51

but then you may also need to modify make_object_catalog() and
_obj_det_match to take into account there are no temporal cells any more.

Perhaps the easiest thing is to change (t0, dt), if you don't want to
spend some time modifying the matching script.

[Eddie Schlafly]

> Going back to your original question, though, would it be reasonable to
> simply _not_ have a temporal key at all, as Bertrand suggested? In
> retrospect, I think that temporal partitioning was a bad idea: most use
> cases I've seen so far usually request the full light curve for every
> object, rather than observations in a given time interval (which is what
> temporal partitioning optimizes for).

For what it's worth, I'm not sure; I at least often do queries of
particular exposures, as well as queries of whole light curves. And
certainly in PS1, that would lead to seriously different MD and 3pi
file sizes---though of course at some level we end up dealing with
this no matter what.