Weave & LSD

[Branimir Sesar]

Dear LSD users,

I'm hoping there's someone among you who has experience using Weave in
Python and LSD.

I've been hacking Mario's "ps1-compute-averages" code to calculate all
sorts of statistics on some light curves. Here is the part where I
define the columns:

out = colgroup.ColGroup(
dtype=[
('obj_id', 'u8'),
('percentiles', '(4,12)f4'),
('periodSearch', '(4,2)f4'),
('matched_to_SDSS', 'u2'),
],
size=len(objs)
)

The part where I'm getting errors is in the "inline" part of my Python
code, more precisely, the assignment given below

PERCENTILES2(row, 0) = s.percentiles;

where "s" is a struct that contains a "percentiles" array of 12
elements. The error I'm getting is

"error: cannot convert �double [12]� to �float� in assignment"

The above assignment works fine if I give a double or a float on the
right hand side. Any ideas on how to fix this problem?

Cheers,

Brani

[Eddie Schlafly]

Forgive the simple-minded response, but it looks like it thinks
s.percentiles is an array while percentiles2(row, 0) is not. I'd
iterate over s.percentiles[i] and do the assignment to percentiles2
element by element.

On Tue, Sep 11, 2012 at 2:40 AM, Branimir Sesar
<bse...@astro.caltech.edu> wrote:
> Dear LSD users,
>
> I'm hoping there's someone among you who has experience using Weave in
> Python and LSD.
>
> I've been hacking Mario's "ps1-compute-averages" code to calculate all sorts
> of statistics on some light curves. Here is the part where I define the
> columns:
>
> out = colgroup.ColGroup(
> dtype=[
> ('obj_id', 'u8'),
> ('percentiles', '(4,12)f4'),
> ('periodSearch', '(4,2)f4'),
> ('matched_to_SDSS', 'u2'),
> ],
> size=len(objs)
> )
>
> The part where I'm getting errors is in the "inline" part of my Python code,
> more precisely, the assignment given below
>
> PERCENTILES2(row, 0) = s.percentiles;
>
> where "s" is a struct that contains a "percentiles" array of 12 elements.
> The error I'm getting is
>

> "error: cannot convert ‘double [12]’ to ‘float’ in assignment"

[Branimir Sesar]

Hi all,

The solution to the problem below is to do the following assignment in
the "inline" (Weave) part of the code:

for(int k = 0; k < 12; k++)
{
PERCENTILES2(row, (0, k)) = s.percentiles[k];
}

The (0, k) part was not obvious and did require some tries and failures :)

Cheers,

Brani

[Branimir Sesar]

Actually, the correct assignment is

for(int k = 0; k < 12; k++)
{

PERCENTILES3(row, 0, k) = s.percentiles[k];
}

B

[Bertrand Goldman]

Hello LSD experts,

running a LSD query in python:

for row in dbdir.query(Parameters + ' FROM
tpijun12(matchedto=hya16k,nmax=1,dmax=1.0), hya16k').iterate():

I'm getting a message which I do not understand but seems to only be a
warning:

No handlers could be found for logger "lsd.pool2"



Independently, that division is not protected against a null value of
self.len:

File "/a41217d5/LSD/stable/lib/python2.7/site-packages/lsd/pool2.py",
line 196, in progress_pct_nnl
pct = 100. * self.at / self.len

I'm not sure what causes that error to happen, but I believe it
happens when the bounds are wrong and there are zero element to search.

Thanks,
Bertrand.


Error message:

Traceback (most recent call last):
File "./astrom-refs.py", line 34, in <module>
for row in dbdir.query(TgParHya + ',' + TgParPM + ' FROM
tpijun12(matchedto=hya16k,nmax=1,dmax=1.0), hya16k').iterate():
File
"/a41217d5/LSD/stable/lib/python2.7/site-packages/lsd/join_ops.py", line
1570, in iterate
_yield_empty=_yield_empty):
File
"/a41217d5/LSD/stable/lib/python2.7/site-packages/lsd/join_ops.py", line
1512, in execute
for result in pool.map_reduce_chain(partspecs.items(), kernels,
progress_callback=progress_callback):
File "/a41217d5/LSD/stable/lib/python2.7/site-packages/lsd/pool2.py",
line 591, in map_reduce_chain
send(obj)
IOError: [Errno 32] Broken pipe
for r in self.imap_unordered(input, K_fun, K_args,
progress_callback=progress_callback, progress_callback_stage=stage):
File "/a41217d5/LSD/stable/lib/python2.7/site-packages/lsd/pool2.py",
line 398, in imap_unordered
progress_callback(progress_callback_stage, 'step', input, k, None)
File "/a41217d5/LSD/stable/lib/python2.7/site-packages/lsd/pool2.py",
line 166, in progress_default
self.dispatch(stage, step, input, index, result)
File "/a41217d5/LSD/stable/lib/python2.7/site-packages/lsd/pool2.py",
line 169, in progress_pct
running = progress_pct_nnl(stage, step, input, index, result)
File "/a41217d5/LSD/stable/lib/python2.7/site-packages/lsd/pool2.py",
line 196, in progress_pct_nnl
pct = 100. * self.at / self.len
ZeroDivisionError: float division by zero

[Eddie Schlafly]

> No handlers could be found for logger "lsd.pool2"

Yeah, this is just a warning, I get it too.

> I'm not sure what causes that error to happen, but I believe it happens
> when the bounds are wrong and there are zero element to search.

Yup, it's figuring out the fraction complete by dividing by the total
amount of work to do, and failing when that's zero. Good catch.

Thanks,

Eddie Schlafly

[Mario Juric]

On 9/20/12 8:04 , Bertrand Goldman wrote:
>
> Independently, that division is not protected against a null value of
> self.len:
>
> File "/a41217d5/LSD/stable/lib/python2.7/site-packages/lsd/pool2.py",
> line 196, in progress_pct_nnl
> pct = 100. * self.at / self.len
>

Hi Bertrand,
Yep, you and Eddie are right, this should be protected. Could you try
replacing the line above with:

pct = 100. * self.at / self.len if self.len else 100.

and see if that fixes it?

If it does, I'll commit the fix to the maintenance branch.

Cheers,
--
Mario Juric,
Data Mgmt. Project Scientist, Large Synoptic Survey Telescope
Web : http://www.cfa.harvard.edu/~mjuric/
Phone : +1 617 744 9003 PGP: ~mjuric/crypto/public.key