Re: [opendap-tech] [fits_handler 1.0.7] fits_handlerTest: 1 3 failed

[Nathan Potter]

Gareth,

I'll have a look. But I need all of the information, OS, compiler, what branch or release of our code you're working from.

Right away I can see some things in the log that I may be able to correct.


Nathan



On Mar 13, 2013, at 7:41 AM, Dave Fulker wrote:

> On Wed, Mar 13, 2013 at 12:17 AM, Gareth Williams <wil...@cherax.hpsc.csiro.au> wrote:
> Hi James (?),
> I hope/think the attached log will be self-explanatory. Any ideas?
>
> Gareth, James will be largely unreachable until the 18th or 19th, but I'm hoping Dan, Nathan or someone else on OPeNDAP Tech can provide useful feedback. I personally am unable to do so; sorry! -- Dave

= = =
Nathan Potter ndp at opendap.org
OPeNDAP, Inc. +1.541.231.3317

[Gareth....@csiro.au]

Thanks Nathan,

I think most of the info is tucked away in the log - nice work there setting that up.

The OS is SLES11
Compiler is gcc
wil240@cherax:~/hyrax/1.8.8/fits_handler-1.0.7> gcc --version
gcc (SUSE Linux) 4.3.4 [gcc-4_3-branch revision 152973]

It is a fresh build of 1.8.8 with a newly compiled cfitsio 3.330

Fits support is not critical for us currently but seems a good idea given the strong astronomy community within CSIRO. I've just built most of the other components first with a few errors on make check - at least a few related to exceeding quota in /tmp

I plan to try out the service today. Is there a nice way of trying it out without putting it in production? I'm figuring on just putting it in production and rolling back if there are problems...

Gareth

[Nathan Potter]

On Mar 13, 2013, at 3:35 PM, <Gareth....@csiro.au> <Gareth....@csiro.au> wrote:

> Thanks Nathan,
>
> I think most of the info is tucked away in the log - nice work there setting that up.
>
> The OS is SLES11
> Compiler is gcc
> wil240@cherax:~/hyrax/1.8.8/fits_handler-1.0.7> gcc --version
> gcc (SUSE Linux) 4.3.4 [gcc-4_3-branch revision 152973]
>
> It is a fresh build of 1.8.8 with a newly compiled cfitsio 3.330

Hmmmm... We built and tested the fits_handler-1.0.7 against cfitsio 3.270 - I wonder if the newer version your using is part of the issue with the failing tests.

>
> Fits support is not critical for us currently but seems a good idea given the strong astronomy community within CSIRO. I've just built most of the other components first with a few errors on make check - at least a few related to exceeding quota in /tmp

You can change where the BES caches things in the bes.conf file.

BES.CacheDir=/tmp
BES.CachePrefix=bes_cache
BES.CacheSize=500


Where cache size is in megabytes.

>
> I plan to try out the service today. Is there a nice way of trying it out without putting it in production? I'm figuring on just putting it in production and rolling back if there are problems...

To disable the fits_handler simply rename $prefix/etc/bes/modules/fits.conf to $prefix/etc/bes/modules/fits.conf.OFF

I'll let you know when I know more.


Nathan

[Gareth....@csiro.au]

Hi Nathan and all,

-snip-

> Hmmmm... We built and tested the fits_handler-1.0.7 against cfitsio 3.270 - I wonder if the newer version your using is part of the issue with the failing tests.

After sorting out other issues I tried again with cfitsio 3.340 and 3.270 both give the same failed tests.

I also realized the fits_handler is needed for the gdal_handler tests, so I'm quite keen to make it work. Can you tell more from the log output provided in the initial email?

-snip-

> You can change where the BES caches things in the bes.conf file.

> BES.CacheDir=/tmp

Thanks, I've started altering bes-testsuite/bes.conf before running 'make check'

-- Gareth

-snip-

[Gallagher James]

On Jun 5, 2013, at 8:45 PM, <Gareth....@csiro.au> wrote:

> Hi Nathan and all,
>
> -snip-
>> Hmmmm... We built and tested the fits_handler-1.0.7 against cfitsio 3.270 - I wonder if the newer version your using is part of the issue with the failing tests.
>
> After sorting out other issues I tried again with cfitsio 3.340 and 3.270 both give the same failed tests.

Our nightly builds are running on the trunk only at this point, but you can see them here: http://test.opendap.org/cgi-bin/build_reader.pl?show=current&sort=yes

The FITS handler is building and passing its tests on both OSX and CentOS 6. Those builds are using cfits 3270.

I will add a build of the Hyrax 1.8 branch to the nightly build set.

>
> I also realized the fits_handler is needed for the gdal_handler tests, so I'm quite keen to make it work.

Hmmm. I must be missing something. Can you point me toward the tests that depend on the fits handler?

> Can you tell more from the log output provided in the initial email?
>
> -snip-
>
>> You can change where the BES caches things in the bes.conf file.
>
>> BES.CacheDir=/tmp
>
> Thanks, I've started altering bes-testsuite/bes.conf before running 'make check'
>
> -- Gareth
>
> -snip-
>
>>>> On Wed, Mar 13, 2013 at 12:17 AM, Gareth Williams
>>> <wil...@cherax.hpsc.csiro.au> wrote:
>>>> Hi James (?),
>>>> I hope/think the attached log will be self-explanatory. Any ideas?

--
James Gallagher
jgallagher at opendap.org
406.723.8663

[Gareth....@csiro.au]

Hi James,

> -----Original Message-----
> From: Gallagher James [mailto:jgall...@opendap.org]
> Sent: Friday, 7 June 2013 1:52 AM
> To: Williams, Gareth (CSIRO IM&T, Docklands)
> Cc: Gallagher James; Potter Nathan; OPeNDAP Support; Patrick West
> Subject: Re: [support] RE: [opendap-tech] [fits_handler 1.0.7]
> fits_handlerTest: 1 3 failed
>
>

> On Jun 5, 2013, at 8:45 PM, <Gareth....@csiro.au> wrote:
>
> > Hi Nathan and all,
> >
> > -snip-
> >> Hmmmm... We built and tested the fits_handler-1.0.7 against cfitsio
> 3.270 - I wonder if the newer version your using is part of the issue
> with the failing tests.
> >
> > After sorting out other issues I tried again with cfitsio 3.340 and
> 3.270 both give the same failed tests.
>
> Our nightly builds are running on the trunk only at this point, but you
> can see them here: http://test.opendap.org/cgi-
> bin/build_reader.pl?show=current&sort=yes
>
> The FITS handler is building and passing its tests on both OSX and
> CentOS 6. Those builds are using cfits 3270.
>
> I will add a build of the Hyrax 1.8 branch to the nightly build set.

Maybe there is something wrong with all our cfitsio installs. It doesn't seem to come with a test suite. :-( I might have to pull out a debugger... I can't see anything wrong with our setup and at least this handler and cfitsio has few dependencies. Any further suggestions would be welcome.

> >
> > I also realized the fits_handler is needed for the gdal_handler
> tests, so I'm quite keen to make it work.
>
> Hmmm. I must be missing something. Can you point me toward the tests
> that depend on the fits handler?

In the test log output I get:

## ---------------------- ##
## Detailed failed tests. ##
## ---------------------- ##

# -*- compilation -*-
1. testsuite.at:67: testing ...
./testsuite.at:67: besstandalone -c $abs_builddir/bes.conf -i $abs_srcdir/gdal/cea.tif.0.bescmd || true
stderr:
Caught plugin exception during initialization of gd module:
libcfitsio.so: cannot open shared object file: No such file or directory
stdout:


I see the gdal problem - libgdal.so depends on libcfitsio.so which is not found and we've not got the gdal build quite right (we use LD_RUN_PATH and rpath). The tests run OK with:
env LD_LIBRARY_PATH=/apps/cfitsio/3.270-gnu/lib make check

So we should fix gdal to either not need cfitsio or to get the dependency right.

Eek. gdal has so many optional dependencies and I guess the person who installed gdal here for me, included cfitsio because they could... sensible enough. They also explicitly included pre-built netcdf, hdf4 and 5, proj4 and jasper, and --with-jpeg=internal --with-gif=internal --with-curl=/usr/bin/curl-config --with-xml2=/usr/bin/xml2-config --with-png=internal --with-expat --with-sqlite2. Would it be straightforward to list the required ones in the gdal-handler README or INSTALL (and/or other recommendations for a minimal gdal build)? I see the 'build a self-contained embedded gdal option' and could also try that. I guess I could even dig in and see what that internal build looks like...

Anyway, it is a long weekend here and I'll let this alone until Tuesday at least.


Gareth

-snip-

[Gallagher James]

I think it's your Tuesday AM. Hope the weekend went well.

As for the embedded gdal option, I took that out since it never worked very well.

Minimal GDAL would be the default gdal library. The GDAL handler is used to return GeoTIFF and JP2000 files but the JP2000 feature never worked the I'd like (i.e., I got output but could never test it, and I suspect it's not always correct). The GeoTIFF files looked good. Fortunately, GDAL contains built in GeoTIFF support.

If you build libgal without support for shared libs, the DAP code will link with the static lib and thus you can avoid the case where the run-time linking loader finds some other copy of the library. In that case, you can build and install the library in a special place (like a subdir in hyrax's root).

HTH

James

>
>
> Gareth
>
> -snip-

[Gareth....@csiro.au]

OK. Well gdal was rebuilt for me and I made the gdal_handler work. I still had to stuff about with rpath options for cfitsio which was handled in eth build setup differently to other gdal dependencies (like jasper).

There was no test data installed as part of 'make install' so I put some from the data directory here:
http://opendap.csiro.au/opendap/test-data/gdal/ and it seems all OK.

> Minimal GDAL would be the default gdal library. The GDAL handler is
> used to return GeoTIFF and JP2000 files but the JP2000 feature never
> worked the I'd like (i.e., I got output but could never test it, and I
> suspect it's not always correct). The GeoTIFF files looked good.
> Fortunately, GDAL contains built in GeoTIFF support.
>
> If you build libgal without support for shared libs, the DAP code will
> link with the static lib and thus you can avoid the case where the run-
> time linking loader finds some other copy of the library. In that case,
> you can build and install the library in a special place (like a subdir
> in hyrax's root).

That sounds OK. Since I have something working I'll let it be for the moment.

That leaves the 'original' fits_handler problem. I'm not planning to act further on that unless I have suggestions from you _or_ actual client demand arises, but it would be good to have it working to take to our astronomers and let them assess the value with something to actually play with!

Thanks!

Gareth

[Gallagher James]

On Jun 10, 2013, at 10:07 PM, <Gareth....@csiro.au> <Gareth....@csiro.au> wrote:

snip

>
> That leaves the 'original' fits_handler problem. I'm not planning to act further on that unless I have suggestions from you _or_ actual client demand arises, but it would be good to have it working to take to our astronomers and let them assess the value with something to actually play with!

Can you remind me what the original FITS problem was?

[Gareth....@csiro.au]

> -----Original Message-----
> From: Gallagher James [mailto:jgall...@opendap.org]
> Sent: Wednesday, 12 June 2013 3:33 AM
> To: Williams, Gareth (CSIRO IM&T, Docklands)
> Cc: Gallagher James; n...@opendap.org; sup...@opendap.org;
> pw...@opendap.org
> Subject: Re: [support] [opendap-tech] [fits_handler 1.0.7]
> fits_handlerTest: 1 3 failed
>
>

> On Jun 10, 2013, at 10:07 PM, <Gareth....@csiro.au>
> <Gareth....@csiro.au> wrote:
>
> snip
>
> >
> > That leaves the 'original' fits_handler problem. I'm not planning to
> act further on that unless I have suggestions from you _or_ actual
> client demand arises, but it would be good to have it working to take
> to our astronomers and let them assess the value with something to
> actually play with!
>
> Can you remind me what the original FITS problem was?
>

gzipped fits_handlerTest.log attached

Gareth

[Gallagher James]

On Jun 12, 2013, at 1:01 AM, <Gareth....@csiro.au> <Gareth....@csiro.au> wrote:

>> -----Original Message-----
>> From: Gallagher James [mailto:jgall...@opendap.org]
>> Sent: Wednesday, 12 June 2013 3:33 AM
>> To: Williams, Gareth (CSIRO IM&T, Docklands)
>> Cc: Gallagher James; n...@opendap.org; sup...@opendap.org;
>> pw...@opendap.org
>> Subject: Re: [support] [opendap-tech] [fits_handler 1.0.7]
>> fits_handlerTest: 1 3 failed
>>
>>
>> On Jun 10, 2013, at 10:07 PM, <Gareth....@csiro.au>
>> <Gareth....@csiro.au> wrote:
>>
>> snip
>>
>>>
>>> That leaves the 'original' fits_handler problem. I'm not planning to
>> act further on that unless I have suggestions from you _or_ actual
>> client demand arises, but it would be good to have it working to take
>> to our astronomers and let them assess the value with something to
>> actually play with!
>>
>> Can you remind me what the original FITS problem was?
>>
>
> gzipped fits_handlerTest.log attached

Gareth, Patrick,

Attribute processing was broken. I have a fix. I'll make this on the trunk and then merge done tot he hyrax 1.8 branch and get a new tar.gz file up.

James

>
> Gareth
>
>>>
>>> Thanks!
>>>
>>> Gareth
>>>
>>>>
>>>> HTH
>>>>
>>>> James
>>>>
>>>>>
>>>>>
>>>>> Gareth
>>>>>
>>>>> -snip-
>>>>
>>>> --
>>>> James Gallagher
>>>> jgallagher at opendap.org
>>>> 406.723.8663
>>>
>>
>> --
>> James Gallagher
>> jgallagher at opendap.org
>> 406.723.8663
>

> <fits_handlerTest.log.gz>

[Gareth....@csiro.au]

> -----Original Message-----
> From: Gallagher James [mailto:jgall...@opendap.org]

> Sent: Thursday, 13 June 2013 3:19 AM
> To: Williams, Gareth (CSIRO IM&T, Docklands); Patrick West
> Cc: Gallagher James; Potter Nathan; OPeNDAP Support

Great. I grabbed the new package from the download site. It passes its own tests and I've installed it serving test data at: http://opendap.csiro.au/opendap/test-data/fits/

However something fairly nasty happens when I access the third test data set.

*** glibc detected *** /apps/hyrax/1.8.8/bin/beslistener: malloc(): memory corruption: 0x00000000008a5390 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x75218)[0x7fbb79159218]
/lib64/libc.so.6(+0x781ff)[0x7fbb7915c1ff]
/lib64/libc.so.6(__libc_malloc+0x77)[0x7fbb7915e2d7]
/usr/lib64/libstdc++.so.6(_Znwm+0x1d)[0x7fbb79974e4d]
/usr/lib64/libstdc++.so.6(_ZNSs4_Rep9_S_createEmmRKSaIcE+0x21)[0x7fbb79952dc1]
/usr/lib64/libstdc++.so.6(+0xa4795)[0x7fbb79953795]
/usr/lib64/libstdc++.so.6(_ZNSsC1EPKcRKSaIcE+0x43)[0x7fbb799538d3]
/apps/hyrax/1.8.8/lib/libdap.so.11(_ZN6libdap6Vector3varERKSsbPSt5stackIPNS_8BaseTypeESt5dequeIS5_SaIS5_EEE+0x64)[0x7fbb785a1794]
/apps/hyrax/1.8.8/lib/libdap.so.11(_ZN6libdap6Vector9set_valueEPii+0x61)[0x7fbb785a03c1]
/apps/hyrax/1.8.8/lib/bes/libfits_module.so(_ZN12fits_handler17process_hdu_imageEP8fitsfileRN6libdap3DDSE+0xac5)[0x7fbb6f748205]
/apps/hyrax/1.8.8/lib/bes/libfits_module.so(_ZN12fits_handler21fits_read_descriptorsERN6libdap3DDSERKSsRSs+0x2d7)[0x7fbb6f749167]
/apps/hyrax/1.8.8/lib/bes/libfits_module.so(_ZN18FitsRequestHandler14fits_build_ddsER23BESDataHandlerInterface+0xd7)[0x7fbb6f74a4e7]
/apps/hyrax/1.8.8/lib/libbes_dispatch.so.8(_ZN21BESRequestHandlerList15execute_currentER23BESDataHandlerInterface+0xee)[0x7fbb7a7b580e]
/apps/hyrax/1.8.8/lib/libbes_dispatch.so.8(_ZN21BESRequestHandlerList12execute_eachER23BESDataHandlerInterface+0x3c)[0x7fbb7a7b4dfc]
/apps/hyrax/1.8.8/lib/bes/libusage_module.so(_ZN23BESUsageResponseHandler7executeER23BESDataHandlerInterface+0x143)[0x7fbb75b07a63]
/apps/hyrax/1.8.8/lib/libbes_dispatch.so.8(_ZN12BESInterface25execute_data_request_planEv+0x6c)[0x7fbb7a77415c]
/apps/hyrax/1.8.8/lib/libbes_dispatch.so.8(_ZN17BESBasicInterface25execute_data_request_planEv+0x301)[0x7fbb7a77cd31]
/apps/hyrax/1.8.8/lib/libbes_xml_command.so.1(_ZN15BESXMLInterface25execute_data_request_planEv+0x42)[0x7fbb7aa26362]
/apps/hyrax/1.8.8/lib/libbes_dispatch.so.8(_ZN12BESInterface15execute_requestERKSs+0x307)[0x7fbb7a7749a7]
/apps/hyrax/1.8.8/bin/beslistener[0x4053c0]
/apps/hyrax/1.8.8/bin/beslistener[0x406ccf]
/apps/hyrax/1.8.8/lib/libbes_ppt.so.4(_ZN9PPTServer14initConnectionEv+0x5a)[0x7fbb7ac6290a]
/apps/hyrax/1.8.8/bin/beslistener[0x409685]
/apps/hyrax/1.8.8/lib/libbes_dispatch.so.8(_ZN10BESBaseApp4mainEiPPc+0x7a)[0x7fbb7a7cab8a]
/apps/hyrax/1.8.8/bin/beslistener[0x408bb2]
/lib64/libc.so.6(__libc_start_main+0xe6)[0x7fbb79102bc6]
/apps/hyrax/1.8.8/bin/beslistener[0x404359]

Gareth

[Gallagher James]

That's bad.

I'll see what I can find out with what remains of today...

[Gallagher James]

On Jun 12, 2013, at 9:13 PM, <Gareth....@csiro.au> <Gareth....@csiro.au> wrote:

>> -----Original Message-----
>> From: Gallagher James [mailto:jgall...@opendap.org]
>> Sent: Thursday, 13 June 2013 9:35 AM
>> To: Williams, Gareth (CSIRO IM&T, Docklands)
>> Cc: Gallagher James
>> Subject: Re: [support] [opendap-tech] [fits_handler 1.0.7]
>> fits_handlerTest: 1 3 failed
>>
>>

>> On Jun 12, 2013, at 5:14 PM, <Gareth....@csiro.au> wrote:
>>
>>> Info response on third link
>>
>> Hmmm... I'll have to look at that tomorrow.
>
> Info response on the 4th link gave this which is probably not helpful:
>
> beslistener: malloc.c:3090: sYSMALLOc: Assertion `(old_top == (((mbinptr) (((char *) &((av)->bins[((1) - 1) * 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) || ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed.
>
> I'm getting nothing in the browser, just this stderr on the console which started the service. I've now looked at logs and there are no errors or other obvious indication of failure there.

True, but I see this now. The tests run with make check don't use all of the data files - I think the problem is in the code that builds a DDS. Should not be too big a deal… (famous last words)

>
> Gareth
>
>
>
>>
>>
>>> ________________________________________
>>> From: Gallagher James [jgall...@opendap.org]
>>> Sent: Thursday, 13 June 2013 9:01 AM

>>> To: Williams, Gareth (CSIRO IM&T, Docklands)
>>> Cc: Gallagher James

>>> Linux?
>>>
>>> BTW, I'm using fits 3270.
>>>
>>> Running the tests with valgrind I'm not seeing anything; what was the
>> query you posed to to the server that got this 'result?'
>>>
>>> James

[Gallagher James]

On Jun 12, 2013, at 9:13 PM, <Gareth....@csiro.au> <Gareth....@csiro.au> wrote:

>> -----Original Message-----
>> From: Gallagher James [mailto:jgall...@opendap.org]
>> Sent: Thursday, 13 June 2013 9:35 AM
>> To: Williams, Gareth (CSIRO IM&T, Docklands)
>> Cc: Gallagher James
>> Subject: Re: [support] [opendap-tech] [fits_handler 1.0.7]
>> fits_handlerTest: 1 3 failed
>>
>>

>> On Jun 12, 2013, at 5:14 PM, <Gareth....@csiro.au> wrote:
>>
>>> Info response on third link
>>
>> Hmmm... I'll have to look at that tomorrow.
>
> Info response on the 4th link gave this which is probably not helpful:
>
> beslistener: malloc.c:3090: sYSMALLOc: Assertion `(old_top == (((mbinptr) (((char *) &((av)->bins[((1) - 1) * 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) || ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed.
>
> I'm getting nothing in the browser, just this stderr on the console which started the service. I've now looked at logs and there are no errors or other obvious indication of failure there.

What about the DAS and DDS links for those? I can see that the info link does weird things, but it seems like the DAS and DDS are working. Backstory: the info response is built by first getting the DAS and DDS and then processing it. This could actually be replaced with XSL processing the DDX response, but that's for another day.

Along these same lines, does the html form work for these on your machine?


>
> Gareth
>
>
>
>>
>>
>>> ________________________________________
>>> From: Gallagher James [jgall...@opendap.org]
>>> Sent: Thursday, 13 June 2013 9:01 AM

>>> To: Williams, Gareth (CSIRO IM&T, Docklands)
>>> Cc: Gallagher James

>>> Linux?
>>>
>>> BTW, I'm using fits 3270.
>>>
>>> Running the tests with valgrind I'm not seeing anything; what was the
>> query you posed to to the server that got this 'result?'
>>>
>>> James
>>>>

>>>> *** glibc detected *** /apps/hyrax/1.8.8/bin/beslistener: malloc():
>>>> memory corruption: 0x00000000008a5390 *** ======= Backtrace:
>>>> ========= /lib64/libc.so.6(+0x75218)[0x7fbb79159218]
>>>> /lib64/libc.so.6(+0x781ff)[0x7fbb7915c1ff]
>>>> /lib64/libc.so.6(__libc_malloc+0x77)[0x7fbb7915e2d7]
>>>> /usr/lib64/libstdc++.so.6(_Znwm+0x1d)[0x7fbb79974e4d]
>>>>

>>>> _requestERKSs+0x307)[0x7fbb7a7749a7]
>>>> /apps/hyrax/1.8.8/bin/beslistener[0x4053c0]
>>>> /apps/hyrax/1.8.8/bin/beslistener[0x406ccf]
>>>>

>> /apps/hyrax/1.8.8/lib/libbes_ppt.so.4(_ZN9PPTServer14initConnectionEv
>>>> +0x5a)[0x7fbb7ac6290a] /apps/hyrax/1.8.8/bin/beslistener[0x409685]
>>>>
>> /apps/hyrax/1.8.8/lib/libbes_dispatch.so.8(_ZN10BESBaseApp4mainEiPPc+

[Patrick West]

Did you get a chance to create the new .tar.gz file yet? I need fits handler for VSTO data access to MLSO data.

Thanks,

Patrick

[Gallagher James]

On Jun 14, 2013, at 11:23 AM, Patrick West wrote:

> Did you get a chance to create the new .tar.gz file yet? I need fits handler for VSTO data access to MLSO data.

I'm close. It took me quite a while to figure it out and then I had to decide to keep the other changes I made in the course of sorting out what was wrong and so on. (short version: memory corruption in cfitsio; possible use of the wrong type on the part of the handler).

more in a bit...