NetCDF and HDF5 support in iRODS 4

[Tony Edgin]

Does iRODS 4.1.3 come standard with NetCDF and HDF5 support these days?

[Jason Coposky]

netcdf and hdf support are packaged separately.  they have not migrated to CI yet but we can provide an engineering preview if it is time critical. 

— 

Jason Coposky

Chief Technologist, iRODS Consortium

RENCI at the University of North Carolina at Chapel Hill

(919) 445-9675

jas...@renci.org

irods.org — Take Control of Your Data

On Jul 21, 2015, at 8:26 PM, Tony Edgin <ted...@iplantcollaborative.org> wrote:

Does iRODS 4.1.3 come standard with NetCDF and HDF5 support these days?

--
--
"iRODS: the Integrated Rule-Oriented Data-management System; A community driven, open source, data grid software solution" https://www.irods.org
 
iROD-Chat: http://groups.google.com/group/iROD-Chat

---
You received this message because you are subscribed to the Google Groups "iRODS-Chat" group.
To unsubscribe from this group and stop receiving emails from it, send an email to irod-chat+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Nirav Merchant]

Hi Jason,

That will be good to have, and key for our migration. 

We make use of both file formats (more netCDF) and the specific icommands:

https://wiki.irods.org/index.php/inc

https://wiki.irods.org/index.php/incattr

There are microservices that go with it as well

https://wiki.irods.org/index.php/NETCDF

Regards,

Nirav

[Tony Edgin]

Jason, when do you guess the engineering preview of the netcdf and hdf support will be ready?

[Jason Coposky]

we will put something together asap if you are ready to put it through its paces. 

— 

Jason Coposky

Chief Technologist, iRODS Consortium

RENCI at the University of North Carolina at Chapel Hill

(919) 445-9675

jas...@renci.org

irods.org — Take Control of Your Data

[Tony Edgin]

We are ready!

[tom.la...@gmail.com]

Hi

I can test this also.

Please send it to me also.

regards

tom

[Jason Coposky]

tom & tony,

I have brought the netcdf code up to date with the latest 4-1-stable development libraries.  

If you need an engineering build we can provide one, otherwise please file any issues you

discover at the project repository and we will get them fixed.

https://github.com/irods/irods_netcdf/issues/

thanks!

—

Jason Coposky
Chief Technologist, iRODS Consortium
RENCI at the University of North Carolina at Chapel Hill
(919)445-9675
jas...@renci.org

irods.org — Take Control of Your Data

---

From: tom.la...@gmail.com <tom.la...@gmail.com>
Sent: Thursday, August 13, 2015 9:40 AM
To: iRODS-Chat
Cc: Jason Coposky
Subject: Re: [iROD-Chat:14157] NetCDF and HDF5 support in iRODS 4

 

[Tom Langborg]

Thanks

I am looking forward to test this for my climate people.

I got a problem with ubuntu14.

I installed irods-dev with ftp://ftp.renci.org/pub/irods/releases/4.1.4/centos6/irods-dev-4.1.4-centos6-x86_64.rpm

when i try to compile netcdf it's still complain that irods-dev packet is missing.

regards

tom

You received this message because you are subscribed to a topic in the Google Groups "iRODS-Chat" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/irod-chat/umR3zqYK-3E/unsubscribe.
To unsubscribe from this group and all its topics, send an email to irod-chat+...@googlegroups.com.

[Jason Coposky]

That should be fixed, please pull.

I ​missed a couple build script updates, the perils of directly running 'make'.

thanks,

—
Jason Coposky
Chief Technologist, iRODS Consortium
RENCI at the University of North Carolina at Chapel Hill
(919)445-9675
jas...@renci.org

irods.org — Take Control of Your Data

---

From: irod...@googlegroups.com <irod...@googlegroups.com> on behalf of Tom Langborg <tom.la...@gmail.com>
Sent: Wednesday, September 2, 2015 7:59 AM
To: irod...@googlegroups.com
Subject: Re: [iROD-Chat:14237] NetCDF and HDF5 support in iRODS 4

 

[Tom Langborg]

Thanks

Now I got an other problem.

As you understand I am outside of my comfort zone now.

But when I try to compile. I got info that curl/curl.h are missing.

its a ref in ../include/ooiCi.hpp for curl.h

My command is ./api/packing/build.sh 

regards

tom

[Jason Coposky]

​tom,

you will need to install the curl development package for  your flavor of linux.

thanks,

—
Jason Coposky
Chief Technologist, iRODS Consortium
RENCI at the University of North Carolina at Chapel Hill
(919)445-9675
jas...@renci.org

irods.org — Take Control of Your Data

From: irod...@googlegroups.com <irod...@googlegroups.com> on behalf of Tom Langborg <tom.la...@gmail.com>

Sent: Wednesday, September 2, 2015 10:21 AM
To: irod...@googlegroups.com
Subject: Re: [iROD-Chat:14240] NetCDF and HDF5 support in iRODS 4

 

[Tom Langborg]

Thanks 

I found a dev packet for curl. Sorry about this simple question.

Now I got an other error instead.

Multiple definition of xx.

Compile plugin librsNcOpen_server.so...

libirods_netcdf_server.a(readAggInfo.o):(.data.rel.local+0x0): multiple definition of `packTypeTable'

/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../lib/libirods_client.a(rcConnect.o):(.data.rel.local+0x840): first defined here

libirods_netcdf_server.a(readAggInfo.o):(.data+0x0): multiple definition of `NumOfPackTypes'

/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../lib/libirods_client.a(rcConnect.o):(.data+0x10): first defined here

libirods_netcdf_server.a(readAggInfo.o):(.data.rel.local+0xa0): multiple definition of `PackConstantTable'

/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../lib/libirods_client.a(rcConnect.o):(.data.rel.local+0x6e0): first defined here

libirods_netcdf_server.a(readAggInfo.o):(.data.rel.ro.local+0x0): multiple definition of `RodsPackTable'

/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../lib/libirods_client.a(rcConnect.o):(.data.rel.ro.local+0x0): first defined here

collect2: error: ld returned 1 exit status

make: *** [obj/librsNcArchTimeSeries_server.so] Error 1

make: *** Waiting for unfinished jobs....

libirods_netcdf_server.a(readAggInfo.o):(.data.rel.local+0x0): multiple definition of `packTypeTable'

/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../lib/libirods_client.a(rcConnect.o):(.data.rel.local+0x840): first defined here

libirods_netcdf_server.a(readAggInfo.o):(.data+0x0): multiple definition of `NumOfPackTypes'

/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../lib/libirods_client.a(rcConnect.o):(.data+0x10): first defined here

libirods_netcdf_server.a(readAggInfo.o):(.data.rel.local+0xa0): multiple definition of `PackConstantTable'

/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../lib/libirods_client.a(rcConnect.o):(.data.rel.local+0x6e0): first defined here

libirods_netcdf_server.a(readAggInfo.o):(.data.rel.ro.local+0x0): multiple definition of `RodsPackTable'

/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../lib/libirods_client.a(rcConnect.o):(.data.rel.ro.local+0x0): first defined here

collect2: error: ld returned 1 exit status

make: *** [obj/librsNcOpen_server.so] Error 1

[Jason Coposky]

​just to be sure we are doing the same thing, how are you building?

something like:

cd api ; ./packaging/build.sh -r 

—
Jason Coposky
Chief Technologist, iRODS Consortium
RENCI at the University of North Carolina at Chapel Hill
(919)445-9675
jas...@renci.org

irods.org — Take Control of Your Data

From: irod...@googlegroups.com <irod...@googlegroups.com> on behalf of Tom Langborg <tom.la...@gmail.com>

Sent: Thursday, September 3, 2015 4:17 AM
To: irod...@googlegroups.com
Subject: Re: [iROD-Chat:14243] NetCDF and HDF5 support in iRODS 4

 

[Tom Langborg]

Hi

I did ./api/packaging/build.sh -r

but  there is no difference in your suggestion.

The files libirods_client.. are coming from irods-dev package at ftp://ftp.renci.org/pub/irods/releases/4.1.4/ubuntu14/irods-dev-4.1.4-ubuntu14-x86_64.deb

learner@learner-vb:~/irods_netcdf/api$ ./packaging/build.sh -r

+------------------------------------+

| iRODS Plugin Build Script          |

+------------------------------------+

Thu Sep  3 07:13:13 EDT 2015

-r detected -- Building for release

Build Environment...

Detected OS                         [Ubuntu]

Detected OS Version                 [14.04]

Detected Plugin Name                [irods_api_plugin_netcdf]

Detected EPM Package Name           [irods-api-plugin-netcdf]

Detected Plugin Version to Build    [1.0]

Detected Plugin Version Integer     [10]

Detected Project Directory          [/home/learner/irods_netcdf/api]

Detected Packaging Directory        [/home/learner/irods_netcdf/api/packaging]

Detected Target Build Directory     [/home/learner/irods_netcdf/api/build]

Detected EPM List File              [/home/learner/irods_netcdf/api/packaging/irods_api_plugin_netcdf.list]

Detected CPUs                       [1]

Compile Command                     [make -j 4]

Building...

Compile plugin librsNcArchTimeSeries_server.so...

Compile plugin librsNcOpen_server.so...

libirods_netcdf_server.a(readAggInfo.o):(.data.rel.local+0x0): multiple definition of `packTypeTable'

/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../lib/libirods_client.a(rcConnect.o):(.data.rel.local+0x840): first defined here

libirods_netcdf_server.a(readAggInfo.o):(.data+0x0): multiple definition of `NumOfPackTypes'

/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../lib/libirods_client.a(rcConnect.o):(.data+0x10): first defined here

libirods_netcdf_server.a(readAggInfo.o):(.data.rel.local+0xa0): multiple definition of `PackConstantTable'

/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../lib/libirods_client.a(rcConnect.o):(.data.rel.local+0x6e0): first defined here

libirods_netcdf_server.a(readAggInfo.o):(.data.rel.ro.local+0x0): multiple definition of `RodsPackTable'

/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../lib/libirods_client.a(rcConnect.o):(.data.rel.ro.local+0x0): first defined here

collect2: error: ld returned 1 exit status

make: *** [obj/librsNcArchTimeSeries_server.so] Error 1

make: *** Waiting for unfinished jobs....

libirods_netcdf_server.a(readAggInfo.o):(.data.rel.local+0x0): multiple definition of `packTypeTable'

/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../lib/libirods_client.a(rcConnect.o):(.data.rel.local+0x840): first defined here

libirods_netcdf_server.a(readAggInfo.o):(.data+0x0): multiple definition of `NumOfPackTypes'

/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../lib/libirods_client.a(rcConnect.o):(.data+0x10): first defined here

libirods_netcdf_server.a(readAggInfo.o):(.data.rel.local+0xa0): multiple definition of `PackConstantTable'

/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../lib/libirods_client.a(rcConnect.o):(.data.rel.local+0x6e0): first defined here

libirods_netcdf_server.a(readAggInfo.o):(.data.rel.ro.local+0x0): multiple definition of `RodsPackTable'

/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../lib/libirods_client.a(rcConnect.o):(.data.rel.ro.local+0x0): first defined here

collect2: error: ld returned 1 exit status

make: *** [obj/librsNcOpen_server.so] Error 1

learner@learner-vb:~/irods_netcdf/api$ 

ls -al /usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../lib/libirods_client_api.a 

-rw-r--r-- 1 root root 857680 Aug  5 12:31 /usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../lib/libirods_client_api.a

 dpkg -S /usr/lib/libirods_client_api.a

irods-dev: /usr/lib/libirods_client_api.a

dpkg -S /usr/lib/libirods_client.a 

irods-dev: /usr/lib/libirods_client.a

Regards

tom

You received this message because you are subscribed to a topic in the Google Groups "iRODS-Chat" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/irod-chat/umR3zqYK-3E/unsubscribe.
To unsubscribe from this group and all its topics, send an email to irod-chat+...@googlegroups.com.

[Tony Edgin]

I'm trying the build on a CentOS 6 machine.

I was able to build the microservices and the api, but I can't build the icommands. 

Here's what happens.

[tedgin@twig icommands]$ packaging/build.sh -r

+------------------------------------+
| iRODS Plugin Build Script          |
+------------------------------------+

Thu Sep  3 17:53:44 MST 2015

-r detected -- Building for release

Build Environment...

Detected OS                         [RedHatCompatible]
Detected OS Version                 [6.6]
Detected Plugin Name                [irods_icommands_netcdf]
Detected EPM Package Name           [irods-icommands-netcdf]

Detected Plugin Version to Build    [1.0]
Detected Plugin Version Integer     [10]

Detected Project Directory          [/home/tedgin/irods_netcdf/icommands]
Detected Packaging Directory        [/home/tedgin/irods_netcdf/icommands/packaging]
Detected Target Build Directory     [/home/tedgin/irods_netcdf/icommands/build]
Detected EPM List File              [/home/tedgin/irods_netcdf/icommands/packaging/irods_icommands_netcdf.list]
Detected CPUs                       [2]
Compile Command                     [make -j 5]

Building...
Compile icommand inc.o...
Compile icommand incarch.o...
Compile icommand incattr.o...
Link icommand inc...
Link icommand incarch...
Link icommand incattr...

Creating Package...
Running EPM :: Generating RedHatCompatible RPMs
epm: Product names should only contain letters and numbers!
Building target platforms: x86_64
Building for target x86_64
error: line 11: Empty tag: Requires:
Packaging failed!

How do I fix this?

Thanks in advance.

[Terrell Russell]

Tom and Tony,

I have just pushed two fixes for these two errors - both the duplicate link error and the empty Requires line.

Please pull and try again.

Terrell

[Tony Edgin]

I got all of the RPMs to build.

To install them on a CentOS 6 machine already running iRODS 4.1.5, I needed to also install the corresponding irods-runtime RPM

 and the libcurl-devel RPM.  Is that expected?

[tom.la...@gmail.com]

Hi

I got all compiled now.

I needed to add libcurl4-nss-dev also.

But the binaries is missing shared libs.

I see the libs in /usr/lib/irods/externals/

I tried with ld_library_path but no success 

_______________________________________________________________________

learner@learner-vb:~$ inc -h

inc: error while loading shared libraries: libnetcdf.so.7: cannot open shared object file: No such file or directory

learner@learner-vb:~$ inca

incarch  incattr  

learner@learner-vb:~$ incarch

incarch: error while loading shared libraries: libjansson.so.4: cannot open shared object file: No such file or directory

learner@learner-vb:~$ incattr 

incattr: error while loading shared libraries: libnetcdf.so.7: cannot open shared object file: No such file or directory

[tom.la...@gmail.com]

Sorry forget my last messages.

I forgot to do export on LD_LIBRARY_PATH.

regards

tom

[tom.la...@gmail.com]

Hi Terrell

I believe that I got all things installed in my iRODS 4.1.5

I trying to read netcdf info from a file that are stored in irods. But got some errors and no info.

The netcdf file was stored in my iRODS as a ordinary file.

inc --header orog_WAS-44_ECMWF-ERAINT_evaluation_r0i0p0_SMHI-RCA4_v2_fx.nc

ERROR: API Entry not found at index 1000 status = -12000 SYS_UNMATCHED_API_NUM

ERROR: ncOperDataObjUtil: rcNcOpen error for /tomZone/home/rods/orog_WAS-44_ECMWF-ERAINT_evaluation_r0i0p0_SMHI-RCA4_v2_fx.nc status = -12000 SYS_UNMATCHED_API_NUM

ERROR: ncUtil: nc error for /tomZone/home/rods/orog_WAS-44_ECMWF-ERAINT_evaluation_r0i0p0_SMHI-RCA4_v2_fx.nc, status = -12000 status = -12000 SYS_UNMATCHED_API_NUM

regards

tom

Den fredag 4 september 2015 kl. 17:16:41 UTC+2 skrev Terrell Russell:

[Jason Coposky]

tom,

does a listing of /var/lib/irods/plugins/api on the client and server side both show the

netcdf api plugins are installed?

thanks,

—
Jason Coposky
Chief Technologist, iRODS Consortium
RENCI at the University of North Carolina at Chapel Hill
(919)445-9675
jas...@renci.org

irods.org — Take Control of Your Data

---

From: irod...@googlegroups.com <irod...@googlegroups.com> on behalf of tom.la...@gmail.com <tom.la...@gmail.com>
Sent: Friday, September 18, 2015 5:15 AM
To: iRODS-Chat
Subject: Re: [iROD-Chat:14285] NetCDF and HDF5 support in iRODS 4

 

[Tony Edgin]

I'm guessing that I didn't build the api correctly.  Whenever I try to do an operation. e.g. iput, I get the following errors, but the operation succeeds.

tedgin@ender:~$ iput sresa1b_ncar_ccsm3-example.nc
ERROR: [-]    iRODS/lib/core/src/apiHandler.cpp:246:init_api_table :  status [PLUGIN_ERROR]  errno [] -- message []
    [-]    iRODS/lib/core/include/irods_load_plugin.hpp:194:load_plugin :  status [PLUGIN_ERROR]  errno [] -- message [failed to get [get_plugin_interface_version] dlerror is [/var/lib/irods/plugins/api/librsNcInqId_client.so: undefined symbol: get_plugin_interface_version]]

ERROR: [-]    iRODS/lib/core/src/apiHandler.cpp:246:init_api_table :  status [PLUGIN_ERROR]  errno [] -- message []
    [-]    iRODS/lib/core/include/irods_load_plugin.hpp:194:load_plugin :  status [PLUGIN_ERROR]  errno [] -- message [failed to get [get_plugin_interface_version] dlerror is [/var/lib/irods/plugins/api/librsNcInqWithId_client.so: undefined symbol: get_plugin_interface_version]]

ERROR: [-]    iRODS/lib/core/src/apiHandler.cpp:246:init_api_table :  status [PLUGIN_ERROR]  errno [] -- message []
    [-]    iRODS/lib/core/include/irods_load_plugin.hpp:194:load_plugin :  status [PLUGIN_ERROR]  errno [] -- message [failed to get [get_plugin_interface_version] dlerror is [/var/lib/irods/plugins/api/librsNcRegGlobalAttr_client.so: undefined symbol: get_plugin_interface_version]]

ERROR: [-]    iRODS/lib/core/src/apiHandler.cpp:246:init_api_table :  status [PLUGIN_ERROR]  errno [] -- message []
    [-]    iRODS/lib/core/include/irods_load_plugin.hpp:194:load_plugin :  status [PLUGIN_ERROR]  errno [] -- message [failed to get [get_plugin_interface_version] dlerror is [/var/lib/irods/plugins/api/librsNcGetAggElement_client.so: undefined symbol: get_plugin_interface_version]]

ERROR: [-]    iRODS/lib/core/src/apiHandler.cpp:246:init_api_table :  status [PLUGIN_ERROR]  errno [] -- message []
    [-]    iRODS/lib/core/include/irods_load_plugin.hpp:194:load_plugin :  status [PLUGIN_ERROR]  errno [] -- message [failed to get [get_plugin_interface_version] dlerror is [/var/lib/irods/plugins/api/librsNcArchTimeSeries_client.so: undefined symbol: get_plugin_interface_version]]

ERROR: [-]    iRODS/lib/core/src/apiHandler.cpp:246:init_api_table :  status [PLUGIN_ERROR]  errno [] -- message []
    [-]    iRODS/lib/core/include/irods_load_plugin.hpp:194:load_plugin :  status [PLUGIN_ERROR]  errno [] -- message [failed to get [get_plugin_interface_version] dlerror is [/var/lib/irods/plugins/api/librsNcInqGrps_client.so: undefined symbol: get_plugin_interface_version]]

ERROR: [-]    iRODS/lib/core/src/apiHandler.cpp:246:init_api_table :  status [PLUGIN_ERROR]  errno [] -- message []
    [-]    iRODS/lib/core/include/irods_load_plugin.hpp:194:load_plugin :  status [PLUGIN_ERROR]  errno [] -- message [failed to get [get_plugin_interface_version] dlerror is [/var/lib/irods/plugins/api/librsNcCreate_client.so: undefined symbol: get_plugin_interface_version]]

ERROR: [-]    iRODS/lib/core/src/apiHandler.cpp:246:init_api_table :  status [PLUGIN_ERROR]  errno [] -- message []
    [-]    iRODS/lib/core/include/irods_load_plugin.hpp:194:load_plugin :  status [PLUGIN_ERROR]  errno [] -- message [failed to get [get_plugin_interface_version] dlerror is [/var/lib/irods/plugins/api/librsNcGetVarsByType_client.so: undefined symbol: get_plugin_interface_version]]

ERROR: [-]    iRODS/lib/core/src/apiHandler.cpp:246:init_api_table :  status [PLUGIN_ERROR]  errno [] -- message []
    [-]    iRODS/lib/core/include/irods_load_plugin.hpp:194:load_plugin :  status [PLUGIN_ERROR]  errno [] -- message [failed to get [get_plugin_interface_version] dlerror is [/var/lib/irods/plugins/api/librsNcGetAggInfo_client.so: undefined symbol: get_plugin_interface_version]]

ERROR: [-]    iRODS/lib/core/src/apiHandler.cpp:246:init_api_table :  status [PLUGIN_ERROR]  errno [] -- message []
    [-]    iRODS/lib/core/include/irods_load_plugin.hpp:194:load_plugin :  status [PLUGIN_ERROR]  errno [] -- message [failed to get [get_plugin_interface_version] dlerror is [/var/lib/irods/plugins/api/librsNcOpenGroup_client.so: undefined symbol: get_plugin_interface_version]]

ERROR: [-]    iRODS/lib/core/src/apiHandler.cpp:246:init_api_table :  status [PLUGIN_ERROR]  errno [] -- message []
    [-]    iRODS/lib/core/include/irods_load_plugin.hpp:194:load_plugin :  status [PLUGIN_ERROR]  errno [] -- message [failed to get [get_plugin_interface_version] dlerror is [/var/lib/irods/plugins/api/librsNcInq_client.so: undefined symbol: get_plugin_interface_version]]

ERROR: [-]    iRODS/lib/core/src/apiHandler.cpp:246:init_api_table :  status [PLUGIN_ERROR]  errno [] -- message []
    [-]    iRODS/lib/core/include/irods_load_plugin.hpp:194:load_plugin :  status [PLUGIN_ERROR]  errno [] -- message [failed to get [get_plugin_interface_version] dlerror is [/var/lib/irods/plugins/api/librsNcOpen_client.so: undefined symbol: get_plugin_interface_version]]

ERROR: [-]    iRODS/lib/core/src/apiHandler.cpp:246:init_api_table :  status [PLUGIN_ERROR]  errno [] -- message []
    [-]    iRODS/lib/core/include/irods_load_plugin.hpp:194:load_plugin :  status [PLUGIN_ERROR]  errno [] -- message [failed to get [get_plugin_interface_version] dlerror is [/var/lib/irods/plugins/api/librsOoiGenServReq_client.so: undefined symbol: get_plugin_interface_version]]

ERROR: [-]    iRODS/lib/core/src/apiHandler.cpp:246:init_api_table :  status [PLUGIN_ERROR]  errno [] -- message []
    [-]    iRODS/lib/core/include/irods_load_plugin.hpp:194:load_plugin :  status [PLUGIN_ERROR]  errno [] -- message [failed to get [get_plugin_interface_version] dlerror is [/var/lib/irods/plugins/api/librsNcClose_client.so: undefined symbol: get_plugin_interface_version]]

[Jason Coposky]

I have adjusted the build of the API plugins, and they should behave themselves.

Please pull and rebuild.

thanks,

—
Jason Coposky
Chief Technologist, iRODS Consortium
RENCI at the University of North Carolina at Chapel Hill
(919)445-9675
jas...@renci.org

irods.org — Take Control of Your Data

---

From: irod...@googlegroups.com <irod...@googlegroups.com> on behalf of Tony Edgin <ted...@iplantcollaborative.org>
Sent: Friday, September 18, 2015 5:03 PM
To: irod...@googlegroups.com
Subject: Re: [iROD-Chat:14292] NetCDF and HDF5 support in iRODS 4

 

[Tony Edgin]

I removed the previous RPMs and then installed the newly build ones. After restarting iRODS, I get the following error when using iput.

[irods@twig ~]$ iput peps.txt
*** glibc detected *** iput: double free or corruption (fasttop): 0x0000000000e377e0 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x75e66)[0x7f5721ad2e66]
/lib64/libc.so.6(+0x7897a)[0x7f5721ad597a]
/usr/lib64/libstdc++.so.6(_ZNSsD1Ev+0x39)[0x7f57223284c9]
/lib64/libc.so.6(exit+0xe2)[0x7f5721a92b22]
/lib64/libc.so.6(__libc_start_main+0x104)[0x7f5721a7bd64]
iput[0x465e19]

I removed the RPMs, and the error stopped occurring. After reinstalling the RPMs, the error reappeared.

[Tony Edgin]

Is there a solution to this problem?  The NetCDF libraries break iput.  See my previous post.

[Zoey Greer]

Hey Tony,
The netcdf plugin was linking against the wrong client library; this has been fixed and they should now work. Let us know if you find anything else!

Thanks,
-Zoey

[Jason Coposky]

​Please note this is only true for the upcoming 4.1.6 release which will most likely be out tomorrow or friday.

thanks,

—
Jason Coposky
Chief Technologist, iRODS Consortium
RENCI at the University of North Carolina at Chapel Hill
(919)445-9675
jas...@renci.org

irods.org — Take Control of Your Data

---

From: irod...@googlegroups.com <irod...@googlegroups.com> on behalf of Zoey Greer <tempo...@gmail.com>
Sent: Wednesday, September 30, 2015 5:24 PM
To: irod...@googlegroups.com; Tony Edgin
Subject: Re: [iROD-Chat:14322] NetCDF and HDF5 support in iRODS 4

 

[Tony Edgin]

After upgrading to the development tools and runtime libraries to version 4.1.6, I pulled the master head of the master branch of irods_netcdf and rebuild everything.  The API and microservices RPMs built.  Unfortunately, the icommands RPM failed to build.  What follows is the build output truncated after the first few error messages. It looks like a library is missing from the link command.

[tedgin@twig icommands]$ packaging/build.sh clean

+------------------------------------+
| iRODS Plugin Build Script          |
+------------------------------------+

Mon Oct  5 13:28:53 MST 2015

Build Environment...
Detected OS                         [RedHatCompatible]

Detected OS Version                 [6.7]

Detected Plugin Name                [irods_icommands_netcdf]
Detected EPM Package Name           [irods-icommands-netcdf]
Detected Plugin Version to Build    [1.0]
Detected Plugin Version Integer     [10]
Detected Project Directory          [/home/tedgin/irods_netcdf/icommands]
Detected Packaging Directory        [/home/tedgin/irods_netcdf/icommands/packaging]
Detected Target Build Directory     [/home/tedgin/irods_netcdf/icommands/build]
Detected EPM List File              [/home/tedgin/irods_netcdf/icommands/packaging/irods_icommands_netcdf.list]

Cleaning...
Cleaning icommands...
Done.

[tedgin@twig icommands]$ packaging/build.sh -r

+------------------------------------+
| iRODS Plugin Build Script          |
+------------------------------------+

Mon Oct  5 13:28:58 MST 2015

-r detected -- Building for release

Build Environment...
Detected OS                         [RedHatCompatible]

Detected OS Version                 [6.7]

Detected Plugin Name                [irods_icommands_netcdf]
Detected EPM Package Name           [irods-icommands-netcdf]
Detected Plugin Version to Build    [1.0]
Detected Plugin Version Integer     [10]
Detected Project Directory          [/home/tedgin/irods_netcdf/icommands]
Detected Packaging Directory        [/home/tedgin/irods_netcdf/icommands/packaging]
Detected Target Build Directory     [/home/tedgin/irods_netcdf/icommands/build]
Detected EPM List File              [/home/tedgin/irods_netcdf/icommands/packaging/irods_icommands_netcdf.list]
Detected CPUs                       [2]
Compile Command                     [make -j 5]

Building...
Compile icommand inc.o...
Compile icommand incarch.o...
Compile icommand incattr.o...
Link icommand inc...

Link icommand incattr...
Link icommand incarch...
obj/inc.o: In function `main':
/home/tedgin/irods_netcdf/icommands/src/inc.cpp:30: undefined reference to `parseCmdLineOpt'
/home/tedgin/irods_netcdf/icommands/src/inc.cpp:43: undefined reference to `rodsLog'
/home/tedgin/irods_netcdf/icommands/src/inc.cpp:48: undefined reference to `getRodsEnv'

Did I miss a step?  I apologize if I've done something stupid.

[Zoey Greer]

No, you're right, those icommands are having linking problems; it appears that in my test I only built the microservices and api plugins. Working on fixing that now.

Thanks again,
-Zoey

[Terrell Russell]

And we'll have them in CI ASAP.

Terrell

[Tony Edgin]

Thanks Zoey and Terrell.  I really appreciate you guys making the extra effort here.

[Zoey Greer]

If you pull now, you should have netcdf icommands that build. Let me know if it isn't working for you.
Thanks again for bringing this to our attention.

-Zoey

[Tony Edgin]

Near success!

After getting the library path set correctly to include /usr/lib/irods/externals, I was able to execute an `inc --header` command and get sensible output.  Unfortunately, there was an error logged in the server logs.

Oct  6 13:46:53 pid:1803 NOTICE: _rsDataObjClose: l3Close of 65536 failed, status = -23000
Oct  6 13:46:53 pid:1803 ERROR: ncCloseDataObj: rcNcClose 3 error, status = -23000
Oct  6 13:46:53 pid:1803 NOTICE: readAndProcClientMsg: received disconnect msg from client
Oct  6 13:46:53 pid:1803 NOTICE: Agent exiting with status = 0

[Terrell Russell]

Thanks.   And for anyone else who finds this later...

$ ierror -23000

irods error: -23000 SYS_BAD_FILE_DESCRIPTOR

Terrell

[Jason Coposky]

can you send me the test data directly?

thanks,

— 
Jason Coposky
Chief Technologist, iRODS Consortium
RENCI at the University of North Carolina at Chapel Hill
(919)445-9675
jas...@renci.org

irods.org — Take Control of Your Data

[Jason Coposky]

​I found some test data and have not run into the issue working directly on the ICAT.

Was there perhaps a redirect with another server in the middle?

—
Jason Coposky
Chief Technologist, iRODS Consortium
RENCI at the University of North Carolina at Chapel Hill
(919)445-9675
jas...@renci.org

irods.org — Take Control of Your Data

---

From: irod...@googlegroups.com <irod...@googlegroups.com> on behalf of Jason Coposky <jas...@renci.org>
Sent: Tuesday, October 6, 2015 8:19 PM
To: irod...@googlegroups.com
Subject: Re: [iROD-Chat:14361] NetCDF and HDF5 support in iRODS 4

 

[Jason Coposky]

I have also just pushed some additional edits to the netcdf repository.

Thanks,

—
Jason Coposky
Chief Technologist, iRODS Consortium
RENCI at the University of North Carolina at Chapel Hill
(919)445-9675
jas...@renci.org

irods.org — Take Control of Your Data

From: irod...@googlegroups.com <irod...@googlegroups.com> on behalf of Jason Coposky <jas...@renci.org>

Sent: Tuesday, October 6, 2015 9:27 PM
To: irod...@googlegroups.com
Subject: Re: [iROD-Chat:14362] NetCDF and HDF5 support in iRODS 4

 

[Tony Edgin]

I was finally able to pull your latest changes.  I still have the same "near success" issue.

For my test configuration, I have an IES with 1 external resource server. The IES is not itself a resource server.  I've installed all three HDF RPMs on both servers and restarted the iRODS services.

As the irods service account user on the IES, I issue the `inc --header` command on the file sresa1b_ncar_ccsm3-example.nc.  I obtained this file from https://www.unidata.ucar.edu/software/netcdf/examples/sresa1b_ncar_ccsm3-example.nc.

The error message is logged by the IES. There is no error message logged by the resource server.

Here are the IES log entries generated by the request.

Oct 23 14:06:45 pid:26089 NOTICE: Agent process 27063 started for puser=rods and cuser=rods from 150.135.93.222
Oct 23 14:06:45 pid:26089 NOTICE: Agent process 27064 started for puser=rods and cuser=rods from 150.135.93.223
Oct 23 14:06:46 pid:27064 NOTICE: readAndProcClientMsg: received disconnect msg from client
Oct 23 14:06:46 pid:27064 NOTICE: Agent exiting with status = 0
Oct 23 14:06:46 pid:26089 NOTICE: Agent process 27064 exited with status 0
Oct 23 14:06:46 pid:26089 NOTICE: Agent process 27065 started for puser=rods and cuser=rods from 150.135.93.223
Oct 23 14:06:46 pid:27065 NOTICE: readAndProcClientMsg: received disconnect msg from client
Oct 23 14:06:46 pid:27065 NOTICE: Agent exiting with status = 0
Oct 23 14:06:46 pid:26089 NOTICE: Agent process 27065 exited with status 0
Oct 23 14:06:46 pid:26089 NOTICE: Agent process 27066 started for puser=rods and cuser=rods from 150.135.93.223
Oct 23 14:06:46 pid:27066 NOTICE: readAndProcClientMsg: received disconnect msg from client
Oct 23 14:06:46 pid:27066 NOTICE: Agent exiting with status = 0
Oct 23 14:06:46 pid:26089 NOTICE: Agent process 27066 exited with status 0
Oct 23 14:06:46 pid:26089 NOTICE: Agent process 27067 started for puser=rods and cuser=rods from 150.135.93.223
Oct 23 14:06:46 pid:27067 NOTICE: readAndProcClientMsg: received disconnect msg from client
Oct 23 14:06:46 pid:27067 NOTICE: Agent exiting with status = 0
Oct 23 14:06:46 pid:27063 NOTICE: _rsDataObjClose: l3Close of 65536 failed, status = -23000
Oct 23 14:06:46 pid:27063 ERROR: ncCloseDataObj: rcNcClose 3 error, status = -23000
Oct 23 14:06:46 pid:27063 NOTICE: readAndProcClientMsg: received disconnect msg from client
Oct 23 14:06:46 pid:27063 NOTICE: Agent exiting with status = 0
Oct 23 14:06:46 pid:26089 NOTICE: Agent process 27063 exited with status 0
Oct 23 14:06:46 pid:26089 NOTICE: Agent process 27067 exited with status 0

Here are the corresponding resource server log entries.

Oct 23 14:06:45 pid:26798 NOTICE: Agent process 27587 started for puser=rods and cuser=rods from 150.135.93.222
Oct 23 14:06:46 pid:27587 NOTICE: readAndProcClientMsg: received disconnect msg from client
Oct 23 14:06:46 pid:27587 NOTICE: Agent exiting with status = 0
Oct 23 14:06:46 pid:26798 NOTICE: Agent process 27587 exited with status 0

I hope this helps.

Cheers,

Tony

[Jason Coposky]

if you run the command connected to the resource server does it succeed?

— 
Jason Coposky
Chief Technologist, iRODS Consortium
RENCI at the University of North Carolina at Chapel Hill
(919)445-9675
jas...@renci.org

irods.org — Take Control of Your Data

[Tony Edgin]

The same error occurs when the command is run from the resource server.

Here are the IES log entries.

Oct 26 09:45:30 pid:26089 NOTICE: Agent process 15243 started for puser=rods and cuser=rods from 150.135.93.223
Oct 26 09:45:30 pid:26089 NOTICE: Agent process 15244 started for puser=rods and cuser=rods from 150.135.93.223
Oct 26 09:45:31 pid:15244 NOTICE: readAndProcClientMsg: received disconnect msg from client
Oct 26 09:45:31 pid:15244 NOTICE: Agent exiting with status = 0
Oct 26 09:45:31 pid:26089 NOTICE: Agent process 15244 exited with status 0
Oct 26 09:45:31 pid:26089 NOTICE: Agent process 15245 started for puser=rods and cuser=rods from 150.135.93.223
Oct 26 09:45:31 pid:15245 NOTICE: readAndProcClientMsg: received disconnect msg from client
Oct 26 09:45:31 pid:15245 NOTICE: Agent exiting with status = 0
Oct 26 09:45:31 pid:26089 NOTICE: Agent process 15245 exited with status 0
Oct 26 09:45:31 pid:26089 NOTICE: Agent process 15246 started for puser=rods and cuser=rods from 150.135.93.223
Oct 26 09:45:31 pid:15246 NOTICE: readAndProcClientMsg: received disconnect msg from client
Oct 26 09:45:31 pid:15246 NOTICE: Agent exiting with status = 0
Oct 26 09:45:31 pid:26089 NOTICE: Agent process 15246 exited with status 0
Oct 26 09:45:31 pid:26089 NOTICE: Agent process 15247 started for puser=rods and cuser=rods from 150.135.93.223
Oct 26 09:45:31 pid:15247 NOTICE: readAndProcClientMsg: received disconnect msg from client
Oct 26 09:45:31 pid:15247 NOTICE: Agent exiting with status = 0
Oct 26 09:45:31 pid:15243 NOTICE: _rsDataObjClose: l3Close of 65536 failed, status = -23000
Oct 26 09:45:31 pid:15243 ERROR: ncCloseDataObj: rcNcClose 3 error, status = -23000
Oct 26 09:45:31 pid:15243 NOTICE: readAndProcClientMsg: received disconnect msg from client
Oct 26 09:45:31 pid:15243 NOTICE: Agent exiting with status = 0
Oct 26 09:45:31 pid:26089 NOTICE: Agent process 15243 exited with status 0
Oct 26 09:45:31 pid:26089 NOTICE: Agent process 15247 exited with status 0

Here are the resource server log entries.

Oct 26 09:45:30 pid:26798 NOTICE: Agent process 12707 started for puser=rods and cuser=rods from 150.135.93.222
Oct 26 09:45:31 pid:12707 NOTICE: readAndProcClientMsg: received disconnect msg from client
Oct 26 09:45:31 pid:12707 NOTICE: Agent exiting with status = 0
Oct 26 09:45:31 pid:26798 NOTICE: Agent process 12707 exited with status 0

[Tony Edgin]

Has any progress been made on this issue?

[Jason Coposky]

please try the netcdf code built against the 4.1.7 release. 

— 
Jason Coposky
Chief Technologist, iRODS Consortium
RENCI at the University of North Carolina at Chapel Hill
(919)445-9675
jas...@renci.org

irods.org — Take Control of Your Data

[Tony Edgin]

I just rebuilt netcdf, and now I get the following error when I execute inc.

? inc --header sresa1b_ncar_ccsm3-example.nc
inc: symbol lookup error: /var/lib/irods/plugins/network/libtcp.so: undefined symbol: operation_rule_execution_manager_factory

[Tony Edgin]

Any ideas?

[Zoey Greer]

Hey Tony,
I built a fresh iRODS 4.1.7 and the tip of the netcdf plugin and installed both. I encountered no errors running inc --header on the file you indicated. You might have an old version of the netcdf plugin, or not be on the 4.1.7 tag of iRODS? If you can confirm the versions, we can move forward from there.
Thanks,
-Zoey

[Tony Edgin]

Hi Zoey.

I just verified that I have iRODS 4.1.7 installed and the latest netcdf plugin.

Let's move forward. ;-)

Tony

[Zoey Greer]

Bringing up a new CentOS6 VM to make sure there aren't any confounding factors on my end. I'll get back to you once I've tested there.

[Zoey Greer]

Apparently there was something in my dev environment that was making this a non-issue for me, but I've reproduced it on this CentOS6 VM. Working on it now, thanks for letting us know about it.

[Zoey Greer]

Hey Tony,
The most recent commit to the netcdf repo should fix your issue. Sorry I didn't see it before, but the linker problem that's causing this appears to be version-specific.
Thanks,
-Zoey

[Tony Edgin]

Thanks Zoey!

I tried it again.  I'm back to the partial failure I was getting with iRODS 4.1.6.  In other words, the command prints the header, but an error is logged when closing a file descriptor.  When you tested it, was an error logged in the rodsLog?

Cheers,

Tony

[Zoey Greer]

Probably, but the machine that it was running on has been destroyed, so I can't check. I've created an issue to track it here:

https://github.com/irods/irods_netcdf/issues/14

[Tony Edgin]

I saw that https://github.com/irods/irods_netcdf/issues/14 has been closed. II rebuilt the irods_netcdf packages and reinstalled them.  I got the missing symbol error that Zoey fixed on 12/15.  A regression had occurred. I then reverted the commit https://github.com/irods/irods_netcdf/commit/457ac0ceb06942044a78450aa92c973f141a73e6 and reinstalled.  It worked this time! 

I'm guessing that the NetCDF library built from head will work once iRODS 4.1.8 is released.

Thanks!

[Jason Coposky]

Tony,

That is correct, I updated the build of netcdf to work for the upcoming 4.1.8 release.

Happy to hear it is working.

—
Jason Coposky
Chief Technologist, iRODS Consortium
RENCI at the University of North Carolina at Chapel Hill
(919)445-9675
jas...@renci.org

irods.org — Take Control of Your Data

From: irod...@googlegroups.com <irod...@googlegroups.com> on behalf of Tony Edgin <ted...@iplantcollaborative.org>

Sent: Tuesday, January 5, 2016 3:49 PM
To: irod...@googlegroups.com
Cc: Tony Edgin
Subject: Re: [iROD-Chat:14683] NetCDF and HDF5 support in iRODS 4

 

[Marco Mancini]

Hi,

i have installed irods 4.1.7 and the netcdf icommands. I got the following error when I use incattr --reg

ERROR: ncattrUtil: ncattr error for /sccZone/home/admin/T2M.nc, status = -43000 status = -43000 SYS_NO_RCAT_SERVER_ERR

When I use a resource based on ceph rados, I got also the following error with inc

ERROR: ncOperDataObjUtil: rcNcOpen error for /sccZone/home/admin/T2M.nc status = -1999998 Unknown iRODS error, Unknown error 998

ERROR: ncUtil: nc error for /sccZone/home/admin/T2M.nc, status = -1999998 status = -1999998 Unknown iRODS error, Unknown error 998

Best Regards,

Marco

...