Problem with Davrods and irule

[Felipe Gutierrez]

Hello,

I am using Davrods to drop files into my irods instance. Every thing is working fine. I can drop images and check through ils -L icommand. I also was working with irules. So I created an irule to check if the file is an image and extract all metadata and insert into irods. My irule works fine when I use iput command, but when I test to drop the images through Davrods I got an error on the httpd server.

#  tail  /etc/httpd/logs/error_log

[Wed Oct 26 10:41:29.282775 2016] [dav:error] [pid 11405] [client 145.117.144.70:59273] Unable to deliver content.  [500, #0]

[Wed Oct 26 10:41:29.282820 2016] [dav:error] [pid 11405] (32)Broken pipe: [client 145.117.144.70:59273] Could not write contents to filter.  [500, #0]

ERROR: [-] iRODS/lib/core/src/procApiRequest.cpp:266:readAndProcApiReply :  status [SYS_HEADER_READ_LEN_ERR]  errno [] -- message []

[-] iRODS/lib/core/src/sockComm.cpp:199:readMsgHeader :  status [SYS_HEADER_READ_LEN_ERR]  errno [] -- message [failed to call 'read header']

[-] libssl.cpp:577:ssl_read_msg_header :  status [SYS_HEADER_READ_LEN_ERR]  errno [] -- message [read 0 expected 4]

[Wed Oct 26 10:41:36.192629 2016] [davrods:error] [pid 11389] [client 145.117.144.70:59275] rcDataObjClose failed: -4000 = SYS_HEADER_READ_LEN_ERR

[Wed Oct 26 10:41:36.192643 2016] [dav:error] [pid 11389] [client 145.117.144.70:59275] Could not close the uploaded resource  [500, #0]

ERROR: [-] iRODS/lib/core/src/rcConnect.cpp:269:rcDisconnect :  status [SYS_HEADER_WRITE_LEN_ERR]  errno [Broken pipe] -- message []

[-] iRODS/lib/core/src/sockComm.cpp:1394:sendRodsMsg :  status [SYS_HEADER_WRITE_LEN_ERR]  errno [Broken pipe] -- message [failed to call 'write body']

[-] libssl.cpp:1058:ssl_send_rods_msg :  status [SYS_HEADER_WRITE_LEN_ERR]  errno [Broken pipe] -- message [Write message header failed.]

[-] iRODS/lib/core/src/sockComm.cpp:473:writeMsgHeader :  status [SYS_HEADER_WRITE_LEN_ERR]  errno [Broken pipe] -- message []

[-] libssl.cpp:1001:ssl_write_msg_header :  status [SYS_HEADER_WRITE_LEN_ERR]  errno [Broken pipe] -- message [Wrote -1 expected 140.]

ERROR: [-] iRODS/lib/core/src/rcConnect.cpp:281:rcDisconnect :  status [SSL_SHUTDOWN_ERROR]  errno [] -- message []

[-] iRODS/lib/core/src/sockComm.cpp:100:sockClientStop :  status [SSL_SHUTDOWN_ERROR]  errno [] -- message [failed to call 'client stop']

[-] libssl.cpp:648:ssl_client_stop :  status [SSL_SHUTDOWN_ERROR]  errno [] -- message [error shutting down the SSL connection]

# tail  /var/lib/irods/iRODS/server/log/rodsLog.2016.10.26

Authenticated

Oct 26 10:40:30 pid:11611 NOTICE: writeLine: inString = Hello world! A image file was uploaded [/var/lib/irods/iRODS/Vault/home/felipe/nave.jpg].

terminate called after throwing an instance of 'Magick::ErrorCorruptImage'

  what():  irodsAgent: Empty input file `/var/lib/irods/iRODS/Vault/home/felipe/nave.jpg' @ error/jpeg.c/JPEGErrorHandler/316

Oct 26 10:40:30 pid:11315 NOTICE: Agent process 11611 exited with status 6

# cat /etc/irods/training_acPostProcForPut.re 

acPostProcForPut {

  if ($filePath like "*.jpg" || $filePath like "*.jpeg" || $filePath like "*.bmp" || $filePath like "*.tif" || $filePath like "*.tiff" || $filePath like "*.rif" || $filePath like "*.gif" || $filePath like "*.png" || $filePath like "*.svg" || $filePath like "*.xpm") {

    writeLine("serverLog", "Hello world! A image file was uploaded [$filePath].");

    msiget_image_meta($filePath, *meta);

    writeLine("serverLog", "image file 1111");

    msiString2KeyValPair(*meta, *meta_kvp);

    writeLine("serverLog", "image file 2222");

    msiAssociateKeyValuePairsToObj(*meta_kvp, $objPath, "-d");

    writeLine("serverLog", "image file 3333");

  } # if

} # acPostProcForPut

[Terrell Russell]

It looks like the davrods code closed the connection too early for some reason and the server was still trying to have a conversation about closing the new file on the disk.

Terrell

--
--
"iRODS: the Integrated Rule-Oriented Data-management System; A community driven, open source, data grid software solution" https://www.irods.org
 
iROD-Chat: http://groups.google.com/group/iROD-Chat

---
You received this message because you are subscribed to the Google Groups "iRODS-Chat" group.
To unsubscribe from this group and stop receiving emails from it, send an email to irod-chat+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Chris Smeele]

Hello Felipe,

I have set up a clean iRODS 4.1.9 installation with Davrods 1.1 and
added the training msi and your postprocforput policy. In this
environment I am unable to reproduce your issue. Putting a JPG file
using the cadaver DAV client adds the image metadata as iRODS metadata
as expected.

Could you please share:

- Your iRODS version
- Your Davrods version
- Which DAV client you are using (Cyberduck, cadaver, Finder, ...?)
- Do other file uploads via Davrods trigger the 'Could not close the
uploaded resource' error message in the error_log? Are the uploaded
files intact?

Additionally, you can enable debug output for Davrods by placing the
following line anywhere in your vhost config's Location block:

LogLevel davrods:debug

If you share the debug info in httpd error_log concerning the upload
request, that may aid in finding the cause of your issue.


FWIW, the output of my test is as follows:

[chris@davrods-centos7 ~]$ cadaver localhost
Authentication required for DAV on server `localhost':
Username: chris
Password:
dav:/> put carrots.jpg carrots2.jpg
Uploading carrots.jpg to `/carrots2.jpg':
Progress: [=============================>] 100,0% of 52006 bytes
succeeded.
dav:/> ls
Listing collection `/': succeeded.
carrots.jpg 52006 okt 30 09:35
carrots2.jpg 52006 okt 30 09:38
dav:/>
Connection to `localhost' closed.
[chris@davrods-centos7 ~]$ imeta ls -d carrots2.jpg
AVUs defined for dataObj carrots2.jpg:
attribute: Format
value: Joint Photographic Experts Group JFIF format
units:
----
attribute: ImageDepth
value: 8
units:
----
attribute: Height
value: 600
units:
----
attribute: CompressionType
value: JPEG
units:
----
attribute: Width
value: 600
units:
----
attribute: Colorspace
value: sRGB
units:


Kind regards,

Chris Smeele

> --

[Felipe Gutierrez]

Hi Chris,

The log with debug enable is at https://justpaste.it/zwff

My irods and davrods version is the same of yours, please check below

[root@fedora20 ~]# yum list installed | grep rod
davrods.x86_64 1.1.0-1.el7.centos installed
irods-database-plugin-mysql.x86_64 1.9-0 installed
irods-icat.x86_64 4.1.9-0 installed
irods-microservice-plugins-curl.x86_64 1.3-0 installed
irods-runtime.x86_64 4.1.9-0 installed

I am using MacOS. Finder > Go > Connect to Server > https://145.117.144.230/

You are right, I tried using cadaver and it works. Here is the log >> https://justpaste.it/zwfk

So the problem is with MacOS Finder. I think Finder is not working very well with SSL connection, since I saw a lot of SSL_SHUTDOWN_ERROR messages.

Thanks for your time. Felipe

[root@fedora20 ~]# cadaver https://localhost
WARNING: Untrusted server certificate presented for `fedora20.ebioscience.amc.nl':
Certificate was issued to hostname `fedora20.ebioscience.amc.nl' rather than `localhost'
This connection could have been intercepted.
Issued to: uva, amc, Amsterdam, Utrech, NL
Issued by: uva, amc, Amsterdam, Utrech, NL
Certificate is valid from Thu, 06 Oct 2016 09:41:46 GMT to Fri, 06 Oct 2017 09:41:46 GMT
Do you wish to accept the certificate? (y/n) y

Authentication required for DAV on server `localhost':

Username: felipe
Password:
dav:/> put carro.jpg carro2.jpg
Uploading carro.jpg to `/carro2.jpg':
Progress: [=============================>] 100.0% of 231259 bytes succeeded.
[root@fedora20 ~]# imeta ls -d carro2.jpg
AVUs defined for dataObj carro2.jpg:

attribute: ImageDepth
value: 8
units:
----

attribute: CompressionType
value: JPEG
units:
----

attribute: Format
value: Joint Photographic Experts Group JFIF format
units:
----

attribute: Colorspace
value: sRGB
units:
----
attribute: Width
value: 2400
units:
----
attribute: Height
value: 1350
units:

[Chris Smeele]

Hi Felipe,

The rodsLog in your previous e-mail shows that the irods agent process
is terminated by an ImageMagick exception. I suspect that this is what
is causing the 'Could not close resource' error on the davrods side (and
all further shutdown errors) - as your policy is likely triggered during
the close of the data object.

I notice that you are testing using two different images. Perhaps the
one you used to test Finder was corrupt? The exception message says the
file is empty.


Kind regards,

Chris