RE: [iROD-Chat:7748] iCommands and SIGPIPE

[schr...@diceresearch.org]

Hello,

I'm not able to reproduce the problem on either MacOS or Linux (Ubuntu):

schroeder@zuri:~$ iquest "select DATA_ID" | wc
     38      76    1463
schroeder@zuri:~$ iquest "select DATA_ID" | head
DATA_ID = 10034
------------------------------------------------------------
DATA_ID = 10049
------------------------------------------------------------
DATA_ID = 17844
------------------------------------------------------------
DATA_ID = 19684
------------------------------------------------------------
DATA_ID = 19685
------------------------------------------------------------
schroeder@zuri:~$ ils -r | wc
     25      28     263
schroeder@zuri:~$ ils -r | head -2
/newZone/home/rods:
  f0
schroeder@zuri:~$

Is there something unique about your platform?

I'm also not sure if changing the SIG_PIPE processing to SIG_IGN would be completely harmless in all cases.  I didn't write this part, but presumably there was need for this on at least some versions of Unix/Linux. 

We can continue discussing this, either in direct email or on the list.  Mike Wan will also have some thoughts on this but he may not have a chance to respond for a while.

 - Wayne -

-------- Original Message --------
Subject: [iROD-Chat:7748] iCommands and SIGPIPE
From: Martin Pollard <martin....@sanger.ac.uk>
Date: Wed, February 29, 2012 9:53 am
To: irod...@googlegroups.com
Cc: martin....@sanger.ac.uk

Hi,

At the Sanger we use shell scripting extensively in our genetics
pipelines and we tend to use pipes | quite a bit. Unfortunately there
is a bug in some of the iCommands whereby if you pipe their output to
something which closes the pipe such as head they flood stderr with
messages claiming that the connection to the server might be broken.
This is because there is currently a SIGPIPE handler which is meant to
catch errors in sending data to the server. Because of the way pipes
work in POSIX environments, this catches all closed pipes including
those created by a shell command. I've had a look through the code and
it appears that the send and receive code is error checked, therefore I
suggest the SIGPIPE handler be replaced with a SIG_IGN. Proposed patch
is posted on Bugzilla:

https://www.irods.org/bugzilla/show_bug.cgi?id=146

Any feedback, suggestions etc are welcomed.

Regards,
Martin




--
The Wellcome Trust Sanger Institute is operated by Genome Research
Limited, a charity registered in England with number 1021457 and a
company registered in England with number 2742969, whose registered
office is 215 Euston Road, London, NW1 2BE.

--
"iRODS: the Integrated Rule-Oriented Data-management System; A community driven, open source, data grid software solution" https://www.irods.org

iROD-Chat: http://groups.google.com/group/iROD-Chat

[Joshua Randall]

Wayne,

Can you try running an iquest that returns more lines of output?

I can confirm this is a problem on Ubuntu, running the same command on
our iRODS system (which produces 244 rather than 38 lines of output).

> $ lsb_release -d
> Description: Ubuntu 10.04.4 LTS
> $ /software/irods/icommands/bin/iquest "select DATA_ID" | wc -l
> 244
> $ /software/irods/icommands/bin/iquest "select DATA_ID" | head
> DATA_ID = 10016
> ------------------------------------------------------------
> DATA_ID = 10080
> ------------------------------------------------------------
> DATA_ID = 10082
> ------------------------------------------------------------
> DATA_ID = 10086
> ------------------------------------------------------------
> DATA_ID = 10087
> ------------------------------------------------------------
> Client Caught broken pipe signal. Connection to server may be down
> Client Caught broken pipe signal. Connection to server may be down

If I instead run the same iquest in a zone that has more entries, I
get more broken pipe signals on STDERR.

> $ /software/irods/icommands/bin/iquest --no-page -z /seq "select DATA_ID where DATA_ID like '102%'" | wc -l
> 743
> $ /software/irods/icommands/bin/iquest --no-page -z /seq "select DATA_ID where DATA_ID like '102%'" | head
> Zone is /seq
> DATA_ID = 10206
> ------------------------------------------------------------
> DATA_ID = 10208
> ------------------------------------------------------------
> DATA_ID = 10209
> ------------------------------------------------------------
> DATA_ID = 10216
> ------------------------------------------------------------
> DATA_ID = 10219
> Client Caught broken pipe signal. Connection to server may be down
> Client Caught broken pipe signal. Connection to server may be down
> Client Caught broken pipe signal. Connection to server may be down
> Client Caught broken pipe signal. Connection to server may be down
> Client Caught broken pipe signal. Connection to server may be down
> Client Caught broken pipe signal. Connection to server may be down
> Client Caught broken pipe signal. Connection to server may be down

And the number of entries seems to be associated with the number of errors:

> $ /software/irods/icommands/bin/iquest --no-page -z /seq "select DATA_ID where DATA_ID like '10%'" | wc -l
> 45839
> $ /software/irods/icommands/bin/iquest --no-page -z /seq "select DATA_ID where DATA_ID like '10%'" | head
> Zone is /seq
> DATA_ID = 10206
> ------------------------------------------------------------
> DATA_ID = 10208
> ------------------------------------------------------------
> DATA_ID = 10209
> ------------------------------------------------------------
> DATA_ID = 10216
> ------------------------------------------------------------
> DATA_ID = 10219
> Client Caught broken pipe signal. Connection to server may be down
> Client Caught broken pipe signal. Connection to server may be down
> Client Caught broken pipe signal. Connection to server may be down
> Client Caught broken pipe signal. Connection to server may be down
> Client Caught broken pipe signal. Connection to server may be down
> Client Caught broken pipe signal. Connection to server may be down
> Client Caught broken pipe signal. Connection to server may be down
> Client Caught broken pipe signal. Connection to server may be down
> Client Caught broken pipe signal. Connection to server may be down
> Client Caught broken pipe signal. Connection to server may be down
> ...(repeats 429 more times)

Fitting a linear model to errors vs lines shows we'd expect about 1
error for every 100 lines of output, so given this data if all else is
equal we'd expect a dataset with 38 lines to have approximately 0
errors.

Josh.

[schr...@diceresearch.org]

Josh,

You're right.  If I do an iquest that results in a larger number of lines, it does produce the broken pipe error messages on linux (ubuntu) and presumably most or all unix/linux systems.

The iRODS client code is linked into the server and we need to be particularly careful about changing something like this so close to the upcoming release.  It would probably be best to address it after 3.1.

We are working toward releasing 3.1 in mid-March, if possible.  We are making good progress toward that, but still have much to do.

[Martin Pollard]

I'll settle for whatever comes after 3.1 just as long as it doesn't
get forgotten. I'm not completely convinced that it's a very risky
changes though. All that the current code is doing is ignoring the
signal and writing to stderr which will probably go to the log if the
server process in question uses your standard daemonize(). Instead
the signal to continue to be ignored but without the write to stderr.
If there are reads and writes which are unchecked then it could
possibly create some slight issues with debugging as we'd miss the
error but other than that I can't see any further issues.

Martin

> "iRODS: the Integrated Rule-Oriented Data-management System; A community driven, open...
>
> read more »