ils not working

[kovid bhardwaj]

Hi ,

The ils command in irods is not working. I am getting below error:

 

remote addresses: <a.b.c.d> ERROR: _rcConnect: connectToRhost error, server on example.xyz.com:1247 is probably down status = -4000 SYS_HEADER_READ_LEN_ERR

we have checked- irods and database service are running .

Thanks

Kovid

[Kory Draughn]

Hi Kovid,

Is there anything in the server log file?

Kory Draughn

Chief Technologist

iRODS Consortium

--
--
The Integrated Rule-Oriented Data System (iRODS) - https://irods.org
 
iROD-Chat: http://groups.google.com/group/iROD-Chat
---
You received this message because you are subscribed to the Google Groups "iRODS-Chat" group.
To unsubscribe from this group and stop receiving emails from it, send an email to irod-chat+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/irod-chat/e556b57f-a9da-4e7b-939c-feae92a30ae0n%40googlegroups.com.

[kovid bhardwaj]

Hi Kory,

We were able to upload file of upto 32MB in size but get below error when uploading a large file > 32MB

terminating with uncaught exception of type std::bad_alloc: std::bad_alloc
Aborted

What could be the reason behind this error ?

[Kory Draughn]

std::bad_alloc means the application's request for more memory could not be satisfied by the OS.

I recommend checking your OS's memory usage.

Kory Draughn

Chief Technologist

iRODS Consortium

To view this discussion on the web visit https://groups.google.com/d/msgid/irod-chat/ac28966a-d93c-4054-8142-2b50e982d56an%40googlegroups.com.

[kovid bhardwaj]

Hi Kory,

Can we define somewhere in irods configuration the memory allocation for file transfer ? 

[my@irodsservr ~]$ iput -R s3res 38MB_file

terminating with uncaught exception of type std::bad_alloc: std::bad_alloc
Aborted

[my@irodsservr  ~]$ free -g
              total        used        free      shared  buff/cache   available
Mem:          7           1           0           0           6                        5
Swap:         0           0           0

Do we need to increase OS memory here?

[Kory Draughn]

Was iput run from a different computer or on the same computer running the iRODS server?

Which computer are those free stats for?

Kory Draughn

Chief Technologist

iRODS Consortium

To view this discussion on the web visit https://groups.google.com/d/msgid/irod-chat/b604d53b-b740-4b17-a2f2-b1f12b2e18den%40googlegroups.com.

[Kory Draughn]

Any updates on this?

Kory Draughn

Chief Technologist

iRODS Consortium

[kovid bhardwaj]

Hi Kory,

The iput and free -g was executed on same computer running the iRODS server?

You received this message because you are subscribed to a topic in the Google Groups "iRODS-Chat" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/irod-chat/tYJyxQH93pM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to irod-chat+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/irod-chat/CAA-7h7mP5Pf4XxM%3DHOSVnzbcXJt20FdWOQ_5HZ%3Djg2ek2AFV4A%40mail.gmail.com.

[Kory Draughn]

Kovid,

Please post the summary information (at the top) produced by top or htop.

How much RAM does that server have? Based on the output of free, it appears to be 7GB. If that's true, I recommend adding more.

Consider running top or htop while doing the transfer to see how the server manages memory.

Kory Draughn

Chief Technologist

iRODS Consortium

To view this discussion on the web visit https://groups.google.com/d/msgid/irod-chat/CAJ4K7GBMOSFNqQWY%3Ddtor4fNfKO4V7nCtZsZ9JDjM743WMU5Sg%40mail.gmail.com.

[Kory Draughn]

Kovid,

Any updates on this?

Kory Draughn

Chief Technologist

iRODS Consortium

[kovid bhardwaj]

Hi Kory,

We were able to upload files upto 2gb in size to s3,  but after that got error :

remote addresses: 1.2.3.4 ERROR: rcPartialDataPut: toWrite 4194304, bytesWritten 0, errno = 32 status = -27032 SYS_COPY_LEN_ERR, Broken pipe

Could you please advise ?

[Kory Draughn]

What changed to allow you to upload 2gb? Did you modify iRODS or add more resources to the machine?

Kory Draughn

Chief Technologist

iRODS Consortium

To view this discussion on the web visit https://groups.google.com/d/msgid/irod-chat/c3ae167a-a899-462c-b24c-5297f00ca3cbn%40googlegroups.com.

[Kory Draughn]

And what version of iRODS are you running?

Kory Draughn

Chief Technologist

iRODS Consortium

[kovid bhardwaj]

Hi Kory,

There was a increase in server memory from 7GB to 32GB , after which we were able to upload files (~2GB) to s3. 

[kovid bhardwaj]

irods version 4.2.8

[Kory Draughn]

Kovid,

Please post more of the log file. Perhaps there are other messages that can help with understanding what's going on.

Also, consider upgrading to a later version of iRODS. A significant number of bug fixes and enhancements have been made since 4.2.8. I recommend 4.2.12 or 4.3.1. 4.3.2 will be released before UGM so that's also a possibility. Before upgrading production systems, please use a test environment to verify the new version satisfies your needs.

Kory Draughn

Chief Technologist

iRODS Consortium

To view this discussion on the web visit https://groups.google.com/d/msgid/irod-chat/0d2fa2d0-50e2-41c4-b9d4-2df151d93ba8n%40googlegroups.com.

[kovid bhardwaj]

Hi Kory,

We were able to upload large files from dev which is outside secure zone , but facing issue while uploading large files from qa server which is in secure zone , 
PFB logs we got while uploading 6gb file from QA to s3  :

remote addresses: 2.2.2.2 ERROR: rcvTranHeader: toread = 24, read = 0
 ERROR: rcvTranHeader: toread = 24, read = 0
remote addresses: 2.2.2.2 ERROR: rcvTranHeader: toread = 24, read = 0
remote addresses: 2.2.2.2 ERROR: rcvTranHeader: toread = 24, read = 0
 ERROR: rcvTranHeader: toread = 24, read = 0
 ERROR: rcvTranHeader: toread = 24, read = 0
remote addresses: 2.2.2.2 ERROR: rcvTranHeader: toread = 24, read = 0
remote addresses: 2.2.2.2 ERROR: rcvTranHeader: toread = 24, read = 0
 ERROR: rcvTranHeader: toread = 24, read = 0
 ERROR: rcvTranHeader: toread = 24, read = 0
 ERROR: rcvTranHeader: toread = 24, read = 0
 ERROR: rcvTranHeader: toread = 24, read = 0
remote addresses: 2.2.2.2 ERROR: rcvTranHeader: toread = 24, read = 0
remote addresses: 2.2.2.2 ERROR: rcPartialDataPut: toWrite 4194304, bytesWritten 255044, errno = 104 status = -27104 SYS_COPY_LEN_ERR, Connection reset by peer
 ERROR: rcvTranHeader: toread = 24, read = 0
remote addresses: 2.2.2.2 ERROR: rcPartialDataPut: toWrite 4194304, bytesWritten 2539008, errno = 104 status = -27104 SYS_COPY_LEN_ERR, Connection reset by peer
 ERROR: [-]     /irods_git_repo/lib/core/src/procApiRequest.cpp:182:int sendApiRequest(rcComm_t *, int, void *, bytesBuf_t *) :  status [SYS_HEADER_WRITE_LEN_ERR]  errno [Broken pipe] -- message [failed to call 'write body']
        [-]     /irods_git_repo/lib/core/src/sockComm.cpp:1257:irods::error sendRodsMsg(irods::network_object_ptr, const char *, bytesBuf_t *, bytesBuf_t *, bytesBuf_t *, int, irodsProt_t) :  status [SYS_HEADER_WRITE_LEN_ERR]  errno [Broken pipe] -- message [failed to call 'write body']
                [-]     /irods_git_repo/plugins/network/ssl/libssl.cpp:1055:irods::error ssl_send_rods_msg(irods::plugin_context &, const char *, bytesBuf_t *, bytesBuf_t *, bytesBuf_t *, int, irodsProt_t) :  status [SYS_HEADER_WRITE_LEN_ERR]  errno [Broken pipe] -- message [Write message header failed.]
                        [-]     /irods_git_repo/lib/core/src/sockComm.cpp:475:irods::error writeMsgHeader(irods::network_object_ptr, msgHeader_t *) :  status [SYS_HEADER_WRITE_LEN_ERR]  errno [Broken pipe] -- message [Wrote -1 expected 139.]
                                [-]     /irods_git_repo/plugins/network/ssl/libssl.cpp:998:irods::error ssl_write_msg_header(irods::plugin_context &, bytesBuf_t *) :  status [SYS_HEADER_WRITE_LEN_ERR]  errno [Broken pipe] -- message [Wrote -1 expected 139.]

 ERROR: putUtil: put error for /ABCZone/home/rods/6gbfile, status = -27000 status = -27000 SYS_COPY_LEN_ERR
 ERROR: [-]     /irods_git_repo/lib/core/src/rcConnect.cpp:285:int rcDisconnect(rcComm_t *) :  status [SSL_SHUTDOWN_ERROR]  errno [] -- message [failed to call 'client stop']
        [-]     /irods_git_repo/lib/core/src/sockComm.cpp:101:irods::error sockClientStop(irods::network_object_ptr, rodsEnv *) :  status [SSL_SHUTDOWN_ERROR]  errno [] -- message [failed to call 'client stop']
                [-]     /irods_git_repo/plugins/network/ssl/libssl.cpp:644:irods::error ssl_client_stop(irods::plugin_context &, rodsEnv *) :  status [SSL_SHUTDOWN_ERROR]  errno [] -- message [error shutting down the SSL connection | error:1409F07F:SSL routines:ssl3_write_pending:bad write retry]

[Terrell Russell]

If I'm reading this correctly.. the iRODS server did not receive the bytes it was 'promised' by the client.

Do you have the client-side error?   If client and server are both blaming the other - then it's probably a networking issue in the middle.

This could be a TCP timeout or reset issue in the network.

Terrell

To view this discussion on the web visit https://groups.google.com/d/msgid/irod-chat/1e28e8fc-4115-4906-9b93-f8ce716af5a7n%40googlegroups.com.

[Kory Draughn]

Agreed. It does look like something terminated the connection.

When other large file uploads fail, do the files share similar sizes? Do they fail after the same amount of time? 

Kory Draughn

Chief Technologist

iRODS Consortium

To view this discussion on the web visit https://groups.google.com/d/msgid/irod-chat/CAFaqtebNzdTWZPmYRyd8V997ySd883QWZ%3D-T72vHqO0xhDSXUA%40mail.gmail.com.