ERROR: readWorkerTask - readStartupPack failed. -4000 in rodsServerLog file

108 views
Skip to first unread message

mauro....@cmcc.it

unread,
Feb 12, 2021, 4:03:57 AM2/12/21
to iRODS-Chat

Dear Users,

I noticed that "ERROR: readWorkerTask - readStartupPack failed. -4000" error message is filling rodsServerLog file.
Do you know the meaning of this message?

I'm using the 4.2.8 irods version on CentOS 7 OS.

Thank you in advance,
Mauro

mauro....@cmcc.it

unread,
Feb 12, 2021, 5:19:21 AM2/12/21
to iRODS-Chat
I forgot to add that I noticed that one of two available iRODS icat servers was not working correctly.
The number of server running processes is less than expected one:

[root@icat01 log]# service irods status
irodsServer :
  Process 81240
irodsReServer :
  Process 81241

I tried to restart the service.

[root@icat01 log]# service irods restart
Stopping iRODS server...
Error encountered in graceful shutdown.
iRODS server processes remain after "irods-grid shutdown".
irodsServer :
  Process 81240
Killing forcefully...
Killing /usr/sbin/irodsServer, pid 81240
iRODS child processes remain after "irods-grid shutdown".
irodsReServer :
  Process 81241
Killing forcefully...
Killing /usr/sbin/irodsReServer, pid 81241
Success
Validating [/var/lib/irods/.irods/irods_environment.json]... Success
Validating [/var/lib/irods/VERSION.json]... Success
Validating [/etc/irods/server_config.json]... Success
Validating [/etc/irods/host_access_control_config.json]... Success
Validating [/etc/irods/hosts_config.json]... Success
Ensuring catalog schema is up-to-date...
Catalog schema is up-to-date.
Starting iRODS server...
Success


iRODS service seems to be ok now, but the errors mentioned in the subject are still there.

Thank you,
Mauro

Terrell Russell

unread,
Feb 12, 2021, 9:50:50 PM2/12/21
to irod...@googlegroups.com
Looks like the main iRODS server process died somewhere along the way.

If there is nothing else in the logs, it's going to be hard to know a reason why.

Terrell


--
--
The Integrated Rule-Oriented Data System (iRODS) - https://irods.org
 
iROD-Chat: http://groups.google.com/group/iROD-Chat
---
You received this message because you are subscribed to the Google Groups "iRODS-Chat" group.
To unsubscribe from this group and stop receiving emails from it, send an email to irod-chat+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/irod-chat/4480e4eb-3a53-4b58-aad7-cc6e99b31784n%40googlegroups.com.

Mauro Tridici

unread,
Feb 13, 2021, 4:55:36 PM2/13/21
to irod...@googlegroups.com
Hi Terrell,

yes, there are no other interesting messages in the rodsServerLog file.
At this moment, the server is running regularly.
Anyway, the same error message is still shown repeatedly (you can find below the last lines I captured from log file). 

Thank you,
Mauro

[root@icat01 log]# service irods status
irodsServer :
  Process 100422
  Process 100423
  Process 114928
irodsReServer :
  Process 100424

Feb 13 22:46:37 pid:100422  ERROR: readWorkerTask - readStartupPack failed. -4000
Feb 13 22:46:39 pid:100422  ERROR: readWorkerTask - readStartupPack failed. -4000
Feb 13 22:46:41 pid:100422  ERROR: readWorkerTask - readStartupPack failed. -4000
Feb 13 22:46:43 pid:100422  ERROR: readWorkerTask - readStartupPack failed. -4000
Feb 13 22:46:45 pid:100422  ERROR: readWorkerTask - readStartupPack failed. -4000
Feb 13 22:46:47 pid:100422  ERROR: readWorkerTask - readStartupPack failed. -4000
Feb 13 22:46:49 pid:100422  ERROR: readWorkerTask - readStartupPack failed. -4000
Feb 13 22:46:51 pid:100422  ERROR: readWorkerTask - readStartupPack failed. -4000
Feb 13 22:46:53 pid:100422  ERROR: readWorkerTask - readStartupPack failed. -4000
Feb 13 22:46:55 pid:100422  ERROR: readWorkerTask - readStartupPack failed. -4000
Feb 13 22:46:57 pid:100422  ERROR: readWorkerTask - readStartupPack failed. -4000
Feb 13 22:46:59 pid:100422  ERROR: readWorkerTask - readStartupPack failed. -4000
Feb 13 22:47:01 pid:100422  ERROR: readWorkerTask - readStartupPack failed. -4000
Feb 13 22:47:03 pid:100422  ERROR: readWorkerTask - readStartupPack failed. -4000
Feb 13 22:47:05 pid:100422  ERROR: readWorkerTask - readStartupPack failed. -4000
Feb 13 22:47:07 pid:100422  ERROR: readWorkerTask - readStartupPack failed. -4000
Feb 13 22:47:09 pid:100422  ERROR: readWorkerTask - readStartupPack failed. -4000
Feb 13 22:47:11 pid:100422  ERROR: readWorkerTask - readStartupPack failed. -4000
Feb 13 22:47:13 pid:100422  ERROR: readWorkerTask - readStartupPack failed. -4000

Coposky, Jason Matthew

unread,
Feb 13, 2021, 8:26:44 PM2/13/21
to irod...@googlegroups.com
Mauro,

This behavior is also exhibited if you have a monitoring system pointed at the iRODS server which is hitting 1247 with a heartbeat which is not properly formed.

Thanks,

------
Jason Coposky
Executive Director, iRODS Consortium
RENCI at the University of North Carolina at Chapel Hill
w: (919)445-9675
m: (919)522-0517
jas...@renci.org


From: irod...@googlegroups.com <irod...@googlegroups.com> on behalf of Mauro Tridici <mauro....@cmcc.it>
Sent: Saturday, February 13, 2021 4:55 PM
To: irod...@googlegroups.com <irod...@googlegroups.com>
Subject: Re: [iROD-Chat:19560] ERROR: readWorkerTask - readStartupPack failed. -4000 in rodsServerLog file
 

Mauro Tridici

unread,
Feb 13, 2021, 8:56:31 PM2/13/21
to irod...@googlegroups.com
Hello Jason,

this is a very important input, I think you hit the mark.
I’m using irods audit plugin and Kibana as monitoring system

Moreover, I’m also using haproxy for high availability and balancing. I have two icat servers instances in the backend.
So, may be I should fix the last part of config file of haproxy service adding something...

  backend servers
balance roundrobin
        option tcp-check
        tcp-check connect
        tcp-check send PING\n
server icat01.cmcc.scc IP1:1247  check
        server icat02.cmcc.scc IP2:1247  check

Do you think that a "correct check interval” could solve the issue?
If yes, could I ask you what is the correct check interval that you suggest?

Many thanks,
Mauro

Coposky, Jason Matthew

unread,
Feb 13, 2021, 8:58:29 PM2/13/21
to irod...@googlegroups.com
Mauro,

According to the documentation you need to send HEARTBEAT to the server, not PING:

That should clear up the connection errors.

Thanks,


------
Jason Coposky
Executive Director, iRODS Consortium
RENCI at the University of North Carolina at Chapel Hill
w: (919)445-9675
m: (919)522-0517
jas...@renci.org


Sent: Saturday, February 13, 2021 8:56 PM
To: irod...@googlegroups.com <irod...@googlegroups.com>
Subject: Re: [iROD-Chat:19562] ERROR: readWorkerTask - readStartupPack failed. -4000 in rodsServerLog file
 

Mauro Tridici

unread,
Feb 13, 2021, 9:03:18 PM2/13/21
to irod...@googlegroups.com
Ops, sorry! I didn’t pay attention to this part of the documentation. Tomorrow morning I will change the haproxy config file.
Thank you very much for your help.
I really appreciated it. 

Kind Regards,
Mauro

Mauro Tridici

unread,
Feb 14, 2021, 5:22:27 AM2/14/21
to irod...@googlegroups.com
Good morning Jason,

I’m trying to modify the haproxy configuration file as you suggested, but I have some doubts about the correct way to do it.

In your opinion, how I should change the "tcp-check send” line shown below?

  backend servers
        balance roundrobin
        option tcp-check
        tcp-check connect
        tcp-check send PING\n
        server icat01.cmcc.scc IP1:1247  check
        server icat02.cmcc.scc IP2:1247  check

Attempt #1:

tcp-check send HEARTBEAT\n

returns the error "Feb 14 10:25:26 pid:100422  ERROR: readWorkerTask - readStartupPack failed. -116104"

Attempt #2:

tcp-check send "\x00\x00\x00\x33<MsgHeader_PI><type>HEARTBEAT</type></MsgHeader_PI>"


returns the error "Feb 14 10:25:38 pid:100422  ERROR: readWorkerTask - readStartupPack failed. -4000"

What am I doing wrong?
I see that Tony Edgin (together with iRODS staff) already solved this issue, but I’m not able to understand how they modified the haproxy configuration file 



Many thanks in advance,
Mauro

mauro....@cmcc.it

unread,
Feb 16, 2021, 3:46:06 AM2/16/21
to iRODS-Chat
Hello everyone,

anyone can help me to add the right "tcp-check send" instruction in haproxy configuration file?

If I execute the example command mentioned in https://docs.irods.org/4.2.8/system_overview/tips_and_tricks/#monitoring-status-of-irods-servers everything seems working as expected,

[root@icat01 ~]# echo -e "\x00\x00\x00\x33<MsgHeader_PI><type>HEARTBEAT</type></MsgHeader_PI>" | nc localhost 1247
HEARTBEAT[root@icat01 ~]#

but I'm not able to make it working in haproxy.
Could you please help me to fix this issue?

Thanks,
Mauro

Terrell Russell

unread,
Feb 16, 2021, 9:56:56 AM2/16/21
to irod...@googlegroups.com
Probably need a bit more escaping around those backslashes...  (same as the work done by the -e flag for echo).


Terrell



To view this discussion on the web visit https://groups.google.com/d/msgid/irod-chat/76c3264a-a524-4709-a2b1-605c8f779be2n%40googlegroups.com.

Tony Edgin

unread,
Feb 16, 2021, 10:10:06 AM2/16/21
to irod...@googlegroups.com
Hi Mauro.

Try the following.  I think you were missing the tcp-check expect part that validates the response from iRODS.

backend server
        balance roundrobin
        option tcp-check
        tcp-check connect
        tcp-check send-binary 00000033
        tcp-check send <MsgHeader_PI><type>HEARTBEAT</type></MsgHeader_PI>
        tcp-check expect string HEARTBEAT
        server icat01.cmcc.scc IP1:1247  check
        server icat02.cmcc.scc IP2:1247  check

Cheers,
Tony

To view this discussion on the web visit https://groups.google.com/d/msgid/irod-chat/76c3264a-a524-4709-a2b1-605c8f779be2n%40googlegroups.com.

Mauro Tridici

unread,
Feb 16, 2021, 10:29:50 AM2/16/21
to irod...@googlegroups.com
Many thanks, Terrell.
I will take a look at this link immediately.

Mauro


-------------------------
Mauro Tridici

Fondazione CMCC
CMCC Supercomputing Center
presso Complesso Ecotekne - Università del Salento -
Strada Prov.le Lecce - Monteroni sn
73100 Lecce  IT
http://www.cmcc.it

mobile: (+39) 327 5630841
email: mauro....@cmcc.it
https://it.linkedin.com/in/mauro-tridici-5977238b

-------------------------

Le informazioni contenute in questo messaggio di posta elettronica e negli allegati se presenti sono riservate e confidenziali: ne è vietata la diffusione in qualsiasi modo o forma (GDPR 2016/679).

Qualora lei non fosse il destinatario del messaggio, la invito a non diffonderlo e ad eliminarlo dandone gentilmente comunicazione al mittente.

 

The information included in this e-mail and any attachments are confidential and may also be privileged (GDPR 2016/679).

If you are not the correct recipient, you are kindly requested to notify the sender immediately, to cancel it and not disclose the contents to any other person.

Mauro Tridici

unread,
Feb 16, 2021, 10:34:23 AM2/16/21
to irod...@googlegroups.com
Hi Tony,

many thanks for your help. I really appreciated it.
I think you solved the issue: error messages stopped flooding the irodsServerLog log file.

Thank you very much for your availability.
Cheers,
Mauro

Terrell Russell

unread,
Feb 23, 2021, 11:15:34 PM2/23/21
to irod...@googlegroups.com
I have updated the docs - thanks Mauro and Tony!


Terrell


To view this discussion on the web visit https://groups.google.com/d/msgid/irod-chat/B0683834-4CB4-4459-ABAC-F63918725A75%40cmcc.it.
Reply all
Reply to author
Forward
0 new messages