out of memory..

[Strikwerda, Ger]

Hello all,

'something' is causing our iRODS (4.2.10) icat-servers to run out of memory:

[wo mei 25 14:06:18 2022] Out of memory: Kill process 32098 (irodsServer) score 175 or sacrifice child
[wo mei 25 14:06:18 2022] Killed process 32098 (irodsServer), UID 998, total-vm:3841276kB, anon-rss:3066772kB, file-rss:224kB, shmem-rss:156kB

We think it is the Cyberduck client sending large files through the icat server instead of directly to the resc-servers. Has anyone else seen this behaviour?
CyberDuck's parallel transfer mode is enabled by default, but appears to connect to the icat servers instead of any of the resource servers, likely causing the problem.

Of course it could be something completely else so, any tips/tricks on how to tackle out of memory issues would also be nice.

Our icat systems have 16 GB of memory and 2 GB swap. Perhaps that is a bit too small. What is considered 'normal'? 

--

Vriendelijke groet, 

Ger Strikwerda
senior expert multidisciplinary enabler 
simple solution architect
Rijksuniversiteit Groningen
CIT/RDMS/HPC 

Smitsborg
Nettelbosje 1
9747 AJ Groningen
Tel. 050 363 9276

"God is hard, God is fair
 some men he gave brains, others he gave hair"

[Terrell Russell]

Hi Ger,

Cyberduck, I believe, is using Java/Jargon's streaming, not parallel transfer...  But even so, that wouldn't/shouldn't go through the provider - I'd expect it to be redirected and make a direct connection to the consumer from the client.

Is the data making it to the correct storage resource (not connected to the provider)?

Did this happen before 4.2.10?

Does it happen if there is no Cyberduck usage?

Terrell

--
--
The Integrated Rule-Oriented Data System (iRODS) - https://irods.org
 
iROD-Chat: http://groups.google.com/group/iROD-Chat
---
You received this message because you are subscribed to the Google Groups "iRODS-Chat" group.
To unsubscribe from this group and stop receiving emails from it, send an email to irod-chat+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/irod-chat/CAC-SFsAtDbMnx24c_dD9%2BcvbYvThwAhs9Ov4h_%2Bhb-cgObG1LA%40mail.gmail.com.

[Strikwerda, Ger]

Hi Terell, 

It used to behave normal, until now.. :-)

So perhaps it is because of a new version of Cyberduck. Also we have a test-environment where we will try to reproduce the same issue (testing with and without Cyberduck in a controlled way)

But I agree, Cyberduck should use/cause a massive load on the icat-servers. This is weird.

To view this discussion on the web visit https://groups.google.com/d/msgid/irod-chat/CAFaqteb2wydx93gxjev%2BB5uBvOjaEVaNKBpu--A%3DSwG4emYD4Q%40mail.gmail.com.

[Strikwerda, Ger]

Cyberduck should *NOT* use/cause a massive load on the icat-servers :-)

[Tony Edgin]

For Cyberduck, parallel transfer means that it transfers multiple files at once. I don't know how many, but I've seen it transfer more than 100 files concurrently. 

To view this discussion on the web visit https://groups.google.com/d/msgid/irod-chat/CAC-SFsAp2R9rdTFQ9BRbT%3DqEiytOtnP77ihjwca90SQwt9DOxQ%40mail.gmail.com.

[Strikwerda, Ger]

aha, we ran into an issue about hitting max-open-files on the icat-servers. And I also suspect Cyberduck from overloading/triggering too many delayed rules/full queue causing an "out of memory". 

To view this discussion on the web visit https://groups.google.com/d/msgid/irod-chat/CACkmaQV2otTC3eS95jOPfhzQuHTqavaoQz25M_qmBXUNncvEog%40mail.gmail.com.

[J.P. Mc Farland]

Indeed!  This is consistent with what we are seeing.  I can't trace the transfers real-time at the moment, but I do see the markers of many new data objects being created from Cyberduck transfers and a simultaneous heavy load on one of the providers.  It appears that at least the arbitration of these transfers is causing this.  It is unclear if the data is being transferred through the provider to the consumers or directly to them.  We are trying to get a handle on that now.

Regardless, in the past I have thrown a much greater load at our setup (up to >2000 connections per second over 3 providers) and suffered severe slowdown, but no iRODS errors or system issues (memory/open files).  Something else is clearly going on, but what?

--John

[Smeele, A.P.M. (Ton)]

John, Ger,

Which protocol is used to connect by Cyberduck?  We have seen similar issues before, it was linked to webdav client action to create many connections in parallel.

If so, you might want to throttle using Apache.

Met vriendelijke groet, kibd regards,

Ton

On 30 May 2022, at 10:57, J.P. Mc Farland <j.p.mc....@rug.nl> wrote:

Indeed!  This is consistent with what we are seeing.  I can't trace the transfers real-time at the moment, but I do see the markers of many new data objects being created from Cyberduck transfers and a simultaneous heavy load on one of the providers.  It appears that at least the arbitration of these transfers is causing this.  It is unclear if the data is being transferred through the provider to the consumers or directly to them.  We are trying to get a handle on that now.

To view this discussion on the web visit https://groups.google.com/d/msgid/irod-chat/eab7d382-2d99-4cb9-800d-123a322ca248n%40googlegroups.com.

[Strikwerda, Ger]

Hello Ton,

Users can use Cyberduck in native mode and/or in webdav-mode. It would be nice if we can set it to native-mode only.. Good idea to throttle apache for max webdav-connection to prevent overloading. We will look into that.

bedankt!

 

To view this discussion on the web visit https://groups.google.com/d/msgid/irod-chat/B8956713-B96B-4276-BCC9-9B04F93556CE%40uu.nl.