Error when uploading big files

[Aan Kurniawan]

Dear all,

I tried to upload a big file (size>100MB) using Jargon and got the following errors:

xception in thread "Thread-0" java.lang.RuntimeException: IOException in thread.
    at edu.sdsc.grid.io.irods.IRODSCommands$TransferThread.run(Unknown Source)
    at java.lang.Thread.run(Thread.java:619)
Caused by: java.net.SocketException: Connection reset
    at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:96)
    at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
    at edu.sdsc.grid.io.irods.IRODSCommands$TransferThread.put(Unknown Source)
    ... 2 more
Exception in thread "Thread-2" java.lang.RuntimeException: IOException in thread.
    at edu.sdsc.grid.io.irods.IRODSCommands$TransferThread.run(Unknown Source)
    at java.lang.Thread.run(Thread.java:619)
Caused by: java.net.SocketException: Connection reset
    at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:96)
    at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
    at edu.sdsc.grid.io.irods.IRODSCommands$TransferThread.put(Unknown Source)
    ... 2 more
Exception in thread "Thread-3" java.lang.RuntimeException: IOException in thread.
    at edu.sdsc.grid.io.irods.IRODSCommands$TransferThread.run(Unknown Source)
    at java.lang.Thread.run(Thread.java:619)
Caused by: java.net.SocketException: Connection reset
    at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:96)
    at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
    at edu.sdsc.grid.io.irods.IRODSCommands$TransferThread.put(Unknown Source)
    ... 2 more
Exception in thread "Thread-4" java.lang.RuntimeException: IOException in thread.
    at edu.sdsc.grid.io.irods.IRODSCommands$TransferThread.run(Unknown Source)
    at java.lang.Thread.run(Thread.java:619)
Caused by: java.net.SocketException: Connection reset
    at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:96)
    at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
    at edu.sdsc.grid.io.irods.IRODSCommands$TransferThread.put(Unknown Source)
    ... 2 more
Exception in thread "Thread-5" java.lang.RuntimeException: IOException in thread.
    at edu.sdsc.grid.io.irods.IRODSCommands$TransferThread.run(Unknown Source)
    at java.lang.Thread.run(Thread.java:619)
Caused by: java.net.SocketException: Broken pipe
    at java.net.SocketOutputStream.socketWrite0(Native Method)
    at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
    at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
    at edu.sdsc.grid.io.irods.IRODSCommands$TransferThread.put(Unknown Source)
    ... 2 more
java.io.IOException: read() -- couldn't read complete packet
    at edu.sdsc.grid.io.irods.IRODSCommands.read(Unknown Source)
    at edu.sdsc.grid.io.irods.IRODSCommands.readHeaderLength(Unknown Source)
    at edu.sdsc.grid.io.irods.IRODSCommands.readHeader(Unknown Source)
    at edu.sdsc.grid.io.irods.IRODSCommands.readMessage(Unknown Source)
    at edu.sdsc.grid.io.irods.IRODSCommands.irodsFunction(Unknown Source)
    at edu.sdsc.grid.io.irods.IRODSCommands.irodsFunction(Unknown Source)
    at edu.sdsc.grid.io.irods.IRODSCommands.operationComplete(Unknown Source)
    at edu.sdsc.grid.io.irods.IRODSCommands.put(Unknown Source)
    at edu.sdsc.grid.io.irods.IRODSFile.copyFrom(Unknown Source)
    at edu.sdsc.grid.io.local.LocalFile.copyTo(Unknown Source)
    at testing.FileOperations.main(FileOperations.java:41)
Caused by: java.net.SocketException: iRODS socket connection is closed.
    ... 11 more

Anyone can help me?

[mw...@diceresearch.org]

Hello Aan,

Our Jargon development is currently in transition. Our new Java/Jargon developer Mike Conway

has just joined the team today. Welcome Mike !! I am sure Mike will address this problem and

other issues once he settles in.

Mike

[Shunde Zhang]

Hi Aan,

Jargon is going to use ports specified in your server config.

# svrPortRangeStart and svrPortRangeEnd - A range of port numbers can be

# specified for the server's parallel I/O communication port.

# svrPortRangeStart specifies the first allowable port number and

# svrPortRangeEnd specifies the end of the range.

# These can also be set via irodssetup which adjusts the

# config/irods.config file (setting SVR_PORT_RANGE_START/END).

# $svrPortRangeStart=20000;

# $svrPortRangeEnd=20199;

I think those are the default value. You can change them to something else. For us, we use "svrPortRangeStart=40000 svrPortRangeEnd=41000"

These ports are only used for multi-threading file transfer. You may need to check if these ports are opened on your firewall.

(It looks like a connection issue.)

Regards,

Shunde Zhang

Australian Research Collaboration Services (ARCS), "Providing collaboration services and tools for the Australian research community"

Email: shunde...@arcs.org.au

eResearch SA (eRSA), "Enabling Discovery, Innovation and Collaboration through eResearch"

Email: shunde...@eresearchsa.edu.au

School of Computer Science, The University of Adelaide, "Life Impact"

Email: shunde...@adelaide.edu.au

Phone: +61 8 83038265

[mconway]

Aan,

Just wanted to let you know I'm concentrating on this right now. I have a
test case I am trying to run against my newly installed local irods. Once
I get it to work or replicate the bug, I'll let you know.

If you are in a really big bind, we can make a quick fix to bump up the max
file size so that your puts don't kick off the parallel transfer code in
Jargon. I think that would serve as a work-around.

Thanks for your patience, I will spin up on this as fast as I can!

Mike

[Danilo Mendonça Oliveira]

I'm having the same problem. Setting the enviroment variables didn't work for me.

[Mike Conway]

Wow that is a really really old version of jargon?

Are you pulling this in from Maven?  Where did it come from?

MC

[Danilo Mendonça Oliveira]

Yes, it's from Maven.

It's the 4.0.2.1 version.