Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss

How do I scp extremely large files

29 views
Skip to first unread message

Mike

unread,
Feb 4, 2012, 9:03:22 PM2/4/12
to
I need to copy extremely large files (30-100G) from a remove server to
a machine where my code will be running. I have code in place now
that uses an sftp connection to scan the file directory watching for
files to show up for me to copy. My question is how do I read in
chunks of these large files and write them out in chunks? I cannot
hold the bytes in memory obviously. Any help or pseudo code is
greatly appreaciated!

Robert Klemme

unread,
Feb 6, 2012, 2:54:16 AM2/6/12
to
I am not sure I understand your question properly. Any Java library
which implements sftp or scp's protocols will have a means to copy
remote files or at least open remote files and obtain an InputStream
or Channel, from which you can read in chunks and store data locally.

If you need to do the transfer on a regularly basis maybe rsync is
better than cooking your own version of it.

Kind regards

robert

Mike

unread,
Feb 6, 2012, 3:54:59 PM2/6/12
to
On Feb 6, 2:54 am, Robert Klemme <shortcut...@googlemail.com> wrote:
> I am not sure I understand your question properly.  Any Java library
> which implements sftp or scp's protocols will have a means to copy
> remote files or at least open remote files and obtain an InputStream
> or Channel, from which you can read in chunks and store data locally.
>
> If you need to do the transfer on a regularly basis maybe rsync is
> better than cooking your own version of it.
>
> Kind regards
>
> robert

Thanks for the response Robert. rsync is not an option. I cannot use
any outside software, open-source or otherwise. I believe you stated
just what I need to do... obtain a channel to read chunks from and
store locally. I'm just a little fuzzy on the details. Haven't done
much with channels or inputStreams for that matter.

Lew

unread,
Feb 6, 2012, 8:21:54 PM2/6/12
to
Mike wrote:
> Thanks for the response Robert. rsync is not an option. I cannot use
> any outside software, open-source or otherwise. I believe you stated
> just what I need to do... obtain a channel to read chunks from and
> store locally. I'm just a little fuzzy on the details. Haven't done
> much with channels or inputStreams for that matter.

I gather Java doesn't count as "outside software".

You can read these to get a start with streams, readers, channels and all that:

<http://docs.oracle.com/javase/tutorial/essential/io/index.html>
<http://docs.oracle.com/javase/tutorial/networking/sockets/index.html>

the tutorials being a great place to start
<http://docs.oracle.com/javase/tutorial/reallybigindex.html>
if you just need the basics.

Oracle has a ton more Java documentation besides the tutorials, of course, as
does IBM Developerworks.

--
Lew

Eric Sosman

unread,
Feb 7, 2012, 8:12:39 AM2/7/12
to
On 2/6/2012 3:54 PM, Mike wrote:
>[...]
> Thanks for the response Robert. rsync is not an option. I cannot use
> any outside software, open-source or otherwise. [...]

How much time have you allotted for writing your own JVM,
Java compiler, operating system, and BIOS?

--
Eric Sosman
eso...@ieee-dot-org.invalid

Robert Klemme

unread,
Feb 7, 2012, 4:50:46 PM2/7/12
to
On 02/06/2012 09:54 PM, Mike wrote:
> On Feb 6, 2:54 am, Robert Klemme<shortcut...@googlemail.com> wrote:
>> I am not sure I understand your question properly. Any Java library
>> which implements sftp or scp's protocols will have a means to copy
>> remote files or at least open remote files and obtain an InputStream
>> or Channel, from which you can read in chunks and store data locally.
>>
>> If you need to do the transfer on a regularly basis maybe rsync is
>> better than cooking your own version of it.

> Thanks for the response Robert. rsync is not an option.

Why?

> I cannot use
> any outside software, open-source or otherwise.

Why aren't you able to use "outside software"? Or did you mean you are
not allowed to?

> I believe you stated
> just what I need to do... obtain a channel to read chunks from and
> store locally. I'm just a little fuzzy on the details. Haven't done
> much with channels or inputStreams for that matter.

IO is pretty basic - in any language. I am surprised you haven't been
exposed to it yet. For the details please follow Lew's advice / links.

Kind regards

robert

Arne Vajhøj

unread,
Feb 7, 2012, 5:02:41 PM2/7/12
to
On 2/7/2012 8:12 AM, Eric Sosman wrote:
> On 2/6/2012 3:54 PM, Mike wrote:
>> [...]
>> Thanks for the response Robert. rsync is not an option. I cannot use
>> any outside software, open-source or otherwise. [...]
>
> How much time have you allotted for writing your own JVM,
> Java compiler, operating system, and BIOS?

:-)

Arne


Arne Vajhøj

unread,
Feb 7, 2012, 5:09:23 PM2/7/12
to
I would assume your SFTP code has some method that allows
reading of data as a stream which will allow you to write as
a stream too.

Arne

Arne Vajhøj

unread,
Feb 7, 2012, 5:10:46 PM2/7/12
to
On 2/4/2012 9:03 PM, Mike wrote:
Note that scp and sftp is not the same, so your first decision
is whether you want to use scp or sftp.

Arne

Arne Vajhøj

unread,
Feb 7, 2012, 5:12:53 PM2/7/12
to
On 2/6/2012 3:54 PM, Mike wrote:
> On Feb 6, 2:54 am, Robert Klemme<shortcut...@googlemail.com> wrote:
>> I am not sure I understand your question properly. Any Java library
>> which implements sftp or scp's protocols will have a means to copy
>> remote files or at least open remote files and obtain an InputStream
>> or Channel, from which you can read in chunks and store data locally.
>>
>> If you need to do the transfer on a regularly basis maybe rsync is
>> better than cooking your own version of it.
>
> Thanks for the response Robert. rsync is not an option. I cannot use
> any outside software, open-source or otherwise. I believe you stated
> just what I need to do... obtain a channel to read chunks from and
> store locally. I'm just a little fuzzy on the details. Haven't done
> much with channels or inputStreams for that matter.

scp and sftp are non trivial to implement.

If reading chunks is a problem, then you will not be
able to implement them.

You need to get whoever made that rule give you
permission to use a library that support one of
those protocols.

Arne



Eric Sosman

unread,
Feb 7, 2012, 8:47:13 PM2/7/12
to
Okay, my question was rather tongue-in-cheek. But the issue it's
intended to raise is the matter of "trust" in software: If the O.P. is
willing to trust an externally-supplied JVM, Java compiler, operating
system, and BIOS, the ban on "outside software" is obviously not
absolute. If it's not absolute, there must be some procedure,
somewhere, that declares "THIS outside software is acceptable; THAT
outside software is not." And (here's the exciting conclusion) if the
allowed/forbidden procedure bans rsync and its sixteen years of
development history in favor of a not-yet-written, not-yet-debugged,
not-even-designed half-hearted home-grown imitation, ... Well, the
procedure may have found an anatomically unlikely place to put its head.

--
Eric Sosman
eso...@ieee-dot-org.invalid

Arne Vajhøj

unread,
Feb 7, 2012, 8:56:05 PM2/7/12
to
I agree.

But weird rules are sometimes seen.

Arne

Mike

unread,
Feb 8, 2012, 7:29:00 PM2/8/12
to
Thanks for the advice Lew. You got me on the right track and I
accomplished what I needed. You were correct, Java was obviously
allowed just not 3rd party libraries. Sorry for the confusion I added
to this thread.

Daniel Pitts

unread,
Feb 9, 2012, 12:19:28 AM2/9/12
to
rsync is not a library, but a full, well supported, mature program. It
is used in many production environments for many different purposes.
Hopefully you're not overlooking it because of political nonsense.

Roedy Green

unread,
Feb 9, 2012, 1:33:14 AM2/9/12
to
On Sat, 4 Feb 2012 18:03:22 -0800 (PST), Mike <mikesm...@gmail.com>
wrote, quoted or indirectly quoted someone who said :

>I need to copy extremely large files (30-100G) from a remove server to
>a machine where my code will be running.

I know you have legal requirement not to use "outside" code.

Perhaps a VPN or WebDav would pass.

http://mindprod.com/jgloss/vpn.html
http://mindprod.com/jgloss/webdav.html
--
Roedy Green Canadian Mind Products
http://mindprod.com
One of the most useful comments you can put in a program is
"If you change this, remember to change ?XXX? too".

Roedy Green

unread,
Feb 9, 2012, 1:37:40 AM2/9/12
to
On Sat, 4 Feb 2012 18:03:22 -0800 (PST), Mike <mikesm...@gmail.com>
wrote, quoted or indirectly quoted someone who said :

If your roll your own system, you might want to consider bundling the
small files and zipping as the Replicator does. You would have to
run a script on the server to unpack them. The Replicator only uploads
files that have changed. See
http://mindprod.com/webstart/replicator.html

If the files have only minor changes, you might consider hosting a
version control system on the server. The server can then check out
files that have changed. You get the advantage of atomic updates,
something you will not get with an FTP system. You also can check in
files from many sources directly to the server.
0 new messages