I need to copy extremely large files (30-100G) from a remove server to
a machine where my code will be running. I have code in place now
that uses an sftp connection to scan the file directory watching for
files to show up for me to copy. My question is how do I read in
chunks of these large files and write them out in chunks? I cannot
hold the bytes in memory obviously. Any help or pseudo code is
greatly appreaciated!
On 5 Feb., 03:03, Mike <mikesmith...@gmail.com> wrote:
> I need to copy extremely large files (30-100G) from a remove server to
> a machine where my code will be running. I have code in place now
> that uses an sftp connection to scan the file directory watching for
> files to show up for me to copy. My question is how do I read in
> chunks of these large files and write them out in chunks? I cannot
> hold the bytes in memory obviously. Any help or pseudo code is
> greatly appreaciated!
I am not sure I understand your question properly. Any Java library
which implements sftp or scp's protocols will have a means to copy
remote files or at least open remote files and obtain an InputStream
or Channel, from which you can read in chunks and store data locally.
If you need to do the transfer on a regularly basis maybe rsync is
better than cooking your own version of it.
On Feb 6, 2:54 am, Robert Klemme <shortcut...@googlemail.com> wrote:
> I am not sure I understand your question properly. Any Java library
> which implements sftp or scp's protocols will have a means to copy
> remote files or at least open remote files and obtain an InputStream
> or Channel, from which you can read in chunks and store data locally.
> If you need to do the transfer on a regularly basis maybe rsync is
> better than cooking your own version of it.
> Kind regards
> robert
Thanks for the response Robert. rsync is not an option. I cannot use
any outside software, open-source or otherwise. I believe you stated
just what I need to do... obtain a channel to read chunks from and
store locally. I'm just a little fuzzy on the details. Haven't done
much with channels or inputStreams for that matter.
Mike wrote:
> Thanks for the response Robert. rsync is not an option. I cannot use
> any outside software, open-source or otherwise. I believe you stated
> just what I need to do... obtain a channel to read chunks from and
> store locally. I'm just a little fuzzy on the details. Haven't done
> much with channels or inputStreams for that matter.
I gather Java doesn't count as "outside software".
You can read these to get a start with streams, readers, channels and all that:
> On Feb 6, 2:54 am, Robert Klemme<shortcut...@googlemail.com> wrote:
>> I am not sure I understand your question properly. Any Java library
>> which implements sftp or scp's protocols will have a means to copy
>> remote files or at least open remote files and obtain an InputStream
>> or Channel, from which you can read in chunks and store data locally.
>> If you need to do the transfer on a regularly basis maybe rsync is
>> better than cooking your own version of it.
> Thanks for the response Robert. rsync is not an option.
Why?
> I cannot use
> any outside software, open-source or otherwise.
Why aren't you able to use "outside software"? Or did you mean you are not allowed to?
> I believe you stated
> just what I need to do... obtain a channel to read chunks from and
> store locally. I'm just a little fuzzy on the details. Haven't done
> much with channels or inputStreams for that matter.
IO is pretty basic - in any language. I am surprised you haven't been exposed to it yet. For the details please follow Lew's advice / links.
> On 2/6/2012 3:54 PM, Mike wrote:
>> [...]
>> Thanks for the response Robert. rsync is not an option. I cannot use
>> any outside software, open-source or otherwise. [...]
> How much time have you allotted for writing your own JVM,
> Java compiler, operating system, and BIOS?
> I need to copy extremely large files (30-100G) from a remove server to
> a machine where my code will be running. I have code in place now
> that uses an sftp connection to scan the file directory watching for
> files to show up for me to copy. My question is how do I read in
> chunks of these large files and write them out in chunks? I cannot
> hold the bytes in memory obviously. Any help or pseudo code is
> greatly appreaciated!
I would assume your SFTP code has some method that allows
reading of data as a stream which will allow you to write as
a stream too.
> I need to copy extremely large files (30-100G) from a remove server to
> a machine where my code will be running. I have code in place now
> that uses an sftp connection to scan the file directory watching for
> files to show up for me to copy. My question is how do I read in
> chunks of these large files and write them out in chunks? I cannot
> hold the bytes in memory obviously. Any help or pseudo code is
> greatly appreaciated!
Note that scp and sftp is not the same, so your first decision
is whether you want to use scp or sftp.
> On Feb 6, 2:54 am, Robert Klemme<shortcut...@googlemail.com> wrote:
>> I am not sure I understand your question properly. Any Java library
>> which implements sftp or scp's protocols will have a means to copy
>> remote files or at least open remote files and obtain an InputStream
>> or Channel, from which you can read in chunks and store data locally.
>> If you need to do the transfer on a regularly basis maybe rsync is
>> better than cooking your own version of it.
> Thanks for the response Robert. rsync is not an option. I cannot use
> any outside software, open-source or otherwise. I believe you stated
> just what I need to do... obtain a channel to read chunks from and
> store locally. I'm just a little fuzzy on the details. Haven't done
> much with channels or inputStreams for that matter.
scp and sftp are non trivial to implement.
If reading chunks is a problem, then you will not be
able to implement them.
You need to get whoever made that rule give you
permission to use a library that support one of
those protocols.
> On 2/7/2012 8:12 AM, Eric Sosman wrote:
>> On 2/6/2012 3:54 PM, Mike wrote:
>>> [...]
>>> Thanks for the response Robert. rsync is not an option. I cannot use
>>> any outside software, open-source or otherwise. [...]
>> How much time have you allotted for writing your own JVM,
>> Java compiler, operating system, and BIOS?
> :-)
Okay, my question was rather tongue-in-cheek. But the issue it's
intended to raise is the matter of "trust" in software: If the O.P. is
willing to trust an externally-supplied JVM, Java compiler, operating
system, and BIOS, the ban on "outside software" is obviously not
absolute. If it's not absolute, there must be some procedure,
somewhere, that declares "THIS outside software is acceptable; THAT
outside software is not." And (here's the exciting conclusion) if the
allowed/forbidden procedure bans rsync and its sixteen years of
development history in favor of a not-yet-written, not-yet-debugged,
not-even-designed half-hearted home-grown imitation, ... Well, the
procedure may have found an anatomically unlikely place to put its head.
> On 2/7/2012 5:02 PM, Arne Vajhøj wrote:
>> On 2/7/2012 8:12 AM, Eric Sosman wrote:
>>> On 2/6/2012 3:54 PM, Mike wrote:
>>>> [...]
>>>> Thanks for the response Robert. rsync is not an option. I cannot use
>>>> any outside software, open-source or otherwise. [...]
>>> How much time have you allotted for writing your own JVM,
>>> Java compiler, operating system, and BIOS?
>> :-)
> Okay, my question was rather tongue-in-cheek. But the issue it's
> intended to raise is the matter of "trust" in software: If the O.P. is
> willing to trust an externally-supplied JVM, Java compiler, operating
> system, and BIOS, the ban on "outside software" is obviously not
> absolute. If it's not absolute, there must be some procedure,
> somewhere, that declares "THIS outside software is acceptable; THAT
> outside software is not." And (here's the exciting conclusion) if the
> allowed/forbidden procedure bans rsync and its sixteen years of
> development history in favor of a not-yet-written, not-yet-debugged,
> not-even-designed half-hearted home-grown imitation, ... Well, the
> procedure may have found an anatomically unlikely place to put its head.
> Mike wrote:
> > Thanks for the response Robert. rsync is not an option. I cannot use
> > any outside software, open-source or otherwise. I believe you stated
> > just what I need to do... obtain a channel to read chunks from and
> > store locally. I'm just a little fuzzy on the details. Haven't done
> > much with channels or inputStreams for that matter.
> I gather Java doesn't count as "outside software".
> You can read these to get a start with streams, readers, channels and all that:
> Oracle has a ton more Java documentation besides the tutorials, of course, as
> does IBM Developerworks.
> --
> Lew
Thanks for the advice Lew. You got me on the right track and I
accomplished what I needed. You were correct, Java was obviously
allowed just not 3rd party libraries. Sorry for the confusion I added
to this thread.
> On Feb 6, 8:21 pm, Lew<lewbl...@gmail.com> wrote:
>> Mike wrote:
>>> Thanks for the response Robert. rsync is not an option. I cannot use
>>> any outside software, open-source or otherwise. I believe you stated
>>> just what I need to do... obtain a channel to read chunks from and
>>> store locally. I'm just a little fuzzy on the details. Haven't done
>>> much with channels or inputStreams for that matter.
>> I gather Java doesn't count as "outside software".
>> You can read these to get a start with streams, readers, channels and all that:
>> Oracle has a ton more Java documentation besides the tutorials, of course, as
>> does IBM Developerworks.
>> --
>> Lew
> Thanks for the advice Lew. You got me on the right track and I
> accomplished what I needed. You were correct, Java was obviously
> allowed just not 3rd party libraries. Sorry for the confusion I added
> to this thread.
rsync is not a library, but a full, well supported, mature program. It is used in many production environments for many different purposes. Hopefully you're not overlooking it because of political nonsense.
On Sat, 4 Feb 2012 18:03:22 -0800 (PST), Mike <mikesmith...@gmail.com>
wrote, quoted or indirectly quoted someone who said :
>I need to copy extremely large files (30-100G) from a remove server to
>a machine where my code will be running. I have code in place now
>that uses an sftp connection to scan the file directory watching for
>files to show up for me to copy. My question is how do I read in
>chunks of these large files and write them out in chunks? I cannot
>hold the bytes in memory obviously. Any help or pseudo code is
>greatly appreaciated!
If your roll your own system, you might want to consider bundling the
small files and zipping as the Replicator does. You would have to
run a script on the server to unpack them. The Replicator only uploads
files that have changed. See
http://mindprod.com/webstart/replicator.html
If the files have only minor changes, you might consider hosting a
version control system on the server. The server can then check out
files that have changed. You get the advantage of atomic updates,
something you will not get with an FTP system. You also can check in
files from many sources directly to the server.
-- Roedy Green Canadian Mind Products
http://mindprod.com One of the most useful comments you can put in a program is "If you change this, remember to change ?XXX? too".
