sending tarball to server using ssh?
Beowulf
PC using ssh. I think this is how I should do it, based on what I read at
http://www.hostlibrary.com/Remotebackupusingsshtarandcron-how-to.html but
I do not understand the reasoning behind some of it and would like to.
Step 1: Create the tarball of a directory
$ tar -zcpf ~/mytarball.tar.gz ~/mydirectory
Step 2: Send tarball to remote PC ssh server
$ cat ~/mytarball.tar.gz | ssh ip_of_remotePC "cd ~;cat > backup.tar.gz"
Step 1 is a no-brainer for me, as is the first part of Step 2. But that
last part of Step 2 eludes me. I get that 'cat ~/mytarball.tar.gz |' is
piping the tarball to the remote PC through SSH. But I do not get what is
going on with "cd ~;cat > backup.tar.gz". So anything in double quotes
are the 'ssh remoteip' is executed up on connection? The last part of
step 2, 'cat > backup.tar.gz' I do not understand--- what is cat using for
sourcefile???
I also assume that I do not need to set up remote keys and all that from
the link above, that that would not necessarily be needed, or am I wrong?
Won't I be prompted for keys/passwords? I understand the usefullness of
setting up remote keys if I wanted to do this as a cron job, but if not
then remote key setup would not be needed, correct?
Any help appreciated.
Chris F.A. Johnson
> I want to create a tarball backup of a directory, then send it to a remote
> PC using ssh. I think this is how I should do it, based on what I read at
> http://www.hostlibrary.com/Remotebackupusingsshtarandcron-how-to.html but
> I do not understand the reasoning behind some of it and would like to.
>
> Step 1: Create the tarball of a directory
> $ tar -zcpf ~/mytarball.tar.gz ~/mydirectory
>
> Step 2: Send tarball to remote PC ssh server
> $ cat ~/mytarball.tar.gz | ssh ip_of_remotePC "cd ~;cat > backup.tar.gz"
Why not use scp?
scp ~/mytarball.tar.gz ip_of_remotePC:
> Step 1 is a no-brainer for me, as is the first part of Step 2. But that
> last part of Step 2 eludes me. I get that 'cat ~/mytarball.tar.gz |' is
> piping the tarball to the remote PC through SSH. But I do not get what is
> going on with "cd ~;cat > backup.tar.gz". So anything in double quotes
> are the 'ssh remoteip' is executed up on connection? The last part of
> step 2, 'cat > backup.tar.gz' I do not understand--- what is cat using for
> sourcefile???
When no file is supplied on the command line, cat reads the
standard input (the output of "cat ~/mytarball.tar.gz").
> I also assume that I do not need to set up remote keys and all that from
> the link above, that that would not necessarily be needed, or am I wrong?
> Won't I be prompted for keys/passwords? I understand the usefullness of
> setting up remote keys if I wanted to do this as a cron job, but if not
> then remote key setup would not be needed, correct?
If you are using it at the command line, password authnetication is
OK (unnnecessary and annoying, but OK).
--
Chris F.A. Johnson, author | <http://cfaj.freeshell.org>
Shell Scripting Recipes: | My code in this post, if any,
A Problem-Solution Approach | is released under the
2005, Apress | GNU General Public Licence
J.O. Aho
> I want to create a tarball backup of a directory, then send it to a remote
> PC using ssh. I think this is how I should do it, based on what I read at
> http://www.hostlibrary.com/Remotebackupusingsshtarandcron-how-to.html but
> I do not understand the reasoning behind some of it and would like to.
>
> Step 1: Create the tarball of a directory
> $ tar -zcpf ~/mytarball.tar.gz ~/mydirectory
>
> Step 2: Send tarball to remote PC ssh server
> $ cat ~/mytarball.tar.gz | ssh ip_of_remotePC "cd ~;cat > backup.tar.gz"
>
> But I do not get what is going on with "cd ~;cat > backup.tar.gz".
'cd ~' makes you to change directory to your remote machines home directory.
';' tells just that it's the end of the first command and do the next even if
the first one fails
'cat > backup.tar.gz' this is used to redirect the stuff from stdin to the
file backup.tar.gz.
> So anything in double quotes are the 'ssh remoteip' is executed up on connection?
The double quotes are there due the ';', if you do the same command without
the double quotes
cat ~/mytarball.tar.gz | ssh ip_of_remotePC cd ~;cat > backup.tar.gz
then the 'cat > backup.tar.gz' would be executed on the local machine and not
on the remote machine, this would lead to an empty new file on your local machine.
> step 2, 'cat > backup.tar.gz' I do not understand--- what is cat using for
> sourcefile???
stdin
> Won't I be prompted for keys/passwords?
Of course you will, if not setup keys.
> I understand the usefullness of
> setting up remote keys if I wanted to do this as a cron job, but if not
> then remote key setup would not be needed, correct?
I can't for sure say, if it will work, I do myself prefer to use sftp when
moving files over ssh.
//Aho
Beowulf
> 'cat > backup.tar.gz' this is used to redirect the stuff from stdin to
> the file backup.tar.gz.
So is stdin in this case the content being piped to ssh, so that "cat >"
is using that piped stream?
Beowulf
> When no file is supplied on the command line, cat reads the
> standard input (the output of "cat ~/mytarball.tar.gz").
Ok. I should have read the above before relying to Aho's reply. I think I
get it now.
> If you are using it at the command line, password authnetication is
> OK (unnnecessary and annoying, but OK).
ok so I do not need to first up keys on each end, I can just do the
commands as I originally indicated, and supply passwords when prompted? I
want to keep it simple at first, skip the keygen stuff, etc., go with
prompted passwords at first to get the basics down.
J.O. Aho
No, the content just passes throe the remote machines standard in for ssh, but
yes with help of the 'cat' the stdin in is directed to stdout and which in it
turn is piped with th '>' to the file.
//Aho
Johan Lindquist
Beowulf was all like, "Dude,
> I want to create a tarball backup of a directory, then send it to a remote
> PC using ssh. I think this is how I should do it, based on what I read at
> http://www.hostlibrary.com/Remotebackupusingsshtarandcron-how-to.html but
> I do not understand the reasoning behind some of it and would like to.
>
> Step 1: Create the tarball of a directory
> $ tar -zcpf ~/mytarball.tar.gz ~/mydirectory
>
> Step 2: Send tarball to remote PC ssh server
> $ cat ~/mytarball.tar.gz | ssh ip_of_remotePC "cd ~;cat > backup.tar.gz"
step 1+2:
tar -cf - yourdirectory | ssh remote "gzip -c > backup.tar.gz"
The cd ~ is done when you login so I'm unsure why that's there, and
you normally don't need -p when creating the archive - only when
extracting.
Or more in line with the original recepie, unless you have something
in particular against using scp tho, it might be less convoluted than
the whole useless use of cat above:
step 2:
scp ~/yourtarball.tar.gz remote:.
--
Time flies like an arrow, fruit flies like a banana. Perth ---> *
20:25:38 up 27 days, 18:07, 6 users, load average: 0.21, 0.12, 0.10
Linux 2.6.18.1 x86_64 GNU/Linux Registered Linux user #261729
Jeroen Geilman
> I want to create a tarball backup of a directory, then send it to a remote
> PC using ssh. I think this is how I should do it, based on what I read at
> http://www.hostlibrary.com/Remotebackupusingsshtarandcron-how-to.html but
> I do not understand the reasoning behind some of it and would like to.
>
> Step 1: Create the tarball of a directory
> $ tar -zcpf ~/mytarball.tar.gz ~/mydirectory
>
> Step 2: Send tarball to remote PC ssh server
> $ cat ~/mytarball.tar.gz | ssh ip_of_remotePC "cd ~;cat > backup.tar.gz"
Step 1:
#tar cz mydirectory | ssh us...@remote.machine "cat > mytarball.tgz"
Zip it up first; it's more efficient on the network.
Note that it may not be /faster/, unless you're sending it over the Internet
and the compression rate would be very high (like ASCII text would be).
Yes, you are in this case sending the raw output of tar gz over the
network - why not ?
On the remote end you just stream it into a file and you're done.
--
All your bits are belong to us.
Davorin Vlahovic
> Beowulf wrote:
>
>> I want to create a tarball backup of a directory, then send it to a remote
>> PC using ssh. I think this is how I should do it, based on what I read at
>> http://www.hostlibrary.com/Remotebackupusingsshtarandcron-how-to.html but
>> I do not understand the reasoning behind some of it and would like to.
>>
>> Step 1: Create the tarball of a directory
>> $ tar -zcpf ~/mytarball.tar.gz ~/mydirectory
>>
>> Step 2: Send tarball to remote PC ssh server
>> $ cat ~/mytarball.tar.gz | ssh ip_of_remotePC "cd ~;cat > backup.tar.gz"
>
> Step 1:
>
> #tar cz mydirectory | ssh us...@remote.machine "cat > mytarball.tgz"
tar cvjf mydirtarred.tar.bz2 mydirectory
scp mydirtarred.tar.bz2 us...@remote.machine:.
--
Uspjesne regije, tvrtke, muskarci i zene znaju da je uvijek bolje biti
prvorazredna verzija sebe nego drugorazredna verzija nekog drugog.
Sybren Stuvel
> 'cd ~' makes you to change directory to your remote machines home
> directory.
Yeah, but don't you end up in your homedir anyway if you ssh to the
machine?
> I can't for sure say, if it will work, I do myself prefer to use
> sftp when moving files over ssh.
I generally use scp.
Sybren
--
Sybren Stüvel
Stüvel IT - http://www.stuvel.eu/
Sybren Stuvel
> ok so I do not need to first up keys on each end, I can just do the
> commands as I originally indicated, and supply passwords when
> prompted? I want to keep it simple at first, skip the keygen stuff,
> etc., go with prompted passwords at first to get the basics down.
If you don't bother with keys, you'll be prompted for a password.
Once you get the hang of things, creating a key and using that is
quite easy:
ssh-keygen -t rsa -b 2048
ssh-copy-id username@someserver
After that, you should be able to log in without password. Once you
get the hang of that, check out ssh-agent. If you want to learn more,
just ask! :)
J.O. Aho
> J.O. Aho enlightened us with:
>> 'cd ~' makes you to change directory to your remote machines home
>> directory.
>
> Yeah, but don't you end up in your homedir anyway if you ssh to the
> machine?
You should end up in your home directory, but thats not necessary that you
will on all systems, specially those that don't have a system that supports
home directories, but has added that with some extra hacks.
>> I can't for sure say, if it will work, I do myself prefer to use
>> sftp when moving files over ssh.
>
> I generally use scp.
Works fine if you moving files in one direction, I tend to end up that I want
to move in both directions.
//Aho
Dan N
> I want to create a tarball backup of a directory, then send it to a remote
> PC using ssh.
You might want to consider using rsync. Google 'rsync backup'.
Dan
Johan Lindquist
J.O. Aho was all like, "Dude,
> Sybren Stuvel wrote:
>> J.O. Aho enlightened us with:
>>> I can't for sure say, if it will work, I do myself prefer to use
>>> sftp when moving files over ssh.
>>
>> I generally use scp.
>
> Works fine if you moving files in one direction, I tend to end up
> that I want to move in both directions.
I've always been able to (and used it to) copy files both to and from
other hosts with scp. Is there any case in which this shouldn't work?
--
Time flies like an arrow, fruit flies like a banana. Perth ---> *
09:43:34 up 28 days, 7:25, 3 users, load average: 0.22, 0.16, 0.27
J.O. Aho
> So anyway, it was like, 09:23 CET Dec 07 2006, you know? Oh, and, yeah,
> J.O. Aho was all like, "Dude,
>> Sybren Stuvel wrote:
>>> J.O. Aho enlightened us with:
>
>>>> I can't for sure say, if it will work, I do myself prefer to use
>>>> sftp when moving files over ssh.
>>> I generally use scp.
>> Works fine if you moving files in one direction, I tend to end up
>> that I want to move in both directions.
>
> I've always been able to (and used it to) copy files both to and from
> other hosts with scp. Is there any case in which this shouldn't work?
When you aren't sure about spellings on the file remotely or unsure about it's
location.
//Aho
Johan Lindquist
Wildcards work on both ends, you know. But I guess what you're driving
at is that with an [s]ftp client you can locate the file manually and
then transfer it, without having to first use ssh and then scp.
This does not in any way mean that scp only "works fine if you['re]
moving files in one direction", however.
--
Time flies like an arrow, fruit flies like a banana. Perth ---> *
11:29:16 up 28 days, 9:10, 3 users, load average: 0.27, 0.18, 0.11
Beowulf
> #tar cz mydirectory | ssh us...@remote.machine "cat > mytarball.tgz"
So the above sends the tarball data directly across the network, rather
than even using local hard drive space? I am guessing the tarball creation
is happening totally in RAM then? or not? (if yes, then would a minimum
amount of RAM be needed to handle very large tarballs using this method?)
Beowulf
> I've always been able to (and used it to) copy files both to and from
> other hosts with scp. Is there any case in which this shouldn't work?
The end goal, perhaps not apparent in the (my) original post, is to be
able to automate with a cron job the tarball creation and sending to the
remote server. Initially I need to figure out how to do this manually.
So, can scp do this too as a cron job? I need to figure out how to do the
ssh keygen stuff for ssh next, as part of the ssh tarball movement across
a network automatically using cron. (or perhaps using scp).
J.O. Aho
It don't require more RAM for a larger tarballs than for small ones, as it
will pipe each piece of the tarball as soon as it has been created. Will just
take longer time.
//Aho
Sybren Stuvel
> about it's location.
That's where public key authentication and bash-completion comes into
play. I can use the tab key to complete the filename, even when it's a
remote file.
To OP: if you want to do things from a cronjob, it's easiest to create
a small shell script that does what you want, and then run that script
from cron. Then you can easily perform multiple commands in sequence,
like tarring a directory and scp'ing it to another machine.
Beowulf
> It don't require more RAM for a larger tarballs than for small ones, as
> it will pipe each piece of the tarball as soon as it has been created.
> Will just take longer time.
>
Ok. Thank you for clarifying that.
Sybren Stuvel
> So the above sends the tarball data directly across the network, rather
> than even using local hard drive space?
Yep.
> I am guessing the tarball creation is happening totally in RAM then?
It is. Tar creates the tarball data, it is then sent to gzip, which is
a "block compressor". In other words: gzip doesn't need the entire
tarball to start compressing. As soon as it has compressed a block of
data, it sends it to stdout, where it is picked up by ssh and sent
through the network.
The memory required for all of this is no more than a few blocks + the
memory required by the running programs themselves.
Johan Lindquist
Beowulf was all like, "Dude,
The one-step example given by myself and others (you have a couple
of variants by now, which one works best for you depends on whether
your particular bottleneck is the network (compress before you send it
or use ssh with compression), the local host (compress on the remote
host) or the remote host (compress on the local host and avoid ssh
compression)) will work just fine from cron.
Regardless of using scp or ssh (it's part of the same package and
uses the same server component to connect to) you'll need to generate
keys to be able to automate it since you can't very easily enter the
passwords from an automated script.
Bear in mind that tar and ssh/scp/sftp isn't the only, and probably
not the most common, way to automate a backup across a network. The
rsync suggestion might be something you should look into rather than
busy yourself trying to figure out how to implement a needlessly
complex routine.
--
Time flies like an arrow, fruit flies like a banana. Perth ---> *
15:20:31 up 28 days, 13:02, 3 users, load average: 0.06, 0.07, 0.08
Steve Ackman
+0100, J.O. Aho wrote:
> Johan Lindquist wrote:
>> I've always been able to (and used it to) copy files both to and from
>> other hosts with scp. Is there any case in which this shouldn't work?
>
> When you aren't sure about spellings on the file remotely or unsure about it's
> location.
In that case, I ssh to the remote machine and scp the
file to where I'm sitting.
Beowulf
world:
..
> Bear in mind that tar and ssh/scp/sftp isn't the only, and probably
> not the most common, way to automate a backup across a network. The
> rsync suggestion might be something you should look into rather than
> busy yourself trying to figure out how to implement a needlessly
> complex routine.
I thought I read somewhere that rsync was not recommended because of
security issues, that it was insecure to use or have an rsync port open or
something?
Beowulf
..
> Once you get the hang of things, creating a key and using that is
> quite easy:
>
> ssh-keygen -t rsa -b 2048
> ssh-copy-id username@someserver
I will give it a try. I do not know what it is, but my brain just freezes
up on the whole keys issues, PGP, public keys, private keys, etc. I am
missing those neurons. I was top dog in Calculus II, I can tell you all
about DNA decryption and encryption, but on keys I go brain dead. I need
to find some website that really explains it like I am an 8 year old.
:)
Sybren Stuvel
> I will give it a try. I do not know what it is, but my brain just
> freezes up on the whole keys issues, PGP, public keys, private keys,
> etc. I am missing those neurons.
Think of a two-key lock. If you lock it with key A you can only unlock
it with key B, and the other way around. You keep key A to yourself,
and you spread key B across the world.
Now if someone finds a lock, and can open it with key B, they are 100%
sure that it was closed with key A. Since you are the only one in
posession of key A, they can be 100% sure you closed the lock.
This is basically how it works. The server that you try to SSH to has
a copy of key B, and thus it can identify you.
I guess you already figured that key A is called your private key, and
key B your public key.
Beowulf
> Think of a two-key lock. If you lock it with key A you can only unlock
> it with key B, and the other way around. You keep key A to yourself,
> and you spread key B across the world.
But key A could also unlock the lock locked with key A, correct? Or would
the person with key A also need to have a copy of key B to unlock the lock
they locked with their key A?
> Now if someone finds a lock, and can open it with key B, they are 100%
> sure that it was closed with key A. Since you are the only one in
> posession of key A, they can be 100% sure you closed the lock.
>
> This is basically how it works. The server that you try to SSH to has
> a copy of key B, and thus it can identify you.
>
> I guess you already figured that key A is called your private key, and
> key B your public key.
...
That is actually perhaps the best explanation I have read, thank you. It
is hard to conceptualize of such a physical lock, I suppose such a lock
exists only in the digital realm-- if someone could make such a physical
lock so I could conceive of it easier, ugh. I know it is a principle I
just need to get down then the light will shine on my brain and it will
make sense, I need to find a way to solidify thw two keys and one lock
principle.
Davorin Vlahovic
> I thought I read somewhere that rsync was not recommended because of
> security issues, that it was insecure to use or have an rsync port open or
> something?
Well, you can always use sshfs :)
Sybren Stuvel
>> Think of a two-key lock. If you lock it with key A you can only
>> unlock it with key B, and the other way around. You keep key A to
>> yourself, and you spread key B across the world.
>
> But key A could also unlock the lock locked with key A, correct?
Nope.
> Or would the person with key A also need to have a copy of key B to
> unlock the lock they locked with their key A?
Yes. Since you create both keys in one go, that's not a practical
limitation, though.
> That is actually perhaps the best explanation I have read, thank
> you.
Thanks!
> It is hard to conceptualize of such a physical lock, I suppose such
> a lock exists only in the digital realm-- if someone could make such
> a physical lock so I could conceive of it easier, ugh.
Think of it as a lock with two keyholes. The left keyhole can only
rotate to the left, and the right hole can only rotate to the right.
I'm sure you can think up the rest of it ;-)
> I know it is a principle I just need to get down then the light will
> shine on my brain and it will make sense, I need to find a way to
> solidify thw two keys and one lock principle.
If you can get your brain around it, RSA encryption is quite
understandable. http://en.wikipedia.org/wiki/RSA has a good
explanation, and I've written a pure-python implementation at
http://www.stuvel.eu/rsa.
Sybren Stuvel
> I thought I read somewhere that rsync was not recommended because of
> security issues, that it was insecure to use or have an rsync port
> open or something?
Nowadays, rsync uses ssh by default ;-)
Beowulf
..
> If you can get your brain around it,...
Well, as Shakespeare said "There's the rub." :)
Beowulf
...
> Think of a two-key lock. If you lock it with key A you can only unlock
> it with key B, and the other way around. You keep key A to yourself, and
> you spread key B across the world.
>
> Now if someone finds a lock, and can open it with key B, they are 100%
> sure that it was closed with key A. Since you are the only one in
> posession of key A, they can be 100% sure you closed the lock.
I just found this analogy also on Wikipedia, helps a bit more, starting to
sink in a *little* but my brain really has a hard time with the whole key
thing; I gotta go take some aspirin... ;)
From http://en.wikipedia.org/wiki/Asymmetric_key_cryptography
..."One analogy is that of a locked store front door with a
mail slot. The mail slot is exposed and accessible to the public; its
location (the street address) is in essence the public key. Anyone knowing
the street address can go to the door and drop a written message through
the slot. However, only the person who possesses the matching private key,
the store owner in this case, can open the door and read the message...."