Copied files take up more space than originals
Dave
The 'source' machine is a Sun Netra T1 with Solaris 10 (03/2005).
The 'destination' machine is a 'zone' configured on a Sun Blade 1000 running
Solaris 10 update 7 (05/2009), using the following command:
drkirkby@kestrel:~$ scp -pr sage-4.2 sage@t2nb:
(The time stamp on the 'scp' command is September of this year, so the version
of Sun's SSH is not that old - it must have been recently patched.)
Executing:
drkirkby@kestrel:~$ du -ks sage-4.2
6923547 sage-4.2
on the 'source' machine, I see I have about 7 GB of data.
Running the same command on the destination machine:
-bash-3.00$ du -ks sage-4.2
10692359 sage-4.2
I see I have around 10 GB, and the copy has not completed yet!
I'm puzzled why this should be. There are no core files, and as far as I am
aware, no sparse files. There are however lots of symbolic links, and one hard
link.
Any thoughts as to why the disk spaced used on the destination is much greater
than on the source? I thought the '-p' option on scp would have copied
everything unchanged. Clearly something is amiss.
I've got 41 GB disk space free, so I hope I'm not going to fill up the whole
disk copying over 7 GB!
Dave
--
I respectfully request that this message is not archived by companies as
unscrupulous as 'Experts Exchange' . In case you are unaware,
'Experts Exchange' take questions posted on the web and try to find
idiots stupid enough to pay for the answers, which were posted freely
by others. They are leeches.
Richard B. Gilbert
> I'm trying to copy a directory from one computer to another using scp.
>
> The 'source' machine is a Sun Netra T1 with Solaris 10 (03/2005).
>
> The 'destination' machine is a 'zone' configured on a Sun Blade 1000
> running Solaris 10 update 7 (05/2009), using the following command:
>
> drkirkby@kestrel:~$ scp -pr sage-4.2 sage@t2nb:
>
> (The time stamp on the 'scp' command is September of this year, so the
> version of Sun's SSH is not that old - it must have been recently patched.)
>
>
> Executing:
>
> drkirkby@kestrel:~$ du -ks sage-4.2
> 6923547 sage-4.2
>
> on the 'source' machine, I see I have about 7 GB of data.
>
> Running the same command on the destination machine:
>
> -bash-3.00$ du -ks sage-4.2
> 10692359 sage-4.2
>
> I see I have around 10 GB, and the copy has not completed yet!
>
> I'm puzzled why this should be. There are no core files, and as far as I
> am aware, no sparse files. There are however lots of symbolic links, and
> one hard link.
>
> Any thoughts as to why the disk spaced used on the destination is much
> greater than on the source? I thought the '-p' option on scp would have
> copied everything unchanged. Clearly something is amiss.
>
> I've got 41 GB disk space free, so I hope I'm not going to fill up the
> whole disk copying over 7 GB!
>
Have you tried comparing file sizes between the originals and the
copies? If they are not the same, what does diff tell you?
Doug Freyburger
>
> I'm trying to copy a directory from one computer to another using scp.
>
>
> 6923547 sage-4.2
>
> on the 'source' machine, I see I have about 7 GB of data.
>
> Running the same command on the destination machine:
>
> -bash-3.00$ du -ks sage-4.2
> 10692359 sage-4.2
>
> I see I have around 10 GB, and the copy has not completed yet!
>
> I'm puzzled why this should be. There are no core files, and as far as I am
> aware, no sparse files. There are however lots of symbolic links, and one hard
> link.
Bingo symbolic links. Tools like tar, cpio and so on understand
symbolic links. Tools like cp consider files to be files. Start
comparing symbolic links and you'll discover they are being expanded to
files at the destination.
> Any thoughts as to why the disk spaced used on the destination is much greater
> than on the source? I thought the '-p' option on scp would have copied
> everything unchanged. Clearly something is amiss.
Permissions and ownerships unchanged is not the same as checking if its
a symbolic link.
unruh
Or use rsync
rsync -av localfilename remotemachinename:/remote/file/name
It also checks to make sure tha tthe copy is the same as the original (
no bit flips during the transfer). It also handles symbolic links and
sparse files properly ( sparse file-- some files with lots of nulls are
stored in a "compressed" format, and if they are expanded out can be
much much larger than the original file. )
Dave
The problem is the symbolic links are not being copied properly:
On the source system, we see there is a file libsqlite3.so.0.8.6, and there are
two symbolic links to it.
lrwxrwxrwx 1 drkirkby other 19 Nov 21 12:22 libsqlite3.so ->
libsqlite3.so.0.8.6
lrwxrwxrwx 1 drkirkby other 19 Nov 21 12:22 libsqlite3.so.0 ->
libsqlite3.so.0.8.6
-rwxr-xr-x 1 drkirkby other 1397028 Nov 21 12:22 libsqlite3.so.0.8.6
On the destination system, I see:
-rwxr-xr-x 1 sage other 1397028 Nov 21 12:22 libsqlite3.so
-rwxr-xr-x 1 sage other 1397028 Nov 21 12:22 libsqlite3.so.0
-rwxr-xr-x 1 sage other 1397028 Nov 21 12:22 libsqlite3.so.0.8.6
So for all practical purposes, this one file has taken up 3x as much space on
the destination as it did on the source.
Is there any way to get scp to not do this?
I've had the same problem using the Solaris 'cp' command. Copying a similar lot
of files and making a tar file is creating one about 50% larger than using the
GNU 'cp' command with the '-a' (archive) option. The '-a' option is not POSIX.
From what I gather, POSIX is not too explicit about how 'cp' handles links -
though it might be only be hard links. Either way, copying files in Solaris
using the Solaris tools, seems to take up a lot more disk space if there are links.
Richard B. Gilbert
I think it did EXACTLY what you told it to do. It looks as if it copied
each file pointed to by the links you gave it. N links gives N
copies. Just what you asked for!
Doug Freyburger
>
> The problem is the symbolic links are not being copied properly:
No. The problem is you are using a tool that copies regular files and
you are using that tool on symbolic links. It's a conceptual problem on
how to use tools not a problem to be fixed by changing the tool.
> Is there any way to get scp to not do this?
( cd source_dir ; tar cf - . ) | ssh dest_host "cd dest_dir ; tar xpf -"
Or similar with cpio, pax, some type of dump. Starting on Solaris 10
even zip and unzip can be used for this purpose.
Using the right tool for the job makes all of the difference. Having a
concept of what each tool does matters.
> Either way, copying files in Solaris
> using the Solaris tools, seems to take up a lot more disk space if there are links.
Exactly as designed for a tool that reads a regular file and writes a
regular file.
Have even more fun - Create a pipe or socket special file in your source
tree and see the difference of how cp and tar behave. cp will correctly
open it as a regular file and read from it until it sends and EOF. In
other words it will hang until some program writes something to the pipe
then closes it. tar will correctly identify if as a special file and
send the special file information to the far end.
unruh
On 2009-12-22, Dave <f...@coo.com> wrote:
> Richard B. Gilbert wrote:
>> Dave wrote:
>>
>> Have you tried comparing file sizes between the originals and the
>> copies? If they are not the same, what does diff tell you?
>
> The problem is the symbolic links are not being copied properly:
>
> On the source system, we see there is a file libsqlite3.so.0.8.6, and there are
> two symbolic links to it.
>
> lrwxrwxrwx 1 drkirkby other 19 Nov 21 12:22 libsqlite3.so ->
> libsqlite3.so.0.8.6
> lrwxrwxrwx 1 drkirkby other 19 Nov 21 12:22 libsqlite3.so.0 ->
> libsqlite3.so.0.8.6
> -rwxr-xr-x 1 drkirkby other 1397028 Nov 21 12:22 libsqlite3.so.0.8.6
>
> On the destination system, I see:
>
> -rwxr-xr-x 1 sage other 1397028 Nov 21 12:22 libsqlite3.so
> -rwxr-xr-x 1 sage other 1397028 Nov 21 12:22 libsqlite3.so.0
> -rwxr-xr-x 1 sage other 1397028 Nov 21 12:22 libsqlite3.so.0.8.6
>
> So for all practical purposes, this one file has taken up 3x as much space on
> the destination as it did on the source.
>
> Is there any way to get scp to not do this?
Not as far as I know. Use something else--eg, rsync ( which uses ssh to
actually transfer the files)
>
> I've had the same problem using the Solaris 'cp' command. Copying a similar lot
> of files and making a tar file is creating one about 50% larger than using the
> GNU 'cp' command with the '-a' (archive) option. The '-a' option is not POSIX.
> From what I gather, POSIX is not too explicit about how 'cp' handles links -
> though it might be only be hard links. Either way, copying files in Solaris
> using the Solaris tools, seems to take up a lot more disk space if there are links.
rsync will also handle hard links properly.
>
> Dave
>
Nico Kadel-Garcia
> Is there any way to get scp to not do this?
No. SCP is based on the behavior of a *much* older tool called 'rcp',
and 'rcp' was never sophisticated enough to handle symlinks as
symlinks rather than as binary files.
I've not explored whether sftp can do this properly. But in general,
if you want efficient transmission of symlinks and/or hardlinks, use
rsync or rsync over SSH.
Dave
> On Dec 22, 4:24 pm, Dave <f...@coo.com> wrote:
>
>> Is there any way to get scp to not do this?
>
> No. SCP is based on the behavior of a *much* older tool called 'rcp',
> and 'rcp' was never sophisticated enough to handle symlinks as
> symlinks rather than as binary files.
On might hope that the developers would look at improving matters in this
respect, with an option which allows links to be copied properly.
> I've not explored whether sftp can do this properly. But in general,
> if you want efficient transmission of symlinks and/or hardlinks, use
> rsync or rsync over SSH.
Thank you.
hume.sp...@bofh.ca
> drkirkby@kestrel:~$ scp -pr sage-4.2 sage@t2nb:
As others have mentioned, scp doesn't act like an archiver, figuring out
what kind of file it is. (It's somewhat logical, since the remote system
might not have an equivalent file type...)
I'd do the following myself:
tar cf - sage-4.2 | ssh sage@t2nb tar xvf -
--
Brandon Hume - hume -> BOFH.Ca, http://WWW.BOFH.Ca/
unruh
> Nico Kadel-Garcia wrote:
>> On Dec 22, 4:24 pm, Dave <f...@coo.com> wrote:
>>
>>> Is there any way to get scp to not do this?
>>
>> No. SCP is based on the behavior of a *much* older tool called 'rcp',
>> and 'rcp' was never sophisticated enough to handle symlinks as
>> symlinks rather than as binary files.
>
> On might hope that the developers would look at improving matters in this
> respect, with an option which allows links to be copied properly.
Why? There are many better file copy commands ( like rsync). Why
reinvent the wheel yet again. What is it about scp that you feel that
you cannot use something else? scp is useful sometimes. Other times it
is not. Do not use it when it is not useful.
Jim Leonard
>
> rsync will also handle hard links properly.
Only if you specify -H as that is not default behavior.
ChrisS
it's a relatively small amount of data 'tar' may be easiest.
I live for rsync these days too. I'm not sure rsync comes with
Solaris 10 by default. I've added it via Blastwave or OpenCSW, but
SunFreeware tools is good for that too.
Good luck.
Chris
Doug Freyburger
> Nico Kadel-Garcia wrote:
>> Dave <f...@coo.com> wrote:
>
>>> Is there any way to get scp to not do this?
>
>> No. SCP is based on the behavior of a *much* older tool called 'rcp',
>> and 'rcp' was never sophisticated enough to handle symlinks as
>> symlinks rather than as binary files.
>
> On might hope that the developers would look at improving matters in this
> respect, with an option which allows links to be copied properly.
No. Why should the manufacturers of horse drawn carriages install all
of the safety features needed for the carriage to operate on an
Interstate highway? If you want to travel on an Interstate highway get
and use a car or truck not a carriage.
As discussed already tar and rsync are there for this use.
The basic concept of UNIX toolsmithing - Make small tools that do one
job well. Converting every tool to have as many options as EMACS is not
consistent with the UNIX philosophy. We already have EMACS and find and
a few other tools that are very complex. We don't need to morph basic
tools into them.
unruh
> On Dec 22, 3:47?pm, unruh <un...@wormhole.physics.ubc.ca> wrote:
>>
>> rsync will also handle hard links properly.
>
> Only if you specify -H as that is not default behavior.
The default behaviour does nothing at all. You must specify various
options in order to run rsync. The usual one is -a (for archive) and -v
( for verbose if you want to see which files are being transfered). As
he says, -H if you want hard links handled properly. There are many
options so skimming the man page is a good idea.
David Kirkby
Semi-related to this, I do need to find a way of copying files and
preserving links, like the '-a' option does on GNU 'cp'. I think
creating a tar archive and then extracting it might be one way, but
I'd like to avoid the need to create an intermediate file if
possible.
Dave
(PS, I am the original poster, also known as f...@coo.com depending on
what computer I happen to use)
Ian Collins
> On Dec 23, 3:21 pm, ChrisS <chris.sca...@gmail.com> wrote:
>> I would definitely use the tar solution or the rsync solution. Since
>> it's a relatively small amount of data 'tar' may be easiest.
>>
>> I live for rsync these days too. I'm not sure rsync comes with
>> Solaris 10 by default. I've added it via Blastwave or OpenCSW, but
>> SunFreeware tools is good for that too.
>>
>> Good luck.
>>
>> Chris
>
> Semi-related to this, I do need to find a way of copying files and
> preserving links, like the '-a' option does on GNU 'cp'. I think
> creating a tar archive and then extracting it might be one way, but
> I'd like to avoid the need to create an intermediate file if
> possible.
If you are using Solaris 10, use zfs pools and send/receive data.
--
Ian Collins
unruh
> On Dec 23, 3:21?pm, ChrisS <chris.sca...@gmail.com> wrote:
>> I would definitely use the tar solution or the rsync solution. ?Since
>> it's a relatively small amount of data 'tar' may be easiest.
>>
>> Solaris 10 by default. ?I've added it via Blastwave or OpenCSW, but
>> SunFreeware tools is good for that too.
>>
>> Good luck.
>>
>> Chris
>
> Semi-related to this, I do need to find a way of copying files and
> preserving links, like the '-a' option does on GNU 'cp'. I think
> creating a tar archive and then extracting it might be one way, but
> I'd like to avoid the need to create an intermediate file if
> possible.
>
> Dave
>
> (PS, I am the original poster, also known as f...@coo.com depending on
> what computer I happen to use)
So use rsync.
>
Nico Kadel-Garcia
> ["Followup-To:" header set to comp.security.ssh.]
> On 2009-12-23, Dave <f...@coo.com> wrote:
>
> > Nico Kadel-Garcia wrote:
> >> On Dec 22, 4:24 pm, Dave <f...@coo.com> wrote:
>
> >>> Is there any way to get scp to not do this?
>
> >> No. SCP is based on the behavior of a *much* older tool called 'rcp',
> >> and 'rcp' was never sophisticated enough to handle symlinks as
> >> symlinks rather than as binary files.
>
> > On might hope that the developers would look at improving matters in this
> > respect, with an option which allows links to be copied properly.
>
> Why? There are many better file copy commands ( like rsync). Why
> reinvent the wheel yet again. What is it about scp that you feel that
> you cannot use something else? scp is useful sometimes. Other times it
> is not. Do not use it when it is not useful.
Because there is a _huge_ deployed codebase out there, there is a
published API which would need significant revision, and making any
backwards incompatible changes to that API is hideously dangerous.
It's also impossible for the client to know, in advance, if the server
does or not support symlinks correctly, and vice versa. The risk is
huge, the workaround of 'rsync" already exists with excellent ssh
integration and well understood behavior, and there's plenty of other
work to do.
It's not that I personally wouldn't *like* to see SSH support
something like the GNU "cp -a": I'm just not that sure it's a wise
investment of deep developer time at this point.
D. Stussy
than at the source.
Fred Mobach
> On Dec 23, 3:21ï¿œpm, ChrisS <chris.sca...@gmail.com> wrote:
>> I would definitely use the tar solution or the rsync solution. ï¿œSince
>> it's a relatively small amount of data 'tar' may be easiest.
>>
>> I live for rsync these days too. ï¿œI'm not sure rsync comes with
>> Solaris 10 by default. ï¿œI've added it via Blastwave or OpenCSW, but
>> SunFreeware tools is good for that too.
>
> Semi-related to this, I do need to find a way of copying files and
> preserving links, like the '-a' option does on GNU 'cp'. I think
> creating a tar archive and then extracting it might be one way, but
> I'd like to avoid the need to create an intermediate file if
> possible.
The intermediate file can be a pipe :
tar zcf - <dir> | ssh <user>@<target> "cd <destdir> && tar zxf -"
--
Fred Mobach - fr...@mobach.nl
website : https://fred.mobach.nl
.... In God we trust ....
.. The rest we monitor ..
Casper H.S. Dik
>Semi-related to this, I do need to find a way of copying files and
>preserving links, like the '-a' option does on GNU 'cp'. I think
>creating a tar archive and then extracting it might be one way, but
>I'd like to avoid the need to create an intermediate file if
>possible.
cpio?
cpio -pmvd < list-of-files
across the network, use cpio -o | cpio -i
Casper
--
Expressed in this posting are my opinions. They are in no way related
to opinions held by my employer, Sun Microsystems.
Statements on Sun products included here are not gospel and may
be fiction rather than truth.
Doug Freyburger
> David Kirkby wrote:
>
>> Semi-related to this, I do need to find a way of copying files and
>> preserving links, like the '-a' option does on GNU 'cp'. I think
>> creating a tar archive and then extracting it might be one way, but
>> I'd like to avoid the need to create an intermediate file if
>> possible.
>
> If you are using Solaris 10, use zfs pools and send/receive data.
Can ZFS really be used to mirror where the receiving end is mounted for
regular use or is this more like the tar method of tunneling the data
into a pipie just using data at a different layer of the stack.
David Kirkby
Thank you Casper.
I was not aware of that. It seems it is also possible to copy using
tar on the same machine
Dave