Why does redirecting stderr prevent file list display?
Dave Dickerson
cpio will display a list of file names as it
writes to tape:
find . -print | cpio -oBcv -O /dev/rStp0
If I use the same command but redirect stderr
to a file, there is no display of file names:
find . -print | cpio -oBcv -O /dev/rStp0 2>/tmp/errors
Why would redirection of stderr prevent cpio (tar, too)
from displaying file names on stdout?
DDinAZ
Jean-Pierre Radley
Perhaps because the file names were written to stderr in the first place?
What's in your /tmp/errors file?
--
JP
Matt Schalit
I think cpio is printing to stderr, not stdout.
Because you redirected it, you can't see it. You
might be able to use the tee command to see the
output and grab the data:
find . -print | cpio -oBcv -O /dev/rStp0 2>&1 | tee /tmp/errors
Good Luck,
Matthew
Dave Dickerson
wrote:
If I were to write a program that was to become a staple utility
of a major operating system, I would, of course, not write my
standard output to stdout, but to stderr. Makes sense (NOT!).
So, where in the heck would I write error messages -- stdin?
:)
And yes, in the example, /tmp/errors would contain cpio's file
listing (along with any error messages, should they be
generated).
So, what does one do if one wants standard output
(file names) written to stdout and error messages (if any)
written to stderr?
DDinAZ
Dave Dickerson
wrote:
Thanks for the tee tip. I'd tried it before but in a different
and unsuccessful way.
However, your example doesn't quite do what I need.
I want error messages (if any) written to /tmp/error and
file listings written to the screen.
Still pondering why cpio or tar standard output is
written to stderr.
DDinAZ
John DuBois
Dave Dickerson <ddi...@hotmail.com> wrote:
>If I were to write a program that was to become a staple utility
>of a major operating system, I would, of course, not write my
>standard output to stdout, but to stderr. Makes sense (NOT!).
>So, where in the heck would I write error messages -- stdin?
-v output does not go to stdout because if you don't use -O,
the archive output goes to stdout.
>So, what does one do if one wants standard output
>(file names) written to stdout and error messages (if any)
>written to stderr?
find ... | tee | cpio -o ...
John
--
John DuBois spc...@armory.com. KC6QKZ/AE http://www.armory.com./~spcecdt/
John DuBois
John DuBois <spc...@deeptht.armory.com> wrote:
>In article <3bf1f03...@news.earthlink.net>,
>Dave Dickerson <ddi...@hotmail.com> wrote:
>>So, what does one do if one wants standard output
>>(file names) written to stdout and error messages (if any)
>>written to stderr?
>
>find ... | tee | cpio -o ...
er, slightly more specifically:
exec 3>&1; find ... | tee /dev/fd/2 2>&3 | cpio -o ...
or
{ find ... | tee /dev/fd/3 | cpio -o ... ;} 3<&1
Bob Meyers
"Dave Dickerson" <ddi...@hotmail.com> wrote in message
news:3bf1f394...@news.earthlink.net...
>
> Still pondering why cpio or tar standard output is
> written to stderr.
So "tar cf - myfiles > tarball.tar" works without sticking error messages in
the archive?
Brian K. White
> I want error messages (if any) written to /tmp/error and
> file listings written to the screen.
tough :)
> Still pondering why cpio or tar standard output is
> written to stderr.
they've explained that already
man tar (and cpio)
the actual tar and cpio archive data itself has to be allowed to go to
stdout for use in pipes, so if the tar data is going to stdout so that it
can be fed into a pipe, then you damned well can't have user display data
going the same place because that would corrupt the tar data.
example:
create a simple tar containing /u/data/*:
tar cvf myfile.tar /u/data
the tar data goes into the file myfile.tar and the filenames go to stderr
which is on the screen. in this case, you are right, the filenames could go
to stdout, but as you'll see next, that would break another very very very
very .....very handy use of tar.
now we want to make a compressed tar..., or send the tar over a network, or
encrypt it, or all of the above at once... you could do all these things by
first creating the tar, then encrypting, then compressing, then ftping, but
suppose the tar is a backup of a big data tree. unless the disk is only 30%
used, you may not have room on the disk to hold a whole uncompressed tar of
the data, plus a compressed copy, plus the live files themselves, which is
what you'd need as a minimum.
so you need to be able to feed tar data directly to other programs, and it
is far more important for actual data to go to stdout instead of some unusal
file descriptor than it is for the filename display. and in this case,
stderr is really not such an unintuitive place either, since the filenames
would fall under the category of "diagnostic output" which is perfectly
reasonable to send to stderr. Especially when you remember that tar by
default has no output, you have to specifically ask for it with the v
option.
tar cvf - /u/data |compress -H |rcmd otherhost -l root "(uncompress |tar
xf -)"
in this case, if the filenames went to stdout, the tar data would be junk,
but since the filenames go to stderr, they show on screen while the tar data
is preserved unsullied on stderr, and the end result is a mirror of a data
tree on another system, that could be:
A) too large to fit on the available space left on the source disk
B) too large to fit on the available space left on the destination disk
D) too large to fit in the 2-Gig filesize limit
E) too large to finish working in the time allowed
if you tried to do the job in stages with seperate files...
such is the case with many unix tools that can be used as filters or
creators or receivers of data streams.
Brian K. White -- br...@aljex.com -- http://www.aljex.com/bkw/
+++++[>+++[>+++++>+++++++<<-]<-]>>+.>.+++++.+++++++.-.[>+<---]>++.
filePro BBx Linux SCO Prosper/FACTS AutoCAD #callahans Satriani
Dave Dickerson
says...
Thank you for the explanation; Now I understand the why of it
and I really appreciate you taking the time to reply.
I no longer ponder the why.
Now I ponder the how.
Let's say I have a script that presents a menu
and one of the menu selections is to backup
/stand using cpio. The backup is performed by another
script called bu_stand. In this scenario, the menu
script calls the bu_stand script.
Let's also say that the menu script logs what
selections are made by the user. In addition,
if any errors are generated by the called script,
they are written to a file called /tmp/errors.
The menu script would have something like this:
bu_stand 2>/tmp/errors
The bu_stand scripts would have something like this:
find /stand -print | cpio -o..yada yada yada
Here comes the conflict:
I want any error messages, and error messages
only, generated by the execution of bu_stand
to be written to /tmp/errors
If the bu_stand script executes successfully,
I want file names displayed on screen as they
are backed up by cpio (/tmp/errors would be
empty).
Is there a way to do this?
DDinAZ
Dave Dickerson
I tried your suggestions, John, and I get a listing of
file names from cpio in the file I want to hold only
error messages.
I'm probably screwing it up -- I don't understand very well
the nuances of 3>&1 and 2>&3.
Thanks for the shot. Maybe I better (or worse) explain
my wants in my reply to Brian White's posting.
Thanks for helping,
DDinAZ
Dave Dickerson
/tmp/error contains a list of file names from cpio output.
I want only error messages, should any be generated by
cpio (or the script that calls cpio).
I guess I'm trying to split errors and file names from
the same stream content. Doesn't look like there's a
way to do that.
DDinAZ
Bob Meyers
> /tmp/error contains a list of file names from cpio output.
> I want only error messages, should any be generated by
> cpio (or the script that calls cpio).
>
> I guess I'm trying to split errors and file names from
> the same stream content. Doesn't look like there's a
> way to do that.
You know... if backup errors are a concern (and they certainly should be)
you should put your efforts into a supertar. Then you could do a real verify
too.
Tony Lawrence
He could do it with awk or perl if he can figure out a regex pattern
that will match errors rather than filenames.
But one of the supertras (http://pcunix.com/Reviews/supertrars.html ) is
indeed what he SHOULD be doing.
--
Tony Lawrence
SCO/Linux Support Tips, How-To's, Tests and more: http://pcunix.com
Brian K. White
nope.
ok ok ok, in practical terms, no.
but in absolute terms, anything is possible. I bet by the time I write down
my first idea or two, I will have thought of a pretty good one and it will
be the last on this list...
1) download the source to gnu tar, modify it to print filenames to fd3
instead of fd1, compile and use it like
tar cvf myfiles.tar myfiles 2>errors.txt 3>&1
2) write a script or program that can parse the output of tar on the fly (as
in: without having to wait for end of file before doing the work) and have
it send errors to file and files to screen.
3) if you don't need it to display the files as they are being processed,
it's easy several different ways. do a tar tvf after the tar is made.
4) use tee to log everything to a file while stripping out the errors for
the screen, then grep the errors from the log to errors.txt and discard the
log.
tar cvf mytar myfiles 2>&1 |tee -u tarlog.txt |grep -v "^tar:\ "
grep "^tar:\ " tarlog.txt >errors.txt
rm -f tarlog.txt
hah! I knew it. enjoy :)
--
John DuBois
Dave Dickerson <ddi...@hotmail.com> wrote:
>In article <3bf224b7$0$79562$8ee...@newsreader.tycho.net>, John DuBois says...
>>
>>or
>>{ find ... | tee /dev/fd/3 | cpio -o ... ;} 3<&1
>
>file names from cpio in the file I want to hold only
>error messages.
Did you give cpio the -v flag? In this method, 'tee' takes the place of -v in
producing the file list on the ouitput.
Those commands *will* send the file list to stdout and any errors to stderr.
For example:
exec 3>&1; find . -print | tee /dev/fd/2 2>&3 | cpio -o -O /tmp/archive.cpio 2>/tmp/errors
File list goes to the screen, cpio errors go to /tmp/errors, archive is
/tmp/archive.cpio. If you also want find errors to go to /tmp/errors, then:
exec 3>&1 2>/tmp/errors; find . -print | tee /dev/fd/2 2>&3 | cpio -o -O /tmp/archive.cpio
If this doesn't work, please post (verbatim) what you try.
Dave Dickerson
wrote:
I think I got your suggestion implemented correctly just before I left
work for home -- but my brain is too mushed after a long, hard day to
recall right now. I did attempt to explain it to my cohort who works
the swing shift. Maybe he'll be able to explain it back to me
tomorrow.
I do remember I'm not using any -v flag in cpio. Since you do not use
that flag in your example, neither do I.
I really appreciate your insights -- thanks!
DDinAZ
Dave Dickerson
FYI to you and Tony,
I do use a super tar product. It was its use via a script that started
this whole thing. It was later discovered the same behavior was
exhibited by cpio and tar. I referenced cpio in my queries in an
attempt to contain the discussions to just one topic.
I think I got a handle on this thing using what I've learned from all
the replies (especially John DuBois).
Thanks to all!
DDinAZ
Bob Meyers
"Dave Dickerson" <ddi...@hotmail.com> wrote in message
news:SdCI7.41423$S4.37...@newsread1.prod.itd.earthlink.net...
> the same stream content. Doesn't look like there's a
> way to do that.
Just curious... would that be because you want to find errors like "path
length exceeded" that are buried deep inside some endless backup file
listing?
And, I don't know how it is done exactly, but isn't what you are asking
exactly how Microlite's summary reports are done? The summary only has the
errors, the listing file has the names.
Or is the peanut gallery topic drifting again :-)
Dave Dickerson
I want to find (log) any errors encountered during the backup
operation because I cannot rely on a remote user to faithfully explain
any problems that may happen. If a backup fails and any resulting
error messages are logged, there's a chance of figuring out what
happened.
I have no idea how the supertar products create their summary logs but
those logs are only relavant if the supertar product itself was able
to run (or at least the command that launches the product was
executed). If the backup fails prior to the command that launches the
backup, I need to capture those error messages as well.
DDinAZ
Dave Dickerson
wrote:
[snip]
>exec 3>&1; find . -print | tee /dev/fd/2 2>&3 | cpio -o -O /tmp/archive.cpio 2>/tmp/errors
[snip]
John,
I understand (mostly) everthing here except the reason for
/dev/fd/2.
Is it a harmless bit bucket or is there more to it?
DDinAZ
Dave Dickerson
wrote:
[snip]
Item 2) looks promising and follows John DuBois' posts. This is the
path I'm following for now and I think it'll work.
Item 4) is along the lines of what I've been trying to do
(unsuccessfully) so far.
The other items are welcomed ideas but I cannot implement for various,
non-technical reasons.
Thanks for the help!
DDinAZ
Brian K. White
/dev/fd/2 is another way of referencing "file descriptor 2" more commonly
known as "stderr". The gimmick here is, tee requires a file name, so this is
a way to send a copy of the data to another file descriptor.
the end result of all that is that 2 gets redirected to 3, and a copy of 2
is send back to 2
now you have two channels, each with a copy of the stream that was
originally just on stderr. so you can do something different with each
stream.
John DuBois
Dave Dickerson <ddi...@hotmail.com> wrote:
>On 15 Nov 2001 01:27:27 GMT, spc...@deeptht.armory.com (John DuBois) wrote:
>
>>exec 3>&1; find . -print | tee /dev/fd/2 2>&3 | cpio -o -O /tmp/archive.cpio 2>/tmp/errors
>
>/dev/fd/2.
You want a file list to appear on stdout (fd 1). The file list is available in
the pipe between find and cpio, so you should be able to tee it off there.
However, file descriptors 0 and 1 are "used up" at that point for the input &
output pipes, and you don't want to use fd 2 (stderr) since you're specifically
trying to avoid mixing the file list with error output. The solution is to
create an extra fd that is a clone of fd 1; that's what the exec 3>&1 does -
after the exec, fd 3 refers to the same thing fd 1 did. Now all you need to do
to accomplish your goal is tee to fd 3. However, there are two obstacles to
this. First, tee requires a filename. That's easily dealt with thanks to the
'dup' driver that has been included since 5.0.0 - /dev/fd/n refers to whatever
fd n does. The second obstacle is that any fds opened via the 'exec' mechanism
other than 0, 1, and 2 are closed when the shell execs a program like tee.
Fortunately, we can get around this by having the shell itself dup fd 3 onto fd
2 (2>&3) before exec'ing, and then tee to /dev/fd/2 (note that we could
just as well use any other fd not in use here, e.g. tee /dev/fd/4 4>&3, since
fds opened in this way are not marked to be closed on exec the way fds > 2
opened via the 'exec' mechanism are).
Dave Dickerson
wrote:
I'm going to print out this post of yours and put it in my bathroom.
That's where I put all my important reading material (no foolin').
I'm going to read it over and over until I understand it fully. This
is good stuff! I've learn alot in the last couple of days from you and
Brian. I really appreciate you guys taking the time to help me out.
I got your previous suggestion implemented and it works great (now,
finally) with cpio. I'm stumbling a little though with tar as there
is no list of file names being fed to it as with cpio so I don't see a
place to tee off of.
I have file names and errors getting displayed on screen and that's
more than I had a few days ago, but cannot figure out how to also get
error messages logged to a file. It's one or the other, screen display
or log file but not both.
I'll be hacking at it some more tomorrow.
Thanks again for the help.
DDinAZ
PS -- fd/2 = file descriptor 2 and not the second floppy drive. What
was I thinking?
Dave Dickerson
wrote:
>
Piece of cake once I realized fd didn't stand for floppy drive.
Thanks again for the education.
DDinAZ
Brian K. White
> Brian. I really appreciate you guys taking the time to help me out.
Please don't use me in the same sentence with John DuBois, I'm mortified!
As far as I can tell, from the by now several years I've been at this, he
knew more unix and shell scripting before I was even born than I will ever
know!
ftp://ftp2.caldera.com/pub/skunkware/osr5/sysadmin/johnd/
Robert Carnegie
> "Dave Dickerson" <ddi...@hotmail.com> wrote in message
> news:3bf3a5e8...@news.earthlink.net...
> > On 15 Nov 2001 01:27:27 GMT, spc...@deeptht.armory.com (John DuBois)
> > wrote:
> >
> > [snip]
> > >exec 3>&1; find . -print | tee /dev/fd/2 2>&3 | cpio -o -O
> /tmp/archive.cpio 2>/tmp/errors
> > [snip]
> >
> > John,
> >
> > I understand (mostly) everthing here except the reason for
> > /dev/fd/2.
> >
> > Is it a harmless bit bucket or is there more to it?
> >
> > DDinAZ
>
> /dev/fd/2 is another way of referencing "file descriptor 2" more
> commonly known as "stderr". The gimmick here is, tee requires a
> file name, so this is a way to send a copy of the data to another
> file descriptor.
So it's like writing "tee >&2" except that "tee /dev/fd/2" is correct
and works ;-)
Another approach to this might be to log file names going in to the
cpio command in a separate file with tee, and then manually strip
those file names entries, in order, out of cpio's verbose log, for
instance with the "diff" utility.
Dave Dickerson
wrote:
OK. Consider yourself thanked by me without ever mentioning
John DuBois.
:)
DDinAZ