will be a good approximation...
++L
how about
du -a | awk '{$1=""; print}'
This does print a leading space but is simple enough,
or perhaps
du -a | while(s=`{read}) echo $s(2-)
which is more accurate but arguably more complex.
-Steve
- erik
> du -a | awk '-F\t' '{print $2}' -
>
lossage on tabs and ' in filenames.
- erik
All this nonsense because the dogmatists refuse to accept
/n/sources/contrib/cross/walk.c into the distribution.
find and walk are about the same program. my version of
find started with andrey's. his find page (http://mirtchovski.com/p9/find/)
is dated 31-jul-2004, predating the given walk.c by ~18 months,
though i don't know which was written first.
the reason i started fiddling with find was to see if it couldn't
go a bit faster than du. (it did.)
my cannonical examples of its use are
find | grep whereisthatfile
and
grep whereisthatfunction `{find /sys/src|grep '\.[chlsy]$'}
i don't think it's that important that it absolutely needs to
be in the distribution; it's a convience.
what seems more important to me is a way to unlimit the size
of argv. otherwise we'll need to go down the hideous xargs path.
(apoligizes to hideous functions everywhere for the slur.)
- erik
I don't have spaces too, but correct script should not make any assumptions.
There is interesting date on http://swtch.com/plan9history/:
March 23, 1999 allow spaces in file names
I think it will be better to just disallow whitespaces (spaces and
tabs) in file names. Looks like that idea with using awk was there
before whitespaces allowed, so there was no problem.
How often have you run up against the current limit? I've yet to hit
it in anything other than contrived tests. And even those took work.
> find and walk are about the same program.
There are a few versions about. Dan's has the exactly right lack of
options to meet my needs. Others might too, but his is the version I
found first.
--lyndon
several times a month; just often enough to be irritating. since storage
is still going exponential, i expect this to get worse.
minooka; grep pattern `{find /sys}
grep: virtual memory allocation failed
- erik
What about
du -a | sed 's/^[0-9]*<tab>//g'
no loss on spaces in filenames.
no loss on tabs in filenames.
Tim Newsham | www.thenewsh.com/~newsham | thenewsh.blogspot.com
-rob
i don't really see why xargs (the idea, not the usual unix implementations)
is inherently such a bad idea. years ago i wrote an ultra simple version
with no options, and it's about 80 lines of code, which i use to grep
through all of /sys/src for example.
if you always split on \n (which is banned in filenames in plan 9) and
don't interpret any other metacharacters, what's the problem?
it's also nice because you often get some results before you've
walked the entire file tree.
that's interesting. my objection to xargs is all about the idea.
the usual way of doing things breaks if there are too many files.
using xargs does work around the problem. but then why not
go all the way and remove ` from rc? after all, ` only works some
of the time?
> if you always split on \n (which is banned in filenames in plan 9) and
> don't interpret any other metacharacters, what's the problem?
>
> it's also nice because you often get some results before you've
> walked the entire file tree.
i'm not sure i understand when and why this would be useful. nobody
has a real worm anymore. i can walk /sys/src in 0.5s. grepping takes
about 12s; saving 1/24th the time (best case) doesn't seem like a
big win. (the ratio is 10:1 on coraid's fileserver.)
- erik
because the limit is big enough that cases that break the
limit almost never happen except in this case?
> i'm not sure i understand when and why this would be useful. nobody
> has a real worm anymore. i can walk /sys/src in 0.5s.
you've got a fast system.
in at least one system i use, du -a of /sys/src takes about 25s.
and /sys/src isn't by any means the largest tree i like to grep
(for instance, searching for lost files with a name i longer remember,
i've been known to search through all the files in my home directory,
~425000 files at last count)
sometimes i think it would be nice if du had a breadth-first option.
we can easily fit all the files in most any system in memory.
why shouldn't that be the limit? see below.
> > i'm not sure i understand when and why this would be useful. nobody
> > has a real worm anymore. i can walk /sys/src in 0.5s.
>
> you've got a fast system.
> in at least one system i use, du -a of /sys/src takes about 25s.
i have a humble 2y.o. single-core 35w celeron as a fileserver.
> and /sys/src isn't by any means the largest tree i like to grep
> (for instance, searching for lost files with a name i longer remember,
> i've been known to search through all the files in my home directory,
> ~425000 files at last count)
>
> sometimes i think it would be nice if du had a breadth-first option.
aren't you contridicting yourself? at 128 characters/file,
that's only 52mb -- 2% of memory on a typical system these days.
why can't it be passed as an argument list?
- erik
really? have you tested this? i've always found the
two to be related.
first, most fileservers have an in-memory block cache.
unless your active set is really big, most directories should
be in the in-memory cache.
when in memory cache, the time it takes to acquire
block cache locks and copy data dominates. for
fossil+venti this factor is multiplied by 2 plus 2 trips
through the kernel. so for memory-cached blocks,
fileserver speed is entirely dependent on network+
cpu. the proportion of memory cache is of course
proportional to one's cache sizes.
sure, you can take this to extremes where the size of
the memory cache is so small, that the memory cache
doesn't matter, or the speed of the network is so slow
(find /n/sources) that nothing other than disk io or
network speed matters.
second, each directory read requires a number of 9p messages.
in the current system, each incurs the full rtt penality.
so the network latency is a really big factor in du
performance.
you can test to see how the du speed is related to
network performance very easily if you have the right
sort of network card. just adjust the interrupt coalesing
values. (Tidv/Tadv in ether82563, or
echo coal $µs>/net/ether$n/clone for etherm10g)
and the cool thing is that especially for tcp, i've
found cpu speed and network latency to be pretty
imporant.
- erik
i'm not saying it can't be passed in an argument list, just that
xargs gives you a lazy evaluation of the walk
of the file tree which can result in a faster result
when the result is found earlier in the file list.
that's why breadth-first might be useful, by putting
shallower files earlier in the search results - i often
do grep foo *.[ch] */*.[ch] */*/*.[ch] to achieve
a similar result, but you have to guess the depth that way.
i have no problem with breadth-first.
my beef with xargs is only that it is used as an excuse
for not fixing exec in unix. it's also used to bolster the
"that's a rare case" argument.
imho "rare case" arguments work best if the downside
is that the "rare case" is slow or awkward. in this case
it's broken.
> that's why breadth-first might be useful, by putting
> shallower files earlier in the search results - i often
> do grep foo *.[ch] */*.[ch] */*/*.[ch] to achieve
> a similar result, but you have to guess the depth that way.
clearly i'm not in your league. my source trees are
smaller than that. no more than two levels. or, it
doesn't matter. on a few machines laying around
the house (details on the poorly-chosen disks here:
http://www.quanstro.net/plan9/fs.html)
i7 2666 0.36u 0.42s 7.35r
Atom 1605 0.83u 1.88s 8.85r
AMD64 2007 0.94u 0.97s 12.51r
and on the fast 10gbe stuff at coraid
Xeon5000 1865 0.45u 0.83s 4.33r 10gbe myricom
PIV/Xeon 3003 0.66u 1.41s 7.32r i82573
(it would be fun to put the i7 with 10gbe
together!)
it's easy to try the completely uncached case at
coraid since the working set is about 7gb and the
cache is only 3.5
Xeon5000 1874 0.50u 0.84s 27.63r
- erik
I often do something like the following:
find . -type f <condition> | xargs grep -l <pattern> | xargs <command>
If by "fixing exec in unix" you mean allowing something like
<command> $(grep -l <pattern> $(find . -type f <condition>))
then <command> would take far too long to even get started.
And can eat up a lot of memory or even run out of it. On a
2+ year old MacBookPro "find -x /" takes 4.5 minutes for 1.6M
files and 155MB to hold paths. My 11 old machine has 64MB
and over a million files on a rather slow disk. Your solution
would run out of space on it. Now granted I should update it
to a more balanced system but mechanisms should continue
working even if one doesn't have an optimal system. At least
xargs gives me that choice.
Basically this is just streams programming for arguments
instead of data. Ideally all the args would be taken from a
stream (and specifying args on a command line would be just a
convenience) but it is too late for that. Often unix
commands have a -r option to walk a file tree but it would've
been nicer to have the tree walk factored out. Then you can
do things like breadth first walk etc. and have everyone
benefit.
modern cat wouldn't fit in core on the early pdps unix was
developed on!
just to be fair, could you fit your 1.6m files on your 11yu machine?
i'm guessing you couldn't.
> Basically this is just streams programming for arguments
> instead of data.
that's fine. but it's no excuse to hobble exec. not unless
you're prepared to be replace argument lists with an argument
fd.
- erik
> that's why breadth-first might be useful, by putting
> shallower files earlier in the search results - i often
> do grep foo *.[ch] */*.[ch] */*/*.[ch] to achieve
> a similar result, but you have to guess the depth that way.
for what it's worth, dan's walk.c has a -d option for limiting search
depth. it's not breadth-first, but is still nice.
No point in gratuitously obsoleting old machines. I am
running FreeBSD-7.2 on it my 11yo machine and so far it has
stood up well enough.
> just to be fair, could you fit your 1.6m files on your 11yu machine?
> i'm guessing you couldn't.
Yes. It's on its third disk. A 6yo 80G IDE disk.
> > Basically this is just streams programming for arguments
> > instead of data.
>
> that's fine. but it's no excuse to hobble exec. not unless
> you're prepared to be replace argument lists with an argument
> fd.
Not sure how exec is hobbled. Given the way Unix programs
behave you can't replace arg list with an arg fd (I used to
carry around a libary to do just that but the problem is all
the standard programs). Anyway, I don't see how xargs can be
gotten rid of.
didn't know this was "unixfans". will keep that in mind.
- erik
just because reviving old threads is fun...
I've just found out about this:
it does not seem to work out of the box (expecting some unix paths), but
since there's a perl port and that thing is supposed to be more or
less self contained (for the standalone version), maybe it's not too
much work for someone interested enough.
Cheers,
Mathieu
"ack is written purely in Perl, and takes advantage
of the power of Perl's regular expressions."
Forgive my ignorance and irrelevance to this topic,
but what are the advantages of Perl's regular
expressions, over the implementation we have
currently in Plan 9?
Thanks,
ak
that would not be the interesting point, if any. it's just that the
tool is already there and (should be) simpler to use than piping
various commands around, as they illustrate below.
> Date: Mon, 3 May 2010 05:18:56 -0700
> From: Akshat Kumar <aku...@mail.nanosouffle.net>
> Subject: Re: [9fans] du and find
> To: Fans of the OS Plan 9 from Bell Labs <9f...@9fans.net>
>
> >From the website:
>
> "ack is written purely in Perl, and takes advantage
> of the power of Perl's regular expressions."
>
> Forgive my ignorance and irrelevance to this topic,
> but what are the advantages of Perl's regular
> expressions, over the implementation we have
> currently in Plan 9?
I had in fact the answer, a long time ago: because they simply do not
know that ed(1) exists, and sed(1) etc.
A group, providing an ISDN router based on Debian, was requiring a lot
of memory and disk space. I asked why??? that much for _that_?!! The
answer: we need perl(1) installed. But what for? Answer: to replace
@@GATEWAY@@ and so on by customized values in a file... (They didn't
even thought of building a distribution on a vulcan to simply install on
the target.)
They didn't know about ed(1). So I tell them regexp were ed(1); and
ed(1) was required by POSIX.2. And tried to make the demonstration... to
see that Debian didn't provide ed(1) by default. I ask Debian, why the
f...?!!? Answer: GNU's Not Unix...
And this day I realized I was not GNU. And switch to *BSD before asking
myself some questions that lead me to Plan9...
--
Thierry Laronde <tlaronde +AT+ polynum +dot+ com>
http://www.kergis.com/
Key fingerprint = 0FF7 E906 FBAF FE95 FD89 250D 52B1 AE95 6006 F40C
Ack looks cute, but I think a fairly simple shell script could do all
of what ack does without requiring perl. I imagine it would be faster
still by not using Perl's broken regular expressions, and bear in mind
on Plan 9 you'd probably want to make a wrapper for grep anyway if you
do a lot of recursive searching.
--
Simplicity does not precede complexity, but follows it. -- Alan Perlis
Or just apply runs grep -r patch...
-Steve
Regexps in Plan9 are on one hand much less powerful than Perl's, on
the other hand they are (thanks to their simplicity) much quicker.
Often one doesn't need Perl's power and in such a case Plan9's regexps
are better. But in sometimes...
Just compare:
http://www.amk.ca/python/howto/regex/
to
regexp(7)
... particularly e.g. Lookahead Assertions, Non-capturing and Named Groups.
It's always been easier for me to use python's/perl's regular
expressions when I needed to process a text file than to use plan9's.
For simple things, e.g. while editing an ordinary text in acme/sam,
plan9's regexps are just fine.
Also read Russ Cox text:
http://swtch.com/~rsc/regexp/regexp1.html
Ruda
don't be silly. russ wrote something like this in pure sh(1) for p9p,
g. i reimplemented it in pure rc(1) and added gh (grep in headers)
and gf (grep function). since g is an engine, you can add other
specialized search functions.
contrib quanstro/g
- erik
i find it hard to think of cases where i would need
such sophistication and where tokenization or
tokenization plus parsing wouldn't be a better idea.
for example, you could write a re to parse the output
of ls -l and or ps. but awk '{print $field}' is so much
easier to write and read.
so in all, i view perl "regular" expressions as a tough sell.
i think they're harder to write, harder to read, require more
and more unstable code, and slower.
one could speculate that perl, by encouraging a
monolithic, rather than tools-based approach;
and cleverness over clarity made perl expressions
the logical next step. if so, i question the assumptions.
- erik
% man 1 grep | grep '\-r'
%
>
> -Steve
s/runs/ron's/
see 9fans passim for the patch.
-Steve
Oh right! Well, if the point of this thread is to talk about something
better than grep -r in the first place...
eh, whateva.
A lot of the `sophisticated' Perl I've seen uses some horrible regexes
when really the job would have been done better and faster by a
simple, job-specific parser.
I've yet to find out why this happens so much, but I think I can
narrow it to a combination of ignorance, laziness, and perhaps that
all-too-frequent assumption `oh, I can do this in 10 lines with perl!'
I guess by the time you've written half a parser in line noise, it's
too late to quit while you're behind.
> On Mon, May 3, 2010 at 10:53 AM, erik quanstrom
> <quan...@quanstro.net> wrote:
>>> It's always been easier for me to use python's/perl's regular
>>> expressions when I needed to process a text file than to use
>>> plan9's.
>>> For simple things, e.g. while editing an ordinary text in acme/sam,
>>> plan9's regexps are just fine.
>>
>> i find it hard to think of cases where i would need
>> such sophistication and where tokenization or
>> tokenization plus parsing wouldn't be a better idea.
>
> A lot of the `sophisticated' Perl I've seen uses some horrible regexes
> when really the job would have been done better and faster by a
> simple, job-specific parser.
>
> I've yet to find out why this happens so much, but I think I can
> narrow it to a combination of ignorance, laziness, and perhaps that
> all-too-frequent assumption `oh, I can do this in 10 lines with perl!'
> I guess by the time you've written half a parser in line noise, it's
> too late to quit while you're behind.
I think it's ignorance and something. I'm not sure what that something
is. I am sure if you tried to suggest writing a parser to many of the
open-sourcers I've talked to you would be treated as if you were
suggesting a big job rather than a small one. "Why Write a Parser,"
they would ask, "when I can just scribble a few little lines of perl?"
Maybe it's humans' natural tendencies toward hierarchy coming into
play. Stuff known by Teachers and Masters easily takes on a bizarre
kind of importance, rank is unconsciously attached, and the student
naturally but unconsciously feels he is not of sufficient rank to
attempt the Master's Way. That explanation does pre-suppose humans
have a very strong natural tendency to hierarchy. I find sufficient
evidence within myself to believe it's true, as unpopular as the idea
may be. Perhaps some people are more strongly inclined that way than
others. Anyway, it's the only explanation I can imagine for the
phenomena.
>
>>
>> for example, you could write a re to parse the output
>> of ls -l and or ps. but awk '{print $field}' is so much
>> easier to write and read.
>>
>> so in all, i view perl "regular" expressions as a tough sell.
>> i think they're harder to write, harder to read, require more
>> and more unstable code, and slower.
>>
>> one could speculate that perl, by encouraging a
>> monolithic, rather than tools-based approach;
>> and cleverness over clarity made perl expressions
>> the logical next step. if so, i question the assumptions.
>>
>> - erik
>>
>>
>
--
On 3 May 2010, at 19:34, Jorden M wrote:I think it's ignorance and something. I'm not sure what that something is. I am sure if you tried to suggest writing a parser to many of the open-sourcers I've talked to you would be treated as if you were suggesting a big job rather than a small one. "Why Write a Parser," they would ask, "when I can just scribble a few little lines of perl?"I've yet to find out why this happens so much, but I think I can
narrow it to a combination of ignorance, laziness, and perhaps that
all-too-frequent assumption `oh, I can do this in 10 lines with perl!'
I guess by the time you've written half a parser in line noise, it's
too late to quit while you're behind.
I can attest that it's not just open-source folk.
> a big job rather than a small one. "Why Write a Parser," they would ask,
> "when I can just scribble a few little lines of perl?"
That phenomenon is true, and if you take it further once that person
is done writing their abominable perl, and point out that they've
written a parser anyway, but poorly (not to mention one that would
have to be totally rewritten to be modified), they look at you
crosseyed and say `whatever.'
>
> Maybe it's humans' natural tendencies toward hierarchy coming into play.
> Stuff known by Teachers and Masters easily takes on a bizarre kind of
> importance, rank is unconsciously attached, and the student naturally but
> unconsciously feels he is not of sufficient rank to attempt the Master's
> Way. That explanation does pre-suppose humans have a very strong natural
> tendency to hierarchy. I find sufficient evidence within myself to believe
> it's true, as unpopular as the idea may be. Perhaps some people are more
> strongly inclined that way than others. Anyway, it's the only explanation I
> can imagine for the phenomena.
>
Pretty much. Like Raschke mentioned, people as students are
conditioned to think that parsers are hard to do because they're a
piece of a compiler, and that Dragon book is too big and scary and
only Gods can write compilers and parsers, etc.. Another function of
the `parsers are too hard' mentality is that people don't recognize
the difference between something that's regular and something that's
CF, and spend days scratching their head wondering why their regexes
break all over the place. Situations often become complicated when
self-proclaimed perl experts drop in and go, `oh here, you just add
this case and that case and you should be fine X% of the time!', where
X is a BS figure pulled out of you know where.
I think what we have here can be construed as a failure of CS
education, which fits right in with the many failures of education at
large.
(about students/trainees and perl)
Being able to recognize what you've studied in your daily work is quite difficult in most places. Also your work objectives are rarely related to the correctness, in the sense of science. I mean something correct or well enough for the business could not be correct or well enough from the science point of view.
Speaking about non programming-related business, for me, it's enough if a student is able to use or ask for a programming language to solve a task perl, vbscript or whatever. I've seen a couple of times students matching two lists of thousands of entries by hand, either in paper or in the original excel format. And I've seen mentors and managers agree with the method. If they can write regexp, even ugly ones, that's enough, you can show them alternatives, suggest other ways, etc.
The fail is not the school, or not completely. The tools are given to you, it is not usual you can choose the tool you want to use to finish a task. In nice places, you might be able to propose one. . .
slds.
gabi
----- Original Message ----
From: Jorden M <jrm...@gmail.com>
To: Fans of the OS Plan 9 from Bell Labs <9f...@9fans.net>
Sent: Tue, May 4, 2010 5:38:35 PM
Subject: Re: [9fans] du and find
This may have to do with failure in the earlier education--I remember
that again, peers could do 'picture frame problems' but without any real
comprehension of the actual algebra.
On the other hand, it may just be a question of what human beings are,
and how few artists there are, proportionately speaking....
K
>>> Gabriel Díaz <gd...@rejaa.com> 04/05/2010 12:56 pm >>>