maildir vs. mbox vs. mh ???
Michael D. Schleif
Considering moving from mbox to maildir a very large (~2GB, ~1000000
messages) email archive. Mostly concerned with the integrity of
receiving messages intact.
Obviously, this will impact performance and inodes used.
Given this brief overview, what ought I to consider?
Are there other options to consider?
How else might I handle this data?
What do you think?
--
Best Regards,
mds
mds resource
888.250.3987
Dare to fix things before they break . . .
Our capacity for understanding is inversely proportional to how much we
think we know. The more I know, the more I know I don't know . . .
--
To UNSUBSCRIBE, email to debian-us...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org
Jeff
>
> Considering moving from mbox to maildir a very large (~2GB, ~1000000
> messages) email archive. Mostly concerned with the integrity of
> receiving messages intact.
>
> Obviously, this will impact performance and inodes used.
>
> Given this brief overview, what ought I to consider?
>
> Are there other options to consider?
>
> How else might I handle this data?
>
> What do you think?
I've not used mh mail boxes before, only maildir and mbox. I use
maildir for my active mailboxes for it's ability to keep messages from
getting messed up on delivery (sorry for the non-tech reason). For
mail boxes that I use for archive and are non-active, I use mbox for
it's ease in backing up.
Performance-wise, I don't think you'll notice a difference in loading
the mail box on a modern machine. On older, slower machines, you
might see the mbox loading faster than the maildir folders.
If I had an archive mail box that size, I'd leave it as an mbox so I
can back it up easier since it's a single file as opposed to a
directory.
my 2 cents,
jc
--
Jeff Coppock Systems Engineer
Diggin' Debian Admin and User
Ricardo B
> Considering moving from mbox to maildir a very large (~2GB, ~1000000
> messages) email archive. Mostly concerned with the integrity of
> receiving messages intact.
> Obviously, this will impact performance and inodes used. Given this
> brief overview, what ought I to consider? Are there other options to
> consider?
> How else might I handle this data?
> What do you think?
Go with Maildir. Don't think it can get any better.
Maildir has two important advantages over MH:
- It is designed for concurrent access (performance and integrity).
- It is more widely suported by software.
Step 1: Create the Maildir
Step 2: If needed, reconfigure your Mail-Delivery-Agent to deliver new
messagens into the Maildir.
Step 3: Convert mbox to Maildir.
--
Ricardo
Michael D. Schleif
Jeff wrote:
>
> Michael D. Schleif, 2003-Feb-09 15:05 -0600:
> >
> > messages) email archive. Mostly concerned with the integrity of
> > receiving messages intact.
> >
> > Obviously, this will impact performance and inodes used.
> >
> > Given this brief overview, what ought I to consider?
> >
> > Are there other options to consider?
> >
> > How else might I handle this data?
> >
> > What do you think?
>
> maildir for my active mailboxes for it's ability to keep messages from
> getting messed up on delivery (sorry for the non-tech reason). For
> mail boxes that I use for archive and are non-active, I use mbox for
> it's ease in backing up.
This is interesting -- is it possible to use two (2) mailbox formats at
once?
Can mutt be used to review incoming mail under maildir, and also to
browse my archived mail under mbox? How?
> Performance-wise, I don't think you'll notice a difference in loading
> the mail box on a modern machine. On older, slower machines, you
> might see the mbox loading faster than the maildir folders.
>
> If I had an archive mail box that size, I'd leave it as an mbox so I
> can back it up easier since it's a single file as opposed to a
> directory.
My biggest concern is number of inodes.
My goal is to have fetchmail go out and grab mail and news from multiple
servers, procmail to sort and process it, mutt to read it and possibly
one of the webmail variants to access it while away from the office.
This is all very clean when the overall universe of mail/news, including
old archives, is small; but, such is not this case.
I am open to all suggestions . . .
--
Best Regards,
mds
mds resource
888.250.3987
Dare to fix things before they break . . .
Our capacity for understanding is inversely proportional to how much we
think we know. The more I know, the more I know I don't know . . .
Vineet Kumar
>
> Jeff wrote:
> > I've not used mh mail boxes before, only maildir and mbox. I use
> > maildir for my active mailboxes for it's ability to keep messages from
> > getting messed up on delivery (sorry for the non-tech reason). For
> > mail boxes that I use for archive and are non-active, I use mbox for
> > it's ease in backing up.
>
> This is interesting -- is it possible to use two (2) mailbox formats at
> once?
>
> Can mutt be used to review incoming mail under maildir, and also to
> browse my archived mail under mbox? How?
Yes. You don't need to do anything special; when you open a mailbox, it
figures out what type it is.
> > Performance-wise, I don't think you'll notice a difference in loading
> > the mail box on a modern machine. On older, slower machines, you
> > might see the mbox loading faster than the maildir folders.
> >
> > If I had an archive mail box that size, I'd leave it as an mbox so I
> > can back it up easier since it's a single file as opposed to a
> > directory.
>
> My biggest concern is number of inodes.
Well, you can tar and compress a maildir, and then it only takes 1, same
as an mbox. That works fine for archiving, though is not as convenient
for active mailboxes. I also don't really buy that a maildir is
difficult to back up (especially if you tar.(gz|bz2) it). I like using
maildirs mostly because mbox feels like a dirty hack, with the whole
"From " thing. I also like the increased scriptability using standard
GNU tools like grep to find and process individual messages, instead of
having to use some sort of mbox-parsing perl module.
good times,
Vineet
--
http://www.doorstop.net/
--
http://www.debian.org/
Cameron Matheson
On Sun, Feb 09, 2003 at 08:00:52PM -0800, Vineet Kumar wrote:
> Well, you can tar and compress a maildir, and then it only takes 1, same
> as an mbox. That works fine for archiving, though is not as convenient
> for active mailboxes. I also don't really buy that a maildir is
> difficult to back up (especially if you tar.(gz|bz2) it). I like using
> maildirs mostly because mbox feels like a dirty hack, with the whole
> "From " thing. I also like the increased scriptability using standard
> GNU tools like grep to find and process individual messages, instead of
> having to use some sort of mbox-parsing perl module.
What do you mean by the "From "-thing? I have always just used mbox
because that is what fetchmail puts my mail into but this thread has
aroused my interest in maildir... Can i use maildir w/ fetchmail/exim?
If not, how does one get maildir, and what are the technical advantages?
Thanks,
Cameron Matheson
Isaac To
Cameron> What do you mean by the "From "-thing?
Send a mail to yourselves, having a line starting with "From ", and see what
is the result. What you will get is this:
--- example ---
>From time to time you get a line starting with the word From.
--- end example ---
If you use an mbox-style mail box, you will see a ">" character there at the
beginning. This is not seen by anyone using maildir, and I didn't type
that in my message.
Cameron> I have always just used mbox because that is what fetchmail
Cameron> puts my mail into but this thread has aroused my interest in
Cameron> maildir... Can i use maildir w/ fetchmail/exim? If not, how
Cameron> does one get maildir,
For exim: yes. For fetchmail: dunno (perhaps not). But you can always ask
fetchmail to put the mail forward to another processor like a local exam, or
procmail, which does support maildir.
Cameron> and what are the technical advantages?
If you don't like the idea that you have mails with "From " lines modified
to "suit" the mailbox, or the idea that mails pile up into huge files so
that it takes forever to delete or move one message, or the idea that to
know how many mails you are having you must read and parse the whole mailbox
file (which might well be counted in megabytes), or the idea that when you
are reading the mailbox the E-mail delivery program must not write into your
mailbox at the same time or the mailbox gets corrupted; while at the same
time you don't afraid that your filesystem suddenly have ten thousand files
because each mail ends up into a file, maildir is for you.
For me, each mail has two copies. I regularly read one of them, the
"regular" copy, which is stored as maildir. I use spamassissin to scan for
junk, let Gnus throw mails away without too much care, etc., so that I can
control the number of files for storing mails (currently around 2000). To
minimize the wastage on filesystem space, I use reiserfs on my mail
partition, although an ext3 partition with tail files enabled should have
similar effect. The other "unprocessed" copy are kept in mbox format for
months or even years, just in case when I accidentally delete a mail that I
want, when I can rush to it, tail it and hope to extract the needed message.
Regards,
Isaac.
Vineet Kumar
> Hi,
>
> On Sun, Feb 09, 2003 at 08:00:52PM -0800, Vineet Kumar wrote:
> > Well, you can tar and compress a maildir, and then it only takes 1, same
> > as an mbox. That works fine for archiving, though is not as convenient
> > for active mailboxes. I also don't really buy that a maildir is
> > difficult to back up (especially if you tar.(gz|bz2) it). I like using
> > maildirs mostly because mbox feels like a dirty hack, with the whole
> > "From " thing. I also like the increased scriptability using standard
> > GNU tools like grep to find and process individual messages, instead of
> > having to use some sort of mbox-parsing perl module.
>
> because that is what fetchmail puts my mail into but this thread has
> aroused my interest in maildir... Can i use maildir w/ fetchmail/exim?
> If not, how does one get maildir, and what are the technical advantages?
An mbox file is a file with all the messages laid out in it end-to-end,
separated by lines beginning with "From ". Any lines in the body of a
message that begin with "From " are then munged into ">From " by the
program delivering into the mbox. That's just a dirty, dirty hack. Add
to that that the file needs to be locked properly(and that file locking
on NFS is not perfect) in order to prevent it from getting corrupted
(and that corrupting the file means corrupting a whole mailbox, not a
single message) and you have a technologically inferior mail storage
design, IMHO. On the other hand, it's the original way of doing things,
so support for it is universal. Also, loading an mbox is generally
faster than loading a maildir.
Exim will happily deliver into maildirs if you uncomment the
"maildir_format" line in the address_directory transport (and somehow
specify the directory you'd like to deliver to, perhaps via .forward or
by making a director that checks if ~/Maildir/ or ~/Inbox/ exists and
delivering there if so). I don't know about fetchmail, but if you have
it configured to deliver via SMTP through the local exim anyway, it
shouldn't be an issue. I do know that procmail and maildrop both
support delivery into maildirs.
As for MUA support, I know mutt does, but I'm not 100% sure of others.
It's fairly widely supported at this point, and I'd guess that the
popular X GUI mailers (sylpheed, kmail, evolution) do support it as
well. I see that you're using mutt, though, so you'll be fine.
good times,
Vineet
--
http://www.doorstop.net/
--
"Under no circumstances will I ever purchase anything offered to me as
the result of an unsolicited email message. Nor will I forward chain
letters, petitions, mass mailings, or virus warnings to large numbers
of others. This is my contribution to the survival of the online
community." - Roger Ebert, "The Boulder Pledge"
Cameron Hutchison
>
> On Sun, Feb 09, 2003 at 08:00:52PM -0800, Vineet Kumar wrote:
> > I like using
> > maildirs mostly because mbox feels like a dirty hack, with the whole
> > "From " thing.
>
The message boundary in a mbox-style mailbox is "\n\nFrom ". If a line
in an email starts with "From ", a ">" is prepended so that it doesn't
appear as an accidental message boundary.
Yven Johannes Leist
> Considering moving from mbox to maildir a very large (~2GB, ~1000000
> messages) email archive. Mostly concerned with the integrity of
> receiving messages intact.
>
> Obviously, this will impact performance and inodes used.
It might not fully apply to the mailclient you're using but I've asked a
similar question on km...@kde.org, and got quite an informative reply:
http://lists.kde.org/?l=kmail&m=103853064326124&w=2
I guess that the inode question depends quite a lot on what type of
fileysystem you're using. If you're using ext2/ext3 you might consider
setting up a dedicated filesystem and experimenting with the --block-size and
--bytes-per-inode setting, I remember quite a big difference in terms of
actual disk usage when I set up a separate partition for a local cddbd
server, (whose data.tar.gz shovels almost half a million files on your
hard-disk) and fine tuned these settings.
Cheers,
Yven
--
Yven Johannes Leist - le...@beldesign.de
http://www.leist.beldesign.de
Michael D. Schleif
Vineet Kumar wrote:
>
> * Michael D. Schleif (m...@helices.org) [030209 19:32]:
> >
> > Jeff wrote:
> > > I've not used mh mail boxes before, only maildir and mbox. I use
> > > maildir for my active mailboxes for it's ability to keep messages from
> > > getting messed up on delivery (sorry for the non-tech reason). For
> > > mail boxes that I use for archive and are non-active, I use mbox for
> > > it's ease in backing up.
> >
> > This is interesting -- is it possible to use two (2) mailbox formats at
> > once?
> >
> > Can mutt be used to review incoming mail under maildir, and also to
> > browse my archived mail under mbox? How?
>
> Yes. You don't need to do anything special; when you open a mailbox, it
> figures out what type it is.
>
> > > Performance-wise, I don't think you'll notice a difference in loading
> > > the mail box on a modern machine. On older, slower machines, you
> > > might see the mbox loading faster than the maildir folders.
> > >
> > > If I had an archive mail box that size, I'd leave it as an mbox so I
> > > can back it up easier since it's a single file as opposed to a
> > > directory.
> >
> > My biggest concern is number of inodes.
>
> as an mbox. That works fine for archiving, though is not as convenient
> for active mailboxes. I also don't really buy that a maildir is
> maildirs mostly because mbox feels like a dirty hack, with the whole
> GNU tools like grep to find and process individual messages, instead of
> having to use some sort of mbox-parsing perl module.
Under netscrape I have directory hierarchies such as:
Lists
|__debian
| |__debian-users
| |__debian-devel
|__tomcat-users
With many lists and so many old mails, a flat list of _all_ Maildirs
will quickly become difficult to navigate . . .
How can this be done with Maildir?
--
Best Regards,
mds
mds resource
888.250.3987
Dare to fix things before they break . . .
Our capacity for understanding is inversely proportional to how much we
think we know. The more I know, the more I know I don't know . . .
Vineet Kumar
> Under netscrape I have directory hierarchies such as:
>
> Lists
> |__debian
> | |__debian-users
> | |__debian-devel
> |__tomcat-users
>
> With many lists and so many old mails, a flat list of _all_ Maildirs
> will quickly become difficult to navigate . . .
>
> How can this be done with Maildir?
Just the same. Wherever you have an mbox file, replace it with a
maildir directory.
You can even go one better by having maildirs which themselves contain
subdirectories, something that is impossible with mbox files.
For example, you could have a maildir called work which contains
unsorted messages, and also sub-maildirs for each client. Each client
maildir can contain messages as well as yet more sub-maildirs for e.g.
certain projects, time periods, contacts, etc.
I don't believe netscape includes support for the maildir format,
though, so you may not be able to do this, unless you do something like
run an IMAP server such as courier and use netscape to access the mail
through the IMAP interface.
good times,
Vineet
--
http://www.doorstop.net/
--
"If we do not believe in freedom of speech for those we despise we do not
believe in it at all." --Noam Chomsky
Michael D. Schleif
Vineet Kumar wrote:
>
> * Michael D. Schleif (m...@helices.org) [030210 12:24]:
> > Under netscrape I have directory hierarchies such as:
> >
> > Lists
> > |__debian
> > | |__debian-users
> > | |__debian-devel
> > |__tomcat-users
> >
> > With many lists and so many old mails, a flat list of _all_ Maildirs
> > will quickly become difficult to navigate . . .
> >
> > How can this be done with Maildir?
>
> Just the same. Wherever you have an mbox file, replace it with a
> maildir directory.
>
> You can even go one better by having maildirs which themselves contain
> subdirectories, something that is impossible with mbox files.
>
> For example, you could have a maildir called work which contains
> unsorted messages, and also sub-maildirs for each client. Each client
> maildir can contain messages as well as yet more sub-maildirs for e.g.
> certain projects, time periods, contacts, etc.
>
> I don't believe netscape includes support for the maildir format,
> though, so you may not be able to do this, unless you do something like
> run an IMAP server such as courier and use netscape to access the mail
> through the IMAP interface.
What I think you are saying is _exactly_ what I want to do!
Perhaps, I misunderstand something?
I have tried this, both via maildirmake [-f] and manually, neither of
which are seen by mutt ?!?!
Is this, then, a deficiency of mutt?
If so, what other than mutt ought I to consider?
What do you think?
Vineet Kumar
>
> Vineet Kumar wrote:
> >
> > * Michael D. Schleif (m...@helices.org) [030210 12:24]:
> > > Under netscrape I have directory hierarchies such as:
> > >
> > > Lists
> > > |__debian
> > > | |__debian-users
> > > | |__debian-devel
> > > |__tomcat-users
> > >
> > > With many lists and so many old mails, a flat list of _all_ Maildirs
> > > will quickly become difficult to navigate . . .
> > >
> > > How can this be done with Maildir?
> >
> > Just the same. Wherever you have an mbox file, replace it with a
> > maildir directory.
Well, it looks like I got a little carried away in my previous message.
The above statment is true: it's easy enough to just have folders which
contain maildirs or folders and maildirs which contain messages.
> >
> > You can even go one better by having maildirs which themselves contain
> > subdirectories, something that is impossible with mbox files.
> >
> > For example, you could have a maildir called work which contains
> > unsorted messages, and also sub-maildirs for each client. Each client
> > maildir can contain messages as well as yet more sub-maildirs for e.g.
> > certain projects, time periods, contacts, etc.
I was misleading (misled?) in saying that you can set up maildirs which
contain messages and maildirs. This is possible with a slightly hacky
extension to maildir which uses '.' as a hierarchy separator. So the
example I gave here would look like a big, ugly, flat list:
INBOX/
INBOX/.work
INBOX/.work.client1
INBOX/.work.client2
INBOX/.work.client2.2003-10
So this only gets nice when you use an access method that recognizes the
dots as hierarchy separators and provides you with a nice hierarchical
presentation, such as courier IMAP and Mozilla mail, for example.
I don't think mutt even really nails the hierarchical presentation bit
when used as an IMAP client. I don't even know that mutt supports IMAP
folder browsing and subscribing, though in fairness I haven't used it
extensively.
Don't give up on it yet, though; it does support the hierarchy you
currently use, with Maildirs in place of mboxes.
good times,
Vineet
--
http://www.doorstop.net/
--
"As we enjoy great advantages from inventions of others, we should be glad
of an opportunity to serve others by any invention of ours; and this we
should do freely and generously." --Benjamin Franklin
Vineet Kumar
> I don't think mutt even really nails the hierarchical presentation bit
> when used as an IMAP client. I don't even know that mutt supports IMAP
> folder browsing and subscribing, though in fairness I haven't used it
> extensively.
Wrong again; it does =)
Mutt supports browsing IMAP directories, indicating them as "IMAP" for
"regular" imap directories or "IMAP +" for directories that contain both
messages and subdirectories. The toggle-subscribed command (bound to
'T' by default) does as its name indicates: toggling between displaying
subscribed folders only or browsing all folders. Mutt allows you to
subscribe and unsubscribe from folders as well. I think its handling of
"IMAP" vs "IMAP +" folders is done very well: select-entry (<Return> by
default) will traverse into an "IMAP +" folder whereas view-file
(<Space> by default) will open an "IMAP +" folder to see the messages it
contains.
Mutt also supports setting the hierarchical separators (defaulting to
"/.", which accommodates courier's maildir subfolders).
So in short, I was wrong to ever doubt it; mutt is one rockin' IMAP
client.
good times,
Vineet
--
http://www.doorstop.net/
--
Rob Weir
> For me, each mail has two copies. I regularly read one of them, the
> "regular" copy, which is stored as maildir. I use spamassissin to scan for
> junk, let Gnus throw mails away without too much care, etc., so that I can
> control the number of files for storing mails (currently around 2000). To
> minimize the wastage on filesystem space, I use reiserfs on my mail
> partition, although an ext3 partition with tail files enabled should have
> similar effect.
This leads to an important point: some FS's do not handle huge
directories well. My d-u folder currently has ~5000 messages in it;
ext2 gets very, very, very slow above about 4000. I've switched to XFS
for everything, so I don't mind too much, but just be aware of this.
I think newer ext impltementations (2.5 at least, possibly 2.4.20 as
well) have much better huge directory handling.
--
Rob Weir <rw...@ertius.org> http://ertius.org/
c...@mindgate.net
Vineet Kumar wrote:
>
> * Cameron Matheson (cmath...@hotpop.com) [030209 22:25]:
> > Hi,
> >
> > > takes 1, same as an mbox. That works fine for archiving,
> > > though is not as convenient for active mailboxes. I also
> > > don't really buy that a maildir is difficult to back up
> > > (especially if you tar.(gz|bz2) it). I like using maildirs
> > > mostly because mbox feels like a dirty hack, with the whole
> > > "From " thing. I also like the increased scriptability
> > > using standard GNU tools like grep to find and process
> > > individual messages, instead of having to use some sort of
> > > mbox-parsing perl module.
> >
> > used mbox because that is what fetchmail puts my mail into
> > but this thread has aroused my interest in maildir... Can i
> > use maildir w/ fetchmail/exim? If not, how does one get
> > maildir, and what are the technical advantages?
>
> An mbox file is a file with all the messages laid out in it
> end-to-end, separated by lines beginning with "From ". Any
> lines in the body of a message that begin with "From " are then
> munged into ">From " by the program delivering into the mbox.
> That's just a dirty, dirty hack. Add to that that the file
> needs to be locked properly(and that file locking on NFS is not
> perfect) in order to prevent it from getting corrupted (and
> that corrupting the file means corrupting a whole mailbox, not
> a single message) and you have a technologically inferior mail
> storage design, IMHO. On the other hand, it's the original way
> of doing things, so support for it is universal. Also, loading
> an mbox is generally faster than loading a maildir.
There are ways of cheating, like having an index.
> Exim will happily deliver into maildirs if you uncomment the
> "maildir_format" line in the address_directory transport (and
> somehow specify the directory you'd like to deliver to, perhaps
> via .forward or by making a director that checks if ~/Maildir/
> or ~/Inbox/ exists and delivering there if so). I don't know
> about fetchmail, but if you have it configured to deliver via
> SMTP through the local exim anyway, it shouldn't be an issue.
> I do know that procmail and maildrop both support delivery into
> maildirs.
The procmail (or maildrop) solution is IMHO better than having
one program to rule them all. One program to fetch(mail), another
to deliver (exim), and another to filter (procmail).
> As for MUA support, I know mutt does, but I'm not 100% sure of
> others. It's fairly widely supported at this point, and I'd
> guess that the popular X GUI mailers (sylpheed, kmail,
> evolution) do support it as well. I see that you're using
> mutt, though, so you'll be fine.
Unless some new feature was added to CVS, Sylpheed(-claws)
supports MH, which isn't quite maildir. KMail supports maildir in
addition to what I believe is its version of mbox. I haven't
tried Evolution in ages. My User-Agent of choice, Wanderlust (an
elisp program running under Emacs) supports all three, mbox, MH
and maildir.
MH uses raw numerals to name each separate message; maildir uses
something fancier. I like MH because to delete mail I can forego
the fussiness of the MUA and just "cd" to the MH folder, type,
say, "rm 12??" and thereby remove messages 1200 to 1299 all in
one go.
Vineet Kumar
> On Sun, 9 Feb 2003 23:46:26 -0800,
> Vineet Kumar wrote:
> > of doing things, so support for it is universal. Also, loading
> > an mbox is generally faster than loading a maildir.
>
> There are ways of cheating, like having an index.
Right. AFAIK, Kmail does this, and I hear it works well. There has
been talk of adding a feature to mutt as well, but I haven't been
following its progress recently.
> > Exim will happily deliver into maildirs if you uncomment the
> > "maildir_format" line in the address_directory transport (and
> > somehow specify the directory you'd like to deliver to, perhaps
> > via .forward or by making a director that checks if ~/Maildir/
> > or ~/Inbox/ exists and delivering there if so). I don't know
> > about fetchmail, but if you have it configured to deliver via
> > SMTP through the local exim anyway, it shouldn't be an issue.
> > I do know that procmail and maildrop both support delivery into
> > maildirs.
>
> The procmail (or maildrop) solution is IMHO better than having
> one program to rule them all. One program to fetch(mail), another
> to deliver (exim), and another to filter (procmail).
Agreed, personally, but TMTOWTDI, and to each his own.
> > As for MUA support, I know mutt does, but I'm not 100% sure of
> > others. It's fairly widely supported at this point, and I'd
> > guess that the popular X GUI mailers (sylpheed, kmail,
> > evolution) do support it as well. I see that you're using
> > mutt, though, so you'll be fine.
>
> Unless some new feature was added to CVS, Sylpheed(-claws)
> supports MH, which isn't quite maildir. KMail supports maildir in
My mistake; that was just a guess without any facts to back it up.
> addition to what I believe is its version of mbox. I haven't
> tried Evolution in ages. My User-Agent of choice, Wanderlust (an
> elisp program running under Emacs) supports all three, mbox, MH
> and maildir.
Another unconfirmed guess here: I believe gnus does as well, for those
who run emacOS =)
good times,
Vineet
--
http://www.doorstop.net/
--
"Extremism in the defense of liberty is no vice.
Moderation in the pursuit of justice is no virtue." -- Barry Goldwater
Karsten M. Self
Good point.
My experience is that ext2/ext3 performance becomes unsatisfactory when
directory entries exceed about 3000 - 6000 files. I've worked with as
many as 128,000 files, and have had some experience with this.
The reason is that ext2/3 both use a list to manage directory entries.
Insert, delete, and search operations involve scanning this list,
serially. At several thousand entries, repeat operations may take many
seconds.
I'm not familiar with XFS, but reiserfs (which I usually use for large
directories) uses a hash table to store entries. Insertion, deletion,
and searches are therefor largely independent of directory size, and
performance for large directories is vastly superior.
I have seen reports that XFS beats both ext3 and reiserfs performance by
a huge factor -- recent Linux Journal article on the recent 64-way SGI
GNU/Linux server.
Peace.
--
Karsten M. Self <kms...@ix.netcom.com> http://kmself.home.netcom.com/
What Part of "Gestalt" don't you understand?
First they came for the Communists, and I didn't speak up, because I
wasn't a Communist. Then they came for the Jews, and I didn't speak
up, because I wasn't a Jew. Then they came for the Catholics, and I
didn't speak up, because I was a Protestant. Then they came for me,
and by that time there was no one left to speak up for me.
-- Rev. Martin Niemoller, 1945
Nathan E Norman
[ snip ]
> I'm not familiar with XFS, but reiserfs (which I usually use for large
> directories) uses a hash table to store entries. Insertion, deletion,
> and searches are therefor largely independent of directory size, and
> performance for large directories is vastly superior.
>
> I have seen reports that XFS beats both ext3 and reiserfs performance by
> a huge factor -- recent Linux Journal article on the recent 64-way SGI
> GNU/Linux server.
XFS is a great filesystem, and seems stable on i386. However, if
you're running debian on a non-i386 platform, don't expect XFS to work
well.
--
Nathan Norman - Incanus Networking mailto:nno...@incanus.net
Liberty may be endangered by the abuses of liberty as well as by
the abuses of power.
-- James Madison
Rob Weir
> On Sun, Feb 16, 2003 at 06:41:28AM +0000, Karsten M. Self wrote:
>
> [ snip ]
>
> > I'm not familiar with XFS, but reiserfs (which I usually use for large
> > directories) uses a hash table to store entries. Insertion, deletion,
> > and searches are therefor largely independent of directory size, and
> > performance for large directories is vastly superior.
> >
> > I have seen reports that XFS beats both ext3 and reiserfs performance by
> > a huge factor -- recent Linux Journal article on the recent 64-way SGI
> > GNU/Linux server.
>
> XFS is a great filesystem, and seems stable on i386. However, if
> you're running debian on a non-i386 platform, don't expect XFS to work
> well.
Really? I'd heard it was far better than (at least) reiserfs in it's
non-x86 stability. Also, it's endian-safe, which reiser isn't. I've at
least had people recommend it to me as the FS of choice on PPC machines.