scaling limits of one-message/one-file mailbox layouts

[Mark Crispin]

It was just claimed to me that, with the exception of nss, Linux
filesystems are limited to 32K objects/directory. That would seem to
impose a per mailbox message limit of 32K in those mailbox formats (such
as netnews spool, mh, maildir, and Cyrus) which store each message in a
separate file.

I'm not sure that I believe this. ISTR heavily spammed newsgroups with
6-digit message counts. I would have thought that our friends at CMU
would have hit a 32K message wall by now in Cyrus if it was real. We're
certainly working with mix mailboxes with many more than 32K messages.

The problem is, we're working in a space in which mix format might
eventually hit a 32K object wall, even with mix's consolidation of
multiple messages into data files. If that wall is real, I will need to
do more aggressive consolidation in mix than I am currently doing.

So, I'm kinda hoping that this wall isn't there, or at least isn't there
with modern versions filesystems...

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.

[Steve Thompson]

On Tue, 1 Dec 2009, Mark Crispin wrote:

> It was just claimed to me that, with the exception of nss, Linux filesystems
> are limited to 32K objects/directory. That would seem to impose a per
> mailbox message limit of 32K in those mailbox formats (such as netnews spool,
> mh, maildir, and Cyrus) which store each message in a separate file.

> [...]

Ext3: I have many maildir-format mailboxes with way more than 32K messages
(100K+). So no, it's not a limit in that sense. The ext3 file system does
have a limit of 32K subdirectories per directory (actually, 31998). As far
as I know, file count is only limited by the number of inodes, but
probably performance will tank way before this in many cases.

Reiser FS: has a limit of 518,701,895 files per directory.

Steve

[Oscar del Rio]

Mark Crispin wrote:
> It was just claimed to me that, with the exception of nss, Linux
> filesystems are limited to 32K objects/directory.

someone in this thread created 500,000 files per directory:

http://www.techtalkz.com/suse-linux/156660-maximum-amount-files-per-directory.html

Wikipedia says that ext3 has a limit of 32000 subdirectories

http://en.wikipedia.org/wiki/Ext4

(ext4) Features:
Break 32,000 subdirectory limit
In ext3 the number of subdirectories that a directory can contain
is limited to 32,000. This limit has been raised to 64,000 in ext4, and
with the "dir_nlink" feature it can go beyond this (although it will
stop increasing the link count on the parent). To allow for continued
performance given the possibility of much larger directories, Htree
indexes (a specialized version of a B-tree) are turned on by default in
ext4. This feature is implemented in Linux kernel 2.6.23. Htree is also
available in ext3 when the dir_index feature is enabled.

[Mark Crispin]

Thanks everybody for the information. The subdirectory limit is an
potential issue, but it's good to hear that the main worry is a non-issue.

Thanks again.

[Grant Taylor]

On 12/1/2009 7:55 PM, Mark Crispin wrote:
> Thanks everybody for the information. The subdirectory limit is an
> potential issue, but it's good to hear that the main worry is a non-issue.

I can see the possibility of running in to a file limit. But I can't
think of a usage case where you might run in to a directory limit. Will
someone please open my eyes and provide a real world example?

Grant. . . .

[Mark Crispin]

On Tue, 1 Dec 2009, Grant Taylor posted:

> I can see the possibility of running in to a file limit. But I can't think
> of a usage case where you might run in to a directory limit. Will someone
> please open my eyes and provide a real world example?

An obvious example is a Flat Directory From Hell which holds all user home
directories (and the obvious fix is not to do it that way...).

I've never been a fan of allowing directories to hold large number of
entries (whether files or subdirectories). The reasons are less important
today than in the past; but you still aren't doing yourself any favors.

The more that I deal with filesystems, the more attractive that solutions
such as Caringo's CAstor looks. It's amazing how little you actually need
a filesystem to do file storage.

[Tim Showalter]

Mark Crispin <m...@panda.com> writes:

> I'm not sure that I believe this. ISTR heavily spammed newsgroups
> with 6-digit message counts. I would have thought that our friends at
> CMU would have hit a 32K message wall by now in Cyrus if it was real.
> We're certainly working with mix mailboxes with many more than 32K
> messages.

You know, you had me a little freaked out for a minute. Then I
discoverd I have 78k messages in a Maildir folder on my Linux box and
I'm feeling better now. (My spam folder, of course.)

Tim