Is Qmail some kind of joke?
vl...@my-deja.com
I'm a Linux newbie so perhaps I just don't have a clue.
Qmail's author either doesn't respond to email or doesn't read his
email.
The link to Qmail RPMS from the qmail.org hasn't worked for weeks. No
one I send mail to from the qmail site seems to respond.
Getting Qmail and WU IMAP and Maildir to work together is a highly
undocumented task.
The patches to IMAP from the Qmail site apparently don't support
Maildir.
The authors of those patches don't respond to email.
Does anyone have docs on getting Qmail and Cyrus working?
Sent via Deja.com http://www.deja.com/
Share what you know. Learn what you don't.
(Scott Schwartz) Scott Schwartz
> Qmail's author either doesn't respond to email or doesn't read his
> email.
There's a very active qmail mailing list, and a web site for updates,
which the qmail documentation refers you to.
> The link to Qmail RPMS from the qmail.org hasn't worked for weeks. No
> one I send mail to from the qmail site seems to respond.
There's no need to use RPMs. Fetch and compile the source; it's just
as easy.
> Getting Qmail and WU IMAP and Maildir to work together is a highly
> undocumented task.
That's because the UW IMAP authors don't want you to use Maildir, for
reasons which are obscure.
> Does anyone have docs on getting Qmail and Cyrus working?
Check the mailing list archives.
Osma Ahvenlampi
> The link to Qmail RPMS from the qmail.org hasn't worked for weeks. No
> one I send mail to from the qmail site seems to respond.
Qmail's license is unfriendly to RPM's - it can not be distributed in
a modified archive. That means that making RPM's is painful (generally
you get the source and a SPEC file separately and build it yourself).
It's easier to just build Qmail from source, although I do the RPM bit
anyway because I prefer the system software to be in the RPM database
for the long run.
> Getting Qmail and WU IMAP and Maildir to work together is a highly
> undocumented task.
Because Maildir sucks performance-wise. A small patch in WU imapd will
allow it to fetch mail from $HOME/Mailbox, and sticking an empty
mbx-format file named INBOX in $HOME as well will guarantee fast imap
service for the inbox.
--
Is a computer language with goto's totally Wirth-less?
Osma Ahvenlampi <oa at iki fi> (damn spammers)
Mark Crispin
> That's because the UW IMAP authors don't want you to use Maildir, for
> reasons which are obscure.
Because Maildir has dreadful performance (worse than any supported mailbox
format) and doesn't scale, that's why. There are other technical (and
non-technical) reasons as well; but performance so wretched that you
wouldn't want to use it for a mailbox with more than a handful of messages
is enough.
Third-party Maildir drivers are available; however if you use them it is
strictly at your own risk (don't even think about sending in bug reports
or complaining about it being slow).
I doubt that you'll find the Cyrus guys any more willing to consider
supporting Maildir. They're probably even less willing than we are,
especially since it would make Cyrus slower than our server.
The recommended mailbox format in the UW IMAP toolkit is mbx, which is
much faster than the standard UNIX mailbox format and allows shared
read/write access.
In the future, we are looking into databases for very high performance
mailbox formats. Cyrus is halfway to being a database; it is not strictly
a one-file/one-message format since it uses an auxillary index that (among
other things) contains preparsed ENVELOPEs and BODYSTRUCTUREs. That's
where Cyrus gets a lot of its performance benefit.
-- Mark --
* RCW 19.190 notice: This email address is located in Washington State. *
* Unsolicited commercial email may be billed $500 per message. *
Science does not emerge from voting, party politics, or public debate.
D. J. Bernstein
performance for writes to a standard mbox.
I measured what was actually going on. The problem, in short, was Pine's
failure to buffer writes properly. One version was even using writev();
to experts, writev() screams ``THIS PROGRAM NEEDS BUFFERING!''
The user tried Mutt, http://www.mutt.org, and immediately noticed the
dramatic improvement in mbox performance.
So I find it richly amusing to see Crispin talking about mailbox speed
as a reason to avoid supporting a popular, NFS-safe, crashproof format
for incoming mail.
---Dan
P.S. Mutt also supports maildir, so users can try it out to see how
maildir actually performs in practice.
(Scott Schwartz) Scott Schwartz
> > That's because the UW IMAP authors don't want you to use Maildir, for
> > reasons which are obscure.
>
> Because Maildir has dreadful performance (worse than any supported mailbox
> format) and doesn't scale, that's why.
The phrase "doesn't scale" is usually a codeword for "works fine in
the usual case". In the limit nothing scales, and, moreover, not
every technique has to be used in every situation (that's why you
support a number of different mailbox formats, right?) Since I'm
experiencing the usual case, and not an extreme one, maildir works
nicely. That's just an empirical fact, not a value judgement.
> There are other technical (and
> non-technical) reasons as well; but performance so wretched that you
> wouldn't want to use it for a mailbox with more than a handful of messages
> is enough.
But see, unless you say what those things are, they remain obscure. :-)
And, not to belabor the obvious, but lots of people *do* use Maildir
and enjoy acceptable performance. That issue's been decided; now it's
just a question of which (if any) imap server to run.
> Third-party Maildir drivers are available; however if you use them it is
> strictly at your own risk (don't even think about sending in bug reports
> or complaining about it being slow).
Ironically, pretty much the only complaint that people have about
maildir is that you won't support it!
Mark Crispin
> I measured what was actually going on. The problem, in short, was Pine's
> failure to buffer writes properly. One version was even using writev();
> to experts, writev() screams ``THIS PROGRAM NEEDS BUFFERING!''
I refuse to get into a pissing contest with this individual. It is a
monumental waste of time.
Here are the facts:
Two issues relate to unix-format mailbox performance; newline conversion
and checkpointing (that is, updating the file on disk). For most users,
newline conversion is in the noise; checkpointing is what generates most
of the complaints. Since that's the only "disk writes" that are done,
the comment above must refer to checkpointing.
There is not a single writev() call in the code, and there has not been
for several years. The old writev() code had considerable buffering and
was timed favorably compared to stdio for the same amount of data. The
issue in that version was memory usage.
The current unix-format code uses stdio disk-block buffering. Buffering
is not the issue in this version either. The issue is that the current
algorithm is safe at the expense of speed; it spends its time writing the
mailbox update twice. You will see a remarkable difference if you have
/tmp mounted on a ramdisk.
An alternate algorithm, which makes the opposite tradeoff (at user
choice), in in the final stages of development and will be released in a
few days. Among other things, this algorithm skips updates of messages
which have unchanged status and position in the maibox. This algorithm
will probably become the default.
Other algorithms exist, but these algorithms have the flaw that are not
globally suitable. The fundamental problem is the design requirement of
the "set hold" mode on the spool file. A great deal of unnecessary is
work is done on non-spool files because of code that fufills the
requirements of the spool file. This requires interaction with the MTA,
and matters are made worse by the fact that some MTAs have a strict
requirement that the inode of the spool file remain unchanged.
mbx format has no need for either newline conversion or checkpointing.
> So I find it richly amusing to see Crispin talking about mailbox speed
> as a reason to avoid supporting a popular, NFS-safe, crashproof format
> for incoming mail.
For a moderate sized mailbox (2000 or so messages) on a DEC-5000, the
following times are obtained for an IMAP select and fetch-fast:
. mbx requires 1.7 user, 2.6 system, 19 real
. unix requires 8.6 user, 7.7 system, 20 real
. maildir requires 61.8 user, 24.4 system, and 1:45 real
Message data was identical in all three cases.
The additional cost for unix over mbx is message boundary hunting and
newline converstion. The additional cost for maildir over unix is in
filesystem operations: directory scan and per-message stat(), open(), and
close().
The differences become more pronounced as the mailbox increases in size;
as the number of files in a directory increases name/inode translation
(used by stat() and open()) becomes progressively more expensive. These
differences are difficult to measure with a 20 message mailbox, and remain
modest at 200 messages. Message counts of 10,000+ are not uncommon with
administrators.
The chief distinguishing factor of Cyrus is that it uses a proto-database
for just about everything except text fetching. Text fetching is
generally not done in bulk. so the filesystem overhead is kept to
managable levels. Because the proto-database maintains permanent
information that has to be reparsed each session, Cyrus is able to pull
ahead of the c-client supported formats. Maildir lacks such a
proto-database, and furthermore it can't have one and still claim to be
"NFS-safe".
A pure database has the capability of doing even better. I haven't timed
Exchange yet, since Exchange 5.5 has a known performance problem with text
sizes that's fixed in a later version. I hope to have some numbers soon.
And yes, we're looking into offering a pure database format in c-client,
as are other groups.
(Scott Schwartz) Scott Schwartz
> For a moderate sized mailbox (2000 or so messages) on a DEC-5000, the
> following times are obtained for an IMAP select and fetch-fast:
> . mbx requires 1.7 user, 2.6 system, 19 real
> . unix requires 8.6 user, 7.7 system, 20 real
> . maildir requires 61.8 user, 24.4 system, and 1:45 real
Are those figures for NFS, or for FFS? A big advantage of maildir is
that it is NFS resistant, and doesn't use file locking (which has
historically been buggy and slow in conjunction with NFS.)
Also, I'd be curious to see the figures for MH. You support that, and
if it turns out to be similar to maildir, then excluding maildir
doesn't seem consistent.
Mark Crispin
> Are those figures for NFS, or for FFS?
Local file system. Neither NFS nor AFS scale well for high performance
email usage.
> A big advantage of maildir is that it is NFS resistant, and doesn't
> use file locking (which has historically been buggy and slow in
> conjunction with NFS.)
The same holds true for the traditional UNIX mailbox format.
Strictly speaking, it is incorrect to claim that Maildir does not use file
locking. It simply shoves the locking problem down into the underlying
filesystem, at much higher system overhead.
> Also, I'd be curious to see the figures for MH. You support that, and
> if it turns out to be similar to maildir, then excluding maildir
> doesn't seem consistent.
mh is slightly faster; it doesn't have to worry about message filenames
being changed because an external process changed a message status.
The justification for supporting mh (albeit strongly discouraged, and with
crippling restrictions) is that it is a legacy format. Maildir has no
such excuse.
Nor do I have to be "consistant" in what I decide to support. You are
welcome to seek out another IMAP server that supports Maildir. The fact
that there doesn't seem to be any such server should suggest something.
Leslie Mikesell
Mark Crispin <m...@CAC.Washington.EDU> wrote:
>For a moderate sized mailbox (2000 or so messages) on a DEC-5000, the
>following times are obtained for an IMAP select and fetch-fast:
> . mbx requires 1.7 user, 2.6 system, 19 real
> . unix requires 8.6 user, 7.7 system, 20 real
> . maildir requires 61.8 user, 24.4 system, and 1:45 real
How do they compare when you delete and expunge a message?
My typical action for a mail message is to read and delete it.
I'd prefer not to incur extra overhead while other stuff in
the same file is re-written to release the space.
>The differences become more pronounced as the mailbox increases in size;
>as the number of files in a directory increases name/inode translation
>(used by stat() and open()) becomes progressively more expensive. These
>differences are difficult to measure with a 20 message mailbox, and remain
>modest at 200 messages. Message counts of 10,000+ are not uncommon with
>administrators.
So why not split the difference and allow mail delivery in one
message-per-file formats where (a) locking contention is
avoided (or at least handled correctly), (b) delivery of the
same copy to multiple recipients could be done with links, and
(c) deletion is cheap - then sweep the messages that aren't deleted
after the first access into a format better suited for long term
storage and subsequent access? The second format wouldn't need
to be known by anything else and would have less need to be
NFS safe.
The one-vs.-many messages per file argument always strikes me as
strange when most places have at least at one time run more
volume through news than mail. While it does have scaling problems
it works in the scale most places need for mail.
Les Mikesell
l...@mcs.com
Mark Crispin
> How do they compare when you delete and expunge a message?
It all depends upon how many messages are expunged, which in turn depends
upon the quality of your client.
If you always delete a single message, then expunge immediately, then a
one-file/one-message format works better because you avoid the mailbox
rewrite for subsequent undeleted messages in return for the overhead of a
file name lookup and iname removal. But only rotten clients (or foolish
users) behave that way.
In IMAP, typically you delete all the messages that you want to delete in
the session, then expunge the lot of them in a single operation. In this
case, the single file tends to win because:
1) in the one-file/one-message format, you must do an individual file
name lookup and iname removal per message.
2) in the flat file format, you only have to rewrite subsequent undeleted
messages (if any) then do a truncate.
> So why not split the difference and allow mail delivery in one
> message-per-file formats where (a) locking contention is
> avoided (or at least handled correctly), (b) delivery of the
> same copy to multiple recipients could be done with links, and
> (c) deletion is cheap - then sweep the messages that aren't deleted
> after the first access into a format better suited for long term
> storage and subsequent access? The second format wouldn't need
> to be known by anything else and would have less need to be
> NFS safe.
What you then have is use of multiple formats simultaneously (in the same
mailbox too?) -- something you generally want to avoid.
NFS is a poor architectural idea for mail data, and taking a massive
performance hit to be "NFS safe" seems to be a poor choice compared to
having a distributed architecture that doesn't depend upon NFS.
You'd get better benefits by moving to a database and not depending upon
the semantics of a filesystem. A database is also where many of the
advanced facilities of IMAP (e.g. ACLs) lead you; it's really hard to
get ACLs and the UNIX permissions model to play ball with each other.
Chris Stratford
vl...@my-deja.com writes:
> Does anyone have docs on getting Qmail and Cyrus working?
Set up Cyrus as normal, then create a .qmail file for each user that
contains something like:
|deliver -a <user> -e <user>
That's all you need.
Chris.
--
Chris Stratford Email: Chris.S...@uk.uu.net
UK Postmaster Voice: +44(0)1223 250690
UUNET Fax: +44(0)1223 250650
An MCI WorldCom Company WWW: http://www.uk.uu.net
Dave Barr
Mark Crispin <m...@CAC.Washington.EDU> wrote:
>Nor do I have to be "consistant" in what I decide to support. You are
>welcome to seek out another IMAP server that supports Maildir. The fact
>that there doesn't seem to be any such server should suggest something.
Sometimes when you fill a vacuum it still sucks.
--Dave
--
http://www.cis.ohio-state.edu/~barr/
ba...@cis.ohio-state.edu
Bennett Todd
>For a moderate sized mailbox (2000 or so messages) on a DEC-5000, the
>following times are obtained for an IMAP select and fetch-fast:
> . mbx requires 1.7 user, 2.6 system, 19 real
> . unix requires 8.6 user, 7.7 system, 20 real
> . maildir requires 61.8 user, 24.4 system, and 1:45 real
Youch. Just took me 28 seconds the first time, 4.71 seconds on the second try
real to fire up mutt, read a 4,827 message Maildir folder, paint the screen,
and exit. This is on a K6 at 300MHz (not sure which species of K6), 64MB RAM,
running Red Hat 5.2.
I can see why you wouldn't like Maildir, with the performance tyou enjoy in
your implementation. But for the rest of us, it's plenty fast:-).
-Bennett
Mark Crispin
mutt does not do the equivalent of an IMAP "fetch fast on all messages" to
open a Maildir folder and paint an initial screen. This is because mutt
is not an IMAP server. It has everything under its control. It doesn't
have to comply with the demands of external software, no matter how
ridiculous those demands might be.
The problem is that many POP and IMAP clients (including certain ones from
Very Big Vendors) do the evil thing. Some do something even worse, a
"fetch all" or "fetch full" on all messages. This is not something that
the POP or IMAP server can control or prevent.
In order to do an equivalent test with mutt, you need to have it run under
a simulation which will make it paint every message in the mailbox (e.g. a
4900 line screen). Once you have such a simulation in place, then get the
numbers between mbx, unix, and Maildir.
On Tue, 8 Jun 1999, Bennett Todd wrote:
> Youch. Just took me 28 seconds the first time, 4.71 seconds on the second try
> real to fire up mutt, read a 4,827 message Maildir folder, paint the screen,
> and exit. This is on a K6 at 300MHz (not sure which species of K6), 64MB RAM,
> running Red Hat 5.2.
>
> I can see why you wouldn't like Maildir, with the performance tyou enjoy in
> your implementation. But for the rest of us, it's plenty fast:-).
-- Mark --
Martin Bligh
(Scott Schwartz))bio.cse.psu.edu (Scott Schwartz <"schwartz+!usenet%"@> wrote:
>And, not to belabor the obvious, but lots of people *do* use Maildir
>and enjoy acceptable performance. That issue's been decided; now it's
>just a question of which (if any) imap server to run.
Interesting plan to choose your storage format before your mail server ...
>> Third-party Maildir drivers are available; however if you use them it is
>> strictly at your own risk (don't even think about sending in bug reports
>> or complaining about it being slow).
>
>Ironically, pretty much the only complaint that people have about
>maildir is that you won't support it!
Hell, you'd better get a refund for that free software right away.
If he won't give you a check for $0, I'd sue him personally.
How dare he decide what to include, and not to include, in software that
he wrote, and chose to give to you for free. Outrageous.
Martin
Leslie Mikesell
Mark Crispin <m...@CAC.Washington.EDU> wrote:
>> How do they compare when you delete and expunge a message?
>If you always delete a single message, then expunge immediately, then a
>one-file/one-message format works better because you avoid the mailbox
>rewrite for subsequent undeleted messages in return for the overhead of a
>file name lookup and iname removal. But only rotten clients (or foolish
>users) behave that way.
If it is the server storing the messages in a format where the
most likely action is inefficient, why is it the user who is
being foolish for doing the nomal thing?
>In IMAP, typically you delete all the messages that you want to delete in
>the session, then expunge the lot of them in a single operation. In this
>case, the single file tends to win because:
> 1) in the one-file/one-message format, you must do an individual file
> name lookup and iname removal per message.
> 2) in the flat file format, you only have to rewrite subsequent undeleted
> messages (if any) then do a truncate.
If you read messages often, that still results in frequent deletions.
And, the flat file turns out to be much more complicated in the
common case of deliveries concurrent with the rewrite operation.
>> So why not split the difference and allow mail delivery in one
>> message-per-file formats where (a) locking contention is
>> avoided (or at least handled correctly), (b) delivery of the
>> same copy to multiple recipients could be done with links, and
>> (c) deletion is cheap - then sweep the messages that aren't deleted
>> after the first access into a format better suited for long term
>> storage and subsequent access? The second format wouldn't need
>> to be known by anything else and would have less need to be
>> NFS safe.
>
>What you then have is use of multiple formats simultaneously (in the same
>mailbox too?) -- something you generally want to avoid.
I can see a small amount of programming complexity in presenting
INBOX as a single entity even though you have optimized the storage
for the short and long term components, but I would think eliminating
the contention between deletion write-backs and new deliveries
would be worth it.
>NFS is a poor architectural idea for mail data, and taking a massive
>performance hit to be "NFS safe" seems to be a poor choice compared to
>having a distributed architecture that doesn't depend upon NFS.
Frequently doing complete rewrites of a potentially huge file to
accomplish deletitions and at the same time dealing with locking
issues for competing deliveries doesn't sound so hot either. Even
without NFS considerations the atomic operations of the filesystem
would be a big help here.
>You'd get better benefits by moving to a database and not depending upon
>the semantics of a filesystem. A database is also where many of the
>advanced facilities of IMAP (e.g. ACLs) lead you; it's really hard to
>get ACLs and the UNIX permissions model to play ball with each other.
Once you give up the possibility of using traditional unix
mailbox-file aware mailers, that all comes under the control of
the server and delivery agent with or without any special database
backend. A database might be able to perform better than the
native filesystem, but my experience is that a filesystem is
actually pretty good at storing files as long as you don't
put too many in the same directory, where 'too many' may be
anywhere from several hundred to several thousand depending
on the filesystem and machine. Consolidating the new deliverys
at the time you perform deletions would normally avoid the
scaling problem.
Les Mikesell
l...@mcs.com
Osma Ahvenlampi
> If it is the server storing the messages in a format where the
> most likely action is inefficient, why is it the user who is
> being foolish for doing the nomal thing?
The client is. For instance, my IMAP client never does DELETE/EXPUNGE
immediately on a message I mark as deletable. It doesn't even do
DELETE on it immediately because I might be forced to use a more
stupid client occassionally. Instead, it uses a special mark for
"expirable", hides the message from me, and will perform a conditional
DELETE/EXPUNGE for all "expirable" messages over n days old. That
basically means two things: the mailbox contains messages I might have
"deleted" two weeks ago, which is good in case I change my mind and
need to see the message again, and a DELETE/EXPUNGE will be performed
once a day, cleaning up more than one message at a time.
It's both efficient and useful.
> native filesystem, but my experience is that a filesystem is
> actually pretty good at storing files as long as you don't
> put too many in the same directory, where 'too many' may be
Most filesystems are bad at storing files that are not at least
roughly four times the size of the block size on average. A file per
message model not only makes directory operations slower, it also
wastes diskspace and inodes (on filesystems where the maximum number
of files is limited, which is most, if not all, UNIX filesystems).
--
Death: to stop sinning suddenly.
Steve Snodgrass
> Most filesystems are bad at storing files that are not at least
> roughly four times the size of the block size on average. A file per
> message model not only makes directory operations slower, it also
> wastes diskspace and inodes (on filesystems where the maximum number
> of files is limited, which is most, if not all, UNIX filesystems).
On the flip side of the coin, there is a potential savings here. Our company
has *lots* of internal mailing lists. This means many users are often
receiving the same message. I wrote a "single store" patch for the Cyrus IMAP
server (which has been previously posted in the newsgroup) that allows Cyrus
to hard-link messages to multiple recipients together. So the "one file per
message" model in that case actually allows us to save lots of disk space and
I/O, since a message to 20 people ends up as one inode with 20 hard links.
Here's a partial listing from my inbox:
-rw------- 1 cyrus mail 2600 Jun 8 15:34 5062.
-rw------- 1 cyrus mail 1071 Jun 9 10:39 5089.
-rw------- 3 cyrus mail 2677 Jun 9 11:45 5096.
-rw------- 3 cyrus mail 33362 Jun 9 13:45 5099.
-rw------- 2 cyrus mail 4079 Jun 9 16:39 5118.
Notice the link counts on the left. I have seen link counts as high as 40 on
our server - the size of sendmail's recipient buffer prevents it from going
much higher since sendmail ends up "chunking" deliveries to lots of
recipients. I could increase that buffer size, but it's probably not worth
the effort.
Of course, as Mark has pointed out, Cyrus also uses index files to get around
the performance penalty imposed by the "one message per file" format. So it
seems that I'm getting the best of both worlds at this point. :-)
--
Steve "Pheran" Snodgrass * ssno...@fore.com * FORE Systems Unix Administrator
Geek Code: GCS d? s: a- C++ US++++$ P+++ L+ w PS+ 5++ b++ DI+ D++ e++ r++ y+*
"I want to take over the world because I'm an egomaniac." --Larry Wall
Leslie Mikesell
Osma Ahvenlampi <oa-ne...@spray.fi> wrote:
>l...@Mars.mcs.net (Leslie Mikesell) writes:
>> If it is the server storing the messages in a format where the
>> most likely action is inefficient, why is it the user who is
>> being foolish for doing the nomal thing?
>
>The client is. For instance, my IMAP client never does DELETE/EXPUNGE
>immediately on a message I mark as deletable. It doesn't even do
>DELETE on it immediately because I might be forced to use a more
>stupid client occassionally. Instead, it uses a special mark for
>"expirable", hides the message from me, and will perform a conditional
>DELETE/EXPUNGE for all "expirable" messages over n days old. That
>basically means two things: the mailbox contains messages I might have
>"deleted" two weeks ago, which is good in case I change my mind and
>need to see the message again, and a DELETE/EXPUNGE will be performed
>once a day, cleaning up more than one message at a time.
>
>It's both efficient and useful.
Yow! You mean the mail store actually has to hold an extra 2
weeks worth of stuff that a person has said they don't ever want
to see again? I'm not sure I consider that a good thing. We
send lots of large attachments around and have other
mechanisms in place to provide backups.
>> native filesystem, but my experience is that a filesystem is
>> actually pretty good at storing files as long as you don't
>> put too many in the same directory, where 'too many' may be
>
>Most filesystems are bad at storing files that are not at least
>roughly four times the size of the block size on average. A file per
>message model not only makes directory operations slower, it also
>wastes diskspace and inodes (on filesystems where the maximum number
>of files is limited, which is most, if not all, UNIX filesystems).
Before you decide that your machine doesn't like putting messages
in single files, ask yourself if you even noticed the fact that
sendmail temporarily stored that same message as 2 files on the
same machine before it was delivered to the mailbox. I've
never seen anyone mention that as a performance problem. The
problem you do see, as with news, is letting vast numbers of
files accumulate in a single directory, which is why I suggest
sweeping the new deliveries into a more efficient long term
storage format as soon as you see that they are likely to be
around a while (i.e. they aren't deleted after the first reading).
A large percentage of our email goes to multiple recipients
(interest group, work group, department, whole company) and
it would be a big win to be able to deliver a single copy
with hard links into the other directories. The sweep into
another storage format might consider the link count in
deciding which way is actually more efficient.
Les Mikesell
l...@mcs.com
Mark Crispin
> Of course, as Mark has pointed out, Cyrus also uses index files to get around
> the performance penalty imposed by the "one message per file" format. So it
> seems that I'm getting the best of both worlds at this point. :-)
Right. One disadvantage of a database (or proto-database such as
Cyrus) is that you lose the ability to access the data outside of POP or
IMAP, because the data can not be owned by an individual user.
You could, in theory, hack up Cyrus to run that way (I don't know how hard
it would be) but that would preclude your use of hard links to share the
data. In my opinion, hard links are the obvious thing to do and I'm
surprised that Cyrus doesn't do it as standard practice.
Lyndon Nerenberg
> In my opinion, hard links are the obvious thing to do and I'm
> surprised that Cyrus doesn't do it as standard practice.
If you inject using the deliver program, you never get more than
one recipient address per injection[*], so you can't hard link even
if you want to. We didn't add hard links to our server until we
switched over to LMTP as the default delivery model.
Even with our current server, if you use the deliver command supplied
with it to do delivery, you don't get hard links. C'est la vie.
[*] Unless you run using a very insane configuration ...
--
The two most common elements in the universe are Hydrogen and stupidity.
-- Harlan Ellison
Rupa Schomaker
> Yow! You mean the mail store actually has to hold an extra 2
> weeks worth of stuff that a person has said they don't ever want
> to see again? I'm not sure I consider that a good thing. We
> send lots of large attachments around and have other
> mechanisms in place to provide backups.
We use the same client (emacs+gnus). You can choose how long these
messages are in limbo and you can also set complex rules for which are
expired sooner than others (really). Say I want to hold messages from
my boss for 2 weeks but want to expire messages from the noisey jokes
list immediate -- I can. IMAP by itself doesn't offer this
flexibility.
Oh, and the messages aren't gone for good, you tell your MUA to show
you these "expired but not deleted" messages. If you want to
explicitly delete a message (and not go through the expire phase) then
there is a mark for that as well (B-del).
> Before you decide that your machine doesn't like putting messages
> in single files, ask yourself if you even noticed the fact that
> sendmail temporarily stored that same message as 2 files on the
> same machine before it was delivered to the mailbox. I've
> never seen anyone mention that as a performance problem.
Umm... Actually it is a big problem with high volume sites. Thats why
many of us run something else like PostFix or qmail.
> The
> problem you do see, as with news, is letting vast numbers of
> files accumulate in a single directory, which is why I suggest
> sweeping the new deliveries into a more efficient long term
> storage format as soon as you see that they are likely to be
> around a while (i.e. they aren't deleted after the first reading).
> A large percentage of our email goes to multiple recipients
> (interest group, work group, department, whole company) and
> it would be a big win to be able to deliver a single copy
> with hard links into the other directories. The sweep into
> another storage format might consider the link count in
> deciding which way is actually more efficient.
No disagreement there...
--
-rupa
Steve Snodgrass
> If you inject using the deliver program, you never get more than
> one recipient address per injection[*], so you can't hard link even
> if you want to. We didn't add hard links to our server until we
> switched over to LMTP as the default delivery model.
Yes, but Cyrus deliver supports LMTP, so there's no reason not to hard link if
you're in LMTP mode, which is what my patch does. It's true that there's no
reasonable way to support hard links without using LMTP, though.
Bennett Todd
>Interesting plan to choose your storage format before your mail server ...
How so? I tend to specify things in roughly descending order of how hard they
are to change. The basic nature of the OS comes first --- Unix. Next you
need to choose the data store --- migrating from one to another is generally
more disruptive than any other change, and the nature of the data store is
going to define the limits on possible growth plans (e.g. buy a horking big
switch and a couple of months production output from Netapp and serve email
to all carbon-based life forms on earth:-). Once you've picked the data store
(Maildir) the particular platform comes next; these days, likely a hotrod PC
with Linux; up until a couple of years back it'd have been a Sun w/ Solaris
2.5 or better; prior to that it'd have been a Sun with SunOS 4.x. Once we've
settled the platform, then you shop for the daemon whose feature set comes
closest to meeting your needs for today. If you've made good choices for the
earlier (harder to change) parts of your decision process, the actual precise
server should be an interchangable part --- and if you've designed to scale to
multiple-of-everything, you might not even need to be running the same server
on all boxes.
-Bennett
Leslie Mikesell
>> Before you decide that your machine doesn't like putting messages
>> in single files, ask yourself if you even noticed the fact that
>> sendmail temporarily stored that same message as 2 files on the
>> same machine before it was delivered to the mailbox. I've
>> never seen anyone mention that as a performance problem.
>
>Umm... Actually it is a big problem with high volume sites. Thats why
>many of us run something else like PostFix or qmail.
It isn't the volume that gets you, it is the number that you can't
deliver immediately causing a large number files to accumulate
in the queue directory at one time. You won't see this problem
on local deliveries where they don't accumulate, and you could
avoid it on a one-message-per-file delivery by letting the IMAP
server move/remove the files instead of letting them accumulate.
If you have people using POP, it would take care of itself.
Les Mikesell
l...@mcs.com
Osma Ahvenlampi
> Yow! You mean the mail store actually has to hold an extra 2
> weeks worth of stuff that a person has said they don't ever want
> to see again? I'm not sure I consider that a good thing. We
Well, I can configure the expiry period on a per folder basis. And I
very rarely have any reason to say I don't ever want to see a message
again. Instead, I say that I have no reason to keep this around for
longer than, depending on the folder, one week to three months. Until
then, it's kept around just in case, and then it goes away without
further action from my part.
The extremely nice thing about it, that I think in the long run SAVES
disk space, is that since I know mail will get cleaned up after it
loses relevance, but not before, I don't have to care about it
myself. I never have to "clean up my INBOX" by hand - many people I
know, most in the place I work, have INBOXes of 2000 messages or more,
mostly because they couldn't delete something immediately and going
back to do so later is too time-involving to do very often (or ever,
as is the case for some). In comparison, my INBOX has less than 500
messages, and has been so for many months. New stuff comes in, but old
stuff gets expunged at about the same rate.
> Before you decide that your machine doesn't like putting messages
> in single files, ask yourself if you even noticed the fact that
> sendmail temporarily stored that same message as 2 files on the
> same machine before it was delivered to the mailbox. I've
> never seen anyone mention that as a performance problem. The
Heh. Sendmail is a huge performance problem, although not necessarily
because of that. You have to realize though that sendmail's queue
serves a totally different purpose than a mail folder - it's
explicitly transitionary data.
A better example is news storage. For years, news servers have stored
every message in a separate file. It's nice for crossposts, but wastes
amazing amounts of space and needs a huge amount of processing
otherwise (profile the expiry process on an INN 1.x system
sometime). Recently, however, high-performance NNTP servers have
started to appear that store all the messages in each group per one
day into a single file that saves a lot of partial blocks and is much
faster to process.
--
Left to themselves, things tend to go from bad to worse.
Martin Bligh
If I wanted a mail 'solution', I'd specify what I wanted it to be able to
do, then go find an application that did it, then choose the OS, then the
hardware. Perhaps one of the original criteria for the application might be
that it runs on UNIX, as there's only expertise on that particular OS within
the organisation.
Why would migrating to a different mailstore format be 'more disruptive than
any other change'? I would have thought the most disruptive change is anything
that requires you to change your email client. If you wanted to move from,
say, POP3 to IMAP, you may have to force all your users to move client
application. Now *that's* disruptive! Changing mailstore should be completely
tranparent to the end user - just requires a conversion tool / script.
My criteria for choice went something along the lines of "must store on
a central server, accessible from everywhere, always keep mail on the server,
and be able to file under different mailboxes on the server, client must
be distructible / changable without any loss of email". The only open standard
that I could find that would do this was IMAP. Thus I wanted an IMAP server.
I really don't care what the backend mail store is - as long as the format is
published, and / or it's accessible to IMAP, I can always convert formats if
I want to. Use whatever makes sense for performance / backup / whatever.
Nor can I understand why anyone would want to access their mailboxes over NFS
when they already have IMAP (as far as I recall, that's how this thread
started, and why people want Maildir). What's the point of implementing a
client-server protocol, then introducing another massive layer of complexity
beneath it, which you don't need?
Martin.
Chris Barrera
on an NFS server even if the sole thing that accesses it is the IMAP server.
Sometimes one will want to separate storage from the host CPU, on a higher
availibility system. If the server host crashes and burns, it is easy to
have a standby system take over within minutes without the mail store itself
being unavailable. Sort of the cheap "high availibility" solution.
There are undoubtedly many other reasons to have the mail store accessible NFS,
say in a mixed IMAP/NFS environment where some users use IMAP and others do
not. I admit this is stupid, but Sun writes their dtmail IMAP client and
their IMAP server support this type of environment. There are even some
weird behaviors that result from this I won't even go into right now...
Chris Barrera
cbar...@msp.sc.ti.com
Martin Bligh <mbl...@sequent.com> wrote:
: Nor can I understand why anyone would want to access their mailboxes over NFS
Mark Crispin
> Sometimes one will want to separate storage from the host CPU, on a higher
> availibility system. If the server host crashes and burns, it is easy to
> have a standby system take over within minutes without the mail store itself
> being unavailable. Sort of the cheap "high availibility" solution.
"Cheap" is a good way of describing it. I would not use certain other
adjectives, e.g. "inexpensive", "robust", "high performing".
> There are undoubtedly many other reasons to have the mail store accessible NFS,
> say in a mixed IMAP/NFS environment where some users use IMAP and others do
> not. I admit this is stupid, but Sun writes their dtmail IMAP client and
> their IMAP server support this type of environment.
I heard that this was over the vehement and frantic objections of the IMAP
software developers at Sun...
I've never met anyone who does IMAP server development and considers NFS
to be a suitable back-end store for an IMAP server, but I've met several
who commiserate about the horrors of such a thing. It's a good topic for
war stories in the hotel bar... ;-)
Andrew Lochart
>
> Because an administrator might still want the server's mail store to exist
> on an NFS server even if the sole thing that accesses it is the IMAP server.
> availibility system. If the server host crashes and burns, it is easy to
> have a standby system take over within minutes without the mail store itself
> being unavailable. Sort of the cheap "high availibility" solution.
True, but why not just set up an IMAP server with high-availability host
(redundant, hot standby CPUs) attached to the disk subsystem? You get
your inexpensive HA and avoid the problems asscoiated with NFS.
Cheers,
Andrew
--
Andrew Lochart Director, Customer Marketing Mirapoint, Inc.
and...@mirapoint.com http://www.mirapoint.com +1-408-517-1326
Mark Crispin
> Intuition (with no empirical evidence to back it up) would lead me to
> suspect that IMAP is mostly IO based, not CPU intensive
This is our experience at UW with our server, and it's even more the case
with Cyrus.
> and thus you'd
> save very little load (if any) on the server by using NFS rather than
> IMAP straight from it.
Yup. I can't believe how many people think that they'd "get better
performance" by having a rack of IMAP server CPUs talking to a single NFS
filesystem.
Then there's the folks who think that they'll "get better reliability",
although it bewilders me why an IMAP server CPU is any more likely to
crash than an NFS server CPU.
As best as I can understand, their thinking goes like this:
Assume that there are 10 CPUs. With 9 IMAP server CPUs going to one NFS
server CPU, the failure of a single CPU has a 90% of simply taking out one
IMAP server IP address but not shutting down availability for anyone.
There is, however, a 10% chance of a CPU failure shutting down
availability for everyone.
With 10 IMAP server CPUs, each with their own disk and user communities,
the failure of a single CPU will take out service to 10% of the user
community.
Our feeling is that (1) we do our damnedest to reduce CPU failures to as
close to zero as we can, and (2) we'd rather have a little failure to deal
with, than run the risk of a catastropic failure.
Martin Bligh
Chris Barrera <cbar...@melon10.msp.sc.ti.com> wrote:
>> Nor can I understand why anyone would want to access their mailboxes over
>> NFS when they already have IMAP (as far as I recall, that's how this
>> thread started, and why people want Maildir). What's the point of
>> implementing a client-server protocol, then introducing another massive
>> layer of complexity beneath it, which you don't need?
>
>on an NFS server even if the sole thing that accesses it is the IMAP server.
>Sometimes one will want to separate storage from the host CPU, on a higher
>availibility system. If the server host crashes and burns, it is easy to
>have a standby system take over within minutes without the mail store itself
>being unavailable. Sort of the cheap "high availibility" solution.
So ... you shift the single point of failure from the IMAP server to
the NFS server, and introduce a pile of complexity, and an ugly manual
failover? Not sure I follow the logic behind this ... are you saying you
can only afford one reliable machine, and aren't willing to run the IMAP
server on it?
Intuition (with no empirical evidence to back it up) would lead me to
suspect that IMAP is mostly IO based, not CPU intensive, and thus you'd
save very little load (if any) on the server by using NFS rather than
IMAP straight from it. The only other reason that springs to mind not
to run IMAP straight from the server is security concerns, but then if
you're running NFS, I don't think you're in much of a position to throw
stones ;-)
So I'm still not sure why you wouldn't run IMAP straight from what is now
your NFS server ... ??? Still a single point of failure either way.
(assuming all along that you don't want to shell out for a clustered machine
with shared disk).
Martin.
Chris Barrera
Oddly enough I don't run IMAP as yet ;)
The motivation I see for using a separate NFS server for the mail store
is to have multiple MX hosts that can deliver into the same mail spool,
say if one loses its marbles the other takes over. That is the reason for
spending the money on a higher availibility NFS server and perhaps using
cheaper but still reasonably sturdy equipment to run sendmail. If the NFS
server it a Network Appliance system, then should somehow, if ever, mailbox
corruption does occur, then an earlier version of the mailbox can be
retrieved from read-only snapshots performed automatically by the filer
itself. This is on top of RAID protection, and of course, backups.
Is it truly worth it? I am sure this itself would make for good discussion,
especially since NFS is such an evil beast in many minds around here ;) I
can say, however, that in 4 years of such an operation I have never seen
mailbox corruption due to NFS locking problems. In fact, mailbox corruption was
always a rare event, and the only times it happened we were able to attribute
it to the user running multiple mail-readers at the same time. This is on a
network of hundreds of UNIX workstations and a few thousand users...
However, when in the future I start running an IMAP server I certainly do
intend to remove all NFS client access to the spool. Security is the main
concern and I still don't like having hundreds of client workstations
and user mail access having such reliance on NFS availibility and performace
to said workstations. I still ponder the possibility of having, say, two
IMAP servers each able to access the same mail store on an NFS server. I
know what I will hear from many vocal and notable people in this newsgroup,
NFS is evil in every circumstance. While I don't doubt others experience,
I have yet to see such nasties for myself. I never discount the possibility
of course, I still do everything possible in the environment to minimize
problems. But if people say this isn't robust, then I say that perhaps
many people haven't exercised enough control over their environment.
Even Oracle under certain circumstances supports the database store being
on one NFS server and the actual database server processes running on another
CPU host. So why not IMAP?
Chris Barrera
cbar...@msp.sc.ti.com
Martin Bligh <mbl...@sequent.com> wrote:
: In article <7jrsla$rro$1...@tilde.csc.ti.com>,
(Scott Schwartz) Scott Schwartz
> I've never met anyone who does IMAP server development and considers NFS
> to be a suitable back-end store for an IMAP server, but I've met several
> who commiserate about the horrors of such a thing. It's a good topic for
> war stories in the hotel bar... ;-)
Look, that's not the point. This isn't about imap, or about
performance. It's about your c-client library. Your mail
applications support MH, but Maildir is just as good, so why not
support that too?
(Scott Schwartz) Scott Schwartz
> As best as I can understand, their thinking goes like this:
>
> Assume that there are 10 CPUs. With 9 IMAP server CPUs going to one NFS
> server CPU, the failure of a single CPU has a 90% of simply taking out one
> IMAP server IP address but not shutting down availability for anyone.
> There is, however, a 10% chance of a CPU failure shutting down
> availability for everyone.
As I understand it, the thinking is,
Assume that there are 10 CPUS. With 4 IMAP server CPUs going to 6 NFS
server CPUS, the failure of a single NFS CPU isn't so bad, and it's a
lot more flexible than using local filesystems for everything,
especially since IMAP isn't our favorite access method anyway.
But that's not the point. The point is, you support MH folders,
so why not Maildir folders too?
(Scott Schwartz) Scott Schwartz
> On 7 Jun 1999, Scott Schwartz wrote:
> > Are those figures for NFS, or for FFS?
>
> Local file system. Neither NFS nor AFS scale well for high performance
> email usage.
But, see, you posted numbers for the wrong scenario, in order to back
up one of those "doesn't scale" assertions. Why not run the right
experiment?
Anyway, the fact is that they do scale more than well enough for the
usage a lot of people care about, so you can't just wish them away
like that.
> > A big advantage of maildir is that it is NFS resistant, and doesn't
> > use file locking (which has historically been buggy and slow in
> > conjunction with NFS.)
>
> The same holds true for the traditional UNIX mailbox format.
Yes, maildir enjoys benefits in both cases, but in the NFS case you
have to do locks over the network, which is slower, and magnifies the
effect.
> Strictly speaking, it is incorrect to claim that Maildir does not use file
> locking. It simply shoves the locking problem down into the underlying
> filesystem, at much higher system overhead.
It doesn't use flock, or lockf, or fcntl. Is rename really more
expensive? It's certainly less buggy, in the network case.
> > Also, I'd be curious to see the figures for MH. You support that, and
> > if it turns out to be similar to maildir, then excluding maildir
> > doesn't seem consistent.
>
> mh is slightly faster; it doesn't have to worry about message filenames
> being changed because an external process changed a message status.
folder -pack
sortm
> The justification for supporting mh (albeit strongly discouraged, and with
> crippling restrictions) is that it is a legacy format. Maildir has no
> such excuse.
Hmm.
> Nor do I have to be "consistant" in what I decide to support. You are
> welcome to seek out another IMAP server that supports Maildir. The fact
> that there doesn't seem to be any such server should suggest something.
The truth is that I couldn't care less about imap (I already have an
authenticated network filesystem, thanks), but since you also produce
pine, which some of my friends use, I thought it would be worth
broaching the issue.
Sasha Mikheev
>On 12 Jun 1999, Martin Bligh wrote:
>
>Yup. I can't believe how many people think that they'd "get better
>performance" by having a rack of IMAP server CPUs talking to a single NFS
>filesystem.
It allows separate machines to act as POP3, IMAP or MX servers. Such
separation of services is a very good idea. It allows better tuning
and in case that one service goes down the rest is still running.
CPU is the issue too. Multiple processes simultaneously writing
to disk will rise load quite a bit.
>
>Then there's the folks who think that they'll "get better reliability",
>although it bewilders me why an IMAP server CPU is any more likely to
>crash than an NFS server CPU.
You use NFS servers which rarely crash and reboot fast. Such as
Netapp filers. They reboot under 2 minutes, rarely crash and have
other nice features such as snapshots. Also newer filers have a
failover options.
Another issue here is price performance. NFS setup scales horizontally
very well. The largest investment is a good NFS server. For cpu nodes
commodity workstations can be used. In my opinion alternatives are
going to be more expensive and more difficult to manage.
>
>As best as I can understand, their thinking goes like this:
>
>Assume that there are 10 CPUs. With 9 IMAP server CPUs going to one NFS
>server CPU, the failure of a single CPU has a 90% of simply taking out one
>IMAP server IP address but not shutting down availability for anyone.
>There is, however, a 10% chance of a CPU failure shutting down
>availability for everyone.
>
You presume that the chance of NFS server going down are the same as
the CPU node. In my expirience this is not true.
--
Sasha Mikheev Linux -- put a penguin in your processor
System Manager at Netvision Israel, Ltd
Tel +972-4-8560600
http://aldan.netvision.net.il/~sasha
Joachim Achtzehnter
>
> But that's not the point. The point is, you support MH folders,
> so why not Maildir folders too?
If you read the whole thread you will find that he has answered this
question already! MH may be (almost) as bad as Maildir in Mark's
opinion, but it is an old, traditional format, hence it is
supported. Maildir is a newcomer, and Mark doesn't want to encourage
yet another bad format.
Joachim
--
joa...@kraut.bc.ca (http://www.kraut.bc.ca)
joa...@mercury.bc.ca (http://www.mercury.bc.ca)
Mark Crispin
> But that's not the point. The point is, you support MH folders,
> so why not Maildir folders too?
Maildir requires an ongoing investment of a significant amount of
resources that are not available at UW. I have a long list of programming
and support project that need to be done that are considerably more
important than crank ideas of what constitutes a mail store.
An organization that wanted MH paid me, personally, a substantial sum of
money to use my own free time to add and support MH to the public c-client
sources. MH would not be offered or supported otherwise.
Maildir has a significant additional support burden above and beyond MH.
One source of additional burden are people who waste my time by arguing
and demanding that I give them something for free.
Jeff Hayward
Martin Bligh <mbl...@sequent.com> wrote:
>Nor can I understand why anyone would want to access their mailboxes over NFS
>when they already have IMAP (as far as I recall, that's how this thread
>started, and why people want Maildir). What's the point of implementing a
>client-server protocol, then introducing another massive layer of complexity
>beneath it, which you don't need?
Reliability, performance and scalability. We can add filers as
needed, we can add IMAP/POP/SMTP frontends as needed. We can fail or
replace a filer or frontend with minimal disruption. We don't have to
go building incredibly expensive monolithic systems just to do email.
The NetApp filer's performance is better than that of local file
systems for our workload.
We (U. Texas) use NetApp filers, intel boxes running BSD/OS, the UW
suite, and qmail to store about 5.5 million (maildir) messages for 80K
mailboxes, with about 500K deliveries per day.
This is the 6th hardware generation of this mail service and the
first to use maildir storage. It is by far the most reliable, best
performing, most scalable configuration we've yet used.
We have used the UW c-client based apps for many years (going on 10, I
think) and greatly appreciate the work and good will of the author.
The existence and continued development of the c-client based apps has
been incredibly valuable to us. We have made some performance
enhancements to the maildir module that are currently not in shape for
use outside our specific environment. I would be motivated to rework
them for wider use if there were any likelyhood that they would be
adopted. That said I don't knock MRC's desire to avoid adding
anything he dislikes to the c-client library. That's clearly his to
decide.
But I want to go on record - we and more than a few others I know of
disagree with MRC on the utility of scalable mail services built
around this paradigm. Mark, if you don't know anyone who has good
things to say about it I think you must not be talking to a
significant part of the enterprise and ISP world.
Regards,
-- Jeff Hayward
--
Jeff Hayward J.Ha...@ots.utexas.edu
UT System/OTS +1 512 471 2432 (v)
UT Austin/ACITS +1 512 471 2449 (f)
Carl S. Gutekunst
>> NFS, say in a mixed IMAP/NFS environment where some users use IMAP and
>> others do not. I admit this is stupid, but Sun writes their dtmail IMAP
>> client and their IMAP server support this type of environment.
>
> I heard that this was over the vehement and frantic objections of the IMAP
> software developers at Sun...
Yep. It was mandated by an internal Sun review committee. The whole issue
came up when we foolishly noted in our documentation that running your Inbox
on an NFS mount is not robust, and was not supported by our IMAP server. The
review committee responded that Sun had been NFS mounting mailboxes for years,
and mandated that we fix the "problem" in our IMAP server. We responded that
NFS access to /var/mail had been non-robust forever, and we didn't introduce
this problem, we just documented it. They responded that they didn't care
whose fault it was, they just wanted Inbox access over NFS to be robust.
So we implemented changes in our locking and a variety of other loose ends,
and documented what we did so that other groups in Sun could make compatible
changes to their /var/mail applications. The only one that did so was DtMail.
<csg>
Carl S. Gutekunst
(Scott Schwartz))bio.cse.psu.edu (Scott Schwartz <"schwartz+!usenet%"@> wrote:
>Look, that's not the point. This isn't about imap, or about
>performance. It's about your c-client library. Your mail
>applications support MH, but Maildir is just as good, so why not
>support that too?
Because he doesn't want to, which is an entirely damn good and sufficient
reason for free software.
<csg>
Mike Davis
>
> In article <7jrppi$p...@scel.sequent.com>,
> Martin Bligh <mbl...@sequent.com> wrote:
> >Nor can I understand why anyone would want to access their mailboxes over NFS
> >when they already have IMAP (as far as I recall, that's how this thread
> >started, and why people want Maildir). What's the point of implementing a
> >client-server protocol, then introducing another massive layer of complexity
> >beneath it, which you don't need?
>
> Reliability, performance and scalability. We can add filers as
> needed, we can add IMAP/POP/SMTP frontends as needed. We can fail or
> replace a filer or frontend with minimal disruption. We don't have to
> go building incredibly expensive monolithic systems just to do email.
> The NetApp filer's performance is better than that of local file
> systems for our workload.
After several years of IMAP support for a University, I can say, without
question, that IO is the biggest bottleneck to IMAP mail service. There
are times when 20 7,200rpm spindles is insufficient for IO. Granted,
those times occur mainly as "Monday Morning Effect," but they dop occur.
Regardless of store method, my life would be absolute misery if we ran
our mailstores over nfs.
If you truly want "reliability, performance, and scalability," the use
of SCSI RAID5 or fibre channel RAID5 combined with cpu failover
protection is a good solution. IMAP is not as cpu intensive as POP (for
more information on this conclusion, see the white papers at
www.imap.org). Our experience has been that cyrus (modified for lmtp and
ldap authentication) provides one of the fastest and most scalable mail
systems available. Simeon may be better, but a lack of source leaves one
at the mercy of the vendor.
All of that said, one might be able to build a mailserver system using a
fast backplane private network for communication, with separate smtp
servers, and a very fast centralized nfs server (or cluster) and get
good performance. But, I doubt that such a configuration could compare
with a fast server with attached RAID in either performance or
cost-of-ownership.
As an aside, we had been told that UT Austin was in the process of
moving to simeon/execmail. Is this not the case?
Mike
--
Mike Davis University Computing Services-MCV
Unix Systems Administrator Virginia Commonwealth University
jmd...@hsc.vcu.edu 804-828-9843 x142 (fax: 804-828-9807)
Mark Crispin
> After several years of IMAP support for a University, I can say, without
> question, that IO is the biggest bottleneck to IMAP mail service.
Yes. I never understand these guys who deploy ultra-fast CPUs with
horrible I/O architecture for IMAP servers. A 68040 class machine could
handle a modern-day IMAP CPU load, just as long as it has enough memory
and excellent I/O.
One of the big things that people overlook is the necessity to have the
mailer queue (/usr/spool/mqueue) on a different spindle from mailboxes.
> Regardless of store method, my life would be absolute misery if we ran
> our mailstores over nfs.
Yes. A few people are sensible enough to realize that it's a bad idea
without trying it first. Most people (including here) have to get their
hands burned before they listen to advice about fire... ;-)
Martin Bligh
>> As best as I can understand, their thinking goes like this:
>>
>> Assume that there are 10 CPUs. With 9 IMAP server CPUs going to one NFS
>> server CPU, the failure of a single CPU has a 90% of simply taking out one
>> IMAP server IP address but not shutting down availability for anyone.
>> There is, however, a 10% chance of a CPU failure shutting down
>> availability for everyone.
I think this is a little misleading - the NFS point is your single point
of failure, no matter how many IMAP server CPUs you had. So the chance of
the NFS designed IMAP "system" failing is equal to the probability of the
NFS server failing, plus the probability of all IMAP servers failing
simultaneously. As the number of IMAP servers increases, the probability
of all IMAP servers failing simultaneously will approach zero, but certainly
never be negative. Thus the hardware reliablilty of an NFS IMAP system is
always worse than that of the NFS server, which you could have just run
the IMAPd process on to start with.
The only other argument for the NFS solution that I can see is performance,
and I'm not sure I believe that - I'll leave this argument to the other
half of this thread.
>Assume that there are 10 CPUS. With 4 IMAP server CPUs going to 6 NFS
>server CPUS, the failure of a single NFS CPU isn't so bad, and it's a
>lot more flexible than using local filesystems for everything,
>especially since IMAP isn't our favorite access method anyway.
By CPUs, do you mean systems, or individual CPUs in an SMP system? If you've
got 6 seperate NFS servers, each serving a seperate filespace, and thus
6 systems, I'd argue that you have six small IMAP systems, not one big one.
If you've got 6 NFS servers all serving the same filespace ... then I'm
confused ;-)
Martin
Martin Bligh
Jeff Hayward <j...@mojo.ots.utexas.edu> wrote:
>In article <7jrppi$p...@scel.sequent.com>,
>Martin Bligh <mbl...@sequent.com> wrote:
>>Nor can I understand why anyone would want to access their mailboxes over NFS
>>when they already have IMAP (as far as I recall, that's how this thread
>>started, and why people want Maildir). What's the point of implementing a
>>client-server protocol, then introducing another massive layer of complexity
>>beneath it, which you don't need?
>
>Reliability, performance and scalability. We can add filers as
>needed, we can add IMAP/POP/SMTP frontends as needed. We can fail or
>replace a filer or frontend with minimal disruption. We don't have to
>go building incredibly expensive monolithic systems just to do email.
>The NetApp filer's performance is better than that of local file
>systems for our workload.
I'm working from a basic assumption here - perhaps it's invalid. The
assumption is that it takes just as much CPU resource / general resource
for a server to service an NFS request to get / write a mail message
from / to a mailbox as it does for it to service an IMAP request to
get / write the same mail message from / to the same mailbox. This is
based on the gut feeling that IMAP isn't a particularly "heavy" or
inefficent protocol, and that mailops are mainly I/O bound.
Now I have no data to back this up, but as I haven't seen any data the
other way, and it sounds like a reasonable assumption, I don't feel too
bad about it ;-) Far be it from me to tell you how to run your mail system,
you are obviously free to do as you please, but I think it's an interesting
discussion to see whether we can bottom out on whether it's a good idea in
general or not. Local administrative setups or whatever may dictate that
you want to do otherwise, but my gut feeling is that people are putting
IMAP over NFS because that's what they're used to and happy with (which is
a fair enough reason in itself), not because it makes sense for IMAP.
As for scalability / expandability, surely if you can add IMAP frontends
or replace NFS filers, you could equally add or replace a combined IMAP
frontend / local filer unit? If the NFS filestore holding a particular
user's data fails, how is this different from his combined mailserver
failing? You still have to retrieve his data from backup, and resetup
a machine ... perhaps I'm missing something in the way you've set this up.
Martin
Mike Davis
terminology, postoffice), it has major performance benefits to segment
this on a spindle apart from the OS disk. If possible, it also helps to
have more than one netscape postoffice location.
In cyrus, we have one mqueue, but it resides on it's own spindle, we
have also set up the system to make use of a separate smtp server (a
nice option when our Nursing School sends messages to a 1000 user list).
Postfix, procmail, or qmail are other alternatives to speed this
delivery processing, but we have yet to begin testing any of these with
our existing configuration. The relative power, low cost and ease of use
of linux hardware as a separate smtp server makes this our current
choice.
I must admit that I use a relatively fast cpu (dual 180mhz R10000 MIPS
in an Origin 200), but the choice of the Origin was largely for
backplane speed and expandability rather than processor speed.
Mike
--
Chris Barrera
Its alot more work to configure the OS properly for a new SMTP/IMAP/POP
frontend system if the original fails than it its to configure an arbitrary
system to serve NFS if your primary filer failer and you have to retrieve
from backup.
Chris Barrera
cbar...@msp.sc.ti.com
Martin Bligh <mbl...@sequent.com> wrote:
: As for scalability / expandability, surely if you can add IMAP frontends
Lyndon Nerenberg
Mike> Our experience has been that cyrus (modified
Mike> for lmtp and ldap authentication) provides one of the
Mike> fastest and most scalable mail systems available. Simeon may
Mike> be better, but a lack of source leaves one at the mercy of
Mike> the vendor.
LMTP is a *big* win. In our current (SMS 2.1) product stream we
use LMTP as the default delivery channel, and include a modified
sendmail (8.9.1) that supports LMTP delivery via a named
pipe to a permanently nailed up LMTP server. Eliminating the
fork/exec overhead on each delivery has noticeably increased
message injection throughput. LMTP also allows us to hardlink
message files on multiple-recipient deliveries, providing a
nice saving in disk usage (and also boosting delivery speed).
We don't support authentication against LDAP, however we do provide
source for the external authentication program (our pwcheck
replacement) to which you can easily add an LDAP authentication
module. We're considering including LDAP authentication in a
future release.
In our next major release (MDstore 3.0) we've pretty well
rewritten the entire server. About the only thing we kept
from the Cyrus code base is the per-folder directory layout for
the message and cachefiles . Our two goals we're to provide *true*
virtual domain support, and to substantially increase the
scalability of the server. We did very well on both, if I
do say so myself :-)
If source code is an issue, make sure your sales rep knows this.
We've considered licensing source, but there hasn't been any
demand for it. If enough people ask it might happen.
--lyndon
The two most common elements in the universe are Hydrogen and stupidity.
-- Harlan Ellison
Leslie Mikesell
Martin Bligh <mbl...@sequent.com> wrote:
>>Reliability, performance and scalability. We can add filers as
>>needed, we can add IMAP/POP/SMTP frontends as needed. We can fail or
>>replace a filer or frontend with minimal disruption. We don't have to
>>go building incredibly expensive monolithic systems just to do email.
>>The NetApp filer's performance is better than that of local file
>>systems for our workload.
>
>I'm working from a basic assumption here - perhaps it's invalid. The
>assumption is that it takes just as much CPU resource / general resource
>for a server to service an NFS request to get / write a mail message
>from / to a mailbox as it does for it to service an IMAP request to
>get / write the same mail message from / to the same mailbox. This is
>based on the gut feeling that IMAP isn't a particularly "heavy" or
>inefficent protocol, and that mailops are mainly I/O bound.
Yes, that would be true if you were using general-purpose computers
to do both the file store and IMAP service. NetApps are
special-purpose things tuned for disk service.
>As for scalability / expandability, surely if you can add IMAP frontends
>or replace NFS filers, you could equally add or replace a combined IMAP
>frontend / local filer unit? If the NFS filestore holding a particular
>user's data fails, how is this different from his combined mailserver
>failing? You still have to retrieve his data from backup, and resetup
>a machine ... perhaps I'm missing something in the way you've set this up.
NetApp claims a very low failure rate (the drives are all raid-clustered
with hot spares in the box, and the processors can be clustered as
well). The system automatically keeps snapshot backups on disk
and a complete re-install of the OS only takes a couple of floppies.
And every time I hear someone say they perform better than local
drives I really wish I could afford one... It does make perfect
sense to separate the details of storage from servicing user-level
protocols.
Les Mikesell
l...@mcs.com
Leslie Mikesell
>We (U. Texas) use NetApp filers, intel boxes running BSD/OS, the UW
>suite, and qmail to store about 5.5 million (maildir) messages for 80K
>mailboxes, with about 500K deliveries per day.
>We have used the UW c-client based apps for many years (going on 10, I
>think) and greatly appreciate the work and good will of the author.
>The existence and continued development of the c-client based apps has
>been incredibly valuable to us. We have made some performance
>enhancements to the maildir module that are currently not in shape for
>use outside our specific environment. I would be motivated to rework
>them for wider use if there were any likelyhood that they would be
>adopted. That said I don't knock MRC's desire to avoid adding
>anything he dislikes to the c-client library. That's clearly his to
>decide.
Are you delivering to a mix of IMAP and file-aware clients, or do
you have some other reason for staying with the UW program instead
of switching to cyrus?
Les Mikesell
l...@mcs.com
Lawrence Greenfield
LMTP is a *big* win. In our current (SMS 2.1) product stream we
use LMTP as the default delivery channel, and include a modified
sendmail (8.9.1) that supports LMTP delivery via a named
pipe to a permanently nailed up LMTP server. Eliminating the
fork/exec overhead on each delivery has noticeably increased
message injection throughput. LMTP also allows us to hardlink
message files on multiple-recipient deliveries, providing a
nice saving in disk usage (and also boosting delivery speed).
Note that our (CMU's) latest version of Cyrus has a couple of nasty
problems in the LMTP support in deliver that will cause problems with
extremely long running deliver processes. These will hopefully be
fixed in our next release of Cyrus.
We're considering adding some amount of authentication to LMTP so it's
possible to talk LMTP over TCP securely (also allowing for
authenticated delivery).
Larry
Ian G Batten
In article <Pine.NXT.4.20.990617...@Tomobiki-Cho.CAC.Washington.EDU>,
Mark Crispin <m...@CAC.Washington.EDU> wrote:
> One of the big things that people overlook is the necessity to have the
> mailer queue (/usr/spool/mqueue) on a different spindle from mailboxes.
I'm just stunned by the quality of doing things `right'. Our mail
system was working just well enough to not get fixed using sendmail and
a single NFS-served /var/mail from our Auspex, a layout that dated back
ten years. The increasing use of POP from PC clients and the desire for
IMAP meant that the load of, say, parsing the mail drop was getting
excessive. I'm now running Cyrus with the mailboxs on a RAID0+1 lump
(two mirrored sets of three striped spindles) and the /var/spool/mqueue
on RAID1 (two mirrored spindles) on an old Ultra 2, and with all the
site's outbound mail, 200 POP users polling at a rate of more than one
call a second _and_ a Virus scanner on all inbound and outbound mail,
the load average doesn't get above 0.1 and the receipt-to-delivery delay
is below a second. Trivial to set up (OK, getting the Trend virus
scanner working wasn't) and phenominal performance.
ian
--
-----BEGIN PGP SIGNATURE-----
Version: PGPfreeware 5.0i for non-commercial use
Charset: noconv
iQB1AwUBN2nybsoy0yij3IvtAQFCEQL/cCaXE2wH92Dk4I4fc92NFB7cKtOeQ49Q
vH+L5eTKtB6hXqRNQAWvI/Y0+wbNW9+mIb8YZ1yjVV6bUAx4VtnVERGNNChVlnHz
4GeJO9nPAqT/QR5MDcKGOawUFsDAXWLm
=GoLB
-----END PGP SIGNATURE-----
Jeff Hayward
Leslie Mikesell <l...@Venus.mcs.net> wrote:
>Are you delivering to a mix of IMAP and file-aware clients, or do
>you have some other reason for staying with the UW program instead
>of switching to cyrus?
We're a closed-box service. When we started this particular service
in 1994 the c-client based stuff was all there was. It was also quite
reliable. We had already developed administration tools for users to
create and maintain their accounts which were not compatible with
Cyrus' toolset. So inertia kept the status quo. About 2 years back I
began to seriously look in to the modular approach and took another
look at Cyrus. I didn't come up with any good modular solutions which
would fit it well. I'd like to hear from anyone who has. I concluded
that the best solution was to use commodity intel machines and NFS,
which precluded using Cyrus and/or Berkeley style mailboxes.
As to other comments on people's experience with these things, I think
we must be doing apples/oranges comparisons. I certainly wouldn't use
NFS storage for mailboxes which were not designed to be used that way.
I suspect, tell me if wrong, that the negative comments are from
people who *have* tried to do that (or haven't tried it at all).
Jeff Hayward
Mike Davis <jmd...@hsc.vcu.edu> wrote:
>All of that said, one might be able to build a mailserver system using a
>fast backplane private network for communication, with separate smtp
>servers, and a very fast centralized nfs server (or cluster) and get
>good performance. But, I doubt that such a configuration could compare
>with a fast server with attached RAID in either performance or
>cost-of-ownership.
Our experience is to the contrary. As I said in another note, we have
many years of experience with the large single system approach using
all the RAID there is. It doesn't scale in a cost effective way, and
the possibilities for long periods of total outage are not acceptable.
System maintenance is a headache, just like the old mainframe days.
Replicating 1:1 for hot/warm standby doubles the cost, naturally.
>As an aside, we had been told that UT Austin was in the process of
>moving to simeon/execmail. Is this not the case?
There may be someone at UT that is doing that but it isn't the main
campus-wide service (mail.utexas.edu).
-- Jeff
Chris Barrera
I have to agree with Jeff.
Separating the storage system-wise from the MTA and the services like
POP and IMAP is far easier to manage in outage/recovery scenarios, cost
effectively. It just gives far more flexibility and allows you to tune
the system better to a particular task.
We have one netapp F330 as our NFS spool. Currently desktops do mount
it for mail reading, but only our MX hosts deliver into it. Maybe once
in a great while we recover a corrupted mail folder from its snapshot.
In years we have had no hardware problems, only just recently had one
failed drive. The thing is a tank and performs for our needs.
Our MX hosts are cheap Ultra 5's maxed on memory with SCSI/Ethernet
expansion cards to add a higher speed mqueue and more network I/O. These
U5's recently replaced SS20's which still weren't breaking a sweat,
except we found that 100bT was a necessity and a little more horsepower
wouldn't hurt.
One MX host is the better preference for mail routing. We have the other
system act as the primary pop3 server (and eventually IMAP). Should the
former go down, mail routing is handled by the other. Should the latter
go down, we re-alias "pop3" and "imap" in DNS to point to the former.
If we want to take either down for maintenance without causing undue
interruption of service, we can.
Yes the NFS filer is a single point of failure. Thats why the most
money is spent on it to:
1) be as simple as possible. NetApp's have by far the simplest
architecture we know of. Ok there might be simpler ones, but
this one is simple enough. More simple, less things that break.
2) Its easy to configure and just forget about.
3) Its RAID4 with hot spares and it performs, and better yet, we
have snapshots in case data is corrupted.
If somehow we lose our filer, we can temporarily configure another
serve NFS and restore from backup. In the mean time, our MX hosts
will just queue the mail in one location rather than it backing up
on hundreds of hosts all over the place. Of course this has never
been a problem, but we are prepared.
It is easy to see why many people use this solution.
Chris Barrera
cbar...@msp.sc.ti.com
Jeff Hayward <j...@mojo.ots.utexas.edu> wrote:
: In article <376955A3...@hsc.vcu.edu>,
Martin Bligh
Leslie Mikesell <l...@Venus.mcs.net> wrote:
>In article <7kbqdj$7...@scel.sequent.com>,
>Martin Bligh <mbl...@sequent.com> wrote:
>
>>I'm working from a basic assumption here - perhaps it's invalid. The
>>assumption is that it takes just as much CPU resource / general resource
>>for a server to service an NFS request to get / write a mail message
>>from / to a mailbox as it does for it to service an IMAP request to
>>get / write the same mail message from / to the same mailbox. This is
>>based on the gut feeling that IMAP isn't a particularly "heavy" or
>>inefficent protocol, and that mailops are mainly I/O bound.
>
>Yes, that would be true if you were using general-purpose computers
>to do both the file store and IMAP service. NetApps are
>special-purpose things tuned for disk service.
OK, this makes more sense, and now I'm prepared to believe that there might
be a case for this is you already had a special-purpose NFS server built
this way ;-)
One could make the same argument for specialist IMAP servers. Someone sent
me some email the other day, and seeing as it's one of the few pieces of
marketing that's failed to annoy me (targeted, interesting and relevant
makes a nice change), I'll take the liberty of passing on their website:
http://www.mirapoint.com/ I dare say other companies do the same thing
(or could), but this is proof it's possible ...
Martin
David A Galbraith CIRT
If you are very curious to check different formats/perfomance things you
can do what I did and put a little chunk of code into the imapd.c
file that times EVERY command and logs it into /tmp to be grabed by
a program and then sorted out for interesting commands and how
long each took..
We continual do this and then send the averaged data back to a
program that selects the fasted responding machine as our mail server
until the next rotation. This of course requires that you have your mail
store on NFS or some other Central type location
In the report below Resp time is the Average time (in seconds) it took for
each IMAP command to complete that I measure (I ignore the silly commands
such as IDLE :)
So this is a TRUE measure of how long it takes a maildir format to respond
... not bad. I was using mbx format (Hacked and Over NFS for a while
but found that Maildir was only slightly slower in performance over
the average) Intial select commands took the same for <500 messages up
to 2 times for >1000 messages all the way up to 4-8 times as long
for >4000 messages... but the most common size of a mailbox here is
<500 messages. other commands (In maildir) were faster so it ended
up not making much difference there. We did notice a difference in the
number of NFS ops the Filers reported. (The "load" on the NFS servers
went up when we moved to Maildir)...
Our mail configuration is currently 2 FAServer Network appliance 330's and
4 Linux boxes (Currently one is out of the rotation for testing) Each
FAServer currently supports 15,000 mailboxes (Can scale to 30,000 on a 330)
and each linux box can run apx. 300 concurrent Imaps. (the ipops are
running there also, I just don't count them here...).. I also don't measure
their response time. mlx1 is the front of the MX for unm.edu so it is
also delivering all (99%) of the incomming mail. (about 140,000-160,000 a day)
THe maildir we run is hacked into storing messages spread though a
series of directories instead of a single directory. (this increased the
speed on large mailboxes for flag changes to make it a reasonable format,
until I did that deleting a message was slow enough to be annoying for
large mailboxes...)
Maildir is definatly slower in the intial load of large mailboxes, but some
of that can be "helped" by spliting it apart into seperate directories so
that it isn't hashing 4000 entires in 1 directory but 400 entires in 10
directories. This also makes flag changes (filename changes) happen in
a reasonble amount of time. Where Maildir kicks mbx's... is when you go
to expunge 20-50 messages spread all over the mailbox... sorry mark but my
numbers below have proved it :)... I tried it and timed it... (I had
to say it was better at something didn't I? :) Changing flags appears to be
slightly slower than mbx, and deliveries of a mail message are almost equal.
(Slightly slower on maildir if you figure delivery into the new directory and
then delivery into the cur/.xx/ directory)
When I say slightly slower, I mean that Resp time used to say 0.0128 instead
of 0.0168 second.. things like that... were talking on average it was so
small it was impossible to notice (Except for Select/Examine which did
change enough to notice...:) To be fair :) Expunges were actualy slightly
faster for Maildir compared to mbx.
Opinion:
users are more forgiving for how long it takes to open your mailbox vs.
how long does it take to mark this message deleted, or fetch this message for
me to read. Once the mailbox is open they want it FAST... opening it can
take an extra 10 second as long as when it is open its fast... (Just an
observation on what I expect when I read email :)
The output from the selecter program looks like this:
Process 140024 opening logfile.mail for count: 1
Cur. 30s Phys Swap TMP Resp
Machine Load Users Imaps (Act) Memory Used Free Time
mlx1: 1.13 1 136 ( 73) 127M 44M 300M 0.0432
mlx2: 0.53 2 194 ( 89) 127M 107M 300M 0.0218
mlx3: 0.30 0 163 ( 51) 127M 52M 300M 0.0090
Machine: mlx3 [129.24.8.189] has the best imap response time: 0.0090
Number of total Imaps: 494
Wed Jun 30 16:45:41 MDT 1999
Process 140024 opening logfile.mail for count: 2
Cur. 30s Phys Swap TMP Resp
Machine Load Users Imaps (Act) Memory Used Free Time
mlx1: 0.80 1 139 ( 83) 127M 44M 300M 0.0714
mlx2: 0.33 2 193 ( 51) 127M 107M 300M 0.0316
mlx3: 0.16 0 160 ( 49) 127M 52M 300M 0.0142
mail -> mlx3 [129.24.8.189] has the best imap response time
Number of total Imaps: 493
Wed Jun 30 16:46:13 MDT 1999
Process 140024 opening logfile.mail for count: 3
Cur. 30s Phys Swap TMP Resp
Machine Load Users Imaps (Act) Memory Used Free Time
mlx1: 1.00 1 137 ( 55) 127M 44M 300M 0.0508
mlx2: 0.20 2 190 ( 52) 127M 107M 300M 0.0117
mlx3: 0.31 0 161 ( 51) 127M 52M 300M 0.1152
Machine: mlx2 [129.24.8.188] has the best imap response time: 0.0117
Number of total Imaps: 489
---------
Individual files might look like this:
22268.FETCH
::::::::::::::
0.001056
0.000843
0.000598
0.000530
0.000532
::::::::::::::
22269.FETCH
::::::::::::::
0.001944
0.038091
0.025964
0.029939
::::::::::::::
22269.SELECT
::::::::::::::
0.369051
-----------End DATA chunk.
On the otherhand I will not argue with Mark that he support
maildir. I can see why he doesn't want to. He HATES NFS and maildir's
biggest (Only?) reason for existing is NFS ... So from his point of view
there is NO reason to support maildir.
ps.
For old times sake I'll also point out that our direction was at one
point local disk/big server. Turns out that it didn't scale for
performance/Availability/Reliability/$$$$$$$/.
In the time we have run this solution (3 years) we have once had the
FAServer down for an unplanned outage. How many people can say that
about their single server machines. From the time (6 months) we have been
using Maildir I have NEVER had email down for ANYBODY... There hasn't been
a single person that has reported that they can't get to their mailbox where
it has been the cause of the Mailserver. I expect that we will continue
to have this kind of availability for years. (or until tonight since
I said something :)... The FAServer is the only thing that can fail and
bring the system down. And they are so reliable it is disgusting :)
With the solution we are doing now, I could see double our usage, spend
$4000, and handle it. (Add 2 more Linux boxes).
pps.
For those who are taking notice, 128M of physical memory per linux box
was all that was used because adding more didn't change the response time
average at all. (I have speculation on this, but its for another conversation)
ppps.
Mark, if the test you ran of the performance of Maildir was on AIX with
all the messages being in a single directory then I am not suprised that
Maildir sucked... Aix's filesystem is extremely slow :) [I know I'm gonna
get it from the Aix lovers now eh :)] ... Try the same test over NFS
to a NetApp box and then tell me what you get *grin*.. (You can leave
locking off the mbx/unix mailbox (don't write into it :) just look for
speed ... I think you will find that they aren't near as far off as your
numbers before suggest)
--
+-----------------------------------------------------------------------+
| David Galbraith dgalb@ University Of New Mexico |
| Systems Analyst unm.edu (505)-277-8499 |
+-----------------------------------------------------------------------+
Mark Crispin
from your tests is:
1) with small enough mailboxes on light enough loads, you don't see mbx
being much faster than Maildir.
2) one exception is when opening the mailbox (Maildir is noticably
slower).
3) another exception is when expuning the mailbox (Maildir is somewhat
faster).
But mailboxes have to be opened in every session (there's no such
requirement for expunge), and all those little bits of slowdown add up in
reduced scalability. That ultimately means worse performance with larger
mailboxes and with larger system loads.
That's a heavy price to pay for faster expunge. I don't think that folks
around here would care to be told that they're limited to 500 messages;
that's a small mailbox by our standards.
You said that you hacked mbx so it would run under NFS. Since mbx uses
O_RDWR and locking (neither of which works well on NFS), that leads to
some questions about the impact of your hacks.
Did you test EXPUNGE on a system with large server loads (e.g. 300+
messages/second being delivered and/or expunged on a single filesystem)? I
think that you'll find that a saturation point gets reached, after which
EXPUNGE in Maildir starts becoming *much* slower than mbx.
Did you test shared EXPUNGE (two or more imapds having the same mailbox
open, and doing an expunge)? Did the other imapds get the proper expunge
notification?
Did you test shared flag manipulation; when two or more imapds have the
same mailbox open and one changes a flag setting, did the other imapds get
the change?
Did you test RFC822.SIZE, e.g.
a001 SELECT INBOX
a002 SEARCH LARGER 4000
What happens with a mailbox with 500 messages? 1000 messages? 5000
messages?
Finally, you should do your tests with local disk, not NFS. NFS has such
an overwhelming impact that it tends to distort or conceal other factors.
David A Galbraith CIRT
Mark Crispin <m...@CAC.Washington.EDU> wrote:
>I'm trying to determine your point. The only conclusion that can be drawn
Ah.. the point is that Maildir works, and so does NFS. Possibly not as
well as mbx for >1000 message mailboxes but well enough to be useable up to
4000 messages (4000 is where people begin to say... "Mail is slow")
For < 1000 messages you won't even notice a difference between mbx/Maildir.
95% of our users are in the < 1000 messages catagory, so those are the
people I try to satisfiy with the limited $/manpower resources I have.
>from your tests is:
> 1) with small enough mailboxes on light enough loads, you don't see mbx
> being much faster than Maildir.
> 2) one exception is when opening the mailbox (Maildir is noticably
> slower).
> 3) another exception is when expuning the mailbox (Maildir is somewhat
> faster).
>
>But mailboxes have to be opened in every session (there's no such
>requirement for expunge), and all those little bits of slowdown add up in
>reduced scalability. That ultimately means worse performance with larger
>mailboxes and with larger system loads.
>
The numbers I posted proved that on average scalablity did not reduce
due to moving from mbx to maildir. That was one of the reasons I
posted them. It is to bad I don't have any old numbers from when we
were in mbx format. It wasn't much different though. If maildir didn't
scale (REAL WORLD USAGE HERE) the numbers there would have told something
different then they did... Select is slower, but other things are faster
on average... enough faster even to make up for select being slower...
>That's a heavy price to pay for faster expunge. I don't think that folks
>around here would care to be told that they're limited to 500 messages;
>that's a small mailbox by our standards.
We have people with 10000+ messages.. they have slow perfomance... tuff
for them ... for 95% of our users with < 1000 they have great performance.
Acutally your paying the price for NFS, which is worth it, when you get
availability/scalability/$$ gains. You begin to lose when the mailboxes
go beyond about 1000 messages. I think if you were to check EVERY INBOX on
your mail system you would find that most of them are 1000 messages or less.
Average on our mailsystem is actually something like 6 messages because alot
of people here have 0 messages in their mailbox :) but when look for
what is the most common size of mailbox it is < 1000.
I does work at 1000+ messages but your response time gets slower. To load a
2000 message mailbox takes about 15-20 seconds. (All other accesses happen
in under a second so loading is the only slow piece). 15-20 seconds for
2000 messages is fairly reasonable access. Once again you have to consider
what the AVERAGE user does. We have VERY few mailboxes over 2000 messages,
I doubt we are abnormal. (I don't have users telling me our mail system
is slow and trust me they bitch if it is... when we ran a single box
imap server long ago it was a nightmare... and that was with 200 concurrent
users... )
>
>You said that you hacked mbx so it would run under NFS. Since mbx uses
>O_RDWR and locking (neither of which works well on NFS), that leads to
>some questions about the impact of your hacks.
>
I undid the O_RDWR, and changed to a lockserver for all locking access. The
only issue I was eventually unable to 100% overcome was some NFS cacheing/
timing issues. I was very close to solving it when I decided to try the
Maildir format to see how its performance really was. Before we moved we would
see mailbox corruption on a fully loaded mail system (800 concurent
readers/35,000 accounts) about once a week. To often)
(To solve the problem I was headed to having the lockserver becoming more
than a lockserver... it was getting complex with redundant lockservers, fail
over... I had VMS styled cluster lock system growing.. it was getting gross
... maildir is so much simpler and it just works) It is a little slower
on average (like .01 seconds slow or something :)... who cares... users
didn't even notice when we moved... that is the real measure of slow. I
actually had people tell me, "it seems faster" when I told them we moved
to a new format... (the numbers said it was slightly slower though.. so
much for users knowing what they are talking about :)
>Did you test EXPUNGE on a system with large server loads (e.g. 300+
>messages/second being delivered and/or expunged on a single filesystem)? I
>think that you'll find that a saturation point gets reached, after which
>EXPUNGE in Maildir starts becoming *much* slower than mbx.
The numbers I posted are on a Fully loaded Mail SYSTEM where one box
is delivering messages and another 3 boxs are reading the email with Imaps.
The stats I gathered were all done using REAL PEOPLE, REAL SYSTEMS... they
aren't speculation about what it would look like. It is run on UNM's real
mail system. It does work... maildir isn't slower than mbx except for
initial loading of the mailbox. (And even then it is the same until 1000+
messages) Expunge for maildir is a hell of alot faster if you want to
talk about large messages (> 2MB) ... try deleting 3 mail messages each
greater than 10 MB under each format and let me know :)
The numbers I posted are actually from 4PM during the summer. Last
semester we ran with only 1 NFS server and 3 Linux boxes, but decided to
upgrade recently to the second NFS server. Durning peak times we have
800+ conncurrent readers, not a huge load but not tiny either. (With the
second Filer I don't expect to see performance problems until we almost
double what we were peaking at during the semester to 1600+ readers and
400,000-600,000 messages a day.)
>
>Did you test shared EXPUNGE (two or more imapds having the same mailbox
>open, and doing an expunge)? Did the other imapds get the proper expunge
>notification?
Are we talking about hacked mbx here? if so yes, thats part of the lockserver
... it knew who was accessing the mailbox and handled it
>
>Did you test shared flag manipulation; when two or more imapds have the
>same mailbox open and one changes a flag setting, did the other imapds get
>the change?
hacked mbx? yes... lockserver handled notification of changes to
other processes even on different machines... I told you it was getting
gross...
>
>Did you test RFC822.SIZE, e.g.
> a001 SELECT INBOX
> a002 SEARCH LARGER 4000
>What happens with a mailbox with 500 messages? 1000 messages? 5000
>messages?
>
>Finally, you should do your tests with local disk, not NFS. NFS has such
>an overwhelming impact that it tends to distort or conceal other factors.
BUT!
How the formats performed on NFS was the whole point of the test (Which
aren't tests at this point by the way.. that was live data :). Local
performance is a moot point since I am using them though NFS. (I'm sure
that to you it isn't moot... the tests were not run for you but for me :)
To make another point...
Lets try a cost analisys. How many machines would it take and at what
cost to handle 40,000 accounts? The NFS solution I propose might be
slower for 5% of our users than a single machine type solution runing
mbx, but for 95% of the users it works as fast if not faster than
single machine solutions.
Before we purchased the solution we currently use we analysized
what UW (and other schools) were doing at the time
(I believe 3 years ago UW was using many AIX machines with a hacked DNS)
It would have cost us Apx. 2 times what the solution we are running cost.
(Today the difference would probaby be smaller since hardware has gotten
cheaper faster than good performing NFS servers). If I were to do it all
again today I would have to relook at it. (Probably would try to get
a "real database" type solution up that works...Might even purchase
something... but... since the solution I currently have works... I don't
need to :)
The availability and Management issues are a major plus to the NFS/Maildir
solution. I can yank mail machines in and out of the pool and nobody even
notices... I can test mail server modifcations on the real data with a group
of test users by simply changing my test cname to point to the test server...
Other considerations for single machine solutions:
the raid disk for each machine you buy, the amount of memory
you'll need for each, and you'd better hope your disks are fast enough
because disk perfomance is EVERYTHING to a single machine solution. And
when you go to upgrade... wow... thats gonna cost you. And when your
single box gets overloaded what a nightmare to move to the next bigger
box. Shut the whole thing down... move it... bring it back...
The thing to consider for the NFS solution is make sure your NFS server
is VERY robust. The boxes around it can be $800-1200 standard PC's since
one failing has little effect on the performance of the whole. NFS
service can be made to be redundant if you want to spend the money, but
then it begins to cost like the other solution... choices choices...
So I could spend 2 (1.5 now?) times and get better performance for the 5%
of our users that have a mailbox greater than 1000 messages? If I
had the money I might consider it, but I don't have a money tree in
my back yard like UW ;) *grin*
ps
To be perfectly clear again. I am 100% happy that you don't want to support
maildir. It doesn't bother me in the least.
Onward to better things such as ldap solutions, calendar solutions, linux/NT
in the Computer Pods(vmware?) now thats interesting... Email has been solved.
It works, it will work if we double usage. Case closed. I have more
intersting things to spend my time on now.
pps.
I do want to say Mark that you have done a GREAT job on the UW server code
I am always amazed when I go in to hack it here and there how well it is
written and lends itself to having other formats plugged in and the code
is actually easy to understand once you've played with it for a while.
(I know everybody else who has looked at it will agree with me ;)
>
>-- Mark --
>
>* RCW 19.190 notice: This email address is located in Washington State. *
>* Unsolicited commercial email may be billed $500 per message. *
>Science does not emerge from voting, party politics, or public debate.
>
>
>
Mark Crispin
> The numbers I posted proved that on average scalablity did not reduce
> due to moving from mbx to maildir.
Correction: "moving from mbx which has been hacked so as to destroy just
about any performance benefit that it has". By your own admission, you
used mbx over NFS, undid O_RDWR, and used a lockserver. Well, golly gee,
if you put kerosine in the gas tank a car will be slower than a bicycle.
> I does work at 1000+ messages but your response time gets slower. To load a
> 2000 message mailbox takes about 15-20 seconds. (All other accesses happen
> in under a second so loading is the only slow piece). 15-20 seconds for
> 2000 messages is fairly reasonable access.
The admins would strangle me if they got a 15-20 second delay when they
opened their mailbox.
> Once again you have to consider
> what the AVERAGE user does. We have VERY few mailboxes over 2000 messages,
> I doubt we are abnormal.
Yes, there are other sites which force people to have tiny mailboxes via
quotas and/or making performance so bad that they're forced to flush old
mail. But it's debatable whether or not that constitutes "normal" user
behavior when the users aren't punished.
> >Did you test shared EXPUNGE (two or more imapds having the same mailbox
> >open, and doing an expunge)? Did the other imapds get the proper expunge
> >notification?
> Are we talking about hacked mbx here? if so yes, thats part of the lockserver
> ... it knew who was accessing the mailbox and handled it
No. Did you test shared EXPUNGE with a Maildir mailbox?
> >Did you test shared flag manipulation; when two or more imapds have the
> >same mailbox open and one changes a flag setting, did the other imapds get
> >the change?
> hacked mbx? yes... lockserver handled notification of changes to
> other processes even on different machines... I told you it was getting
> gross...
No. Did you test shared flag manipulation with a Maildir mailbox?
> >Did you test RFC822.SIZE, e.g.
> > a001 SELECT INBOX
> > a002 SEARCH LARGER 4000
> >What happens with a mailbox with 500 messages? 1000 messages? 5000
> >messages?
Well, did you test this with Maildir?
Unless you did a lot of hacking, the answer to the previous three is
"Maildir doesn't work work a damn doing that." You've already established
that it doesn't work worth a damn when the mailbox has more than 1000
messages.
> Lets try a cost analisys. How many machines would it take and at what
> cost to handle 40,000 accounts?
We do twice as many.
> (I believe 3 years ago UW was using many AIX machines with a hacked DNS)
> It would have cost us Apx. 2 times what the solution we are running cost.
> (Today the difference would probaby be smaller since hardware has gotten
> cheaper faster than good performing NFS servers).
We used to do the "good performing NFS server" route. So did a number of
other organizations. We gave it up for what we do now because what seemed
"good enough" initially turned out not to be all that good.
Just as the cost of hardware decreases over time, so the system load
increases over time. You can try to force people to keep their demands
down, such as arbitrarily saying "don't have 1000+ message mailboxes".
But it just isn't a viable strategy in the long run.
Another problem with using an NFS server for your mail back end is that it
puts all your eggs in one basket. You have a single point of failure: the
NFS server.
Finally, it simply does not make sense to use IMAP over NFS. If you
really want to use NFS, why not just have the clients talk directly to the
NFS server? You're not getting anything with using IMAP this way.
Sasha Mikheev
>
>Another problem with using an NFS server for your mail back end is that it
>puts all your eggs in one basket. You have a single point of failure: the
>NFS server.
single point of failure. Or even spread mailboxes across multiple
NFS servers.
>Finally, it simply does not make sense to use IMAP over NFS. If you
>really want to use NFS, why not just have the clients talk directly to the
>NFS server? You're not getting anything with using IMAP this way.
IMHO, having array of cheap boxes sharing same storage makes perfect
sense. This acrhitecture is simple, it scales well, it is easy to
maintain and it has enough redundancy. Not to mention that this
way it is possible to split SMTP, POP3 and IMAP daemons.
>
>-- Mark --
>
>* RCW 19.190 notice: This email address is located in Washington State. *
>* Unsolicited commercial email may be billed $500 per message. *
>Science does not emerge from voting, party politics, or public debate.
>
--
Terry Gray
On 1 Jul 1999, David A Galbraith CIRT wrote:
> Before we purchased the solution we currently use we analysized
> what UW (and other schools) were doing at the time
> (I believe 3 years ago UW was using many AIX machines with a hacked DNS)
> It would have cost us Apx. 2 times what the solution we are running cost.
> (Today the difference would probaby be smaller since hardware has gotten
> cheaper faster than good performing NFS servers). If I were to do it all
> again today I would have to relook at it. (Probably would try to get
> a "real database" type solution up that works...Might even purchase
> something... but... since the solution I currently have works... I don't
> need to :)
David (et al),
I apologize for replying to your message three weeks late, but I did want
to comment on this particular point in your argument.
I believe that you looked at the UW architecture, assumed that we had the
same goals/requirements that you did, and observed that you could see a
less expensive way to meet those goals. What's missing from that analysis
is that a very important goal for us was to minimize what I call "fault
zones". (Sorry, I grew up in L.A. :) The idea is that when there is a
failure, you want to reduce the number of people affected to a relatively
small number. We were willing to put money into that goal, in the form of
using more servers than we "needed" to --especially having the conviction
that the design was essentially infinitely scalable and that the economics
of using commodity platforms was getting better and better. (We had also
already developed management tools to make it easy to deal with large
numbers of similar boxes, so having large numbers of servers around was
not a concern.)
NFS solutions can obviously be made to work, but I don't believe they
offer any cost advantage over the strategy we adopted *if* there are
similar goals with respect to how many eggs you're willing to have in one
basket.
The one and only thing that NFS approaches have going for them, as far as
I can see, is that they obviate the service location problem. We chose to
solve that problem via a DNS server hack, and it's worked well for us.
-teg
Andrew Lochart
>
> On 1 Jul 1999, David A Galbraith CIRT wrote:
>
> > what UW (and other schools) were doing at the time
> > (I believe 3 years ago UW was using many AIX machines with a hacked DNS)
> > It would have cost us Apx. 2 times what the solution we are running cost.
> > (Today the difference would probaby be smaller since hardware has gotten
> > cheaper faster than good performing NFS servers). If I were to do it all
> > again today I would have to relook at it. (Probably would try to get
> > a "real database" type solution up that works...Might even purchase
> > something... but... since the solution I currently have works... I don't
> > need to :)
>
> I apologize for replying to your message three weeks late, but I did want
> to comment on this particular point in your argument.
>
> I believe that you looked at the UW architecture, assumed that we had the
> same goals/requirements that you did, and observed that you could see a
> less expensive way to meet those goals. What's missing from that analysis
> is that a very important goal for us was to minimize what I call "fault
> zones". (Sorry, I grew up in L.A. :) The idea is that when there is a
> failure, you want to reduce the number of people affected to a relatively
> small number. We were willing to put money into that goal, in the form of
> using more servers than we "needed" to --especially having the conviction
> that the design was essentially infinitely scalable and that the economics
> of using commodity platforms was getting better and better. (We had also
> already developed management tools to make it easy to deal with large
> numbers of similar boxes, so having large numbers of servers around was
> not a concern.)
>
> NFS solutions can obviously be made to work, but I don't believe they
> offer any cost advantage over the strategy we adopted *if* there are
> similar goals with respect to how many eggs you're willing to have in one
> basket.
>
> The one and only thing that NFS approaches have going for them, as far as
> I can see, is that they obviate the service location problem. We chose to
> solve that problem via a DNS server hack, and it's worked well for us.
>
> -teg
Terry,
Yes, yes yes! It's always good to find a kindred spirit. You're making
the same point you laid out in your paper, "Integrating Personal
Computers in a Distributed Client/server environment," and I think it's
a good one.
This is why Mirapoint has developed an IMAP server that does exactly
that -- minimize "fault zones". (We're in the Bay Area and are also
sensitive to faults!) And we've done it by using inexpensive,
off-the-shelf h/w, so it's simple to create a distributed environment
that minimizes fault zones while providing high-performance to end-users.
Regardless of whether you buy a commercial product, or roll your own, we
feel that this design philosophy is a sound one, and a useful
alternative to NFS storage.
Cheers,
Andrew
--
Andrew Lochart Director, Customer Marketing Mirapoint, Inc.
and...@mirapoint.com http://www.mirapoint.com +1-408-517-1326