Mail Quota restrictions in postfix
Rakesh
I am trying to set up a POP server (Postfix+Qpopper with MailDir
support) and i just wanted that the mailbox quota restrictions to be
taken care by postfix itself.
I knew about
virtual_mailbox_limit
but that wld take care of the entire setup but i wanted userwise or
domain wise.
Our best friend google.com suggested a site
(http://web.onda.com.br/nadal/) which said that individual mailbox quota
can be taken care of in postfix itself but only after applying patch
suggested by the author.
The
virtual_mailbox_limit then become
virtual_mailbox_limit_maps = hash:/etc/postfix/vquota
and in the hash file i can specify the quotas for individual mailboxes
or domains.
I just wanted to know is there any other best to get this implemented in
postfix. I mean do the new versions of postfix have this kind of feature
as really i don't want to jump to patch work (however I don't mind doing
that, but an inbuilt feature will be the best for me) and also i i don't
wanted to have system related quotas.
Please lemme know whether we have anything like this in postfix as an in
built feature.
regards
Rakesh
Wietse Venema
> Hi guys,
>
> I am trying to set up a POP server (Postfix+Qpopper with MailDir
> support) and i just wanted that the mailbox quota restrictions to be
> taken care by postfix itself.
UNIX systems support file system quotas in a better way.
Wietse
Mike Horwath
That's relative.
For large email systems, using a single ID, Maildir, and a quota
system ala maildir++ works far better in my experience.
There is more load on the disk with the maildir++ lookups, but
handling 100K users is far easier without having to have a HUGE
password map and the like.
Just an opinion...
--
Mike Horwath
drec...@gmail.com
Wietse Venema
> On Fri, 25 Jun 2004 10:39:13 -0400 (EDT), Wietse Venema
> <wie...@porcupine.org> wrote:
> >
> > Rakesh:
> > > Hi guys,
> > >
> > > I am trying to set up a POP server (Postfix+Qpopper with MailDir
> > > support) and i just wanted that the mailbox quota restrictions to be
> > > taken care by postfix itself.
> >
> > UNIX systems support file system quotas in a better way.
>
> That's relative.
>
> For large email systems, using a single ID, Maildir, and a quota
> system ala maildir++ works far better in my experience.
No it doesn't. None of the maildir quota patches for Postfix have
a satisfactory answer for the questions below.
> There is more load on the disk with the maildir++ lookups, but
> handling 100K users is far easier without having to have a HUGE
> password map and the like.
Quotas are done in the kernel. The kernel does not care if
a user exists in /etc/passwd.
Wietse
Does the code enforce mailbox file quotas or is it maildir only?
How does the code address race conditions when multiple processes
attempt to deliver to the same mailbox file? If 10 processes deliver
a 1MB message and the quota is only 2MB, what happens? Will the
user receive 10 messages of 1MB?
How does the code address race conditions when multiple processes
attempt to deliver to the same maildir directory? If 10 processes
deliver a 1MB message and the quota is only 2MB, what happens? Will
the user receive 10 messages of 1MB?
How does the code address NFS related race conditions that can
affect mailbox delivery?
How does the code address NFS related race conditions that can
affect maildir delivery?
Maildir delivery over NFS is safer than mailbox delivery over NFS;
why is this still true with the maildir quota code?
How does the code enforce maildir quota when a maildir contains 10
million files? Will it do 10 million file system operations? If
not, how does the code address race conditions when multiple
processes attempt to update the file with maildir totals? Why is
this method safe over NFS?
How does the code avoid race conditions when multiple processes
add AND remove messages to the mailbox file? What does the code do
in order to address NFS related race conditions in this respect?
How does the code avoid race conditions when multiple processes
add AND remove messages to the maildir file? What does the code do
in order to address NFS related race conditions in this respect?
How does the code recover when some operation is interrupted due
to a system crash?
Mike Horwath
<wie...@porcupine.org> wrote:
> Mike Horwath:
> > That's relative.
> >
> > For large email systems, using a single ID, Maildir, and a quota
> > system ala maildir++ works far better in my experience.
>
> No it doesn't. None of the maildir quota patches for Postfix have
> a satisfactory answer for the questions below.
I'll try to answer.
As a user of Postfix since before it was Postfix, I am amazed at your
curt comments.
> > There is more load on the disk with the maildir++ lookups, but
> > handling 100K users is far easier without having to have a HUGE
> > password map and the like.
>
> Quotas are done in the kernel. The kernel does not care if
> a user exists in /etc/passwd.
And that was the issue, how?
> Does the code enforce mailbox file quotas or is it maildir only?
Maildir only, but I think you already knew that.
> How does the code address race conditions when multiple processes
> attempt to deliver to the same mailbox file? If 10 processes deliver
> a 1MB message and the quota is only 2MB, what happens? Will the
> user receive 10 messages of 1MB?
The maildir++ patches will rewrite the maildirsize file every 15 minutes.
There is a chance that a user can go over quota for a period of time,
but not for very long.
> How does the code address race conditions when multiple processes
> attempt to deliver to the same maildir directory? If 10 processes
> deliver a 1MB message and the quota is only 2MB, what happens? Will
> the user receive 10 messages of 1MB?
Duplicated....
> How does the code address NFS related race conditions that can
> affect mailbox delivery?
No locking... Or are you thinking of another race condition?
> How does the code address NFS related race conditions that can
> affect maildir delivery?
As the person who had a cluster running with over 40K mailboxes using
Maildir formatted mailboxes - I did not notice any NFS related race
conditions.
And that cluster is still going, 4 years later.
> Maildir delivery over NFS is safer than mailbox delivery over NFS;
> why is this still true with the maildir quota code?
There is a single file that is read and written to, even if the file
were to become corrupted it is rewritten if it is over 15 minutes old
(if I remember right).
By default, lines are only appended to it as required for delivery, or
lines are appended once a mail is deleted, but with a negative value.
> How does the code enforce maildir quota when a maildir contains 10
> million files? Will it do 10 million file system operations? If
> not, how does the code address race conditions when multiple
> processes attempt to update the file with maildir totals? Why is
> this method safe over NFS?
The maildir++ specification allows the admin to set both disk space
quotas and # of message quotas.
That cluser I referenced above has 8M messages right now, serviced by
a NetApp F760, and again, after 4 years, still running.
> How does the code avoid race conditions when multiple processes
> add AND remove messages to the mailbox file? What does the code do
> in order to address NFS related race conditions in this respect?
Addressed above.
> How does the code avoid race conditions when multiple processes
> add AND remove messages to the maildir file? What does the code do
> in order to address NFS related race conditions in this respect?
Another duplicate.
> How does the code recover when some operation is interrupted due
> to a system crash?
haven't seen that happen yet, but it would be addressed as soon as 15
minutes has gone by and another process comes into read the mailbox.
I need to review the code, but I think only the POP and IMAP daemons
rewrite the file in a clean fashion. I think the delivery agents just
append lines as things happen.
Thanks for asking the hard questions, and I'll even endeavor to answer
the best I can.
--
Mike Horwath
drec...@gmail.com
Wietse Venema
> > > There is more load on the disk with the maildir++ lookups, but
> > > handling 100K users is far easier without having to have a HUGE
> > > password map and the like.
> >
> > Quotas are done in the kernel. The kernel does not care if
> > a user exists in /etc/passwd.
>
> And that was the issue, how?
See above. Search for "HUGE password map".
> > Does the code enforce mailbox file quotas or is it maildir only?
>
> Maildir only, but I think you already knew that.
Sorry, features with exceptions are not acceptable for Postfix.
Either it works or it does not work.
> > How does the code address race conditions when multiple processes
> > attempt to deliver to the same mailbox file? If 10 processes deliver
> > a 1MB message and the quota is only 2MB, what happens? Will the
> > user receive 10 messages of 1MB?
>
> The maildir++ patches will rewrite the maildirsize file every 15 minutes.
So the user will receive 10 messages of 1 MB, exceeding
their quota by 500 percent.
> There is a chance that a user can go over quota for a period of time,
> but not for very long.
Sorry, can go over quota by 500 percent for as long as they
wish to leave that mail in the spool.
> > How does the code address race conditions when multiple processes
> > attempt to deliver to the same maildir directory? If 10 processes
> > deliver a 1MB message and the quota is only 2MB, what happens? Will
> > the user receive 10 messages of 1MB?
>
> Duplicated....
>
> > How does the code address NFS related race conditions that can
> > affect mailbox delivery?
>
> No locking... Or are you thinking of another race condition?
I'm asking what the assurances are that the code actually works.
As opposed to some piece of crap that just happens to work by
accident most of the time. Proof by "I never had a problem"
is NOT ACCEPTABLE.
> > Maildir delivery over NFS is safer than mailbox delivery over NFS;
> > why is this still true with the maildir quota code?
>
> There is a single file that is read and written to, even if the file
> were to become corrupted it is rewritten if it is over 15 minutes old
> (if I remember right).
How does the code address race conditions when multiple
processes attempt to update the file with maildir totals?
Why is this method safe over NFS?
> > How does the code avoid race conditions when multiple processes
> > add AND remove messages to the mailbox file? What does the code do
> > in order to address NFS related race conditions in this respect?
>
> Addressed above.
I have not seen an "address" except for "anecdotal evidence".
This is not satisfactory. The satisfactory answer shows how the
code is designed to deal with such problems.
> > How does the code avoid race conditions when multiple processes
> > add AND remove messages to the maildir file? What does the code do
> > in order to address NFS related race conditions in this respect?
>
> Another duplicate.
What a wonderful piece of junk.
Wietse
Mike Horwath
<wie...@porcupine.org> wrote:
>
> Mike Horwath:
> > > > There is more load on the disk with the maildir++ lookups, but
> > > > handling 100K users is far easier without having to have a HUGE
> > > > password map and the like.
> > >
> > > Quotas are done in the kernel. The kernel does not care if
> > > a user exists in /etc/passwd.
> >
> > And that was the issue, how?
>
> See above. Search for "HUGE password map".
You don't seem to want to actually discuss this issue.
> > > Does the code enforce mailbox file quotas or is it maildir only?
> >
> > Maildir only, but I think you already knew that.
>
> Sorry, features with exceptions are not acceptable for Postfix.
> Either it works or it does not work.
I never asked that these patches are rolled in. I don't mind patching
them in myself.
This does work, and it works very well, whether you wish to see that or not.
> > > How does the code address race conditions when multiple processes
> > > attempt to deliver to the same mailbox file? If 10 processes deliver
> > > a 1MB message and the quota is only 2MB, what happens? Will the
> > > user receive 10 messages of 1MB?
> >
> > The maildir++ patches will rewrite the maildirsize file every 15 minutes.
>
> So the user will receive 10 messages of 1 MB, exceeding
> their quota by 500 percent.
Test it yourself, I have, and I can't get a 500% overquota to occur.
Scaremongering?
> > There is a chance that a user can go over quota for a period of time,
> > but not for very long.
>
> Sorry, can go over quota by 500 percent for as long as they
> wish to leave that mail in the spool.
No, you brought up 500% out of thin air.
I have *never* seen that happen. I didn't roll out my original
cluster (or the new cluster I am putting together right now for
another venture) and not test out items such as that.
If you can create that kind of race condition *in a real world
situation*, I would love to see it.
> > > How does the code address NFS related race conditions that can
> > > affect mailbox delivery?
> >
> > No locking... Or are you thinking of another race condition?
>
> I'm asking what the assurances are that the code actually works.
> As opposed to some piece of crap that just happens to work by
> accident most of the time. Proof by "I never had a problem"
> is NOT ACCEPTABLE.
Why the anger?
*I* have tried to break it. I have not broken it, and I have not seen
it happen.
My opinions and my experience is worth something, whether you wish to
acknowledge it or not.
> > > Maildir delivery over NFS is safer than mailbox delivery over NFS;
> > > why is this still true with the maildir quota code?
> >
> > There is a single file that is read and written to, even if the file
> > were to become corrupted it is rewritten if it is over 15 minutes old
> > (if I remember right).
>
> How does the code address race conditions when multiple
> processes attempt to update the file with maildir totals?
>
> Why is this method safe over NFS?
I answered this already, it is appending the file only.
> > > How does the code avoid race conditions when multiple processes
> > > add AND remove messages to the mailbox file? What does the code do
> > > in order to address NFS related race conditions in this respect?
> >
> > Addressed above.
>
> I have not seen an "address" except for "anecdotal evidence".
> This is not satisfactory. The satisfactory answer shows how the
> code is designed to deal with such problems.
If you don't wish to look at it yourself, then it won't matter, will it?
> > > How does the code avoid race conditions when multiple processes
> > > add AND remove messages to the maildir file? What does the code do
> > > in order to address NFS related race conditions in this respect?
> >
> > Another duplicate.
>
> What a wonderful piece of junk.
Yep, which is why I mentioned it, twice.
I'll continue to use Postfix, I'll continue to patch virtual and local
to support maildir++ because it works and *I* have not seen any race
conditions in production or in a test lab.
But I don't see why I deserve your angry comments, but it is just a
mailing list, so who really cares, eh?
--
Mike Horwath
drec...@gmail.com
Wietse Venema
> On Fri, 25 Jun 2004 23:14:25 -0400 (EDT), Wietse Venema
> <wie...@porcupine.org> wrote:
> >
> > Mike Horwath:
> > > > > There is more load on the disk with the maildir++ lookups, but
> > > > > handling 100K users is far easier without having to have a HUGE
> > > > > password map and the like.
> > > >
> > > > Quotas are done in the kernel. The kernel does not care if
> > > > a user exists in /etc/passwd.
> > >
> > > And that was the issue, how?
> >
> > See above. Search for "HUGE password map".
>
> You don't seem to want to actually discuss this issue.
The above is clear enough: there is no need for a user to exist
in the password file.
Wietse
> > > > Does the code enforce mailbox file quotas or is it maildir only?
> > >
> > > Maildir only, but I think you already knew that.
> >
> > Sorry, features with exceptions are not acceptable for Postfix.
> > Either it works or it does not work.
>
> I never asked that these patches are rolled in. I don't mind patching
> them in myself.
>
> This does work, and it works very well, whether you wish to see that or not.
>
> > > > How does the code address race conditions when multiple processes
> > > > attempt to deliver to the same mailbox file? If 10 processes deliver
> > > > a 1MB message and the quota is only 2MB, what happens? Will the
> > > > user receive 10 messages of 1MB?
> > >
> > > The maildir++ patches will rewrite the maildirsize file every 15 minutes.
> >
> > So the user will receive 10 messages of 1 MB, exceeding
> > their quota by 500 percent.
>
> Test it yourself, I have, and I can't get a 500% overquota to occur.
Sorry, software is not proven correct by testing. Software is
designed and built to solve a problem correctly, or it is not.
How does maildir++ avoid parallel deliveries from happening?
If it doesn't, then multiple messages will be delivered.
> > > > How does the code address NFS related race conditions that can
> > > > affect mailbox delivery?
> > >
> > > No locking... Or are you thinking of another race condition?
> >
> > I'm asking what the assurances are that the code actually works.
> > As opposed to some piece of crap that just happens to work by
> > accident most of the time. Proof by "I never had a problem"
> > is NOT ACCEPTABLE.
>
> Why the anger?
Software is not proven correct by anecdotal evidence. Software is
not proven correct by testing. Software is designed and built to
solve a problem correctly, or it is not.
I try to maintain a minimal quality level with Postfix. "has not
failed" is not sufficient assurance that it will work correctly
under difficult conditions.
Wietse
Ralf Hildebrandt
> The maildir++ patches will rewrite the maildirsize file every 15 minutes.
>
> There is a chance that a user can go over quota for a period of time,
> but not for very long.
So, if a mailbox is being mailbombed (joe jobbed, whatever), it may go
WAAAY beyond the quota limit. And that's exactly why the quota was
imposed in the first place.
--
Ralf Hildebrandt Ralf.Hil...@charite.de
my current spamtrap spam...@charite.de
http://www.arschkrebs.de/postfix/ Tel. +49 (0)30-450 570-155
UNIX is an operating system, OS/2 is half an operating system, Windows
is a shell, and DOS is a boot partition virus." -- Peter H. Coffin
Tony Earnshaw
> > The maildir++ patches will rewrite the maildirsize file every 15 minu=
tes.
I support mike right up to the hilt, so do the hundreds or thousands of
mailadmins using Courier maildrop for maildir++. As far as those 15
minutes go, they don't apply to maildrop - maildrop will recalculate
(and even recreate if it's deleted) Maildir/maildirsize on each
delivery, on the basis of the admin's quota directive.
> >=20
> > There is a chance that a user can go over quota for a period of time,
> > but not for very long.
>=20
> So, if a mailbox is being mailbombed (joe jobbed, whatever), it may go
> WAAAY beyond the quota limit. And that's exactly why the quota was
> imposed in the first place.
Each maildrop process (there could be many) will open maildirsize for
writing on delivery. A second process can not open it, so cannot write
until it has been written and closed by the previous process. Dupes do
not happen, over-quota does not happen - not for 15 minutes, 15 seconds
or 15 hundredths of a second.
I'd much rather see Sam Varshavchik responding on this thread - his
temperament's about the same as Wietse's. Either they'd end up by going
for a beer together, or they'd annihilate one another.
Me, I'll keep on using maildrop and maildir++ whoever says what ;) Just
as I do Postfix.
--Tonni
--=20
We make out of the quarrel with others rhetoric
but out of the quarrel with ourselves, poetry.
Wietse Venema
[ Charset ISO-8859-15 unsupported, converting... ]
> l_r, 26.06.2004 kl. 14.05 skrev Ralf Hildebrandt:
>
> > > The maildir++ patches will rewrite the maildirsize file every 15 minutes.
>
> I support mike right up to the hilt, so do the hundreds or thousands of
> mailadmins using Courier maildrop for maildir++. As far as those 15
> minutes go, they don't apply to maildrop - maildrop will recalculate
> (and even recreate if it's deleted) Maildir/maildirsize on each
> delivery, on the basis of the admin's quota directive.
How does the code scale when each maildirsize file is recreated on
each delivery (meaning: scan the entire directory and tally up all
the file sizes) on a busy server that handles mail for thousands
of accounts? What stops the code from keeping the file system busy
doing nothing but directory scanning to recreate maildirsize files?
And people still claim that this utter kludge is superior to kernel
based quota.
> > > There is a chance that a user can go over quota for a period of time,
> > > but not for very long.
> >
> > So, if a mailbox is being mailbombed (joe jobbed, whatever), it may go
> > WAAAY beyond the quota limit. And that's exactly why the quota was
> > imposed in the first place.
>
> Each maildrop process (there could be many) will open maildirsize for
> writing on delivery. A second process can not open it, so cannot write
> until it has been written and closed by the previous process. Dupes do
> not happen, over-quota does not happen - not for 15 minutes, 15 seconds
> or 15 hundredths of a second.
What measures have been taken to make this reliable when delivering
to an NFS mounted file system that is shared among multiple clients?
All these are simple questions that I have seen no other answer
for than "it hasn't broken on my box" or "I was unable to break it".
Wietse
> I'd much rather see Sam Varshavchik responding on this thread - his
> temperament's about the same as Wietse's. Either they'd end up by going
> for a beer together, or they'd annihilate one another.
>
> Me, I'll keep on using maildrop and maildir++ whoever says what ;) Just
> as I do Postfix.
>
> --Tonni
>
> --
>
Roger B.A. Klorese
>And people still claim that this utter kludge is superior to kernel
>based quota.
>
>
>
I won't defend this particular implementation, but:
- it's clear that an operating system should not need to know about
application-specific users such as mailbox owners, where those users
have no validity in any context outside the application
- my earlier argument that a message store manager is the central
authority for all client implementations and is the correct place to
implement quotas is given wieght by your argument -- you just seem to
want to make the kernel do extra and application-specific duty as a
message store manager
Wietse Venema
> Wietse Venema wrote:
>
> >And people still claim that this utter kludge is superior to kernel
> >based quota.
>
> I won't defend this particular implementation, but:
> - it's clear that an operating system should not need to know about
> application-specific users such as mailbox owners, where those users
> have no validity in any context outside the application
Indeed, the kernel does not know about user names. Kernel quota
operations are expressed in terms of numerical UID values. I thought
that people were clueful enough if they engage in this discussion.
The kernel can maintain the per-UID byte counts without needing
kludges to add up file sizes throught the time a file system is
being used, and without kludges to avoid race conditions.
Wietse
Roger B.A. Klorese
>Indeed, the kernel does not know about user names. Kernel quota
>operations are expressed in terms of numerical UID values. I thought
>that people were clueful enough if they engage in this discussion.
>
>
You're ignoring the point. I never said a thing about usernames (or
password files, or any of the other gunkola you keep dragging into this
discussion).
If a server is providing email services for a million authenticating
mail users ONLY FOR EMAIL SERVICES, it's silly to use a unique UID and a
bunch of kernel services based on those UIDs. Those functions should be
provided by the email application. Rely on kernel services only for
multi-purpose entities.
>The kernel can maintain the per-UID byte counts without needing
>kludges to add up file sizes throught the time a file system is
>being used, and without kludges to avoid race conditions.
>
>
But it's not the kernel's business. Having separate UIDs for each of a
million email accounts moves authentication, security, and many other
management concerns out of the mail system where it belongs, and into
the kernel where it doesn't. It also means that superuser privileges
are required for many email management functions where they shouldn't be.
Wietse Venema
[ Charset ISO-8859-1 unsupported, converting... ]
> Wietse Venema wrote:
>
> >Indeed, the kernel does not know about user names. Kernel quota
> >operations are expressed in terms of numerical UID values. I thought
> >that people were clueful enough if they engage in this discussion.
>
> You're ignoring the point. I never said a thing about usernames (or
> password files, or any of the other gunkola you keep dragging into this
> discussion).
At the start of this thread:
There is more load on the disk with the maildir++ lookups, but
handling 100K users is far easier without having to have a HUGE
password map and the like.
HUGE password maps were used as an excuse against kernel-based quota.
> >The kernel can maintain the per-UID byte counts without needing
> >kludges to add up file sizes throught the time a file system is
> >being used, and without kludges to avoid race conditions.
>
> But it's not the kernel's business.
File system quota are the kernel's business, for the same reason
that file systems are the kernel's business. Now, if Postfix were
to maintain an MS-Exchange like email database, where the mail
system is responsible for how storage is assigned to users, then
it would also be responsible for quota enforcement. But Postfix
does not assign storage to users. That's the job of the file system.
> Having separate UIDs for each of a
> million email accounts moves authentication, security, and many other
> management concerns out of the mail system where it belongs, and into
> the kernel where it doesn't.
Authentication moved into the kernel? You mis-underestimate how
UNIX works. UNIX kernels don't authenticate users. That is the root
cause of why UNIX has such a weak security model. Any process with
sufficient privilege pokes a UID into the kernel and *poof* it has
the rights of that user. Authentication is completely optional.
Wietse
Roger B.A. Klorese
>HUGE password maps were used as an excuse against kernel-based quota.
>
>
>
You're using "map" as the uid-to-name translation -- I'm using it, if at
all, as the simple existence of all of those uid's. A map is still a
map even if the locations aren't labelled.
>File system quota are the kernel's business, for the same reason
>that file systems are the kernel's business. Now, if Postfix were
>to maintain an MS-Exchange like email database, where the mail
>system is responsible for how storage is assigned to users, then
>it would also be responsible for quota enforcement. But Postfix
>does not assign storage to users. That's the job of the file system.
>
>
But file systems are only one thing that Postfix delivers to. It also
delivers to LDA's. By itself, Postfix uses one of a few primitive
translations to implement a white-box message store manager, hoping the
access servers will do so in a complementary manner. A decent LDA is
integrated with access servers and message store management. Cyrus,
Sendmail's hosting products, and many others are examples that use
black-box message store management. Postfix can deliver to those as well.
>Authentication moved into the kernel? You mis-underestimate how
>UNIX works.
>
Misspoken, granted. You move authorization, based on exernally
authenticatable entities, into the kernel. For a mail-only system, or
for mail-only entities in a mixed-use system, that belongs only in the
message store manager,
Roger B.A. Klorese
>With mail stored as individual files, the file system is
>responsible for sotrage management.
>
>
At the lowest level, yes. But some stores -- the production-quality
ones, I think -- use files only as an underlying containering mechanism,
and do a higher level of storage management too.
Tony Earnshaw
> > > > The maildir++ patches will rewrite the maildirsize file every 15 =
minutes.
> >=20
> > I support mike right up to the hilt, so do the hundreds or thousands =
of
> > mailadmins using Courier maildrop for maildir++. As far as those 15
> > minutes go, they don't apply to maildrop - maildrop will recalculate
> > (and even recreate if it's deleted) Maildir/maildirsize on each
> > delivery, on the basis of the admin's quota directive.
>=20
> How does the code scale when each maildirsize file is recreated on
> each delivery (meaning: scan the entire directory and tally up all
> the file sizes) on a busy server that handles mail for thousands
> of accounts?
The file is only recreated if it is deleted. Normally it doesn't get
deleted. The total volume of the each Maildir sub directory is noted,
and each delivery is simply a plus or a minus to that total. So the
above hardly ever happens. I'm talking about Courier's maildrop and IMAP
server, not any Postfix patches, of which I have no knowledge and would
never use, given the use of maildrop.=20
> What stops the code from keeping the file system busy
> doing nothing but directory scanning to recreate maildirsize files?
It doesn't. Simple as that.
> And people still claim that this utter kludge is superior to kernel
> based quota.
Kludge? As I said, I'd like to have Sam Varshavchik answer in this
thread; he's the code's creator and maintainer. He has to put up with a
ton of Mark Crispin's sneering as well, so he's used to coping with
suchlike. In the mean time, Courier IMAP and maildrop are being used by
many large sites.
> > > > There is a chance that a user can go over quota for a period of t=
ime,
> > > > but not for very long.
> > >=20
> > > So, if a mailbox is being mailbombed (joe jobbed, whatever), it may=
go
> > > WAAAY beyond the quota limit. And that's exactly why the quota was
> > > imposed in the first place.
> >=20
> > Each maildrop process (there could be many) will open maildirsize for
> > writing on delivery. A second process can not open it, so cannot writ=
e
> > until it has been written and closed by the previous process. Dupes d=
o
> > not happen, over-quota does not happen - not for 15 minutes, 15 secon=
ds
> > or 15 hundredths of a second.
>=20
> What measures have been taken to make this reliable when delivering
> to an NFS mounted file system that is shared among multiple clients?
Sam could tell you more about that than I can ;)
> All these are simple questions that I have seen no other answer
> for than "it hasn't broken on my box" or "I was unable to break it".
>=20
> Wietse
>=20
> > I'd much rather see Sam Varshavchik responding on this thread - his
> > temperament's about the same as Wietse's. Either they'd end up by goi=
ng
> > for a beer together, or they'd annihilate one another.
> >=20
> > Me, I'll keep on using maildrop and maildir++ whoever says what ;) Ju=
st
> > as I do Postfix.
--Tonni
--=20
Tony Earnshaw
> >And people still claim that this utter kludge is superior to kernel
> >based quota.
> I won't defend this particular implementation, but:
> - it's clear that an operating system should not need to know about
> application-specific users such as mailbox owners, where those users
> have no validity in any context outside the application
> - my earlier argument that a message store manager is the central
> authority for all client implementations and is the correct place to
> implement quotas is given wieght by your argument -- you just seem to
> want to make the kernel do extra and application-specific duty as a
> message store manager
I use file system hard quotas. But I agree entirely with the points
above. My user administration is almost entirely contained in LDAP
records. This includes all mail administration parameters for individual
users. I find it convenient to include quotas in that. It is often
desirable that the admin gives individual users their own quota and
LDAP/maildir++ is a convenient way to do this. It is part of my message
store management, I shall not drop this method; it works well in
practice and i have no negative experience with it.
--Tonni
--
Wietse Venema
[ Charset ISO-8859-15 unsupported, converting... ]
>
> > > > > The maildir++ patches will rewrite the maildirsize file every 15 minutes.
> > >
> > > I support mike right up to the hilt, so do the hundreds or thousands of
> > > mailadmins using Courier maildrop for maildir++. As far as those 15
> > > minutes go, they don't apply to maildrop - maildrop will recalculate
> > > (and even recreate if it's deleted) Maildir/maildirsize on each
> > > delivery, on the basis of the admin's quota directive.
> >
> > each delivery (meaning: scan the entire directory and tally up all
> > the file sizes) on a busy server that handles mail for thousands
> > of accounts?
>
> The file is only recreated if it is deleted. Normally it doesn't get
> deleted. The total volume of the each Maildir sub directory is noted,
> and each delivery is simply a plus or a minus to that total. So the
> above hardly ever happens. I'm talking about Courier's maildrop and IMAP
> server, not any Postfix patches, of which I have no knowledge and would
> never use, given the use of maildrop.
What measures have been taken to avoid corruption of the maildirsize
file when multiple processes attempt to deliver/delete mail?
What measures have been taken to avoid corruption of the maildirsize
file when the file system is shared via NFS?
> Sam could tell you more about that than I can ;)
Surely SAM has written up a document with the measures that were
taken to avoid corruption of the maildirsize file.
Wietse
Mike Horwath
<wie...@porcupine.org> wrote:
> Tony Earnshaw:
> [ Charset ISO-8859-15 unsupported, converting... ]
> >
> >
> > I support mike right up to the hilt, so do the hundreds or thousands of
> > mailadmins using Courier maildrop for maildir++. As far as those 15
> > minutes go, they don't apply to maildrop - maildrop will recalculate
> > (and even recreate if it's deleted) Maildir/maildirsize on each
> > delivery, on the basis of the admin's quota directive.
>
> How does the code scale when each maildirsize file is recreated on
> each delivery (meaning: scan the entire directory and tally up all
> the file sizes) on a busy server that handles mail for thousands
> doing nothing but directory scanning to recreate maildirsize files?
If you would get off of your high hourse and look at things, you would
see that the filesize is encoded in the filename, so it is just a
directory open and listing.
Sorry if I seem to be taking the same attitude you are towards this,
but it is frustrating that you won't take the time at all to have a
discussion, but instead just want to attack people.
Do you have a test suite written that I can test with?
If I give you access to my systems that are not in production yet,
would you take me up on it to test out the patches for yourself?
How about instead of just being negative on the whole thing, why not
work with one of us other admins of mail systems to see if any of our
comments have any weight.
> And people still claim that this utter kludge is superior to kernel
> based quota.
Who says that?
That'd be stupid.
--
Mike Horwath
drec...@gmail.com
Mike Horwath
<wie...@porcupine.org> wrote:
> At the start of this thread:
>
> There is more load on the disk with the maildir++ lookups, but
> handling 100K users is far easier without having to have a HUGE
> password map and the like.
>
No, there was no excuse.
You just latched onto this one item.
My first full cluster system was done via UIDs, password maps, and
kernel quotas.
As the other poster stated, to allow support or non-admin staff do
anything requires the ability to become superuser, something I am not
comfortable with.
Also, using static:1003 (or whatever UID you want to use) is just one
less lookup item required of the database.
--
Mike Horwath
drec...@gmail.com
Victor....@morganstanley.com
> If you would get off of your high horse and look at things, you would
> see that the filesize is encoded in the filename, so it is just a
> directory open and listing.
>
> Sorry if I seem to be taking the same attitude you are towards this,
> but it is frustrating that you won't take the time at all to have a
> discussion, but instead just want to attack people.
>
I suggest we close this thread for a while. Personal attacks are strong
evidence that the thread is no longer productive.
The real sticking point is that the users of "maildir++" don't want robust
quota code that is guaranteed to work all the time! They are willing to
trade robustness (quota state files never need to be rebuilt, user always
under quota, ...) for ease of use (no filesystem quota interface to worry
about, quotas can be soft allowing for configurable limited functionality
when over quota).
My corporate Cyrus IMAP mailbox, for example, has a non-kernel based quota
mechanism that blocks my ability to refile mail out of the INBOX when I am
over quota. There is no quota on new mail, but if I want to be able to
save it (old INBOX messages are auto-expired) I need to get under quota.
Such bells and whistles are not possible within a filesystem based quota
system. On the other hand the flexible semantics require careful agreement
betweent the delivery code and the IMAP server. In this case both are
Cyrus code, and the MTA is none the wiser.
The Cyrus quota system, Cyrus mailbox indices, ... are not by design
crash-proof, messages are not lost, but occasional recovery of mailbox
metadata is necessary. Cyrus administrators accept this tradeoff, but
Wietse Venema does not. Fragile features are taboo in the Postfix design.
This design goal benefits all Postfix users, because they can safely
assume that the features that are supported will work all the time. Yes,
some useful features get left out until a robust design is identified or
forever if no such design appears.
There is a clear boundary between Postfix and third-party extensions,
unsafe designs that offer a reasonable tradeoff acceptable to the user are
on the other side of the boundary.
The pipe(8) and lmtp(8) delivery agents allow you to plug mailstore
specific delivery mechanisms into Postfix, by all means use them.
The most prudent strategy for maildir users who want a non filesystem
based quota system is to use a third-party virtual delivery agent known to
implement something compatible with with their chosen POP or IMAP server
and observed to be robust enough in practice in their target environment.
The most popular delivery agent in this space is Courier "maildrop". By
all means use it if you want soft quotas.
The only feature maildrop lacks relative to virtual(8) is catchall mailbox
support. This can be emulated with virtual(5) and identity mappings for
non-catchall users. Also there is a performance cost due to the
per-delivery fork/exec. This IMHO is not an excessive price to pay, so
instead of patching virtual(8) use "maildrop".
Soft quotas for maildir are not coming to Postfix any time soon. To those
who are disappointed: you can surely deal with it, the world is imperfect.
> Do you have a test suite written that I can test with?
>
It is not Wietse's code, why would you expect him to have a test-suite?
The author of the code should have a test-suite.
> If I give you access to my systems that are not in production yet, would
> you take me up on it to test out the patches for yourself?
>
Wietse's point is that code he is willing to include in Postfix needs to
be demonstrably robust by *design* on every supported system and in every
supported configuration, as well as typically robust in practice. Somehow
that point is not becoming clearer.
Though it may not be apparent to the casual user, the current Postfix
feature set is robust in this way.
> How about instead of just being negative on the whole thing, why not
> work with one of us other admins of mail systems to see if any of our
> comments have any weight.
>
The burden of proof when claiming design robustness always lies with the
author of the code. If the problem is clearly a difficult one to solve
"correctly" in every case, the sceptic need not prove anything. The
sceptic is entitled to demand an a proof that demostrates the correctness
claim "in principle" as well as tests that demonstrate it in practice,
neither alone is sufficient.
--
Viktor.
Disclaimer: off-list followups get on-list replies or get ignored.
Please do not ignore the "Reply-To" header.
To unsubscribe from the postfix-users list, visit
http://www.postfix.org/lists.html or click the link below:
<mailto:majo...@postfix.org?body=unsubscribe%20postfix-users>
Mike Horwath
victor....@morganstanley.com <victor....@morganstanley.com>
wrote:
>
> On Sun, 27 Jun 2004, Mike Horwath wrote:
>
> > If you would get off of your high horse and look at things, you would
> > see that the filesize is encoded in the filename, so it is just a
> > directory open and listing.
> >
> > Sorry if I seem to be taking the same attitude you are towards this,
> > but it is frustrating that you won't take the time at all to have a
> > discussion, but instead just want to attack people.
>
> I suggest we close this thread for a while. Personal attacks are strong
> evidence that the thread is no longer productive.
I am going to agree with this as well, this will be my last response
on this thread.
> The real sticking point is that the users of "maildir++" don't want robust
> quota code that is guaranteed to work all the time! They are willing to
> trade robustness (quota state files never need to be rebuilt, user always
> under quota, ...) for ease of use (no filesystem quota interface to worry
> about, quotas can be soft allowing for configurable limited functionality
> when over quota).
And you can achieve most of that via in-kernel UID quotas as well.
Of course not every UNIX supports it, and if you listen to NetApp,
they claim that soft quotas are a hack. Of course, after years of
hearing that they have implemented this as of DataOntap 6.4 and above
:)
> My corporate Cyrus IMAP mailbox, for example, has a non-kernel based quota
> mechanism that blocks my ability to refile mail out of the INBOX when I am
> over quota. There is no quota on new mail, but if I want to be able to
> save it (old INBOX messages are auto-expired) I need to get under quota.
>
> Such bells and whistles are not possible within a filesystem based quota
> system. On the other hand the flexible semantics require careful agreement
> betweent the delivery code and the IMAP server. In this case both are
> Cyrus code, and the MTA is none the wiser.
I am not a user of Cyrus for storage, so bear with me when I ask this
set of questions for clarification:
In your setup, delivery of email will continue, but you can not move
email from your INBOX to another mailbox until you have resolved your
quota issues.
If a mailbomb were to occur towards your account, you could
effectively create a DOS for the system/filesystem because the quota
doesn't disallow new email reception.
Are my assumptions right or wrong?
New thread for discussion so I can honor my statement of not
responding to this thread after this message? :)
> This design goal benefits all Postfix users, because they can safely
> assume that the features that are supported will work all the time. Yes,
> some useful features get left out until a robust design is identified or
> forever if no such design appears.
Actually, the words I read were that if these 'features' don't work
for both mbox and Maildir storage, then what is the point?
But that could be just as easily my current attitude in this thread...
> The pipe(8) and lmtp(8) delivery agents allow you to plug mailstore
> specific delivery mechanisms into Postfix, by all means use them.
using either pipe(8) or lmtp(8) doesn't necessarily bounce email if a
user is over quota, creating async bounces.
Using local Postfix delivery agents does not create async bounces.
The async bounce issue is a large issue, especially for ISPs and mail providers.
> Soft quotas for maildir are not coming to Postfix any time soon. To those
> who are disappointed: you can surely deal with it, the world is imperfect.
I don't think anyone asked for soft quotas...
> > Do you have a test suite written that I can test with?
>
> It is not Wietse's code, why would you expect him to have a test-suite?
> The author of the code should have a test-suite.
I was thinking of a generic test suite that would be used to look for
the types of corruption alluded to by him.
I am not looking for a test suite that goes against maildir++ type
setups, but a general test system that will exercise everything.
> > If I give you access to my systems that are not in production yet, would
> > you take me up on it to test out the patches for yourself?
>
> Wietse's point is that code he is willing to include in Postfix needs to
> be demonstrably robust by *design* on every supported system and in every
> supported configuration, as well as typically robust in practice. Somehow
> that point is not becoming clearer.
Why would you say that?
That part was made clear very early on, the maildir++ system is robust
by design, and is supported by any and all software that has been
written/patched to include the correct functions.
I am researching the issue to help show that maildir++ is a robust
design, failure resilient, and capable at high loads and transactions.
> Though it may not be apparent to the casual user, the current Postfix
> feature set is robust in this way.
Those of us building clusters using Postfix as the MTA (and LDA) do
realize that. :)
--
Mike Horwath
drec...@gmail.com
Wietse Venema
> > > Do you have a test suite written that I can test with?
> >
> > It is not Wietse's code, why would you expect him to have a test-suite?
> > The author of the code should have a test-suite.
Sorry, code is not proven correct by testing.
Code must be DESIGNED in order to solve the probem at hand.
So far, not a single person has been able to point to a document
that describes how maildir++ is DESIGNED to address the scalability
and concurrency requirements that I have expressed here, and how
it is DESIGNED to make the solution robust in an NFS environment.
I do maintain some minimal quality requirements. "Works for me"
is not sufficient.
Wietse
Tony Earnshaw
[...]
> Wietse's point is that code he is willing to include in Postfix needs t=
o
> be demonstrably robust by *design* on every supported system and in eve=
ry
> supported configuration, as well as typically robust in practice. Someh=
ow
> that point is not becoming clearer.
>=20
> Though it may not be apparent to the casual user, the current Postfix
> feature set is robust in this way.
>=20
> > How about instead of just being negative on the whole thing, why not
> > work with one of us other admins of mail systems to see if any of our
> > comments have any weight.
> >
>=20
> The burden of proof when claiming design robustness always lies with th=
e
> author of the code. If the problem is clearly a difficult one to solve
> "correctly" in every case, the sceptic need not prove anything. The
> sceptic is entitled to demand an a proof that demostrates the correctne=
ss
> claim "in principle" as well as tests that demonstrate it in practice,
> neither alone is sufficient.
A lucid and understandable explanation of a philosophy - thanks. Shan't
make use of third-party products as arguments for or against Postfix
design policy again.
--Tonni
--=20
J. Ryan Earl
>* Mike Horwath <drec...@gmail.com>:
>
>
>
>>The maildir++ patches will rewrite the maildirsize file every 15 minutes.
>>
>>There is a chance that a user can go over quota for a period of time,
>>but not for very long.
>>
>>
>
>So, if a mailbox is being mailbombed (joe jobbed, whatever), it may go
>WAAAY beyond the quota limit. And that's exactly why the quota was
>imposed in the first place.
>
>
I think the "update every 15 minutes" if not true. Mine gets updated
immediately, there is no possibility of going over quota, the first
message over quota will be bounced back. I was under the impression
that the maildirsize file was locked on a per user basis preventing
parallel delivery to the same user, thus avoiding all NFS race
conditions; NFS should block on paralle delivery until the maildirsize
file is closed by the previous delivery process. When the blocked
process finally opens the NFS maildirsize file for writing, it will see
the user is overquota and bounce the message back, thus avoiding any
mail spool overhead.
This is the obvious answer is it not?
-ryan
Victor....@morganstanley.com
> I think the "update every 15 minutes" if not true. Mine gets updated
> immediately, there is no possibility of going over quota, the first
> message over quota will be bounced back. I was under the impression
> that the maildirsize file was locked on a per user basis preventing
> parallel delivery to the same user, thus avoiding all NFS race
> conditions; NFS should block on paralle delivery until the maildirsize
> file is closed by the previous delivery process. When the blocked
> process finally opens the NFS maildirsize file for writing, it will see
> the user is overquota and bounce the message back, thus avoiding any
> mail spool overhead.
>
This thread is dead. You will be unsubscribed. You may resubscribe if you
please, but please don't revive dead threads. Also don't just guess. The
first hit on google when searching for '"maildir++" specification' is:
http://www.inter7.com/courierimap/README.maildirquota.html
This explains the lack of locks, the 15 minute resyncs, the possibility of
going substantially over quota for a short time, lots of caveats and yes
no-one disputes that it typically works in practice. The "typically works
in practice" part is not good enough to get it included in the main
Postfix distribution, there will not be compromise on this. So the thread
is closed. Do not re-open this thread.
J. Ryan Earl
>How does the code scale when each maildirsize file is recreated on
>each delivery (meaning: scan the entire directory and tally up all
>the file sizes) on a busy server that handles mail for thousands
>of accounts? What stops the code from keeping the file system busy
>doing nothing but directory scanning to recreate maildirsize files?
>
>
That I don't know, but I would be HIGHLY surprised if it rescanned. If
my maildirsize file gets corrupted, I have to rebuild it manually,
leading me to believe maildirsize is updated. I'll audit the code when
I have time.
>And people still claim that this utter kludge is superior to kernel
>based quota.
>
>
Does this method give user friendly bounce messages?
>
>
>>>>There is a chance that a user can go over quota for a period of time,
>>>>but not for very long.
>>>>
>>>>
>>>So, if a mailbox is being mailbombed (joe jobbed, whatever), it may go
>>>WAAAY beyond the quota limit. And that's exactly why the quota was
>>>imposed in the first place.
>>>
>>>
>>Each maildrop process (there could be many) will open maildirsize for
>>writing on delivery. A second process can not open it, so cannot write
>>until it has been written and closed by the previous process. Dupes do
>>not happen, over-quota does not happen - not for 15 minutes, 15 seconds
>>or 15 hundredths of a second.
>>
>>
>
>What measures have been taken to make this reliable when delivering
>to an NFS mounted file system that is shared among multiple clients?
>
>
Is this not inherent to the NFS protocol? Isn't there something similar
to a RTS/CTS packet sent, though I guess it would be RTW/CTW (Request To
Write/Clear To Write). What's the point of NFS if it doesn't handle
this sort of thing? The NFS server should keep track of what clients
have requested and currently have a write lock to particular files.
>All these are simple questions that I have seen no other answer
>for than "it hasn't broken on my box" or "I was unable to break it".
>
>
Yes, well, C/C++ isn't the best language for Formal Method Analysis, on
top of which you could define "Correctness" incorrectly. You have to be
careful what you prove. Having been involved in Formal Method analysis
ala Dr. Moore's lab
(http://www.cs.utexas.edu/users/moore/atp/index.html) I can tell you
without a doubt that Proving Software Correct will never replace Testing
Software because it's hard to get the definition of "Correctness"
right. Not to mention the cost of Formal Methods versus Testing/QA.
You have to use both, but I feel you on the correctness stuff.
-ryan