Multiple queue directories, multiple queue runners?
Nik Clayton
The Sendmail documentation (sections 2.3.1 and 2.3.5 of the Operations
Guide) says that if you have multiple queue directories (e.g.,
/var/spool/mqueue/qdir00 through /var/spool/mqueue/qdir04), and a
QUEUE_DIR definition that includes them, such as:
define(`QUEUE_DIR', `/var/spool/mqueue/qdir*')
then running something like this:
sendmail -q30m
should start five queue runners, one for every queue directory, and then
another five every 30 minutes.
This is also mentioned in various other sources, including Nick
Christenson's book "Sendmail Performance Tuning".
The problem is, I don't see this (Sendmail 8.13.7). No matter how many
queue directories I have Sendmail only ever starts one queue runner
every <interval>. I've described this in a bit more detail at:
http://jc.ngo.org.uk/blog/2006/08/19/day-41-of-60-multiple-queues-multiple-queue-runners-pt-2/
If you've got a similar configuration, could you take the time to verify
this on your system, and either confirm or deny this? I'd greatly
appreciate it.
Thanks,
N
Per Hedeland
<n...@ngo.org.uk> writes:
>
>The Sendmail documentation (sections 2.3.1 and 2.3.5 of the Operations
>Guide) says that if you have multiple queue directories (e.g.,
>/var/spool/mqueue/qdir00 through /var/spool/mqueue/qdir04), and a
>QUEUE_DIR definition that includes them, such as:
>
> define(`QUEUE_DIR', `/var/spool/mqueue/qdir*')
>
>then running something like this:
>
> sendmail -q30m
>
>should start five queue runners, one for every queue directory, and then
>another five every 30 minutes.
>
>This is also mentioned in various other sources, including Nick
>Christenson's book "Sendmail Performance Tuning".
>
>The problem is, I don't see this (Sendmail 8.13.7). No matter how many
>queue directories I have Sendmail only ever starts one queue runner
>every <interval>.
Did you increase MaxRunnersPerQueue above the default of 1? I'm afraid
the addition of queue groups (in 8.12 - multiple queue directories was
added in 8.10) has made some things in the config and docs a bit
ambiguous at best in the "queue vs queue group" area - e.g. I believe
MaxRunnersPerQueue should probably be called MaxRunnersPerQueueGroup
(though it was added in 8.12, so it could have gotten the right name
from the start:-).
Looking at the code in queue.c, it does seem to do the "one runner per
directory" thing "To match older behavior (8.10/8.11)", but only in the
default case of there being only one queue group called "mqueue". And,
as far as I can see, it is still capped by the per-queuegroup-max, which
defaults to MaxRunnersPerQueue. I don't know if the number of runners
will be capped to the number of directories if you set
MaxRunnersPerQueue to a huge number, or if you may need to set it to
match the number of directories you have, but I think it will at least
start out with one runner per directory even if MaxRunnersPerQueue is
higher.
Either way, if you have serious performance issues to deal with and are
looking at multiple queues for the solution, you should probably go for
a modern multiple-queue-group setup rather than sticking with the old
and more simplistic "just create a bunch of directories" stuff - I think
things will start to make more sense then.
--Per Hedeland
p...@hedeland.org
Nik Clayton
> Did you increase MaxRunnersPerQueue above the default of 1? I'm afraid
> the addition of queue groups (in 8.12 - multiple queue directories was
> added in 8.10) has made some things in the config and docs a bit
> ambiguous at best in the "queue vs queue group" area - e.g. I believe
> MaxRunnersPerQueue should probably be called MaxRunnersPerQueueGroup
> (though it was added in 8.12, so it could have gotten the right name
> from the start:-).
I didn't, I've just tried that.
define(`confMAX_QUEUE_CHILDREN', `5')
define(`confMAX_RUNNERS_PER_QUEUE', `5')
With 5 queue directories (and one queuegroup, the default 'mqueue'), running
sendmail -q30s
still only starts one queue runner every 30 seconds, not 5. However, it
does only start a maximum of 5 children (i.e., no more queue runners are
created after 120 seconds), so this has had an effect on Sendmail's
behaviour, just not the effect that I wanted.
> Looking at the code in queue.c, it does seem to do the "one runner per
> directory" thing "To match older behavior (8.10/8.11)", but only in the
> default case of there being only one queue group called "mqueue".
I'm configured as that default case. No explicit queue groups defined,
so I should be using the default 'mqueue'.
Could you try this, and see if you can duplicate the results I'm seeing?
> Either way, if you have serious performance issues to deal with and are
> looking at multiple queues for the solution, you should probably go for
> a modern multiple-queue-group setup rather than sticking with the old
> and more simplistic "just create a bunch of directories" stuff - I think
> things will start to make more sense then.
I know how to configure multiple queue groups -- that's not the point of
this at the moment[1]. Either this is a fairly serious Sendmail
documentation bug, or it's a fairly serious Sendmail behaviour bug.
N
[1] My wider efforts are targeted at attempting to provide empirical
evidence to answer questions like "How many queues should I have?",
"What difference do the different sort orders have?" and so on.
Existing documentation in this area is very much of the "You'll have to
try it and see" variety, which is not very useful.
Anonyme
--
0% de pub! Que du bonheur et des vrais adhérents !
Vous aussi inscrivez-vous sans plus tarder!!
Message posté à partir de http://www.gyptis.org, BBS actif depuis 1995.
Per Hedeland
writes:
>
>Could you try this, and see if you can duplicate the results I'm seeing?
Yes, it's definitely broken - the 8.10/8.11 backward compatibility code
(in queue.c/makeworkgroups()) that is supposed to set up the number of
queue runners based on the number of queues (i.e. directories) in the
"mqueue" group is run long before the code (in queue.c/setup_queues())
that figures out how many directories there are...
>> Either way, if you have serious performance issues to deal with and are
>> looking at multiple queues for the solution, you should probably go for
>> a modern multiple-queue-group setup rather than sticking with the old
>> and more simplistic "just create a bunch of directories" stuff - I think
>> things will start to make more sense then.
>
>I know how to configure multiple queue groups -- that's not the point of
>this at the moment[1]. Either this is a fairly serious Sendmail
>documentation bug, or it's a fairly serious Sendmail behaviour bug.
Well, my point is that since 8.12 (released almost 5 years ago), a much
more powerful tool to parallelize the queue processing is available,
namely queue groups - so there is little point in fiddling with the
old/primitive 8.10/8.11 mechanism. In those versions, creating multiple
queue directories was the only knob that you could turn, queue groups
offer many more - and there is no obvious reason that the number of
queue runners should equal the number of queue directories.
I.e. it's definitely a bug since the intent is to preserve that old
behaviour in the case where no queue groups have been configured - but I
would hardly call it serious.
>[1] My wider efforts are targeted at attempting to provide empirical
>evidence to answer questions like "How many queues should I have?",
>"What difference do the different sort orders have?" and so on.
>Existing documentation in this area is very much of the "You'll have to
>try it and see" variety, which is not very useful.
That's great, but I think that those that need to ask those questions
should probably already be looking at current functionality - i.e. queue
groups - and your efforts would be better spent exploring that. And I
hope you realize that one reason that there are no simple answers is
that those questions are hard to answer, due to the large number of
variables involved - some artificial tests on a single system will not
produce the definitive answer. You'll have a hard time doing better
than Nick's book...
--Per Hedeland
p...@hedeland.org
Nik Clayton
> In article <44E85B0A...@ngo.org.uk> Nik Clayton <n...@ngo.org.uk>
> writes:
>> Could you try this, and see if you can duplicate the results I'm seeing?
>
> Yes, it's definitely broken - the 8.10/8.11 backward compatibility code
> (in queue.c/makeworkgroups()) that is supposed to set up the number of
> queue runners based on the number of queues (i.e. directories) in the
> "mqueue" group is run long before the code (in queue.c/setup_queues())
> that figures out how many directories there are...
Yeah -- I looked at the code and came to the same conclusions that you
did. Then I checked this newsgroup for replies. I could've saved
myself 30 minutes :-)
>>> Either way, if you have serious performance issues to deal with and are
>>> looking at multiple queues for the solution, you should probably go for
>>> a modern multiple-queue-group setup rather than sticking with the old
>>> and more simplistic "just create a bunch of directories" stuff - I think
>>> things will start to make more sense then.
>>
>> I know how to configure multiple queue groups -- that's not the point of
>> this at the moment[1]. Either this is a fairly serious Sendmail
>> documentation bug, or it's a fairly serious Sendmail behaviour bug.
>
> Well, my point is that since 8.12 (released almost 5 years ago), a much
> more powerful tool to parallelize the queue processing is available,
> namely queue groups - so there is little point in fiddling with the
> old/primitive 8.10/8.11 mechanism. In those versions, creating multiple
> queue directories was the only knob that you could turn, queue groups
> offer many more - and there is no obvious reason that the number of
> queue runners should equal the number of queue directories.
Yes, save that queue groups have their own problems, such as the dire
warnings about moving messages between different queue groups, or that
(by default) they work on the recipient address, not other aspects of
the message. Even Nick's book (which you aptly recommend later) says:
At first glance, even more complex strategies seem appealing.
However, further consideration will reveal that moving too far down
this path leads to madness. While notions such as one queue per
domain or hierarchical queues might seem intriguing, how do they
affect the number of queue runners per queue? How does one implement
the mailq command? How can person track what's going on in all parts
of the "queue" without sophisticated tools? The use of multiple
queues [* Here I think he means multiple queue directories *] to
increase parallelism and queue rotation are excellent strategies
that busy sites should adopt. Sites with a dispraportionate amount
of traffic going to a single domain or sites with a large volume of
mailing list and non-mailing list traffic may want to use queue groups
to help minimize how performance problems in one area affect another
area, but increasing complexity in one's queue structure to obtain a
marginal gain in performance probably is not advisable. Keep a
server's queue structure as simple as possible.
-- Nick Christenson, Sendmail Performance Tuning, p146
>> [1] My wider efforts are targeted at attempting to provide empirical
>> evidence to answer questions like "How many queues should I have?",
>> "What difference do the different sort orders have?" and so on.
>> Existing documentation in this area is very much of the "You'll have to
>> try it and see" variety, which is not very useful.
>
> That's great, but I think that those that need to ask those questions
> should probably already be looking at current functionality - i.e. queue
> groups - and your efforts would be better spent exploring that. And I
> hope you realize that one reason that there are no simple answers is
> that those questions are hard to answer, due to the large number of
> variables involved - some artificial tests on a single system will not
> produce the definitive answer.
Which is exactly why the disclaimer on the site (which I link to every
time I post any 'conclusions' from the testing) says that the reader is
encouraged not to take the figures that I present and assume that they
apply directly to their production environment.
Instead, they should take the tools that I've provided (patches to
Sendmail that enabled DTrace instrumentation, numerous DTrace scripts
that gather and report on data, ministat for understanding the
statistical significance of the data) and run it on their own production
systems to get figures that are applicable to them.
The results that I've obtained are not going to be directly applicable,
but the methodology and the tools are.
> You'll have a hard time doing better than Nick's book...
Indeed -- which recommends against the use of complex queuing strategies.
Either way, I've reported this problem to Claus and co. through the
Sendmail mailing lists (with slightly more detailed info then I've put
here so far), so we'll see whether or not it's fixed in a later release.
N
Dennis Peterson
> Per Hedeland wrote:
> > You'll have a hard time doing better than Nick's book...
>
> Indeed -- which recommends against the use of complex queuing strategies.
>
He actually says going too far can be maddening but leaves the door open
to discover how far "far enough" (quotes mine) is. Many things done to
excess can obscure the benefits of a particular concept.
dp
Per Hedeland
>Per Hedeland wrote:
>>
>> Well, my point is that since 8.12 (released almost 5 years ago), a much
>> more powerful tool to parallelize the queue processing is available,
>> namely queue groups - so there is little point in fiddling with the
>> old/primitive 8.10/8.11 mechanism. In those versions, creating multiple
>> queue directories was the only knob that you could turn, queue groups
>> offer many more - and there is no obvious reason that the number of
>> queue runners should equal the number of queue directories.
>
>Yes, save that queue groups have their own problems, such as the dire
>warnings about moving messages between different queue groups, or that
>(by default) they work on the recipient address, not other aspects of
>the message.
Well, you could configure a single queue group and send everything there
- that would be the same as only having the default 'mqueue' group,
except that you would have full control over all the parameters, and
e.g. be able to specify the number of directories and the number of
queue runners independently.
> Even Nick's book (which you aptly recommend later) says:
>
> At first glance, even more complex strategies seem appealing.
> However, further consideration will reveal that moving too far down
> this path leads to madness.
I don't think I've suggested anything that will make you go crazy:-),
and clearly Nick isn't advising to stay away from queue groups
altogether.
--Per Hedeland
p...@hedeland.org