MSMQ permissions problem between W2003 servers cross-domain, please help!!
Adam Jones
problem on and off now for a few weeks and it's driving me crazy, so any
advice would be greatly appreciated! :-) Please note I'm a bit of an
MSMQ newbie (and I'm not a Developer), so please go easy.
Basically our company have just finished migration from NT4 -> 2003
(yeah I know, we're very behind) and still has servers spanning two
trusted NT4 domains along with our new AD2003 domain (also with full
two-way trusts to both NT4 domains).
We have an application that makes heavy use of MSMQ (note: not something
written in-house or for which I have access to the source code)
installed on servers in all the domains. It's installed in
Workgroup/non-AD integration mode (a requirement of the app, or so I'm
told). Anyway here's the problem: when a server in the AD2003 domain
(running Windows Server 2003 R2 SP1) tries to send a transactional
message to a server in one of the NT4 domains (also running Windows
Server 2003 - I've tried base version and SP1), it doesn't receive an
acknowledgement and this results in a dead-letter message being
generated. The error message I keep seeing is:
"The sender does not have send access rights on the queue".
The problem APPEARS to be that the W2003 server in the NT4 domain cannot
send messages or acknowledgements (or "talk MSMQ" in any way) to the
server in the AD2003 domain. By "talk MSMQ" I mean any MSMQ interactions
at all - there is another symptom. If I open an MMC console on the NT4
W2003 server and try to open any queue on the server in the AD2003
domain I get the following error (the other way round works fine):
"The list of messages cannot be retrieved. Error: A security package
specific error occurred".
Now the permissions are all in place, as far as I can see. I have even
gone as far as to make the domain account from the "other" system a
local administrator on both sides - no difference. I've also tried
messing around with the "anonymous" account (adding it in with full
access in various places), with no luck.
I believe I've ruled out Group Policy by putting the AD2003 server into
an OU with no GPOs applied, and doing a gpupdate /force and rebooting.
This made no difference.
Here's the really weird thing... (maybe it gives you experts a clue) -
in the second NT4 domain there is a Windows 2000 server (SP4) obviously
running MSMQ 2.0 - this can exchange messages with the W2003 servers
across the NT4 domains, and also with the servers in the AD2003 domain,
without any issues!! It is also able to view all other queues through an
MMC console. So Windows 2000->Windows 2003 (NT4->AD2003) is fine, but
Windows 2003->Windows 2003 (NT4->AD2003) is not?! What can that mean?
Please help! Sorry for writing such an essay, but I'm reaching my wits
end with this problem and this is something that I quite desperately
need to get sorted.
Many thanks in advance,
Adam.
Frank Boyne
news:Xns9B23BCC...@69.16.176.253...
> Please note I'm a bit of an
> MSMQ newbie (and I'm not a Developer), so please go easy.
Not really pertinent to your specific question but a few good sources of
information are:
John Breakwell's Blog and the other MSMQ blogs John points to
http://blogs.msdn.com/johnbreakwell/default.aspx
The Message Queuing section of the MSDN Library
http://msdn.microsoft.com/en-us/library/ms711472(VS.85).aspx
The downloadable MSMQ FAQ document
http://download.microsoft.com/download/F/C/9/FC9989A2-DA75-4D96-B654-4BD29CF6AEE1/MSMQ_faq_updated.doc
It's also often worth using the Web Search Engine of your choice to hunt
around for information. However, you have to keep in mind that MSMQ has
evolved over the years. Articles written back in the MSMQ 1.0 days may
not accurately reflect how MSMQ 3.0 (or later) behaves.
> It's installed in
> Workgroup/non-AD integration mode (a requirement of the app, or so I'm
> told).
Again this isn't really relevant, but Workgroup mode probabluy isn't an
application requirement. Workgroup mode is a subset of AD mode - it's
the subset of all the things that work without an Active Directory. Any
code that can operate in Workgroup mode should operate equally well on
systems with MSMQ installed in AD mode. The only exception would be
code that detects the MSMQ installed mode and then behaves differently
depending on that mode.
> Anyway here's the problem: when a server in the AD2003 domain
> (running Windows Server 2003 R2 SP1) tries to send a transactional
> message to a server in one of the NT4 domains (also running Windows
> Server 2003 - I've tried base version and SP1), it doesn't receive an
> acknowledgement and this results in a dead-letter message being
> generated. The error message I keep seeing is:
>
> "The sender does not have send access rights on the queue".
So the original transactional message arrives okay in the queue on the
server in the NT4 domain but the positive acknowledgement message back
to the admin queue on the server in the AD2003 domain ends up in the
Dead Letter Queue?
Or does the original message fail to reach the queue on the server in
the NT4 domain with a negative acknowledgement message being sent back
to the admin queue?
Which server's Dead Letter Queue does the failure end up in?
> Now the permissions are all in place, as far as I can see.
What permissions are in place and what is the SenderId of the failing
message?
>I have even
> gone as far as to make the domain account from the "other" system a
> local administrator on both sides - no difference.
I wouldn't expect that to have much effect (unless you've granted
special permisions to the local Administrators group).
> I've also tried
> messing around with the "anonymous" account (adding it in with full
> access in various places), with no luck.
That's interesting, anonymous senders is one possible explanation for at
least some of the symptoms you describe.
> Here's the really weird thing... (maybe it gives you experts a clue) -
> in the second NT4 domain there is a Windows 2000 server (SP4)
> obviously
> running MSMQ 2.0 - this can exchange messages with the W2003 servers
> across the NT4 domains, and also with the servers in the AD2003
> domain,
> without any issues!!
Does that include the server in the AD2003 being able to send
transactional messages to a queue on the Windows 2000 server?
Assuming that it does, we have an experiment where we've changed two or
three variables - we've changed which NT4 domain we are using and we've
changed MSMQ version and OS version on the destination system. If there
are subtle differences in the way the AD2003 domain trusts the two NT4
domains that might affect our experiment. Likewise changes in the the
way security is implemented between Windows 2000 and Windows 2003 might
explain the problem or differences in the way MSMQ works between MSMQ
2.0 and MSMQ 3.0 might explain the problem.
Ideally we'd like to eliminate some of this variation in our experiment.
So, for example, if a Windows 2003 server in the second NT4 domain also
works (and/or a Windows 2000 server in the first NT4 domain does _not_
work) then we can probably pin the blame on differences between the two
NT4 domains.
On the other hand, if a Windows 2003 server in the second NT4 domain
also fails (and/or a Windows 2000 server in the first NT4 domain works)
then we can probably pin the blame on differences between the two
destination server types: Windows 2003/MSMQ 3.0 fails while Windows
2000/MSMQ 2.0 works.
> [the Windows 2000 server] is also able to view all other queues
> through an MMC console.
Your problems with an MMC console running on Windows Server 2003 sound
like they might be due to a change introduced in MSMQ 3.0. See question
5.25 in the downloadable FAQ above and this KB article:
http://support.microsoft.com/default.aspx?scid=kb;EN-US;840146.
I wouldn't expect this to be the cause of your delivery problem - but I
could be wrong.
Adam Jones
for the helpful reply - much appreciated. Answers/responses in-line:
"Frank Boyne" <frank...@unisys.com> wrote in
news:u6cy4YpH...@TK2MSFTNGP02.phx.gbl:
> Not really pertinent to your specific question but a few good sources
> of information are:
>
> John Breakwell's Blog and the other MSMQ blogs John points to
> http://blogs.msdn.com/johnbreakwell/default.aspx
>
> The Message Queuing section of the MSDN Library
> http://msdn.microsoft.com/en-us/library/ms711472(VS.85).aspx
>
> The downloadable MSMQ FAQ document
> http://download.microsoft.com/download/F/C/9/FC9989A2-DA75-4D96-B654-
4B
> D29CF6AEE1/MSMQ_faq_updated.doc
>
> It's also often worth using the Web Search Engine of your choice to
> hunt around for information. However, you have to keep in mind that
> MSMQ has evolved over the years. Articles written back in the MSMQ
> 1.0 days may not accurately reflect how MSMQ 3.0 (or later) behaves.
Thanks for the links - the document will be particularly handy, as I
didn't even know it existed(!) I have been doing lots of hunting around
trying to find an answer to this. The most likely thing I've happened
upon appeared to be this link:
http://support.microsoft.com/kb/902951/
...but unfortunately I've tried the mentioned hotfix, with no luck.
> Again this isn't really relevant, but Workgroup mode probabluy isn't
> an application requirement. Workgroup mode is a subset of AD mode -
> it's the subset of all the things that work without an Active
> Directory. Any code that can operate in Workgroup mode should operate
> equally well on systems with MSMQ installed in AD mode. The only
> exception would be code that detects the MSMQ installed mode and then
> behaves differently depending on that mode.
Very useful to know - thanks. From what I've gathered the use of AD
integration with MSMQ is discouraged by the Devs of this application due
to performance implications. I have wondered if there's much truth in
this I must admit! I can't really imagine the performance hit would be
that great (and we are looking at some pretty beefy servers here)...
> So the original transactional message arrives okay in the queue on the
> server in the NT4 domain but the positive acknowledgement message back
> to the admin queue on the server in the AD2003 domain ends up in the
> Dead Letter Queue?
>
> Or does the original message fail to reach the queue on the server in
> the NT4 domain with a negative acknowledgement message being sent back
> to the admin queue?
>
> Which server's Dead Letter Queue does the failure end up in?
I was pretty convinced that it was the first case (NT4 -> AD2003 failing
to send the acknowledgement) but I've done some testing and actually
you're exactly right with the second possibility. I was getting confused
because no negative acknowledgement message ever seemed to arrive at the
Admin queue, but I have since discovered (by examining log files created
by the application) that this is most likely because these messages are
being "handled" by the application - which I can only assume means they
are being deleted (I have seen it do this before, so I am quite sure
that's what is happening). I expect if it did not do this, a dead-letter
message would land into the Transactional Dead-Letter messages queue on
the sending/AD2003 server?
The error message in the negative acknowledgement is:
"The sender does not have send access rights on the queue."
I thought that I had seen some dead-letter messages on the server in the
NT4 domain, but couldn't reproduce that today in my testing so I must've
been thinking of something else (either that or they take time to reach
the dead-letter queue... maybe they'll be there tomorrow). Another
observation is that an outgoing queue is "created" (I know they're a
transient thing) on the sending server AND the receiving server, where
the message tends to sit with the "unacknowledged" flag equalling the
"number of messages". Outgoing queues to other servers (including the
W2000 servers in the second NT4 domain) have "unacknowledged" = 0.
>
> What permissions are in place and what is the SenderId of the failing
> message?
>
Right I didn't explain this bit very well - my apologies. The
application uses a domain (service) account for all of its operations...
including sending MSMQ messages. This account has explicit permissions
on the target queue, and is the owner of all private queues on the
source... (sorry I also forgot to mention that all the queues are
private). This permission is what is required to make the message sends
work; certainly for the Windows 2000 servers! As I mentioned I have
tried cross-permissioning these accounts as Local Admins on either side,
with no luck. The sender of these messages shows as being "AD2003DOM
\ACCNAME" (hope this is what you meant?)
> That's interesting, anonymous senders is one possible explanation for
> at least some of the symptoms you describe.
I tried this because I didn't know whether, for whatever reason, the
domain credentials were not being passed correctly (and therefore the
target sees the transaction as being anonymous?) I realise this was a
bit of a long shot! :-) Do you think there is any way that impersonation
is getting in the way here?
>
> Does that include the server in the AD2003 being able to send
> transactional messages to a queue on the Windows 2000 server?
Yes, definitely.
> Assuming that it does, we have an experiment where we've changed two
> or three variables - we've changed which NT4 domain we are using and
> we've changed MSMQ version and OS version on the destination system.
> If there are subtle differences in the way the AD2003 domain trusts
> the two NT4 domains that might affect our experiment. Likewise
> changes in the the way security is implemented between Windows 2000
> and Windows 2003 might explain the problem or differences in the way
> MSMQ works between MSMQ 2.0 and MSMQ 3.0 might explain the problem.
>
> Ideally we'd like to eliminate some of this variation in our
> experiment. So, for example, if a Windows 2003 server in the second
> NT4 domain also works (and/or a Windows 2000 server in the first NT4
> domain does _not_ work) then we can probably pin the blame on
> differences between the two NT4 domains.
> On the other hand, if a Windows 2003 server in the second NT4 domain
> also fails (and/or a Windows 2000 server in the first NT4 domain
> works) then we can probably pin the blame on differences between the
> two destination server types: Windows 2003/MSMQ 3.0 fails while
> Windows 2000/MSMQ 2.0 works.
>
Right. I believe I have now eliminated the possibility of problems
between the two NT4 domains by testing it with a Windows 2003 server in
the same NT4 domain as the "working" Windows 2000 servers. The exact
same problem occurs when trying to send a message from AD2003 to this
server, along with the "MMC" symptom. I guess this means it's an OS/MSMQ
3.0 problem (or a setting, local to those servers, amiss somewhere - our
standard server build is quite heavily customised)?
>
> Your problems with an MMC console running on Windows Server 2003 sound
> like they might be due to a change introduced in MSMQ 3.0. See
> question 5.25 in the downloadable FAQ above and this KB article:
> http://support.microsoft.com/default.aspx?scid=kb;EN-US;840146.
>
> I wouldn't expect this to be the cause of your delivery problem - but
> I could be wrong.
>
Thanks for the link - I hadn't seen that one! Unfortunately I've now
tried the mentioned fix (for source and target servers) and it makes no
noticable difference to either the message transmission problem or the
MMC error.
Thanks again for helping me with this problem. Any advice you can give
about where's best to go with this problem from here would be
appreciated.
P.S. Just one other thing: I have now been able to reproduce this
problem in my test lab, and have been able to prove that joining the NT4
2003 server to the AD domain DOES fix both problems (and causes a few
others!) However this will not be a viable option in Live/Production for
some months, so another solution (if one exists!) would be marvellous...
Regards,
Adam.
John Breakwell (MSFT)
> Very useful to know - thanks. From what I've gathered the use of AD
> integration with MSMQ is discouraged by the Devs of this application due
> to performance implications. I have wondered if there's much truth in
> this I must admit! I can't really imagine the performance hit would be
> that great (and we are looking at some pretty beefy servers here)...
No, there is no performance implication. A cost implication, yes, as you
need at least one domain controller.
You can develop an application that runs on MSMQ in workgroup mode using
private queues and this will run exactly the same way on MSMQ in AD
integrated mode with zero difference in performance. As you are not using
any AD-integrated functionality then this is to be expected.
On the other hand, you can develop an application that runs on MSMQ in AD
integrated using the extra functionality that is exposed, such as being able
to search for public queues, and so on. This application will probably run
slower compared to an application written for Workgroup mode MSMQ but it is
not a like-for-like comparison as the AD-integrated application cannot run
on workgroup mode MSMQ and the workgroup mode application won't have the
same functionality as the AD-integrated application.
I would suggest that the worries of your developers are probably
misinterpretations of things they may have read about AD-integrated mode
MSMQ.
I don't recommend AD-integrated mode for the sake of simplicity and cost -
if your application needs it, use it. If it doesn't, don't.
I notice from your original post:
"It's installed in Workgroup/non-AD integration mode (a requirement of the
app, or so I'm told)."
They're telling you wrong ;-)
"Now the permissions are all in place, as far as I can see. I have even gone
as far as to make the domain account from the "other" system a local
administrator on both sides - no difference. I've also tried messing around
with the "anonymous" account (adding it in with full access in various
places), with no luck."
MSMQ sends messages with the SID of the sending account. The SID has to
match one of the accounts/groups in the permissions list on the queue OR
anonymous logon needs to be there. If you have added anonymous and it
doesn't work then it is NOT a permissions issue.
"Basically our company have just finished migration from NT4 -> 2003 (yeah I
know, we're very behind) and still has servers spanning two trusted NT4
domains along with our new AD2003 domain (also with full two-way trusts to
both NT4 domains)."
Now, as I understand it (and I could be wrong), this means you are operating
something similar to having multiple trusted forests. Each Windows NT4
domain is in itself a forest of one domain. So the security databases are
not shared between each domain even though there are trusts. I've put
together a couple of blog posts for cross-forest MSMQ but unfortunately
neither seem to cover problems when anonymous logon has been enabled:
http://blogs.msdn.com/johnbreakwell/archive/2008/02/14/how-do-i-send-msmq-messages-between-domains.aspx
http://blogs.msdn.com/johnbreakwell/archive/2007/01/15/msmq-3-0-too-secure-for-you.aspx
The MMC issue ("The list of messages cannot be retrieved. Error: A security
package specific error occurred") may be resolved by the
NewRemoteReadServerAllowNoneSecurityClient registry value.
I'll have another look at your problem after I've had a lie down :-)
Cheers
John Breakwell (MSFT)
Adam Jones
Responses below (and it looks like we might be making some progress...)
"John Breakwell \(MSFT\)" <Green...@msn.com> wrote in
news:eR#9IbjIJ...@TK2MSFTNGP05.phx.gbl:
Thanks - that confirms my suspicions. To be honest the main reason for
my mentioning AD integration was in case you guys saw any mileage in
switching the mode of operation. I'm more than happy to leave it in
Workgroup mode for simplicity's sake alone! :)) But in all seriousness
it's good to know that I can use AD-integrated mode if I need to in the
future.
>
> "Now the permissions are all in place, as far as I can see. I have
> even gone as far as to make the domain account from the "other" system
> a local administrator on both sides - no difference. I've also tried
> messing around with the "anonymous" account (adding it in with full
> access in various places), with no luck."
>
> MSMQ sends messages with the SID of the sending account. The SID has
> to match one of the accounts/groups in the permissions list on the
> queue OR anonymous logon needs to be there. If you have added
> anonymous and it doesn't work then it is NOT a permissions issue.
>
This certainly agrees with my findings. I've been thinking it must be
something to do with a difference in behaviour between MSMQ 3.0 and 2.0
(given that it works fine with a W2000 server in the NT domain). What
exactly that is, though, I wish I knew!
>
> "Basically our company have just finished migration from NT4 -> 2003
> (yeah I know, we're very behind) and still has servers spanning two
> trusted NT4 domains along with our new AD2003 domain (also with full
> two-way trusts to both NT4 domains)."
>
> Now, as I understand it (and I could be wrong), this means you are
> operating something similar to having multiple trusted forests. Each
> Windows NT4 domain is in itself a forest of one domain. So the
> security databases are not shared between each domain even though
> there are trusts. I've put together a couple of blog posts for
> cross-forest MSMQ but unfortunately neither seem to cover problems
> when anonymous logon has been enabled:
>
> http://blogs.msdn.com/johnbreakwell/archive/2008/02/14/how-do-i-send-
ms
> mq-messages-between-domains.aspx
> http://blogs.msdn.com/johnbreakwell/archive/2007/01/15/msmq-3-0-too-
sec
> ure-for-you.aspx
>
> The MMC issue ("The list of messages cannot be retrieved. Error: A
> security package specific error occurred") may be resolved by the
> NewRemoteReadServerAllowNoneSecurityClient registry value.
>
Many thanks for the links to your blog entries, and the suggestion.
Reading about that registry change (and as both blog entries mention
it), I was quite hopeful so I've given it a shot. It's exactly as you
thought - using that DWORD (cor what a long name - I had to check it
twice!) fixes the MMC issue when used in conjunction with the anonymous
account (ie it worked once I'd re-added ANONYMOUS_LOGON to the
permissions on the queues), but it didn't help with the message sending
problem... despite adding "send message" permission on the target queue
for the anonymous account, I still get the same negative acknowledgement
message generated (and the messages are still being marked in the
outgoing queue as "unacknowledged"). Still from what you say this is
exactly what you expected, and it seems like we might be getting near to
the cause of the problem! (Or am I getting my hopes up prematurely? ;-))
> I'll have another look at your problem after I've had a lie down :-)
>
Thanks again, John - I really appreciate your help. Hope you had a nice
rest! :-)
> Cheers
> John Breakwell (MSFT)
>
>
Regards,
Adam.
John Breakwell (MSFT)
The MMC fix was the easy one.
Now to the sending problem.
Looking through the support archives, I can't find any problems that were
not solved by using anonymous logon.
If the destination queue has given send permissions to "anonymous logon"
then I don't see any issues with security (as there is no security) so I
wouldn't expect you to see any negative ACKs for
MQMSG_CLASS_NACK_ACCESS_DENIED ("The sender does not have send access rights
on the queue").
Q1 Try giving Full Control to the queue to anonymous logon
Q2 On the server, do you get any MSMQ events (such as 2177 "The Message
Queuing service rejects incoming messages when it is unable to check whether
the sender is allowed access to the queue for sending messages.")
Q3 Reboot sender and destination machines
Cheers
John Breakwell
"Adam Jones" <adamN...@npower.com> wrote in message
news:Xns9B28DC6...@69.16.176.253...
Adam Jones
That's it!! Granted Full Control to the anonymous logon account on the
destination queue, and it all sprang into life! I knew it was working
the second I saw the "$ORDER_QUEUE" appear in the outgoing queues list
(literally jumped out of the chair! :-)) I had tried this level of
permission before, but that was before putting in the "allow none
security clients" registry value (so I guess that did the trick in the
end). That's really awesome - thanks so much for your help! I owe you
beer, or cake, or both! :-D
Adam.
"John Breakwell \(MSFT\)" <Green...@msn.com> wrote in
news:6C9E0AB2-9AA4-48DD...@microsoft.com: