Lessons: MSMQ & NAT Firewalls
Jeff Dunmall
A while back I posted a rather frantic question about a problem I was having
running MSMQ in a production environment. Let me share the two lessons I
learned since then in hopes of saving everyone else some headache:
1) NAT firewalls and MSMQ transacted messages don't mix
If you try and send a transacted MSMQ message from machine A to machine B,
you'll see strange behaviour depending on your firewall configuration. As a
general rule of thumb, if there is a NAT firewall between the two machines,
the message may not get sent, and Microsoft won't support the configuration.
Sometimes it's hard to know if you are in this situation, as it comes up
most often when you're moving to a production environment.
The reasoning has to do with sending acknowledgements back - the receiving
machine sends the acknowledgements back to the source IP address of the
connection through port 1801; it does not use the machine name of the source
machine. If a NAT firewall is translating that source IP address to
something else, the receiving machine will send the acknowledgement back to
the NAT'ed address. In some cases this works, and in others it does not.
There are two main symptoms if there is a NAT firewall in the way. First,
you'll see very slow message delivery - messages that would normally arrive
in 100 ms will be taking upwards of 10 minutes. In some cases, the messages
won't arrive at all. The fact that some of them do arrive will make the
problem difficult to diagnose. If the network topology is complex, it may
be hard to tell if a firewall is doing NAT. Being able to 'ping' each
machine is only the first step!
The second symptom is visible using perfmon and monitoring the outbound
queues (MSMQ 1), or using the outbound queue list (MSMQ 2). Outbound
acknowledgement messages are sent to the administration order_queue - it's
pretty easy to see the connection. If you see a connection to an IP address
that isn't in your list for the domain, then it is likely that IP address is
one that a firewall has translated. Make sure you can identify all of the
outbound IP addresses in the list.
Messages that are 'stuck' will be visible in outbound queues for a very long
time if you don't change the default timeout for messages in a domain (90
days). I recommend changing this value to something more reasonable. Ask
yourself, if this message doesn't appear for X days, would I still want it?
At least the messages appear in the transacted dead letter queue and you can
recover them. Otherwise, they're stuck in MSMQ 'limbo' and aren't
retrievable.
2) Beware of MSMQ's ability to get the message there
MSMQ is very good at delivering messages, network conditions permitting.
Even if it takes days or weeks for those conditions to appear. If you are
setting up multiple testing environments (dev, system test, production,
etc), make sure it is physically impossible to get a MSMQ message into your
production environment. You never know when one, two, or several thousand
messages might appear.
3) Microsoft's MSMQ support team is very good
Kudo's to the MSMQ support team for spending their Saturday helping me
diagnose this problem. If only they could come up with a KB article about
this issue. Until then, there's always this post and dejanews.
Jeff Dunmall
imason.com inc - web technology that works
Yoel Arnon
comment on item #2 (messages hanging arround for a long time):
To avoid having MSMQ messages hanging around for days, weeks or months,
always use timeouts (time to reach queue or time to be received). If you
don't, all messages will have the default timeout, which is three months.
I still need to check item 1. I think there is a hot fix available for that
problem, but I'm not sure.
Best regards,
Yoel
"Jeff Dunmall" <jeff.d...@imason.com> wrote in message
news:OfikY17ZAHA.236@tkmsftngp02...
Jeff Dunmall
> always use timeouts (time to reach queue or time to be received). If you
> don't, all messages will have the default timeout, which is three months.
Good point Yoel. I believe you can also set the default timeout for the
domain to less than the three month default. A three month default seems
extremely excessive in all but the most unusual of applications.
> I still need to check item 1. I think there is a hot fix available for
that
> problem, but I'm not sure.
I'd be very interested to find out that there is a hotfix for #1.
Steve Cargill
The enterprise timeout can be set through MSMQ explorer ( MSMQ 1 ) to a
minimum of one hour. The timouts for individual messages can be set on
a per message bases down to one second.
Steve.
In article <#NMO1SmdAHA.1268@tkmsftngp02>, Jeff Dunmall
<jeff.d...@imason.com> writes
--
Steve Cargill
Tom Olson
idea due to the timing of the exactly once delivery mechanism. For
recoverable and express messages, one second is OK. One might argue that an
interval like that probably means that a synchronous model, rather than
MSMQ, should be used.
(I know that Steve knows this, but I'm posting for everyone else that
doesn't)
Steve Cargill <St...@cargills.demon.co.uk> wrote in message
news:sJIbWJAC...@cargills.demon.co.uk...
Yoel Arnon
About item 1 (NAT firewalls and MSMQ transacted messages don't mix).
I made some inquiries and I am sorry, but there is no hotfix.
The reason that transactional messages do not work through NAT / Firewall is
that in a DIRECT transactional session, the source sends its IP address to
the destination and waits for internal acknowledge. Naturally, this will not
work over NAT or Firewall.
I heard of people who were able to work with transactional messages through
firewall by using PUBLIC queues (this of course raises the issue of working
with PEC/PSC or W2K DC over the firewall). I didn't experience it myself and
I don't know exactly what they did. Anyway I understood that the "official"
Microsoft's statement is that indeed NAT firewalls and MSMQ transacted
messages don't mix.
MSMQ 3.0 should do a much better job in this issue, however.
Sorry,
Yoel
"Jeff Dunmall" <jeff.d...@imason.com> wrote in message
news:#NMO1SmdAHA.1268@tkmsftngp02...