Named properties quota exceeded?

[jim]

One of our developers is setting up a system whereby when an employee makes
a vacation request on our intranet, the approval is emailed to the user
along with a calendar item they can add to their Outlook calendar (an .ics
file). She's using ColdFusion with a java script that does the actual
emailing.

Shortly after she started testing it, the messages stopped being delivered.
I tracked the failed messages and they're all coming up with this error:

550 5.2.0 STOREDRV.Deliver: The Microsoft Exchange Information Store service
reported an error. The following information should help identify the cause
of this error: "MapiExceptionNamedPropsQuotaExceeded"

I found a technet article that describes how to increase 'Named Properties'
quota, but it doesn't seem to be recommended by Microsoft as a good
solution. It's far better to fix the application that's using Exchange than
to manipulate Exchange to accommodate the app.

That said, how can i determine the number of named properties she's sending
so that she can adjust them on her end? Is there some logging i need to
turn up on the MB server?

We're entirely Ex2k7 SP1, rollup 2.

Here's the full error if that helps:

550 5.2.0 STOREDRV.Deliver: The Microsoft Exchange Information Store service
reported an error. The following information should help identify the cause
of this error: "MapiExceptionNamedPropsQuotaExceeded:16.18969:BE000000,
17.27161:00000000D4000000000000000000000000000000, 255.23226:31000000,
255.27962:7A000000, 255.27962:56000000, 255.17082:00090480,
0.16993:80030400, 4.21921:00090480, 255.27962:FA000000, 255.1494:00000000,
255.26426:56000000, 4.6363:0F010480, 2.31229:00000000, 4.6363:0F010480,
2.17597:00000000, 2.22787:00000000, 2.22787:00000000, 2.22957:00000000,
2.19693:00000000, 2.17917:00000000, 2.27341:00000000, 2.22787:00000000,
4.5415:00090480, 4.7867:00090480, 4.4475:00090480, 4.4603:00090480,
4.5323:00090480, 255.1750:00000000, 0.26849:2F000000, 255.21817:00090480,
0.24529:00000000, 4.18385:00090480".

[Dgoldman [MSFT]]

You dont want to increase the limit because down the road all you are going
to do is hit it and have the same problem. You can download a copy of
MFCMapi and logon to the information store and dump out the Named Props
table. Essentially the only way to fix it (the correct way) is move the
users to a new information store and also to stop the program from creating
so many named props.

--
This posting is provided "AS IS" with no warranties, and confers no rights.

Dgoldman
http://blogs.msdn.com/dgoldman
Download OABInteg
(http://gotdotnet.com/Community/UserSamples/Download.aspx?SampleGuid=A2338E73-F521-4071-9B1D-AAF49C346ACD)

"jim" <j...@nospam.com> wrote in message
news:eggFGXmw...@TK2MSFTNGP06.phx.gbl...

[Dan Mitchell]

"jim" <j...@nospam.com> wrote in
news:eggFGXmw...@TK2MSFTNGP06.phx.gbl:

> That said, how can i determine the number of named properties she's
> sending so that she can adjust them on her end?

The important thing is to look in the code, basically. You said:

>She's using ColdFusion with a java script that does the actual
>emailing.

but if all it's doing is sending mail with attachments, I can't think of
any reason it would be using named properties. Those are for custom
properties for non-standard things -- so normal mail wouldn't need them,
and even if you did, I can't think of any reason that the code would be
creating many different named properties.

I'd guess there's a bug in the code somewhere, or at least a very strange
design decision, but without knowing what the code does it's hard to
suggest anything more.

-- dan

[Dgoldman [MSFT]]

You can add custom headers to emails that have the ability to use named
props. Faxing software packages are common for this as an example.

--
This posting is provided "AS IS" with no warranties, and confers no rights.

Dgoldman
http://blogs.msdn.com/dgoldman
Download OABInteg
(http://gotdotnet.com/Community/UserSamples/Download.aspx?SampleGuid=A2338E73-F521-4071-9B1D-AAF49C346ACD)

"Dan Mitchell" <djmit...@yahoo.com> wrote in message
news:Xns9AB167E5A1AA3d...@207.46.248.16...

[Karl Mitschke]

Hello Dgoldman [MSFT],

> You dont want to increase the limit because down the road all you are
> going to do is hit it and have the same problem. You can download a
> copy of MFCMapi and logon to the information store and dump out the
> Named Props table. Essentially the only way to fix it (the correct
> way) is move the users to a new information store and also to stop the
> program from creating so many named props.
>

> Dgoldman
> http://blogs.msdn.com/dgoldman
> Download OABInteg
> (http://gotdotnet.com/Community/UserSamples/Download.aspx?SampleGuid=A
> 2338E73-F521-4071-9B1D-AAF49C346ACD)

Are you kidding me?

We get 1 million mail messages a month. If only a very small percentage of
thes have a rogue header like "x-david-mailscanner-spamcheck" and soon we
will have to move all 14000 users to new databases?

That's insane!

[Stephen Griffin]

Yes - it sucks. We're all in agreement on this. But it's the way things are,
and as long as MAPI property values are 32 bit numbers with only 16 bits
available for the unique identifier, that's the way things will continue to
be. It's a bad situation and we're doing our best to help everyone cope with
it.

This scenario (and others) are part of why we don't recommend placing so
many users in a single database. You can put them all on the same server -
just break up the databases so you have far fewer users per database. The
named prop issue will take much longer to appear, migration to a new
database if it does appear will be much faster, and backup/restore times
will be much shorter. I'm sure someone around here's got a link to our
official guidance if you want some numbers to help.

Steve

"Karl Mitschke" <kmit...@somestate.gov> wrote in message
news:d66cd4c240ce58...@msnews.microsoft.com...

[Karl Mitschke]

Hello Steve,

The 2 databases that have had this problem (both have had the namedproperties
increased twice now) have less than 700 mailboxes.

We have 1 database with slightly more mailboxes that isn't even logging warning
yet

I'm perfectly willing to add more databases, I guess I will look for the
documentation you reference.

Karl

[Hunter Coleman]

Steve-

Yes, we agree that it sucks. And while it likely doesn't change our options
for fixing the situation, I respectfully disagree that you (Microsoft) are

"doing our best to help everyone cope with it."

Check through your Exchange 2007 Planning and Architecture documentation,
and Deployment documentation, and Operations documentation. I have yet to
find a reference to the named properties exhaustion problem. There is a
bunch of information about sizing databases for I/O, backup and restore,
etc, but it all looks like a ghost town with tumbleweeds rolling through
when it comes to MAPI named properties. Where is the information about this
on the EHLO blog? Try searching for "MapiExceptionNamedPropsQuotaExceeded"
at support.microsoft.com; not much there, eh? Could you provide some details
on how to "download a copy of MFCMapi and logon to the information store and
dump out the Named Props table."?

My impression (opinion only) is that you (Microsoft) have a very
uncomfortable situation and are hoping that the spotlight doesn't shine in
this area. I mean, you have a great product in terms of stability,
scalability, fault tolerance, clustering and such, but the very real
possibility that any deployment receiving reasonably large amounts of
internet email is going to start dropping messages. Not exactly a strong
selling point, is it? But your customers need to be aware of this up front
during the design stage of their upgrade/rollout if the best you can offer
for mitigation is to move mailboxes to a new store. From an operational and
administrative standpoint, that's a remarkably poor solution and we need
something better.

--
Hunter

"Stephen Griffin" <sgri...@microsoft.com> wrote in message
news:%23WtICK5...@TK2MSFTNGP06.phx.gbl...

[Stephen Griffin]

Here's the EHLO article from three years ago talking about the problem:
http://msexchangeteam.com/archive/2005/08/08/408796.aspx.
Note that acknowledgement that named property exhaustion could be used as a
basis for a Denial of Service attack. We're not trying to hide this.

I also found these articles on technet and support.microsoft.com:
Cannot create any more MAPI named properties -
http://technet.microsoft.com/en-us/library/bb218047(EXCHG.80).aspx
Understanding the Impact of Named Property and Replica Identifier Limits on
Exchange Databases -
http://technet.microsoft.com/en-us/library/bb851492(EXCHG.80).aspx
How to configure quota settings for named properties and for replica
identifiers in Exchange Server 2003 and in Exchange Server 2007 -
http://support.microsoft.com/?kbid=820379
Events 9666, 9667, 9668, and 9669 Received When Named Properties or Replica
Identifiers Are Depleted for An Exchange Database -
http://technet.microsoft.com/en-us/library/bb851495.aspx

MFCMAPI can be downloaded from here:
http://codeplex.com/mfcmapi

If you're looking to dump the named property table, make sure you're using
an uncached profile (else you'll just be looking at the named prop table for
the OST), then open your mailbox and run "Property Pane"/"Find All Named
Props (SLOW)"

Steve

"Hunter Coleman" <glaci...@yahoo.com> wrote in message
news:C3EB4069-2F3D-45A5...@microsoft.com...

[Hunter Coleman]

Thank you for those links.

Are there any perfmon counters that will show what the named property count
is for a store?

When I dump the named property table for one store, I see entries such as
<NamedPropName>sz: "x-antivirus-konceptio" and
<NamedPropName>sz: "x-sender-reputation"
that are fairly obvious as to their source. However, I'm also seeing several
thousand PS_INTERNET_HEADERS formatted like:
<NamedPropName>sz: "m+c@x("tr."xt,"!41`hh16yd"
<NamedPropName>sz: "m(#(n.3(@5$0**"`p+c`p*2!4"
<NamedPropName>sz: "m+c0p(#(n.3(@5$0**"`p+c`p*2!4"
etc.

I want to identify the internet messages that have those 'randomly
generated' X-headers and look into the system/gateway that is creating them;
how can I associate those named property entries with the message(s) that
contain the corresponding X-headers?

--
Hunter

"Stephen Griffin" <sgri...@microsoft.com> wrote in message

news:uPF8i171...@TK2MSFTNGP04.phx.gbl...

[Stephen Griffin]

I don't think there's a perf counter for this - someone admin focused might
correct me.

The random headers: We've seen those, and not everything is understood about
them. We think they come from spam mail, but it's also possible they're
coming from non-english mail that may have used foreign character sets (like
asian character sets) in the headers. Headers like that did lead to a
problem we issued a hotfix for:
http://support.microsoft.com/kb/941060

This fix reduces the number of those randomly generated headers that can get
into the database. The cache referred to in the article is not actually the
named prop cache, but the fix has an effect on both. With that fix applied
(or any fix that puts exmime.dll >= 6.5.7653.20) you should see a reduction
in the number of those type of headers getting promoted to the named prop
cache. Of course, the fix won't clean up the ones that are already there -
to do that you'd need to move the mailboxes. :)

Steve

"Hunter Coleman" <glaci...@yahoo.com> wrote in message

news:172298CB-BAAD-44C1...@microsoft.com...

[Hunter Coleman]

We're on E2k7 SP1 RU2, so I'm assuming that we're already getting any relief
that came with the exmime.dll hotfix.

The perfmon counter would be helpful for baselining and long-term growth
monitoring. As it is, we don't seem to have any way of knowing whether the
named prop cache is growing in a steady fashion, or random/bursty.

And there's no way to correlate a particular X-header (or any other entry)
in the cache to the message(s) that contain that header? So if we had an
ill-designed application generating custom named properties, or an external
SMTP host causing lots of custom named properties to generate, how should we
go about identifying that source?

--
Hunter

"Stephen Griffin" <sgri...@microsoft.com> wrote in message

news:%23SBRU2I...@TK2MSFTNGP04.phx.gbl...

[Dgoldman [MSFT]]

You will not see a much better fix until Exchange 14. Just to add a little
more information to this as well. If you have Public Folders that are mail
enabled and addressable from the Internet this would lead to this problem..
Mailbox stores tend to get this more since it has to promote all props for
all emails coming in to the mailbox. Custom forms with custom props can also
cause this problem as well.

1. It is important to understand how Named Property gets added to the
information store. Named props do not come from a specific mailbox; they are
common to the mailbox store as a whole. Every time a mailbox in the store
gets a new message with a unique X-Header that's not in the props table, the
store will add it to the props table.

2. Most customers do add additionally 3rd party software such as spam
filtering software which also adds X-Headers and those will also get added
to the named props table.

3. Finally any 3rd party applications which create additional MAPI
properties, those properties are also added to the table.

4. There is no way to clear out this table unless you create a new mailbox
store!! As explained before you can move mailboxes off this store to a new
store and have a new named props table which is not full. Remember that you
will only have 32K of space available for named props.

To answer your question about the performance counter, you can monitor the
number of rows in the named props table by looking at the following
performance counter: MSExchangeIS Mailbox -> Rows in NamedProps Table ->
<Name of store>.

You will have to enabled advanced store counters to expose this value and
you can look at the following link on how to do this:
http://support.microsoft.com/kb/821912

Bare in mind that this performance counter will *only* help you monitor how
close to the edge you are getting as far as availability of rows in the
named props table but it will *NOT* tell you the origin of those props.

Steve gave you the best way to dump out these properties by using MFCMapi so
you can find out which X-Headers or MAPI properties take up the most space.

If you have X-Headers coming from the Internet then you may also want to use
a spam filtering solution to remove these X-Headers before they even reach
the Exchange organization.

--
This posting is provided "AS IS" with no warranties, and confers no rights.

Dgoldman

http://blogs.msdn.com/dgoldman
Download OABInteg
(http://gotdotnet.com/Community/UserSamples/Download.aspx?SampleGuid=A2338E73-F521-4071-9B1D-AAF49C346ACD)

"Hunter Coleman" <glaci...@yahoo.com> wrote in message

news:ukcUq0J2...@TK2MSFTNGP03.phx.gbl...

[dwatson80]

I am experiencing the same problems discussed in this post. I wish there was
a tool or something that would allow deletion of junk Named Properties. I
tried deleting on from MFCMapi but got some cryptic message afterwards and
the property was still there. I am also interested in a way to stop them or
find out where they are coming from. I can see on our PerfLog counter that
they are still going up. I am at 9103 today and was at 9071 yesterday.

[Dgoldman [MSFT]]

The best thing you can do is dump out the table to see the properties and
examine them. Find out which application they are coming from and get that
changed. Having the ability to delete them from the table every so often
will never fix your problem.

--
This posting is provided "AS IS" with no warranties, and confers no rights.

Dgoldman

http://blogs.msdn.com/dgoldman
Download OABInteg
(http://gotdotnet.com/Community/UserSamples/Download.aspx?SampleGuid=A2338E73-F521-4071-9B1D-AAF49C346ACD)

"dwatson80" <dwat...@discussions.microsoft.com> wrote in message
news:7885CB57-A01E-4FEC...@microsoft.com...

[Karl Mitschke]

Hello Dgoldman [MSFT],

I hate to rain on your parade, but the problem we are having is not "our
problem".

We are receiving a million email messages every month from the internet.
Some percentage of these have new x-headers, which get added to the non-mapi
named properties table.

Even if we COULD figure out what application is creating them, there is no
way we could "get that changed"

Having the ability to delete them from the table every so often would indeed
fix "our problem".

Now, on a conference call with another Microsoft Exchange Engineer, we were
informed that SP1 doesn't actually drop messages when the non-mapi table
is full., it just fails to add the new x-header. In testing, this SEEMS correct,
but since no one can actually confirm this, or point to an actual Microsoft
document that states this we are still a little skeptical. However, we will
be bumping our non-mapi quota's to 32000 soon, so we can stop seeing erroneous
"errors" in our log files.

Can you confirm that SP1 no longer drops messages when the non-mapi quota
is reached?

Regards

Karl

> The best thing you can do is dump out the table to see the properties
> and examine them. Find out which application they are coming from and
> get that changed. Having the ability to delete them from the table
> every so often will never fix your problem.
>

> Dgoldman
> http://blogs.msdn.com/dgoldman
> Download OABInteg
> (http://gotdotnet.com/Community/UserSamples/Download.aspx?SampleGuid=A
> 2338E73-F521-4071-9B1D-AAF49C346ACD)

[Andy David {MVP}]

On Wed, 16 Jul 2008 15:18:14 +0000 (UTC), Karl Mitschke
<kmit...@somestate.gov> wrote:

>Hello Dgoldman [MSFT],
>
>I hate to rain on your parade, but the problem we are having is not "our
>problem".
>
>We are receiving a million email messages every month from the internet.
>Some percentage of these have new x-headers, which get added to the non-mapi
>named properties table.
>
>Even if we COULD figure out what application is creating them, there is no
>way we could "get that changed"
>
>Having the ability to delete them from the table every so often would indeed
>fix "our problem".
>
>Now, on a conference call with another Microsoft Exchange Engineer, we were
>informed that SP1 doesn't actually drop messages when the non-mapi table
>is full., it just fails to add the new x-header. In testing, this SEEMS correct,
>but since no one can actually confirm this, or point to an actual Microsoft
>document that states this we are still a little skeptical. However, we will
>be bumping our non-mapi quota's to 32000 soon, so we can stop seeing erroneous
>"errors" in our log files.
>
>Can you confirm that SP1 no longer drops messages when the non-mapi quota
>is reached?

Be aware that this was released on codeplex:

http://www.codeplex.com/HeaderFilterAgent
"HeaderFilterAgent for Exchange 2007 "

[Stu Jackson]

Is there somewhere I can go for a quick tutorial on how to do this? I'd like
to believe I know what I am doing in Exchange but I started poking around
with MFCMapi and I feel like I'm reading Greek.

I'm managed to eliminate most the souces of the x- headers but would like to
examine the few remaining that show up. I tried to find them, but my head
almost exploded trying to digest what I am reading. AFter I make best effort
to find the source of my named properties I would like to dump contents of
the named properties table. How do I do that?

Thanks